CN116821102B

CN116821102B - Data migration method, device, computer equipment and storage medium

Info

Publication number: CN116821102B
Application number: CN202311079590.7A
Authority: CN
Inventors: 尹超; 张瑞安
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2023-08-25
Filing date: 2023-08-25
Publication date: 2023-11-17
Anticipated expiration: 2043-08-25
Also published as: CN116821102A

Abstract

The present application relates to a data migration method, apparatus, computer device, storage medium and computer program product. The method comprises the following steps: acquiring a cold data set to be migrated based on the data migration request; cloud data identification conversion is carried out on the data identification in the cold data set to obtain a target cold data set; performing object data conversion based on the target cold data set to obtain a cloud object data file, and determining target metadata and target index information corresponding to the cloud object data file; storing target metadata and target index information in an associated manner, wherein the target metadata and the target index information are used for accessing cloud object data files in a cloud object data storage space; and carrying out data migration on the cloud object data file, the target metadata and the target index information to a cloud object data storage space, and deleting the cold data set when the data migration is completed. The method can ensure the access performance and greatly reduce the storage cost.

Description

Data migration method, device, computer equipment and storage medium

Technical Field

The present application relates to the field of computer technology, and in particular, to a data migration method, apparatus, computer device, storage medium, and computer program product.

Background

With the development of computer technology, database technology has emerged, and database technology is generally used to store business data, for example, data generated by online business is generally stored using a high-performance storage device. The data generated by the online service generally comprises cold data, wherein the cold data refers to data which is not frequently accessed or used, and the data state of the cold data is not active. The use of high performance storage devices to store cold data results in higher storage costs. Currently, in order to save storage costs, cold data is generally stored using a low-cost storage device, however, storing cold data using a low-cost storage device may result in a decrease in access performance of cold data.

Disclosure of Invention

In view of the foregoing, it is desirable to provide a data migration method, apparatus, computer device, computer readable storage medium, and computer program product that can greatly reduce storage costs while ensuring access performance.

In a first aspect, the present application provides a data migration method. The method comprises the following steps:

acquiring a data migration request, and acquiring a cold data set to be migrated based on the data migration request;

Cloud data identification conversion is carried out on the data identification in the cold data set to obtain a target cold data set;

performing object data conversion based on the target cold data set to obtain a cloud object data file, and determining target metadata and target index information corresponding to the cloud object data file, wherein the target metadata are used for describing a cloud data identification range corresponding to the cloud object data file, and the target index information is used for determining the position of the cloud object data file in a cloud object data storage space;

storing target metadata and target index information in an associated manner, wherein the target metadata and the target index information are used for accessing cloud object data files in a cloud object data storage space;

and carrying out data migration on the cloud object data file, the target metadata and the target index information to a cloud object data storage space, and deleting the cold data set when the data migration is completed.

In a second aspect, the application further provides a data migration device. The device comprises:

the cold data acquisition module is used for acquiring a data migration request and acquiring a cold data set to be migrated based on the data migration request;

the identifier conversion module is used for carrying out cloud data identifier conversion on the data identifiers in the cold data set to obtain a target cold data set;

The data conversion module is used for carrying out object data conversion based on the target cold data set to obtain a cloud object data file, determining target metadata and target index information corresponding to the cloud object data file, wherein the target metadata are used for describing a cloud data identification range corresponding to the cloud object data file, and the target index information is used for determining the position of the cloud object data file in a cloud object data storage space;

the information storage module is used for storing the target metadata and the target index information in an associated mode, and the target metadata and the target index information are used for accessing cloud object data files in the cloud object data storage space;

and the data migration module is used for carrying out data migration on the cloud object data file, the target metadata and the target index information to the cloud object data storage space, and deleting the cold data set when the data migration is completed.

In a third aspect, the present application also provides a computer device. The computer device comprises a memory storing a computer program and a processor which when executing the computer program performs the steps of:

In a fourth aspect, the present application also provides a computer-readable storage medium. The computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of:

In a fifth aspect, the present application also provides a computer program product. The computer program product comprises a computer program which, when executed by a processor, implements the steps of:

The above data migration method, apparatus, computer device, storage medium and computer program product by obtaining a cold data set to be migrated; cloud data identification conversion is carried out on the data identification in the cold data set to obtain a target cold data set; performing object data conversion based on the target cold data set to obtain a cloud object data file, and determining target metadata and target index information corresponding to the cloud object data file, wherein the target metadata are used for describing a cloud data identification range corresponding to the cloud object data file, and the target index information is used for determining the position of the cloud object data file in a cloud object data storage space; storing target metadata and target index information in an associated manner, wherein the target metadata and the target index information are used for accessing cloud object data files in a cloud object data storage space; and carrying out data migration on the cloud object data file, the target metadata and the target index information to a cloud object data storage space, and deleting the cold data set when the data migration is completed. The cold data set is migrated to the cloud object data storage space and is accessed by using the stored target metadata and target index information, so that the storage cost can be greatly reduced while the access performance is ensured.

Drawings

FIG. 1 is a diagram of an application environment for a data migration method in one embodiment;

FIG. 2 is a flow chart of a method of data migration in one embodiment;

FIG. 3 is a schematic diagram of an information file structure in an embodiment;

FIG. 4 is a flow diagram of data merging in one embodiment;

FIG. 5 is a schematic diagram of a framework for data migration in one embodiment;

FIG. 6 is a flow chart of data migration in one embodiment;

FIG. 7 is a flow diagram of data polling in one embodiment;

FIG. 8 is a schematic diagram of an architecture for access through proxy services in one embodiment;

FIG. 9 is a flow chart of data polling in one embodiment;

FIG. 10 is a flow chart of data statistics in one embodiment;

FIG. 11 is a flow diagram of data deletion in one embodiment;

FIG. 12 is a flow diagram of data update in one embodiment;

FIG. 13 is a flow chart of data migration in another embodiment;

FIG. 14 is a schematic diagram of a data migration application scenario in an embodiment;

FIG. 15 is a block diagram of a data migration apparatus in one embodiment;

FIG. 16 is an internal block diagram of a computer device in one embodiment;

Fig. 17 is an internal structural view of a computer device in another embodiment.

Detailed Description

The present application will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present application more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the application.

The data migration method provided by the embodiment of the application can be applied to an application environment shown in figure 1. The terminal 102 communicates with the server 104 through a network, and the server 104 communicates with the cloud object storage space through the network. The data storage system may store data, and the data storage system may be integrated on the server 104 or may be separately provided. The server 104 acquires a data migration request sent by the terminal 102, and acquires a cold data set to be migrated based on the data migration request; the server 104 performs cloud data identification conversion on the data identification in the cold data set to obtain a target cold data set; the server 104 performs object data conversion based on the target cold data set to obtain a cloud object data file, and determines target metadata and target index information corresponding to the cloud object data file, wherein the target metadata is used for describing a cloud data identification range corresponding to the cloud object data file, and the target index information is used for determining the position of the cloud object data file in a cloud object data storage space; the server 104 stores target metadata and target index information in an associated manner, wherein the target metadata and the target index information are used for accessing cloud object data files in a cloud object data storage space; the server 104 performs data migration on the cloud object data file, the target metadata, and the target index information to the cloud object data storage space 106, and deletes the cold data set when the data migration is completed. The terminal 102 may be, but not limited to, various desktop computers, notebook computers, smart phones, tablet computers, internet of things devices, and portable wearable devices, where the internet of things devices may be smart speakers, smart televisions, smart air conditioners, smart vehicle devices, and the like. The portable wearable device may be a smart watch, smart bracelet, headset, or the like. The server 104 may be implemented as a stand-alone server or as a server cluster of multiple servers. Cloud object storage 106 may be a cloud server that provides cloud computing services. The terminal and the server may be directly or indirectly connected through wired or wireless communication, and the present application is not limited herein.

In one embodiment, as shown in fig. 2, a data migration method is provided, where the method is applied to the server in fig. 1 for illustration, it is understood that the method may also be applied to a terminal, and may also be applied to a system including the terminal and the server, and implemented through interaction between the terminal and the server. In this embodiment, the method includes the steps of:

s202, acquiring a data migration request, and acquiring a cold data set to be migrated based on the data migration request.

The data migration request refers to a request for migrating cold data. The cold data set to be migrated refers to a set of cold data to be migrated, which may be various types of data, such as image type data, text type data, voice type data, and numerical type data, etc. The cold data may also be data in various service scenarios, such as an instant messaging service scenario, a live broadcast service scenario, a network transaction service scenario, a video sharing service scenario, and so on. The cold data may also be data in different languages, which may be real languages of different countries, such as chinese, japanese, english, russian, etc. The cold data may also be data of a different storage format, etc.

Specifically, the server may obtain a data migration request uploaded by the terminal, and then obtain a cold data set to be migrated according to the data migration request and a preset cold data range. The server can also analyze the data migration request, and acquire the cold data set to be migrated according to the cold data identification range to be migrated carried in the data migration request. In one embodiment, the server may find the cold data set to be migrated directly from the data stored in the database according to the cold data identification range to be migrated. In one embodiment, the data may be stored in a distributed manner, and the server may obtain the cold data set to be migrated from a distributed node, which may be a server.

In one embodiment, the server may trigger a data migration instruction according to the trigger event, and obtain the cold data set to be migrated according to the data migration instruction. The triggering event is a preset event for performing cold data migration, for example, when the server detects that the current time point is a preset cold data migration time point, a data migration instruction is triggered. When the server detects that the current data storage space reaches a preset used threshold value, a data migration instruction is triggered. The server detects that the data volume reaches the preset upper limit, and triggers a data migration instruction.

S204, carrying out cloud data identification conversion on the data identification in the cold data set to obtain a target cold data set.

Wherein the data identification is used to identify the corresponding cold data in the cold dataset. The types of the different cold data corresponding data identifiers can be different, for example, the type of the cold data corresponding data identifier can be a numerical value type, a character string type and the like. The cloud data identification is used for uniquely identifying the corresponding cold data, and the cloud data identification types of all the cold data are consistent.

Specifically, the server converts the data identifier of the cold data in the cold data set, wherein the type of the data identifier is converted, that is, the type of the data identifier is converted into the type of the cloud data identifier, for example, the data identifiers of different types can be converted into the character string type. The server may also convert the data identifier itself, that is, may perform unique conversion on the data identifier after type conversion, for example, may obtain the converted cloud data identifier by adding a unique character string, where the unique character string may be preset sequence information, time information, and the like, and may set specific content of the unique character string according to needs. And finally, the server obtains a cloud data identifier corresponding to the cold data through conversion, and obtains a target cold data set according to the cloud data identifier and the corresponding cold data.

S206, converting object data based on the target cold data set to obtain a cloud object data file, determining target metadata and target index information corresponding to the cloud object data file, wherein the target metadata are used for describing a cloud data identification range corresponding to the cloud object data file, and the target index information is used for determining the position of the cloud object data file in a cloud object data storage space.

The cloud object data file refers to a file of cloud object data obtained by converting cold data according to a data format of object storage, and the cloud object data is object data which can be stored in a cloud object storage space. The cloud object data file stores each cloud object data and cloud data identifiers corresponding to each cloud object data. Cloud object data is obtained by converting cold data in a cold data set through object data. Object storage refers to a technique for storing and managing data in an unstructured format (referred to as an object). The target metadata is a cloud data identification range corresponding to the cloud object data file, that is, the cloud data identification range in which the cloud object data stored in the cloud object data file is stored in the target cloud data, and the cloud data identification range can be represented by an upper limit value and a lower limit value of the cloud data identification in the cloud object data file. The target index information is used to determine a location of the cloud object data file in the cloud object data storage space. The location of the cloud object data file in the cloud object data storage space may include a size and an offset.

Specifically, the server converts each cold data in the target cold data set into corresponding cloud object data, and then the server obtains a cloud object data file according to the converted cloud object data. The server can determine the quantity of cloud object data stored in the cloud object data file according to the preset file size, and then distribute all the cloud object data according to the preset file size to obtain each cloud object data file, wherein the cloud object data in each cloud object data file are arranged according to the sequence of cloud data identification. And then generating target metadata according to the cloud data identification range corresponding to each cloud object data file, determining the position of the cloud object data file in the cloud object data storage space according to the sequence of the cloud data identifications in the cloud object data file and the file size, and further generating target index information corresponding to all the cloud object data files.

And S208, storing the target metadata and the target index information in an associated manner, wherein the target metadata and the target index information are used for accessing the cloud object data file in the cloud object data storage space.

Specifically, the server stores target metadata and target index information corresponding to the cloud object data file in an associated manner, and then the server can use the target metadata and the target index information to access the cloud object data file in the cloud object data storage space, for example, search the cloud object data file stored in the cloud object data storage space by using the target metadata and the target index information, count the cloud object data stored in the cloud object data storage space by using the target metadata and the target index information, update and delete the cloud object data stored in the cloud object data storage space by using the target metadata and the target index information, and the like.

And S210, performing data migration on the cloud object data file, the target metadata and the target index information to a cloud object data storage space, and deleting the cold data set when the data migration is completed.

Specifically, the server uploads the cloud object data file, the target metadata and the target index information to the cloud object data storage space, and the cloud object data storage space acquires and stores the cloud object data file, the target metadata and the target index information. The server then detects that the data migration is complete and may delete the cold data set stored in the database. The server may also delete the cold data set stored by the distributed node. In one embodiment, when there are at least two cloud object data files, the server may delete the cold data set after all cloud object data files have been data migrated. The server may delete the cold data corresponding to the cloud object data in the cloud object data file after completing the data migration of any one cloud object data file.

According to the data migration method, the cold data set to be migrated is obtained; cloud data identification conversion is carried out on the data identification in the cold data set to obtain a target cold data set; performing object data conversion based on the target cold data set to obtain a cloud object data file, and determining target metadata and target index information corresponding to the cloud object data file, wherein the target metadata are used for describing a cloud data identification range corresponding to the cloud object data file, and the target index information is used for determining the position of the cloud object data file in a cloud object data storage space; storing target metadata and target index information in an associated manner, wherein the target metadata and the target index information are used for accessing cloud object data files in a cloud object data storage space; and carrying out data migration on the cloud object data file, the target metadata and the target index information to a cloud object data storage space, and deleting the cold data set when the data migration is completed. The cold data set is migrated to the cloud object data storage space and is accessed by using the stored target metadata and target index information, so that the storage cost can be greatly reduced while the access performance is ensured.

In one embodiment, S204, performing cloud data identifier conversion on the data identifier in the cold dataset to obtain the target cold dataset, including the steps of:

converting the data types corresponding to the data identifiers in the cold data set to obtain the data identifiers of the target data types; acquiring a sequence identifier, and combining the sequence identifier with a data identifier of a target data type to obtain a cloud data identifier; and updating the data identifier in the cold data set into a cloud data identifier to obtain the target cold data set.

The target data type refers to a preset data type of the cloud data identifier, for example, the data type may be a character string type, a numerical value type, a binary type, or the like. The sequence identification is used for uniquely identifying the cloud object data which are arranged in sequence, and is determined according to the arrangement sequence of the cloud object data in the cloud object data file.

Specifically, the server converts the data type corresponding to the data identifier in the cold dataset to obtain the data identifier of the target data type, that is, the server may normalize all the data types corresponding to the data identifier to be string (character string) types, for example, there are two documents storing cold data, the data identifier in one document is double type, and the data identifier in the other document is string type, and at this time, all the data identifiers may be converted to string types. And then the server can acquire the sequence identifier, and combine the sequence identifier with the data identifier of the target data type, wherein the data identifier of the target data type and the sequence identifier can be spliced end to obtain the cloud data identifier. And finally, the server updates the data identifier in the cold data set into the cloud data identifier to obtain the target cold data set. In one embodiment, the server may also obtain identification information, for example, the identification information may be time information, attribute information of the cold data, the attribute information may be a cold data size, a cold data amount, and a maximum value and a minimum value of the cold data, and so on. And then splicing the acquired identification information with the data identification of the target data type and the sequence identification end to obtain the target cloud data identification.

In a specific embodiment, the double type data identifier obtained by the server may include: cold data with uuid (universal unique identifier) of "1" and cold data with uuid of "2", and then converting the type of the data identifier to obtain the data identifier of the character string type, wherein the data identifier of the character string type comprises "+_002\u004" corresponding to uuid of "1", and uuid is "+_004\u004" corresponding to "2". The data identification that can be obtained for the string type can then include: cold data with uuid "1" and cold data with uuid "CMongo". Then converting the type of the data identifier, wherein the obtained data identifier comprises the following steps of converting the data identifier, and obtaining the data identifier of the character string type comprises "<1\u000\u004" corresponding to uuid "1", and "< CMongo\u000\u004" corresponding to uuid "CMongo".

In the above embodiment, the cloud data identifier is obtained by converting the data type corresponding to the data identifier and combining the data type with the sequence identifier, and then the data identifier in the cold data set is updated to the cloud data identifier to obtain the target cold data set, so that the uniqueness of the generated cloud data identifier is ensured.

In one embodiment, S206, determining target metadata and target index information corresponding to the cloud object data file, includes the steps of:

Acquiring a cloud data identification upper limit value and a cloud data identification lower limit value in a cloud object data file, and determining target metadata corresponding to the cloud object data file based on the cloud data identification upper limit value and the cloud data identification lower limit value; and acquiring the position information of the cloud object data file in the cloud object data storage space, and determining target index information corresponding to the cloud object data file based on the position information.

The cloud data identification upper limit value is used for representing the largest cloud data identification in the cloud object data file. The cloud data identification lower limit value is used for representing the minimum cloud data identification in the cloud object data file.

Specifically, the server may directly find the cloud data identifier upper limit value and the cloud data identifier lower limit value from the cloud object data file, where the cloud object data files are arranged according to the cloud data identifier sequence, and may directly obtain the start cloud data identifier and the end cloud data identifier in the cloud object data file, so as to obtain the cloud data identifier upper limit value and the cloud data identifier lower limit value. And then determining target metadata corresponding to the cloud object data file according to the data identification upper limit value and the cloud data identification lower limit value. In one embodiment, the server may further obtain file attribute information of the cloud object data file, for example, the number of cold data in the file, the total size of the file, and the like, and use the file attribute information as target metadata corresponding to the object data file. And then the server acquires the position information of the cloud object data file in the cloud object data storage space, wherein the offset and the size of the cloud object data file in the cloud object data storage space can also be acquired, so that the position information of the cloud object data file is acquired, and then the position information of the cloud object data file is used as target index information corresponding to the cloud object data file. In one embodiment, the offset and the size of the cloud data identifier in the cloud object data file may also be acquired, so as to obtain the position information of the cloud object data, and the position information of the cloud object data may also be used as index information.

In the above embodiment, by acquiring the cloud data identification upper limit value and the cloud data identification lower limit value in the cloud object data file, determining target metadata corresponding to the cloud object data file based on the cloud data identification upper limit value and the cloud data identification lower limit value; and acquiring the position information of the cloud object data file in the cloud object data storage space, and determining target index information corresponding to the cloud object data file based on the position information, so that the accuracy of the obtained target metadata and the target index information is ensured.

In one embodiment, the data migration method further comprises the steps of:

hash mapping is carried out on cloud data identifiers in the cloud object data file, and hash values corresponding to the cloud data identifiers are obtained; generating a target bit array corresponding to the cloud data identifier based on each hash value, and determining the identifier presence detection information corresponding to the cloud object data file based on the target bit array; and storing the identification presence detection information, the target metadata and the target index information in an associated manner.

Wherein hash mapping refers to mapping using a hash function. Different hash functions are used for carrying out hash mapping on the same cloud data identifier, and different hash values can be obtained. The identification presence detection information is used to characterize that cloud data is identified in a corresponding cloud object data file. The identification presence detection information can detect whether any cloud data identification is in the cloud object data file or not, and can be used for filtering cloud data identifications which are not in the cloud object data file. The hash function refers to a function that maps key values of elements in a hash table to element storage locations.

Specifically, the server performs hash mapping on cloud data identifiers in the cloud object data file by using preset hash functions to obtain hash values corresponding to the cloud data identifiers. For example, the cloud data identifier may be converted into an integer, then the integer pair 10 is left to obtain a hash value, the ASCII (a character encoding standard) code value of each character in the cloud data identifier may be added, then the sum 10 is left to obtain a hash value, the character string may be flipped, then the flipped character string is converted into an integer, and then the integer pair 10 is left to obtain a hash value. And generating a target bit array corresponding to the cloud data identifier by using each hash value, wherein the initialized bit array can be acquired firstly, then after each hash value is obtained, determining the initial element position in the bit array according to each hash value, and then modifying the initial element in the initial element position into a target element, so that the target bit array is obtained. And finally, the server can generate a target bit array corresponding to each cloud data identifier in the cloud object data file, and all the target bit arrays are used as the identifier existence detection information corresponding to the cloud object data file. And finally, the server stores the identification presence detection information, the target metadata and the target index information in an associated manner.

In the embodiment, the cloud data identifier in the cloud object data file is subjected to hash mapping to obtain each hash value corresponding to the cloud data identifier, then a target bit array corresponding to the cloud data identifier is generated based on each hash value, the identifier presence detection information corresponding to the cloud object data file is determined based on the target bit array, and finally the identifier presence detection information, the target metadata and the target index information are associated and stored, namely, the data stored in the cloud object storage space can be accessed through the identifier presence detection information, the target metadata and the target index information, so that the access performance is improved.

In a specific embodiment, after object data conversion is performed on the cold dataset, four types of files are obtained, including a cloud object data file, a target metadata file, a target index information file and an identification presence detection information file. As shown in fig. 3, a schematic structure diagram of an information file obtained after object Data conversion is shown, in which a Data table storing cold Data is converted to obtain a cloud object Data file, i.e., a Data file, a target metadata file, i.e., a Meta file, a target Index information file, i.e., an Index file, and a bloom filter file, which is a presence detection information file. The Data file is a BSON file, which is a computer Data exchange format, mainly used as a Data storage and network transmission format in a mongo db (a database based on distributed file storage) database. The Data file stores sequentially arranged BSON format cold Data, including bson.0 (cold Data 0), bson.1 (cold Data 1), …, bson.n (cold Data n), each BSON (cold Data) storing specific object Data bsonraw. The Meta file is a minimum value and a maximum value of metadata identification in the storage internal Data file through the BSON format, and can also store the quantity of cold Data, the total size of the cold Data, the index of metadata identification and the like. The bloom filter file is used for judging whether the metadata identifier is in the Data file, and comprises bloom filter information corresponding to each metadata identifier, namely bf.0 (filter 0), bf.1 (filter 1), … and bf.n (filter n). The Index file is a file in SST format, which is a file format for storing data, and which is a file format used by a storage engine based on LSM tree (Log-Structured Merge Tree). The Index file stores the location in the Data file corresponding to each metadata identification, including sst.0 (Index 0), sst.1 (Index 1), …, sst.n (Index n). Each sst in the Index file is composed of a key-value pair key, which is a specific metadata identification, and a value, which is a specific location, including offset (offset) and size (length). The position of the cloud Data identification in the Data file can be found through the Index file. The cold Data stored in the cloud storage space can be accessed quickly by using the Data file, the Meta file, the Index file, and the bloom filter file.

In one embodiment, S210, the deletion of the cold data set includes the steps of:

and updating and checking the cold data in the cold data set, and deleting the cold data set through the parallel asynchronous thread when the cold data in the cold data set is not updated.

Wherein, the update check refers to checking whether the cold data is changed. The parallel asynchronous thread is a preset thread for deleting cold data, and the cold data is deleted in a parallel and asynchronous mode.

Specifically, the server performs update verification on the cold data in the cold data set, for example, the server may detect whether the obtained cold data is identical to the cold data identified by the same data stored in the database, and when the obtained cold data is not identical to the cold data identified by the same data, it indicates that the cold data is updated, where updating the cold data may be performing operations such as modifying, deleting, and using the cold data. When there is cold data in the cold data set that has been updated, the server may treat the cold data that has been updated as non-cold data, i.e., the non-cold data will not undergo data migration. When the cold data in the cold data set is not updated, the cold data set is deleted through the parallel asynchronous thread, namely, the server simultaneously generates a plurality of deletion tasks to delete different cold data. The deletion of the cold data set may also be performed by a batch deletion method, for example, by a mongadb batch deletion method.

In the above embodiment, the update verification is performed on the cold data in the cold data set, the non-updated cold data is deleted through the update verification, the deletion of non-cold data is avoided, and then when the cold data in the cold data set is not updated, the cold data set is deleted through the parallel asynchronous thread, so that the efficiency of deleting the cold data is improved.

In one embodiment, the cloud object data file includes at least two;

as shown in fig. 4, after S210, after performing data migration on the cloud object data file, the target metadata, and the target index information to the cloud object data storage space, and when the data migration is completed, deleting the cold data set, further includes:

s402, detecting target metadata corresponding to at least two cloud object data files, and acquiring each range coincidence file from a cloud object data storage space based on the target metadata corresponding to at least two cloud object data files and target index information corresponding to at least two cloud object data files when cloud data identification ranges in the target metadata coincide.

The overlapping of the cloud data identification ranges means that the cloud data identification ranges corresponding to the cloud object data files are repeated, that is, the cloud object data files with the same cloud data identification are different, and the cloud data identification of some cloud data can be the same, or all cloud data identification can be the same. The range coincidence file refers to a cloud object data file with the cloud data identification identical to the cloud data identifications in other cloud object data files. I.e. the same cloud data identification is found in the individual range recall files.

Specifically, the server detects target metadata corresponding to at least two cloud object data files, and when cloud data identification ranges in the target metadata are overlapped, the server needs to perform duplication removal processing on the cloud object data files with the overlapped cloud data identification ranges. The server acquires each range coincidence file from the cloud object data storage space by using target metadata corresponding to at least two cloud object data files and target index information corresponding to at least two cloud object data files. The server determines each range coincidence file identifier from each cloud object data file according to the cloud data identifier range in the target metadata, and then acquires each range coincidence file from the cloud object data storage space according to the target metadata and the target index information corresponding to each range coincidence file identifier. For example, the upper limit value of the cloud data identification range corresponding to the cloud object data file a is 1, the lower limit value is 100, the upper limit value of the cloud data identification range corresponding to the cloud object data file B is 150, the lower limit value is 50, at this time, the cloud object data file a and the cloud object data file B have the overlapping of the cloud data identification ranges, and the range of the recall is 50 to 100, at this time, the cloud object data file a and the cloud object data file B are the range recall files.

S404, merging the range overlapping files to obtain a merged file, and determining merging metadata and merging index information corresponding to the merged file.

The merging file is a cloud object data file obtained by merging all the range overlapping files, and the cloud data identification in the merging file is obtained by de-duplicating the repeated cloud data identification. The merging metadata refers to target cloud data corresponding to the merging files, the upper limit value of the cloud data identification in the merging metadata is the largest cloud data identification in each range overlapping file, and the lower limit value of the cloud data identification in the merging metadata is the smallest cloud data identification in each range overlapping file. Merging index information refers to

Specifically, the server merges the overlapping files of each range, wherein the cloud object data identified by the same cloud data are de-duplicated, the cloud object data identified by single same cloud data are reserved, and then all the cloud object data are arranged according to the sequence of the cloud data identification, so that the merged file is obtained. And then the server generates merging metadata and merging index information corresponding to the merging file. In one embodiment, the overlapping files of each range are combined to obtain at least two combined files, wherein the data size in the combined files is preset, and then each combined file is sequentially obtained according to the data size of the combined file during combination. For example, the combined data size is 10GB (unit of storage, gigabytes). And if the data size of each merging file which can be stored is 1GB, merging all the range overlapping files to obtain 10 merging files. The cloud data identification ranges in the 10 merged files do not coincide.

S406, storing the merging metadata and the merging index information in an associated mode, and performing data migration on the merging file to the cloud object data storage space.

Specifically, the server stores the merging metadata and the merging index information in an associated manner, and the merging files stored in the cloud object data storage space can be accessed through the merging metadata and the merging index information. And meanwhile, the server performs data migration on the combined file to the cloud object data storage space, namely the combined file can be uploaded to the cloud object data storage space for storage.

In the above embodiment, by detecting the target metadata corresponding to at least two cloud object data files, when the cloud data identification ranges in the target metadata are overlapped, each range overlapping file is obtained from the cloud object data storage space based on the target metadata corresponding to at least two cloud object data files and the target index information corresponding to at least two cloud object data files. And then merging the overlapping files in each range to obtain a merged file, and carrying out data migration on the merged file to the cloud object data storage space while storing the merged metadata and the merged index information, so that the storage space can be saved, the query efficiency can be improved, and the access performance is improved.

In a specific embodiment, as shown in fig. 5, a framework diagram of data migration is provided, specifically: the server firstly acquires a cold Data set from a Data cluster according to a preset cold Data rule, then generates a Data (object Data) file, a Meta (metadata) file, an Index (Index information) file and a bloom filter file corresponding to the cold Data set, then stores the Meta file, the Index file and the bloom filter file, and uploads the Data file, the Meta file, the Index file and the bloom filter file to a cloud object Data storage space which can be a cloud server, and meanwhile, the server can generate a queue to be deleted of cold Data in the Data migration process, namely when one piece of cold Data migration is completed, the Data identification of the piece of cold Data is written into the queue to be deleted. And then after the data migration is completed, the server can delete the cold data in the data cluster according to the queue to be deleted, and the deletion efficiency can be improved by batch deletion, namely, the cold and hot data in the data cluster are layered through the cooling service, namely, the data migration. And then the server can periodically detect the stored target metadata, when the cloud Data identification range is found to be coincident in the target metadata, the server acquires an original Data file with the cloud Data identification range being coincident from the cloud server, then merges the original Data files with the cloud Data identification range being coincident, so as to obtain a new Data file with a new cloud Data identification range being not coincident, generates a new Meta file, a new Index file and a new bloom filter file corresponding to the new Data file with the cloud Data identification range being not coincident, stores the new Meta file, the new Index file and the new bloom filter file, and uploads the new Data file, the new Meta file, the new Index file and the new bloom filter file to the cloud server. In a specific embodiment, as shown in fig. 6, a flow diagram of data migration is provided, in which: and the server acquires a cold data set from the database according to a preset cold data range, and then converts the type of the data identifier in the cold data set into a character string type to obtain the cloud data identifier. Then generating a file corresponding to the cold data set, including: data files, a Meta file, an Index file and a BloomFilter file are subjected to Data migration to a cloud server, and the Meta file, the Index file and the BloomFilter file are stored in a memory or a cache of the server. After the Data migration is completed, the server concurrently deletes the Data which has been subjected to the Data migration from the database in batches, and then deletes local files in the server, such as a cold Data set, a Data file and the like, so that the storage space of the server itself can be saved, the storage cost can be saved, and meanwhile, the Meta file, the Index file and the BloomFilter file are used for Data access, and the Data access performance is ensured.

In one embodiment, the cloud object data file includes at least two;

as shown in fig. 7, after S210, that is, after performing data migration on the cloud object data file, the target metadata, and the target index information to the cloud object data storage space, and deleting the cold data set when the data migration is completed, the method further includes:

s702, acquiring a data query request, wherein the data query request carries a query data identifier, and converting the query data identifier into a cloud data identifier to obtain the cloud query identifier.

The query data identification refers to identification of data to be queried, and the cloud query identification refers to identification obtained by converting the query data identification according to the type of the cloud data identification.

Specifically, the server may obtain a data query request from the request end, and then parse the data query request to obtain the carried query data identifier. In one embodiment, the data query request may carry at least two query data identifiers, and may also carry a range of data identifiers to be queried. And then the server performs cloud data identification conversion on the query data identification to obtain a cloud query identification. For example, the query data identifier is a numeric identifier, and the numeric identifier can be converted into a binary string identifier to obtain a cloud query identifier.

S704, determining a cloud query file identification corresponding to the cloud query identification from the at least two cloud object data files based on target cloud data corresponding to the at least two cloud object data files.

The cloud query file identifier refers to an identifier of a cloud query file, and is used for uniquely identifying a corresponding cloud query file, where the cloud query file identifier may be a number, a character string, or the like.

Specifically, the server determines a cloud query file identifier corresponding to the cloud query identifier according to the stored target cloud data. Comparing the cloud query file identifications with cloud query file identifications in each target cloud data, and when the cloud data identifications consistent with the cloud query file identifications exist in the target cloud data, indicating that the data to be queried are in cloud object data files corresponding to the target cloud data, wherein at the moment, the server takes the cloud object data file identifications corresponding to the target cloud data as cloud query file identifications corresponding to the cloud query identifications.

S706, acquiring a cloud query file from the cloud object data storage space based on target index information corresponding to the cloud query file identification, and querying corresponding target cloud object data in the cloud query file based on the cloud query identification.

The cloud query file refers to a cloud object data file with cloud query identifiers obtained by query. The target cloud object data refers to data stored in an object storage format corresponding to a cloud query identifier in a cloud query file.

Specifically, the server uses the target index information corresponding to the stored cloud query file identifier to obtain the cloud query file from the cloud object data storage space, namely, downloads the cloud query file from the cloud object data storage space according to the position information of the cloud query file to obtain the cloud query file. And then inquiring from the cloud inquiry file to obtain target cloud object data corresponding to the cloud inquiry identifier. In one embodiment, when there are multiple query data identifications, the server may determine a cloud query file identification corresponding to each query data identification. And then acquiring cloud query files corresponding to each cloud query file identifier from the cloud object storage space, and querying data stored in an object storage format corresponding to each query data identifier from the cloud query files.

S708, restoring the target cloud object data to obtain target query data corresponding to the query data identification, and returning the target query data to a request end corresponding to the data query request.

The target query data refers to data corresponding to the query data identification obtained by query. The request end corresponding to the data query request refers to a service end for requesting the data query, and the service end can be a terminal or a server.

Specifically, the server restores the target cloud object data, namely restores the storage format of the target cloud object data to the storage format of the query data, so as to obtain target query data corresponding to the query data identifier. And finally, the server returns the target query data obtained by query to a request end corresponding to the data query request, and the target query data is displayed in the request end.

In one embodiment, the server can also directly return the target cloud object data to the request end corresponding to the data query request, and the data query efficiency is further improved without restoration.

In the embodiment, the cloud query file identification can be obtained by quickly querying by using the target cloud data and the target index information after the data query request is obtained, further the cloud query file corresponding to the cloud query file identification is directly obtained from the cloud object storage space, the target query data corresponding to the query data identification is obtained from the cloud query file, thus the scanning in the cloud object storage space is not needed, the data query efficiency is improved, the access performance is ensured,

In one embodiment, the data migration method further comprises the steps of:

when the query data identifier is a cloud-stored data identifier, performing cloud data identifier conversion on the query data identifier through a preset proxy service to obtain a cloud query identifier; determining a cloud query file identifier corresponding to the cloud query identifier from at least two cloud object data files by using target cloud data corresponding to the at least two cloud object data files through a preset proxy service; and acquiring the cloud query file from the cloud object data storage space by using target index information corresponding to the cloud query file identification through a preset proxy service.

The preset proxy service is a preset proxy service for accessing the cloud object storage space.

Specifically, when the server receives the data query request, and when judging that the query data identifier carried in the data query request is the data identifier stored in the cloud object storage space, the server performs data query through a preset proxy service. The cloud data identification conversion is carried out on the query data identification through the preset proxy service, so that a cloud query identification is obtained; and determining a cloud query file identifier corresponding to the cloud query identifier from the at least two cloud object data files by using target cloud data corresponding to the at least two cloud object data files through a preset proxy service. The cloud query file is acquired from the cloud object data storage space by using target index information corresponding to the cloud query file identification through a preset proxy service, then target object data corresponding to the cloud query identification is acquired from the cloud query file, the target object data is restored to obtain target query data, and the target query data is returned to the request end. The preset proxy service may be implemented by a proxy cluster.

In the above embodiment, when the query data identifier is a cloud-stored data identifier, the query data identifier is subjected to cloud data identifier conversion by a preset proxy service to obtain a cloud query identifier; determining a cloud query file corresponding to the cloud query identifier from at least two cloud object data files by using target cloud data corresponding to the at least two cloud object data files through a preset proxy service; the cloud query file is acquired from the cloud object data storage space by using the target index information corresponding to the cloud query file through the preset proxy service, namely the cloud server is accessed through the proxy service, so that the cloud server is prevented from being accessed and scanned, the access performance of the data is ensured on the basis of saving the storage cost, and the reduction of the access performance is avoided.

In a specific embodiment, as shown in fig. 8, an architecture diagram of access through a proxy service is shown, where a server is deployed with a database MongoDB of distributed file storage, and can access data stored in a cloud server through a proxy service compatible with the MongoDB protocol. When the server obtains an access command through the command interface, for example, the access command may be a data query request, a data statistics request, a data deletion request, a request for displaying all databases, a request for displaying all sets, etc. And judging whether the data to be accessed is stored in a cloud server according to the access data identifier carried by the access command, wherein the cloud server stores a cloud object data file, target metadata, target index information and a bloom filter file. When the data to be accessed is not stored in the cloud server, the server can query the database MongoDB of the distributed file storage for the data corresponding to the data identification to be accessed through a default storage engine (WiredTiger engine) of the MongoDB. When data to be accessed is stored in a cloud server, the server needs to query the data through a cloud service storage engine, i.e., needs to access the data in the cloud server through a proxy service compatible with the MongoDB protocol. The proxy service firstly converts the data identifier to be accessed to obtain a cloud data identifier, then queries the cloud object data file identifier to be accessed by using target metadata in cloud server storage data stored in a server through the cloud data identifier, and then determines the position of the cloud object data file identifier to be accessed in the cloud server according to target index information in the cloud server storage data stored in the server. And then, according to the position, acquiring a cloud object Data file to be accessed, namely a Data file, from a cloud server through threads in a thread pool, acquiring cloud object Data to be accessed from the cloud object Data file, and restoring the cloud object Data by proxy service to obtain target Data to be accessed. And finally, the server returns the target data inquired through the proxy service to the request end, namely, the cloud server is accessed through the proxy service, and access scanning on the cloud server is avoided, so that the access performance of the data is ensured on the basis of saving the storage cost, and the reduction of the access performance is avoided. The server may also detect stored cloud server storage data, such as Meta files, etc., by refreshing timing tasks, and also provide an object storage SDK (software toolkit).

In one embodiment, S702, that is, determining, from at least two cloud object data files, a cloud query file identifier corresponding to the cloud query identifier based on target cloud data corresponding to the at least two cloud object data files, includes the steps of:

searching each candidate cloud object data file identification of cloud query identifications in a cloud data identification range corresponding to the cloud object data files from at least two cloud object data files; and acquiring identification presence detection information corresponding to at least two cloud object data files, and filtering each candidate cloud object data file identification through a cloud query identification based on the identification presence detection information to obtain a cloud query file identification corresponding to the cloud query identification.

The candidate cloud object data file identifiers refer to cloud object data file identifiers needing further screening, and the cloud object data file identifiers are used for uniquely identifying corresponding cloud object data files.

Specifically, the server may store the identifier of each cloud object data file in association with the corresponding target metadata, the corresponding target index information, and the corresponding identifier presence detection information. And the server determines each target metadata containing the cloud query identifier according to the cloud data identifier range in the target metadata corresponding to each cloud object data file, and then takes the cloud object data file identifier corresponding to each target metadata as each candidate cloud object data file identifier. The server acquires identification existence detection information corresponding to each candidate cloud object data file identification, calculates identification existence detection information corresponding to cloud query identification, compares the identification existence detection information corresponding to the cloud query identification with the identification existence detection information corresponding to each candidate cloud object data file identification, filters the candidate cloud object data file identifications inconsistent in the identification existence detection information, so as to obtain filtered candidate cloud object data file identifications, and then uses the filtered candidate cloud object data file identifications as cloud query file identifications corresponding to the cloud query identifications.

In the embodiment, each candidate cloud object data file identifier is determined according to the cloud data identifier range corresponding to the cloud object data file, and then the cloud query file identifier corresponding to the cloud query identifier is obtained by filtering the identifier presence detection information from each candidate cloud object data file identifier, so that the efficiency of obtaining the cloud query file identifier is improved.

In one embodiment, S706, obtaining a cloud query file from a cloud object data storage space based on target index information corresponding to the cloud query file identifier, and querying corresponding target cloud object data in the cloud query file based on the cloud query identifier, includes the steps of:

acquiring file index information corresponding to the cloud query file identification from a cloud object data storage space based on target index information corresponding to the cloud query file identification, wherein the file index information comprises each cloud data identification and an object data storage position corresponding to each cloud data identification; searching a query object data storage position corresponding to the cloud query identifier from the file index information, and acquiring target cloud object data corresponding to the cloud query identifier from the cloud object data storage space based on the query object data storage position.

The file index information refers to index information of cloud object data in the cloud object data file, and the file index information is stored in a key value pair mode. The file index information includes respective cloud data identifications and respective object data storage locations corresponding to the respective cloud data identifications, which may include offsets and sizes of the object data in the cloud object data file. Querying the object data storage location refers to the storage location of the object data to be queried.

Specifically, the server may acquire file index information corresponding to the cloud query file identifier from the cloud object data storage space using target index information corresponding to the cloud query file identifier. The file index information comprises each cloud data identifier and an object data storage position corresponding to each cloud data identifier. And then the server searches the query object data storage position corresponding to the cloud query identifier from the file index information, namely compares the cloud query identifier with the cloud data identifier in the file index information, searches the consistent cloud data identifier, and then takes the object data storage position corresponding to the consistent cloud data identifier as the query object data storage position corresponding to the cloud query identifier. And then the server acquires target cloud object data corresponding to the cloud query identifier from the cloud object data storage space by using the cloud query identifier and the query object data storage position.

In the above embodiment, the file index information corresponding to the cloud query file identifier is obtained from the cloud object data storage space through the target index information, then the query object data storage position corresponding to the cloud query identifier is searched from the file index information, and the target cloud object data corresponding to the cloud query identifier is obtained from the cloud object data storage space based on the query object data storage position, so that the target cloud object data corresponding to the cloud query identifier can be directly obtained from the cloud object data storage space, the cloud object data stored in the cloud object data file can be directly obtained without obtaining the cloud object data file, and the data query efficiency is improved.

In a specific embodiment, as shown in fig. 9, a flow chart of a data query is provided, specifically: the server acquires a data query request, and the data identifier carried by the data query request can be UUID 100. At this time, the server converts the UUID of 100 into a cloud data identification. And then acquiring a file identification list through a Map (key value pair container) of a memory, namely acquiring a cloud object Data file identification list through a maximum value and a minimum value of cloud Data identifications in a stored Meta table, filtering through a bloom filter file, namely calculating bloom filter information corresponding to the cloud Data identifications to be queried, comparing the bloom filter information with the bloom filter information in the bloom filter file, filtering cloud object Data file identifications corresponding to the bloom filter information which are not consistent with each other from the cloud object Data file identification list to acquire a filtered cloud object Data file identification, acquiring target Index information corresponding to the filtered cloud object Data file identification, namely an Index file, acquiring an Index file through a Footer (Footer) Data table, wherein an Index block in the Index file stores the offset and the size of each DataBlock in a cloud server, and downloading corresponding DataDataIndex file from the cloud server according to the Index file and the cloud Data identification, namely the Index file is the Index value of which is the big key value of the cloud, and the Index value of the Index file is the big key value of the Data. The server acquires the offset and the size of the object Data in the Data file from the DataBlock according to the cloud Data identifier to be queried, and then the cloud server downloads the object Data corresponding to the cloud Data identifier to be queried according to the offset and the size of the object Data in the Data file, wherein the object Data is in a Bson format. The server can directly return the object data to the request end to be queried, and can also use delete data to filter the files in Bson format of the object data obtained by query, namely, the deleted files are filtered, and finally, the filtered files are returned to the request end of query. Namely, object data stored in the cloud server is queried through a Meta file, a bloom filter file and an index file stored in the server, so that the data query efficiency can be improved.

In one embodiment, as shown in fig. 10, after S210, after performing data migration on the cloud object data file, the target metadata, and the target index information to the cloud object data storage space, and when the data migration is completed, deleting the cold data set, further includes:

s1002, acquiring a data statistics request, wherein the data statistics request carries a data identifier to be counted, and performing cloud data identifier conversion on the data identifier to be counted to obtain a cloud statistics identifier.

The identification of the data to be counted refers to the identification of the data for counting calculation. The cloud statistics identification refers to a cloud data identification corresponding to the data identification to be counted.

Specifically, the server may obtain a data statistics request from the request end, then analyze the data statistics request to obtain carried data identifiers to be counted, where the data identifiers to be counted may include at least two data identifiers, then perform cloud data identifier conversion on each data identifier to be counted, and may convert both a data type and a data storage format to obtain a corresponding cloud statistics identifier.

S1004, determining a cloud statistics file identifier corresponding to the cloud statistics identifier based on target cloud data corresponding to the cloud object data file.

S1006, acquiring a cloud statistics file from the cloud object data storage space based on target index information corresponding to the cloud statistics file identification, and searching cloud object data corresponding to the cloud statistics identification in the cloud statistics file.

The cloud statistics file identification refers to an identification of a cloud statistics file in which the cloud statistics identification is located. The cloud statistics file refers to a cloud object data file containing cloud statistics identifications.

Specifically, the server determines target metadata corresponding to the cloud statistics identifications according to the target metadata of the stored cloud object data file, and can determine the cloud data identification range where the cloud statistics identifications are located according to the cloud data identification range stored in the target metadata, so as to obtain the target metadata corresponding to the cloud statistics identifications. And then taking the cloud object data file identifier corresponding to the target metadata where the cloud statistics identifier is located as the cloud statistics file identifier. And then, acquiring a cloud statistics file from the cloud object data storage space by using target index information corresponding to the cloud statistics file identification, and searching cloud object data corresponding to the cloud statistics identification in the cloud statistics file.

S1008, restoring the cloud object data corresponding to the cloud statistics identification to obtain data to be counted corresponding to the cloud statistics identification, performing statistics calculation based on the data to be counted to obtain a statistics result, and returning the statistics result to a request end corresponding to the data statistics request.

Specifically, restoring cloud object data corresponding to the cloud statistics identification, and restoring the data format of the cloud object data into the data format of server storage data to obtain data to be counted. And then carrying out statistical calculation by using the data to be counted to obtain a statistical result, wherein the statistical calculation can comprise counting, calculating the mean value, the median, the mode, the variance, the standard check and the like. In one embodiment, the server may also directly use the cloud object data to perform statistical computation to obtain a statistical structure. And finally, the server returns the statistical result to the request end corresponding to the data statistical request. In one embodiment, after the server obtains the data statistics request, the server may obtain a cloud statistics file from the cloud object data storage space according to the data statistics request through a preset proxy service, and search cloud object data corresponding to the cloud statistics identifier in the cloud statistics file.

In the embodiment, the cloud object data can be quickly obtained from the cloud object storage space by obtaining the data statistics request, then using the target cloud data and the target index information, then using the cloud object data to perform data statistics to obtain the statistics result, and returning the statistics result to the request end corresponding to the data statistics request, thereby improving the efficiency of data statistics.

In one embodiment, as shown in fig. 11, after S210, that is, after performing data migration on the cloud object data file, the target metadata, and the target index information to the cloud object data storage space, and deleting the cold data set when the data migration is completed, the method further includes:

s1102, acquiring a data deleting request, wherein the data deleting request carries a data identifier to be deleted, and converting the data identifier to be deleted into a cloud data identifier to obtain the cloud deleting identifier.

S1104, determining a cloud deletion file identifier corresponding to the cloud deletion identifier based on target cloud data corresponding to the cloud object data file, and acquiring a cloud deletion file from the cloud object data storage space based on target index information corresponding to the cloud deletion file identifier.

The data identifier to be deleted refers to the identifier of the data to be deleted. The cloud deleting identifier is a cloud data identifier corresponding to the data identifier to be deleted. The cloud deletion file identifier refers to an identifier of a cloud deletion file in which the cloud deletion identifier is located. The cloud deleting file refers to a cloud object data file comprising a cloud deleting identifier.

Specifically, the server acquires a data deletion request, analyzes the data deletion request to obtain a data identifier to be deleted, and when judging that the data identifier to be deleted is the data identifier already stored in the cloud object data storage space, the server converts the cloud data identifier to be deleted to obtain a cloud deletion identifier, wherein whether the data identifier to be deleted is the identifier of cold data is judged according to a preset identifier range of the cold data. And then the server judges the target cloud data corresponding to the cloud deletion identification, and judges whether the cloud deletion identification is within the range according to the cloud data identification range in the target cloud data. And then, acquiring a cloud object data file identifier corresponding to the target cloud data as a cloud deletion file identifier corresponding to the cloud deletion identifier, and downloading the cloud object data file identifier from the cloud object data storage space to a cloud deletion file according to the offset and the size in the target index information corresponding to the cloud deletion file identifier.

And S1106, deleting cloud object data corresponding to the cloud deletion identification in the cloud deletion file to obtain a cloud update file, and determining update metadata and update index information corresponding to the cloud update file.

The cloud update file is a cloud object data file obtained by deleting cloud object data corresponding to a cloud deletion identifier in the cloud deletion file. The updating metadata refers to target metadata corresponding to the cloud updating file, and the updating index information refers to target index information corresponding to the cloud updating file.

Specifically, the server deletes cloud object data corresponding to the cloud deletion identifier in the cloud deletion file to obtain a cloud update file, and determines update metadata and update index information corresponding to the cloud update file.

In one embodiment, the server may obtain a plurality of data identifiers to be deleted, and then delete cloud object data corresponding to the plurality of data identifiers to be deleted, where when the cloud data identifiers corresponding to the plurality of data identifiers to be deleted are in the same cloud deletion file, deletion is directly performed in the cloud deletion file, and when the cloud data identifiers corresponding to the plurality of data identifiers to be deleted are in different cloud deletion files, various corresponding cloud deletion files are obtained and deleted. In one embodiment, the server may store each object data to be deleted in the file to be deleted, then obtain a cloud object file data corresponding to the cloud object storage space, and then combine the cloud object file with the file to be deleted, so as to obtain an updated object data file, where the updated object data file is obtained by deleting each object data in the file to be deleted from the cloud object data file.

S1108, storing the update metadata and the update index information in an associated mode, and performing data migration on the cloud update file to the cloud object data storage space.

Specifically, the server performs association storage on the identification, the update metadata and the update index information of the cloud update file, and then performs data migration on the cloud update file to the cloud object data storage space.

In the above embodiment, by acquiring the data deletion request, then using the target cloud data and the target index information, the cloud deletion file can be quickly acquired from the cloud object storage space, then the cloud object data corresponding to the cloud deletion identifier in the cloud deletion file is deleted, so as to obtain the cloud update file, and the update metadata and the update index information corresponding to the cloud update file are determined. And finally, the update metadata and the update index information are stored in an associated mode, and the cloud update file is subjected to data migration to the cloud object data storage space, so that the data deleting efficiency is improved.

In one embodiment, as shown in fig. 12, after S210, that is, after performing data migration on the cloud object data file, the target metadata, and the target index information to the cloud object data storage space, and deleting the cold data set when the data migration is completed, the method further includes:

S1202, acquiring a data update request, wherein the data update request carries a data identifier to be updated and update data, converting the data identifier to be updated into cloud data identifier to obtain cloud update identifier, and converting the update data into object data to obtain cloud update file.

The to-be-updated data identifier refers to an identifier of stored data to be updated, and the updated data refers to data to be replaced by to-be-updated data corresponding to the to-be-updated data identifier. The cloud update identifier refers to a cloud data identifier corresponding to the data identifier to be updated. The cloud update file is a file storing object data corresponding to update data.

Specifically, the server may obtain a data update request from the request end, and then parse the data update request to obtain the carried data identifier to be updated and the update data. When the server judges that the data identifier to be updated is the data identifier stored in the cloud object data storage space, the data identifier to be updated is subjected to cloud data identifier conversion to obtain a cloud update identifier, and the update data is subjected to object data conversion to obtain a cloud update file.

S1204, determining a cloud storage file identifier corresponding to the cloud update identifier based on target cloud data corresponding to the cloud object data file, and acquiring a cloud storage file from the cloud object data storage space based on target index information corresponding to the cloud storage file identifier.

The cloud storage file identification refers to an identification of a cloud storage file in which the cloud update identification is located, and the cloud storage file refers to a cloud object data file containing the cloud update identification.

Specifically, the server determines a cloud data identification range in which the cloud update identification is located according to the cloud data identification range in the stored target metadata, so as to obtain the target metadata in which the cloud update identification is located. And then taking the cloud object data file identifier corresponding to the target metadata where the cloud update identifier is located as a cloud storage file identifier. And then, acquiring the cloud storage meter file from the cloud object data storage space by using target index information corresponding to the cloud storage file identification.

S1206, combining the cloud updating file and the cloud storage file to obtain a target updating file, and determining target updating metadata and target updating index information corresponding to the target updating file;

s1208, storing the target update metadata and the target update index information in an associated mode, and performing data migration on the target update file to the cloud object data storage space.

The target update file is a cloud object data storage file obtained by replacing object data corresponding to the cloud update identifier in the cloud storage file with object data corresponding to the update data. The target update metadata refers to target metadata corresponding to the target update file, and the target update index information refers to target index information corresponding to the target update file.

Specifically, the server merges the cloud update file and the cloud storage file, that is, the object data corresponding to the cloud update identifier in the cloud storage file is replaced by the object data in the cloud update file to obtain the target update file, then the target update metadata and the target update index information corresponding to the target update file are determined, and the identifier of the target update file, the target update metadata and the target update index information are associated and stored. And then the server performs data migration on the target update file to the cloud object data storage space.

In the embodiment, the cloud storage file can be quickly obtained from the cloud object storage space by obtaining the data update request, then using the target cloud data and the target index information, then combining the cloud update file and the cloud storage file to obtain the target update file, so that the efficiency of obtaining the target update file is improved, finally, the target update metadata and the target update index information are stored in an associated mode, and the target update file is subjected to data migration to the cloud object data storage space, so that the data update efficiency is improved.

In a specific embodiment, as shown in fig. 13, a data migration method is provided, which specifically includes the following steps:

S1302, acquiring a data migration request, acquiring a cold data set to be migrated based on the data migration request, converting a data type corresponding to a data identifier in the cold data set to obtain a data identifier of a target data type, acquiring a sequence identifier, combining the sequence identifier with the data identifier of the target data type to obtain a cloud data identifier, and updating the data identifier in the cold data set to the cloud data identifier to obtain the target cold data set.

And S1304, performing object data conversion based on the target cold data set to obtain a cloud object data file, acquiring a cloud data identification upper limit value and a cloud data identification lower limit value in the cloud object data file, and determining target metadata corresponding to the cloud object data file based on the cloud data identification upper limit value and the cloud data identification lower limit value.

S1306, acquiring position information of a cloud object data file in a cloud object data storage space, determining target index information corresponding to the cloud object data file based on the position information, performing hash mapping on cloud data identifiers in the cloud object data file to obtain hash values corresponding to the cloud data identifiers, generating a target bit array corresponding to the cloud data identifiers based on the hash values, and determining identification existence detection information corresponding to the cloud object data file based on the target bit array.

S1308, storing the identification presence detection information, the target metadata and the target index information in association, performing data migration on the cloud object data file, the target metadata and the target index information to the cloud object data storage space, and deleting the cold data set when the data migration is completed.

S1310, acquiring a data query request, wherein the data query request carries a query data identifier, and when the query data identifier is a cloud-stored data identifier, performing cloud data identifier conversion on the query data identifier through a preset proxy service to obtain a cloud query identifier.

S1312, searching each candidate cloud object data file identifier of cloud query identifiers within a cloud data identifier range corresponding to the cloud object data file from at least two cloud object data files through a preset proxy service, acquiring identifier presence detection information corresponding to the at least two cloud object data files, and filtering each candidate cloud object data file identifier through the cloud query identifier based on the identifier presence detection information to obtain a cloud query file identifier corresponding to the cloud query identifier.

S1314, acquiring file index information corresponding to the cloud query file identification from the cloud object data storage space based on target index information corresponding to the cloud query file identification through a preset proxy service, and searching a query object data storage position corresponding to the cloud query identification from the file index information.

S1316, acquiring target cloud object data corresponding to the cloud query identifier from the cloud object data storage space based on the query object data storage position through the preset proxy service. And restoring the target cloud object data to obtain target query data corresponding to the query data identifier, and returning the target query data to a request end corresponding to the data query request.

In the embodiment, the identification presence detection information, the target metadata and the target index information are stored during data migration, and the stored identification presence detection information, the target metadata and the target index information are used for acquiring the queried data from the cloud object storage space during data query, so that the storage cost can be greatly reduced while the access performance is ensured.

In a specific embodiment, the data migration method is applied in a scenario of a database service platform, as shown in fig. 14, which is an overall structural schematic diagram of the application scenario, specifically: a user can manage metadata through a management and control system of the database service platform, namely, the metadata management module can realize the management of cooling rules and the conversion of the identification. The cooling rule management may be a management cooling rule for determining a cold data set from data stored in a data cluster. The cooling rule may be a preset range of cold data, and the range of cold data may be determined according to the identification of the data, for example, the identification of the data from 1 to 100 is determined as cold data. The migration of the cold data is then performed through a cooling down service, and the cloud object data files in which the data is repeated after the cold data migration are merged. The cooling service acquires a cold data set from the distributed data set according to a cooling rule, and then converts the data identifier of the cold data set to obtain a cloud data identifier. And then generating a Data file, a Meta file, an Index file and a BloomFilter file corresponding to the cold Data set, storing the Meta file, the Index file and the BloomFilter file, and uploading the Data file, the Meta file, the Index file and the BloomFilter file to a cloud server. Then, when a data query is performed, a query for a single data identifier, which may be a key in a key-value pair, may be performed, and the data to be queried may be a value in the key-value pair. And then pulling the object data to be queried from the cloud server through the proxy cluster compatible MongoDB protocol, namely converting the data identifier to obtain a cloud data identifier, querying the object data position to be queried from the saved Meta file, index file and BloomFilter file according to the cloud data identifier, acquiring the queried object data from the cloud server by using the object data position, and returning the queried object data to the query terminal. The database service platform can also download the whole amount of data from the cloud server through the proxy cluster, namely, when the database service platform obtains the whole amount of object data downloading request, all object data stored in the cloud server can be downloaded.

Then, access performance test can be performed, and during the test, a YCSB (database performance test tool mainly used for cloud or server side) test tool is adopted to perform performance test, the test model uses YCSB random reading, test data is 16TB (terabyte), and metadata is 2.8GB. After the test data is migrated, only 2.8GB of metadata is saved, the storage space is saved, and then the access performance is tested, and the obtained test result is shown in the following table 1.

Table 1 comparison of properties

Wherein by adjusting the concurrency, the QPS (Querys per second, query rate per second, used to measure throughput capacity of the service), average delay and P99 (used to measure worst response delay in the case of 99%) delay are recorded. When the concurrency is between 200 and 400, the average delay is about 100 milliseconds, namely the access performance is not obviously reduced, and the access performance is ensured. Then, according to the pressure measurement of the long-time interface, the success rate of the long-time pressure measurement interface is 99.999969%, namely the usability reaches 6 pieces of 9.

It should be understood that, although the steps in the flowcharts related to the embodiments described above are sequentially shown as indicated by arrows, these steps are not necessarily sequentially performed in the order indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in the flowcharts described in the above embodiments may include a plurality of steps or a plurality of stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of the steps or stages is not necessarily performed sequentially, but may be performed alternately or alternately with at least some of the other steps or stages.

Based on the same inventive concept, the embodiment of the application also provides a data migration device for realizing the above related data migration method. The implementation of the solution provided by the device is similar to the implementation described in the above method, so the specific limitation in one or more embodiments of the data migration device provided below may refer to the limitation of the data migration method hereinabove, and will not be repeated herein.

In one embodiment, as shown in fig. 15, a data migration apparatus 1500 is provided, comprising a cold data acquisition module 1502, an identification conversion module 1504, a data conversion module 1506, an information storage module 1508, and a data migration module 1510, wherein:

a cold data acquisition module 1502, configured to acquire a data migration request, and acquire a cold data set to be migrated based on the data migration request;

the identifier conversion module 1504 is configured to perform cloud data identifier conversion on data identifiers in the cold dataset to obtain a target cold dataset;

the data conversion module 1506 is configured to perform object data conversion based on the target cold data set to obtain a cloud object data file, and determine target metadata and target index information corresponding to the cloud object data file, where the target metadata is used to describe a cloud data identification range corresponding to the cloud object data file, and the target index information is used to determine a position of the cloud object data file in the cloud object data storage space;

The information saving module 1508 is configured to store target metadata and target index information in association, where the target metadata and the target index information are used to access a cloud object data file in the cloud object data storage space;

the data migration module 1510 is configured to perform data migration on the cloud object data file, the target metadata, and the target index information to the cloud object data storage space, and delete the cold data set when the data migration is completed.

In one embodiment, the identifier conversion module 1504 is further configured to convert a data type corresponding to the data identifier in the cold dataset to obtain a data identifier of the target data type; acquiring a sequence identifier, and combining the sequence identifier with a data identifier of a target data type to obtain a cloud data identifier; and updating the data identifier in the cold data set into a cloud data identifier to obtain the target cold data set.

In one embodiment, the data conversion module 1506 is further configured to obtain an upper limit value of cloud data identifier and a lower limit value of cloud data identifier in the cloud object data file, and determine target metadata corresponding to the cloud object data file based on the upper limit value of cloud data identifier and the lower limit value of cloud data identifier; and acquiring the position information of the cloud object data file in the cloud object data storage space, and determining target index information corresponding to the cloud object data file based on the position information.

In one embodiment, the data migration apparatus 1500 further comprises:

the detection information determining module is used for carrying out hash mapping on cloud data identifiers in the cloud object data file to obtain hash values corresponding to the cloud data identifiers; generating a target bit array corresponding to the cloud data identifier based on each hash value, and determining the identifier presence detection information corresponding to the cloud object data file based on the target bit array; and storing the identification presence detection information, the target metadata and the target index information in an associated manner.

In one embodiment, the data migration module 1510 is further configured to update and verify cold data in the cold dataset, and delete the cold dataset by the parallel asynchronous thread when the cold data in the cold dataset is not updated.

In one embodiment, the cloud object data file includes at least two; the data migration apparatus 1500 further includes:

the merging module is used for detecting target metadata corresponding to at least two cloud object data files, and acquiring each range coincidence file from the cloud object data storage space based on the target metadata corresponding to the at least two cloud object data files and target index information corresponding to the at least two cloud object data files when the cloud data identification ranges in the target metadata coincide; merging the overlapping files of each range to obtain a merged file, and determining merging metadata and merging index information corresponding to the merged file; and storing the merging metadata and the merging index information in an associated mode, and carrying out data migration on the merging file to a cloud object data storage space.

the query module is used for acquiring a data query request, wherein the data query request carries a query data identifier, and performing cloud data identifier conversion on the query data identifier to obtain a cloud query identifier; determining a cloud query file identifier corresponding to the cloud query identifier from the at least two cloud object data files based on target cloud data corresponding to the at least two cloud object data files; acquiring a cloud query file from a cloud object data storage space based on target index information corresponding to the cloud query file identification, and querying corresponding target cloud object data in the cloud query file based on the cloud query identification; and restoring the target cloud object data to obtain target query data corresponding to the query data identifier, and returning the target query data to a request end corresponding to the data query request.

In one embodiment, the data migration apparatus 1500 further comprises:

the agent module is used for carrying out cloud data identification conversion on the query data identification through a preset agent service when the query data identification is the cloud stored data identification, so as to obtain a cloud query identification; determining a cloud query file identifier corresponding to the cloud query identifier from at least two cloud object data files by using target cloud data corresponding to the at least two cloud object data files through a preset proxy service; and acquiring the cloud query file from the cloud object data storage space by using target index information corresponding to the cloud query file identification through a preset proxy service.

In one embodiment, the query module is further configured to search at least two cloud object data files for each candidate cloud object data file identifier of the cloud query identifier within a cloud data identifier range corresponding to the cloud object data file; and acquiring identification presence detection information corresponding to at least two cloud object data files, and filtering each candidate cloud object data file identification through a cloud query identification based on the identification presence detection information to obtain a cloud query file identification corresponding to the cloud query identification.

In one embodiment, the query module is further configured to obtain file index information corresponding to the cloud query file identifier from the cloud object data storage space based on target index information corresponding to the cloud query file identifier, where the file index information includes each cloud data identifier and an object data storage location corresponding to each cloud data identifier; searching a query object data storage position corresponding to the cloud query identifier from the file index information, and acquiring target cloud object data corresponding to the cloud query identifier from the cloud object data storage space based on the query object data storage position.

In one embodiment, the data migration apparatus 1500 further comprises:

the statistics module is used for acquiring a data statistics request, wherein the data statistics request carries a data identifier to be counted, and performing cloud data identifier conversion on the data identifier to be counted to obtain a cloud statistics identifier; determining a cloud statistics file identifier corresponding to the cloud statistics identifier based on target cloud data corresponding to the cloud object data file; acquiring a cloud statistics file from a cloud object data storage space based on target index information corresponding to the cloud statistics file identification, and searching cloud object data corresponding to the cloud statistics identification in the cloud statistics file; and restoring the cloud object data corresponding to the cloud statistics identification to obtain to-be-counted data corresponding to the cloud statistics identification, carrying out statistics calculation based on the to-be-counted data to obtain a statistics result, and returning the statistics result to a request end corresponding to the data statistics request.

In one embodiment, the data migration apparatus 1500 further comprises:

the deleting module is used for acquiring a data deleting request, wherein the data deleting request carries a data identifier to be deleted, and converting the data identifier to be deleted into a cloud data identifier to obtain the cloud deleting identifier; determining a cloud deletion file identifier corresponding to a cloud deletion identifier based on target cloud data corresponding to the cloud object data file, and acquiring a cloud deletion file from a cloud object data storage space based on target index information corresponding to the cloud deletion file identifier; deleting cloud object data corresponding to the cloud deletion identification in the cloud deletion file to obtain a cloud update file, and determining update metadata and update index information corresponding to the cloud update file; and storing the update metadata and the update index information in an associated manner, and carrying out data migration on the cloud update file to the cloud object data storage space.

In one embodiment, the data migration apparatus 1500 further comprises:

the updating module is used for acquiring a data updating request, wherein the data updating request carries a data identifier to be updated and updating data, performing cloud data identifier conversion on the data identifier to be updated to obtain a cloud updating identifier, and performing object data conversion on the updating data to obtain a cloud updating file; determining a cloud storage file identifier corresponding to a cloud update identifier based on target cloud data corresponding to the cloud object data file, and acquiring a cloud storage file from a cloud object data storage space based on target index information corresponding to the cloud storage file identifier; combining the cloud updating file and the cloud storage file to obtain a target updating file, and determining target updating metadata and target updating index information corresponding to the target updating file; and storing the target update metadata and the target update index information in an associated manner, and performing data migration on the target update file to the cloud object data storage space.

The various modules in the data migration apparatus described above may be implemented in whole or in part by software, hardware, and combinations thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.

In one embodiment, a computer device is provided, which may be a server, and the internal structure of which may be as shown in fig. 16. The computer device includes a processor, a memory, an Input/Output interface (I/O) and a communication interface. The processor, the memory and the input/output interface are connected through a system bus, and the communication interface is connected to the system bus through the input/output interface. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of the computer device is used for storing target index information, target metadata, thermal data and the like. The input/output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a data migration method.

In one embodiment, a computer device is provided, which may be a terminal, and the internal structure thereof may be as shown in fig. 17. The computer device includes a processor, a memory, an input/output interface, a communication interface, a display unit, and an input means. The processor, the memory and the input/output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input/output interface. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The input/output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used for carrying out wired or wireless communication with an external terminal, and the wireless mode can be realized through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. The computer program is executed by a processor to implement a data migration method. The display unit of the computer equipment is used for forming a visual picture, and can be a display screen, a projection device or a virtual reality imaging device, wherein the display screen can be a liquid crystal display screen or an electronic ink display screen, the input device of the computer equipment can be a touch layer covered on the display screen, can also be a key, a track ball or a touch pad arranged on a shell of the computer equipment, and can also be an external keyboard, a touch pad or a mouse and the like.

It will be appreciated by those skilled in the art that the structures shown in fig. 16 or 17 are merely block diagrams of portions of structures associated with the present inventive arrangements and are not limiting of the computer device to which the present inventive arrangements may be implemented, and that a particular computer device may include more or fewer components than shown, or may be combined with certain components, or may have different arrangements of components.

In an embodiment, there is also provided a computer device comprising a memory and a processor, the memory having stored therein a computer program, the processor implementing the steps of the method embodiments described above when the computer program is executed.

In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored which, when executed by a processor, carries out the steps of the method embodiments described above.

In an embodiment, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the steps of the method embodiments described above.

It should be noted that, the user information (including but not limited to user equipment information, user personal information, etc.) and the data (including but not limited to data for analysis, stored data, presented data, etc.) related to the present application are information and data authorized by the user or sufficiently authorized by each party, and the collection, use and processing of the related data need to comply with the related laws and regulations and standards of the related country and region.

Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, database, or other medium used in embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high density embedded nonvolatile Memory, resistive random access Memory (ReRAM), magnetic random access Memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric Memory (Ferroelectric Random Access Memory, FRAM), phase change Memory (Phase Change Memory, PCM), graphene Memory, and the like. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, and the like. By way of illustration, and not limitation, RAM can be in the form of a variety of forms, such as Static Random access memory (Static Random access memory AccessMemory, SRAM) or dynamic Random access memory (Dynamic Random Access Memory, DRAM), and the like. The databases referred to in the embodiments provided herein may include at least one of a relational database and a non-relational database. The non-relational database may include, but is not limited to, a blockchain-based distributed database, and the like. The processor referred to in the embodiments provided in the present application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, or the like, but is not limited thereto.

The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.

The foregoing examples illustrate only a few embodiments of the application and are described in detail herein without thereby limiting the scope of the application. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the application, which are all within the scope of the application. Accordingly, the scope of the application should be assessed as that of the appended claims.

Claims

1. A method of data migration, the method comprising:

converting the type of the data identifier in the cold data set into the type of the cloud data identifier, and performing unique conversion on the data identifier after the type conversion to obtain a target cold data set;

Performing object data conversion based on the target cold data set to obtain a cloud object data file, and determining target metadata and target index information corresponding to the cloud object data file, wherein the target metadata are used for describing a cloud data identification range corresponding to the cloud object data file, and the target index information is used for determining the position of the cloud object data file in the cloud object data storage space;

storing the target metadata and target index information in an associated manner, wherein the target metadata and the target index information are used for accessing cloud object data files in the cloud object data storage space;

performing data migration on the cloud object data file, the target metadata and the target index information to a cloud object data storage space, and deleting the cold data set when the data migration is completed;

acquiring a data query request, wherein the data query request carries a query data identifier, and converting the query data identifier into a cloud data identifier to obtain a cloud query identifier;

determining a cloud query file identifier corresponding to the cloud query identifier from at least two cloud object data files based on target cloud data corresponding to the at least two cloud object data files;

Acquiring the cloud query file from the cloud object data storage space based on target index information corresponding to the cloud query file identification, and querying corresponding target cloud object data in the cloud query file based on the cloud query identification;

and restoring the target cloud object data to obtain target query data corresponding to the query data identifier, and returning the target query data to a request end corresponding to the data query request.

2. The method of claim 1, wherein converting the type of the data identifier in the cold dataset into the type of the cloud data identifier, and performing unique conversion on the type-converted data identifier to obtain the target cold dataset, comprises:

converting the data types corresponding to the data identifiers in the cold data set to obtain the data identifiers of the target data types;

acquiring a sequence identifier, and combining the sequence identifier with the data identifier of the target data type to obtain a cloud data identifier;

and updating the data identifier in the cold data set into the cloud data identifier to obtain the target cold data set.

3. The method of claim 1, wherein determining target metadata and target index information corresponding to the cloud object data file comprises:

Acquiring a cloud data identification upper limit value and a cloud data identification lower limit value in the cloud object data file, and determining target metadata corresponding to the cloud object data file based on the cloud data identification upper limit value and the cloud data identification lower limit value;

and acquiring the position information of the cloud object data file in the cloud object data storage space, and determining target index information corresponding to the cloud object data file based on the position information.

4. The method according to claim 1, characterized in that the method further comprises:

hash mapping is carried out on cloud data identifiers in the cloud object data file, and hash values corresponding to the cloud data identifiers are obtained;

generating a target bit array corresponding to the cloud data identifier based on the hash values, and determining the identifier presence detection information corresponding to the cloud object data file based on the target bit array;

and storing the identification presence detection information, the target metadata and the target index information in an associated mode.

5. The method of claim 1, wherein said deleting the cold data set comprises:

and updating and checking the cold data in the cold data set, and deleting the cold data set through a parallel asynchronous thread when the cold data in the cold data set is not updated.

6. The method of claim 1, wherein the cloud object data file comprises at least two;

after the cloud object data file, the target metadata and the target index information are subjected to data migration to a cloud object data storage space, and the cold data set is deleted when the data migration is completed, the method further comprises the steps of:

detecting target metadata corresponding to the at least two cloud object data files, and acquiring each range coincidence file from the cloud object data storage space based on the target metadata corresponding to the at least two cloud object data files and target index information corresponding to the at least two cloud object data files when cloud data identification ranges in the target metadata coincide;

merging the range overlapping files to obtain a merged file, and determining merging metadata and merging index information corresponding to the merged file;

and storing the merging metadata and the merging index information in an associated mode, and performing data migration on the merging file to the cloud object data storage space.

7. The method according to claim 1, characterized in that the method further comprises:

When the query data identifier is a cloud-stored data identifier, performing cloud data identifier conversion on the query data identifier through a preset proxy service to obtain a cloud query identifier;

determining a cloud query file identifier corresponding to the cloud query identifier from at least two cloud object data files by using target cloud data corresponding to the at least two cloud object data files through a preset proxy service;

and acquiring the cloud query file from the cloud object data storage space by using target index information corresponding to the cloud query file identification through a preset proxy service.

8. The method of claim 1, wherein the determining, from at least two cloud object data files, the cloud query file identifier corresponding to the cloud query identifier based on the target cloud data corresponding to the at least two cloud object data files, comprises:

searching each candidate cloud object data file identification of the cloud query identification in a cloud data identification range corresponding to the cloud object data file from the at least two cloud object data files;

and acquiring identification presence detection information corresponding to the at least two cloud object data files, and filtering the candidate cloud object data file identifications through the cloud query identifications based on the identification presence detection information to obtain cloud query file identifications corresponding to the cloud query identifications.

9. The method of claim 1, wherein the obtaining the cloud query file from the cloud object data storage space based on the target index information corresponding to the cloud query file identification, and querying the cloud query file for the corresponding target cloud object data based on the cloud query identification, comprises:

acquiring file index information corresponding to the cloud query file identification from the cloud object data storage space based on target index information corresponding to the cloud query file identification, wherein the file index information comprises each cloud data identification and an object data storage position corresponding to each cloud data identification;

searching a query object data storage position corresponding to the cloud query identifier from the file index information, and acquiring target cloud object data corresponding to the cloud query identifier from the cloud object data storage space based on the query object data storage position.

10. The method of claim 1, further comprising, after said migrating the cloud object data file, the target metadata, and the target index information to a cloud object data storage space and deleting the cold data set when the data migration is completed:

Acquiring a data statistics request, wherein the data statistics request carries a data identifier to be counted, and performing cloud data identifier conversion on the data identifier to be counted to obtain a cloud statistics identifier;

determining a cloud statistics file identifier corresponding to the cloud statistics identifier based on target cloud data corresponding to the cloud object data file;

acquiring the cloud statistics file from the cloud object data storage space based on target index information corresponding to the cloud statistics file identification, and searching cloud object data corresponding to the cloud statistics identification in the cloud statistics file;

and restoring the cloud object data corresponding to the cloud statistics identification to obtain data to be counted corresponding to the cloud statistics identification, performing statistics calculation based on the data to be counted to obtain a statistics result, and returning the statistics result to a request end corresponding to the data statistics request.

11. The method of claim 1, further comprising, after said migrating the cloud object data file, the target metadata, and the target index information to a cloud object data storage space and deleting the cold data set when the data migration is completed:

Acquiring a data deleting request, wherein the data deleting request carries a data identifier to be deleted, and converting the data identifier to be deleted into a cloud data identifier to obtain a cloud deleting identifier;

determining a cloud deletion file identifier corresponding to the cloud deletion identifier based on target cloud data corresponding to the cloud object data file, and acquiring the cloud deletion file from the cloud object data storage space based on target index information corresponding to the cloud deletion file identifier;

deleting cloud object data corresponding to the cloud deletion identification in the cloud deletion file to obtain a cloud update file, and determining update metadata and update index information corresponding to the cloud update file;

and storing the update metadata and the update index information in an associated mode, and performing data migration on the cloud update file to the cloud object data storage space.

12. The method of claim 1, further comprising, after said migrating the cloud object data file, the target metadata, and the target index information to a cloud object data storage space and deleting the cold data set when the data migration is completed:

Acquiring a data updating request, wherein the data updating request carries a data identifier to be updated and updating data, performing cloud data identifier conversion on the data identifier to be updated to obtain a cloud updating identifier, and performing object data conversion on the updating data to obtain a cloud updating file;

determining a cloud storage file identifier corresponding to the cloud update identifier based on target cloud data corresponding to the cloud object data file, and acquiring the cloud storage file from the cloud object data storage space based on target index information corresponding to the cloud storage file identifier;

combining the cloud updating file and the cloud storage file to obtain a target updating file, and determining target updating metadata and target updating index information corresponding to the target updating file;

and storing the target updating metadata and the target updating index information in an associated mode, and performing data migration on the target updating file to the cloud object data storage space.

13. A data migration apparatus, the apparatus comprising:

The identifier conversion module is used for converting the type of the data identifier in the cold data set into the type of the cloud data identifier, and carrying out unique conversion on the data identifier after the type conversion to obtain a target cold data set;

the data conversion module is used for carrying out object data conversion based on the target cold data set to obtain a cloud object data file, and determining target metadata and target index information corresponding to the cloud object data file, wherein the target metadata are used for describing a cloud data identification range corresponding to the cloud object data file, and the target index information is used for determining the position of the cloud object data file in the cloud object data storage space;

the data migration module is used for carrying out data migration on the cloud object data file, the target metadata and the target index information to a cloud object data storage space, and deleting the cold data set when the data migration is completed;

The query module is used for acquiring a data query request, wherein the data query request carries a query data identifier, and performing cloud data identifier conversion on the query data identifier to obtain a cloud query identifier; determining a cloud query file identifier corresponding to the cloud query identifier from at least two cloud object data files based on target cloud data corresponding to the at least two cloud object data files; acquiring the cloud query file from the cloud object data storage space based on target index information corresponding to the cloud query file identification, and querying corresponding target cloud object data in the cloud query file based on the cloud query identification; and restoring the target cloud object data to obtain target query data corresponding to the query data identifier, and returning the target query data to a request end corresponding to the data query request.

14. The apparatus of claim 13, wherein the identifier conversion module is further configured to convert a data type corresponding to the data identifier in the cold dataset to obtain a data identifier of a target data type; acquiring a sequence identifier, and combining the sequence identifier with the data identifier of the target data type to obtain a cloud data identifier; and updating the data identifier in the cold data set into the cloud data identifier to obtain the target cold data set.

15. The apparatus of claim 13, wherein the data conversion module is further configured to obtain a cloud data identifier upper limit value and a cloud data identifier lower limit value in the cloud object data file, and determine target metadata corresponding to the cloud object data file based on the cloud data identifier upper limit value and the cloud data identifier lower limit value; and acquiring the position information of the cloud object data file in the cloud object data storage space, and determining target index information corresponding to the cloud object data file based on the position information.

16. The apparatus of claim 13, wherein the apparatus further comprises:

the detection information determining module is used for carrying out hash mapping on cloud data identifiers in the cloud object data file to obtain hash values corresponding to the cloud data identifiers; generating a target bit array corresponding to the cloud data identifier based on the hash values, and determining the identifier presence detection information corresponding to the cloud object data file based on the target bit array;

17. The apparatus of claim 13, wherein the data migration module is further configured to update and verify cold data in the cold dataset, and delete the cold dataset by a parallel asynchronous thread when cold data in the cold dataset is not updated.

18. The apparatus of claim 13, wherein the cloud object data file comprises at least two;

the device further comprises:

the merging module is used for detecting target metadata corresponding to the at least two cloud object data files, and acquiring each range coincidence file from the cloud object data storage space based on the target metadata corresponding to the at least two cloud object data files and target index information corresponding to the at least two cloud object data files when cloud data identification ranges in the target metadata coincide; merging the range overlapping files to obtain a merged file, and determining merging metadata and merging index information corresponding to the merged file; and storing the merging metadata and the merging index information in an associated mode, and performing data migration on the merging file to the cloud object data storage space.

19. The apparatus of claim 13, wherein the apparatus further comprises:

the agent module is used for carrying out cloud data identification conversion on the query data identification through a preset agent service when the query data identification is a cloud-stored data identification, so as to obtain a cloud query identification; determining a cloud query file identifier corresponding to the cloud query identifier from at least two cloud object data files by using target cloud data corresponding to the at least two cloud object data files through a preset proxy service; and acquiring the cloud query file from the cloud object data storage space by using target index information corresponding to the cloud query file identification through a preset proxy service.

20. The apparatus of claim 13, wherein the query module is further configured to search the at least two cloud object data files for each candidate cloud object data file identification of the cloud query identification within a cloud data identification range corresponding to the cloud object data file; and acquiring identification presence detection information corresponding to the at least two cloud object data files, and filtering the candidate cloud object data file identifications through the cloud query identifications based on the identification presence detection information to obtain cloud query file identifications corresponding to the cloud query identifications.

21. The apparatus of claim 13, wherein the query module is further configured to obtain file index information corresponding to the cloud query file identifier from the cloud object data storage space based on target index information corresponding to the cloud query file identifier, the file index information including each cloud data identifier and an object data storage location corresponding to each cloud data identifier; searching a query object data storage position corresponding to the cloud query identifier from the file index information, and acquiring target cloud object data corresponding to the cloud query identifier from the cloud object data storage space based on the query object data storage position.

22. The apparatus of claim 13, wherein the apparatus further comprises:

the statistics module is used for acquiring a data statistics request, wherein the data statistics request carries a data identifier to be counted, and the data identifier to be counted is subjected to cloud data identifier conversion to obtain a cloud statistics identifier; determining a cloud statistics file identifier corresponding to the cloud statistics identifier based on target cloud data corresponding to the cloud object data file; acquiring the cloud statistics file from the cloud object data storage space based on target index information corresponding to the cloud statistics file identification, and searching cloud object data corresponding to the cloud statistics identification in the cloud statistics file; and restoring the cloud object data corresponding to the cloud statistics identification to obtain data to be counted corresponding to the cloud statistics identification, performing statistics calculation based on the data to be counted to obtain a statistics result, and returning the statistics result to a request end corresponding to the data statistics request.

23. The apparatus of claim 13, wherein the apparatus further comprises:

the deleting module is used for acquiring a data deleting request, wherein the data deleting request carries a data identifier to be deleted, and converting the data identifier to be deleted into a cloud data identifier to obtain the cloud deleting identifier; determining a cloud deletion file identifier corresponding to the cloud deletion identifier based on target cloud data corresponding to the cloud object data file, and acquiring the cloud deletion file from the cloud object data storage space based on target index information corresponding to the cloud deletion file identifier; deleting cloud object data corresponding to the cloud deletion identification in the cloud deletion file to obtain a cloud update file, and determining update metadata and update index information corresponding to the cloud update file; and storing the update metadata and the update index information in an associated mode, and performing data migration on the cloud update file to the cloud object data storage space.

24. The apparatus of claim 13, wherein the apparatus further comprises:

the updating module is used for acquiring a data updating request, wherein the data updating request carries a data identifier to be updated and updating data, performing cloud data identifier conversion on the data identifier to be updated to obtain a cloud updating identifier, and performing object data conversion on the updating data to obtain a cloud updating file; determining a cloud storage file identifier corresponding to the cloud update identifier based on target cloud data corresponding to the cloud object data file, and acquiring the cloud storage file from the cloud object data storage space based on target index information corresponding to the cloud storage file identifier; combining the cloud updating file and the cloud storage file to obtain a target updating file, and determining target updating metadata and target updating index information corresponding to the target updating file; and storing the target updating metadata and the target updating index information in an associated mode, and performing data migration on the target updating file to the cloud object data storage space.

25. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any one of claims 1 to 12 when the computer program is executed.

26. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 12.