CN116166636A - Data migration method, device, equipment and readable storage medium - Google Patents

Data migration method, device, equipment and readable storage medium Download PDF

Info

Publication number
CN116166636A
CN116166636A CN202310175116.8A CN202310175116A CN116166636A CN 116166636 A CN116166636 A CN 116166636A CN 202310175116 A CN202310175116 A CN 202310175116A CN 116166636 A CN116166636 A CN 116166636A
Authority
CN
China
Prior art keywords
transaction
operation information
data
data blocks
log files
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202310175116.8A
Other languages
Chinese (zh)
Inventor
洪少辉
翁世清
刘飞
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
China Construction Bank Corp
CCB Finetech Co Ltd
Original Assignee
China Construction Bank Corp
CCB Finetech Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by China Construction Bank Corp, CCB Finetech Co Ltd filed Critical China Construction Bank Corp
Priority to CN202310175116.8A priority Critical patent/CN116166636A/en
Publication of CN116166636A publication Critical patent/CN116166636A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21—Design, administration or maintenance of databases
    • G06F16/214—Database migration support
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0638—Organizing or formatting or addressing of data
    • G06F3/064—Management of blocks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
    • G06F3/0647—Migration mechanisms
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
    • G06F3/0656—Data buffering arrangements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/0671—In-line storage system
    • G06F3/0673—Single storage device
    • G06F3/0674—Disk device
    • G06F3/0676—Magnetic disk device
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Databases & Information Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The application provides a data migration method, a device, equipment and a readable storage medium, and relates to the technical field of big data. The method comprises the following steps: acquiring a plurality of log files for operating a source database; analyzing the log files to obtain a plurality of pieces of operation information, wherein the operation information comprises operation sentences, and identifiers and global sequence numbers of transactions corresponding to the operation sentences; sorting the plurality of pieces of operation information according to the global sequence number to obtain sorted pieces of operation information, and determining at least one piece of operation information corresponding to each transaction in the sorted pieces of operation information; generating a transaction file according to at least one piece of operation information corresponding to each transaction, and sending the transaction file to target equipment. And the migration efficiency of the log file is improved.

Description

Data migration method, device, equipment and readable storage medium
Technical Field
The present disclosure relates to the field of big data technologies, and in particular, to a data migration method, apparatus, device, and readable storage medium.
Background
As data storage requirements expand, databases are of increasing types, and if the data storage database changes, the data in the source database needs to be migrated to the target device.
In the process of database migration, log files in the source database also need to be migrated to the target device. In the related art, the log file may be migrated through the structured data replication software of the log, however, when the log file is migrated through the structured data replication software, more other components need to be deployed, resulting in lower migration efficiency of the log file.
Disclosure of Invention
The application provides a data migration method, a device, equipment and a readable storage medium, which are used for solving the problem of low migration efficiency of log files.
In a first aspect, the present application provides a data migration method, including:
acquiring a plurality of log files for operating a source database;
analyzing the log files to obtain a plurality of pieces of operation information, wherein the operation information comprises operation sentences, and identifiers and global sequence numbers of transactions corresponding to the operation sentences;
sorting the plurality of pieces of operation information according to the global sequence number to obtain sorted pieces of operation information, and determining at least one piece of operation information corresponding to each transaction in the sorted pieces of operation information;
Generating a transaction file according to at least one piece of operation information corresponding to each transaction, and sending the transaction file to target equipment.
In one possible implementation, obtaining a plurality of log files operating on a source database includes:
acquiring a log file downloading progress;
and downloading a plurality of data blocks in the source database according to the log file downloading progress, and storing the plurality of data blocks in a disk in the form of log files to obtain the plurality of log files.
In one possible implementation, the downloading a plurality of data blocks in the source database and storing the plurality of data blocks in a disk in the form of a log file to obtain the plurality of log files includes:
downloading the plurality of data blocks in parallel through a plurality of downloading threads, and storing the plurality of data blocks into a data block queue;
the method comprises the steps that a plurality of data blocks are obtained from a data block queue through an ordering thread, and are subjected to ordering processing according to data block information of the plurality of data blocks, so that a plurality of ordered data blocks are obtained, wherein the data block information comprises identifications of log files corresponding to the data blocks and serial numbers of the data blocks in the corresponding log files;
And storing the plurality of ordered data blocks to a disk in the form of log files to obtain the plurality of log files.
In one possible implementation, storing the plurality of ordered data blocks to disk in the form of log files to obtain the plurality of log files includes:
storing the data blocks into a cache according to the sequence of the data blocks, and storing the data blocks in the cache into a disk when the data amount in the cache is greater than or equal to a first threshold value until the log files are obtained.
In one possible implementation manner, after determining at least one operation information corresponding to each transaction in the sequenced pieces of operation information, the method further includes:
for any one transaction, if the memory occupied by at least one piece of operation information corresponding to the transaction is larger than or equal to a second threshold value, storing the at least one piece of operation information corresponding to the transaction to a disk in a file form to obtain a transaction file corresponding to the transaction;
and if the memory occupied by at least one piece of operation information corresponding to the transaction is smaller than the second threshold value, storing the at least one piece of operation information corresponding to the transaction into a cache until the data volume in the cache is larger than or equal to a third threshold value, and storing the operation information in the cache into a disk in a file form to obtain a corresponding transaction file.
In one possible embodiment, the method further comprises:
determining a first transaction to be rolled back;
deleting the first transaction in the cache if the first transaction is in the cache;
and if the first transaction is located in the disk, deleting the first transaction in the disk.
In a second aspect, the present application provides a data migration apparatus, including an acquisition module, a first processing module, a second processing module, a first determining module, a generating module, and a sending module, where:
the acquisition module is used for acquiring a plurality of log files for operating the source database;
the first processing module is used for analyzing the log files to obtain a plurality of pieces of operation information, wherein the operation information comprises operation sentences, and identifiers and global sequence numbers of transactions corresponding to the operation sentences;
the second processing module is used for sorting the plurality of pieces of operation information according to the global sequence number to obtain sorted pieces of operation information,
the first determining module is used for determining at least one piece of operation information corresponding to each transaction in the sequenced pieces of operation information;
The generating module is used for generating a transaction file according to at least one piece of operation information corresponding to each transaction,
the sending module is used for sending the transaction file to target equipment.
In one possible implementation manner, the acquiring module is specifically configured to:
acquiring a log file downloading progress;
and downloading a plurality of data blocks in the source database according to the log file downloading progress, and storing the plurality of data blocks in a disk in the form of log files to obtain the plurality of log files.
In one possible implementation manner, the acquiring module is specifically configured to:
downloading the plurality of data blocks in parallel through a plurality of downloading threads, and storing the plurality of data blocks into a data block queue;
the method comprises the steps that a plurality of data blocks are obtained from a data block queue through an ordering thread, and are subjected to ordering processing according to data block information of the plurality of data blocks, so that a plurality of ordered data blocks are obtained, wherein the data block information comprises identifications of log files corresponding to the data blocks and serial numbers of the data blocks in the corresponding log files;
and storing the plurality of ordered data blocks to a disk in the form of log files to obtain the plurality of log files.
In one possible implementation manner, the acquiring module is specifically configured to:
storing the data blocks into a cache according to the sequence of the data blocks, and storing the data blocks in the cache into a disk when the data amount in the cache is greater than or equal to a first threshold value until the log files are obtained.
In one possible embodiment, the apparatus further comprises a first storage module and a second storage module, wherein:
the first storage module is specifically configured to store, for any one transaction, at least one operation information corresponding to the transaction in a file form to a disk if a memory occupied by at least one operation information corresponding to the transaction is greater than or equal to a second threshold, so as to obtain a transaction file corresponding to the transaction;
the second storage module is specifically configured to store, if the memory occupied by at least one operation information corresponding to the transaction is smaller than the second threshold, at least one operation information corresponding to the transaction to the cache until the data amount in the cache is greater than or equal to a third threshold, and store the operation information in the cache to the disk in a file form to obtain a corresponding transaction file.
In a possible implementation manner, the apparatus further includes a second determining module, a first deleting module, and a second deleting module, where:
the second determining module is specifically configured to determine a first transaction to be rolled back;
the first deleting module is specifically configured to delete the first transaction in the cache if the first transaction is in the cache;
the second deleting module is specifically configured to delete the first transaction in the disk if the first transaction is located in the disk.
In a third aspect, an embodiment of the present application provides an electronic device, including: a memory and a processor, wherein the memory is configured to store,
the memory stores computer-executable instructions;
the processor executing computer-executable instructions stored in the memory causes the processor to perform the data migration method of any one of the first aspects.
In a fourth aspect, embodiments of the present application provide a computer-readable storage medium having stored therein computer-executable instructions for implementing the data migration method of any one of the first aspects when the computer-executable instructions are executed by a processor.
In a fifth aspect, embodiments of the present application provide a computer program product comprising a computer program which, when executed by a processor, implements the data migration method of any one of the first aspects.
According to the data migration method, the device, the equipment and the readable storage medium, a plurality of log files for operating the source database are obtained, analysis processing is carried out on the plurality of log files, the obtained pieces of operation information are subjected to ordering processing according to the global sequence number, at least one piece of operation information corresponding to each transaction is determined in the ordered pieces of operation information, a transaction file is generated, and the transaction file is sent to the target equipment, so that migration efficiency of the log files is improved.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and together with the description, serve to explain the principles of the application.
Fig. 1 is a schematic diagram of an application scenario provided in an embodiment of the present application;
fig. 2 is a schematic flow chart of a data migration method according to an embodiment of the present application;
fig. 3 is a schematic flow chart of determining a data segment to be downloaded according to an embodiment of the present application;
FIG. 4 is a schematic flow chart of acquiring multiple data blocks in a disk according to an embodiment of the present application;
FIG. 5 is a schematic flow chart of a parsing thread according to an embodiment of the present disclosure;
FIG. 6 is a schematic flow chart of sending a transaction file according to an embodiment of the present application;
FIG. 7 is a flowchart illustrating another data migration method according to an embodiment of the present disclosure;
FIG. 8 is a schematic flow chart of data downloading according to an embodiment of the present application;
FIG. 9 is a schematic flow chart of data block ordering according to an embodiment of the present application;
FIG. 10 is a flowchart illustrating another data migration method according to an embodiment of the present disclosure;
FIG. 11 is a schematic flow chart of a sorting process according to an embodiment of the present disclosure;
fig. 12 is a schematic structural diagram of a data migration device according to an embodiment of the present application;
fig. 13 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Specific embodiments thereof have been shown by way of example in the drawings and will herein be described in more detail. These drawings and the written description are not intended to limit the scope of the inventive concepts in any way, but to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments.
Detailed Description
Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The implementations described in the following exemplary examples are not representative of all implementations consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present application as detailed in the accompanying claims.
Fig. 1 is a schematic diagram of an application scenario provided in an embodiment of the present application. Referring to fig. 1, the system includes a database 101, a processing device 102, and a target device 103.
The database 101 may be used to store service data, the log file may record the variation of the service data in the database 101, and the data storage location may be determined in the database according to the log file.
The processing device 102 may migrate data in the database 101 to the target device 103, migrate data by using multiple threads, parse the log file according to the log file of the database 101, sort the parsed operation information, determine the operation information corresponding to each transaction, and finally send the transaction file to the target device 103. The target device 103 may obtain the transaction file according to the requirement, and the target device 103 may include a database, a terminal device, and the like.
In the related art, the log file may be migrated through the structured data replication software of the log, however, when the log file is migrated through the structured data replication software, more other components need to be deployed, resulting in lower migration efficiency of the log file.
In the embodiment of the application, a plurality of log files operating on a source database can be acquired, the log files are analyzed, the acquired operation information is subjected to sorting according to a global sequence number, at least one piece of operation information corresponding to each transaction is determined in the sorted operation information, a transaction file is generated, and the transaction file is sent to a target device. In the process, the migration efficiency of the log file is improved.
The method shown in the present application will be described below by way of specific examples. It should be noted that the following embodiments may exist alone or in combination with each other, and for the same or similar content, the description will not be repeated in different embodiments.
Fig. 2 is a flow chart of a data migration method according to an embodiment of the present application. Referring to fig. 2, the method may include:
s201, acquiring a plurality of log files for operating a source database.
The execution body of the embodiment of the application may be a processing device, or may be a data migration device disposed in the processing device. The data migration device may be implemented by software, or may be implemented by a combination of software and hardware.
A plurality of log files operating on a source database may be obtained according to the following: acquiring a log file downloading progress; and downloading a plurality of data blocks in a source database according to the log file downloading progress, and storing the plurality of data blocks in a disk in the form of log files to obtain a plurality of log files.
The operation process can be continuously recorded in the source database through the log file, the processing equipment needs to judge the downloading progress of the log file in the process of acquiring the log file, and the data blocks which are not downloaded are determined according to the downloading progress of the log file.
The log file may include a plurality of data blocks therein.
For example, assuming that 3 log files, namely log files 1-3, are generated in the source database at time 1, and at this time, the log file downloading progress is 2, it may be determined that a plurality of data blocks corresponding to the log files 1 and 2 have been downloaded, and that a plurality of data blocks corresponding to the log file 3 have not been downloaded.
In order to facilitate the description of the embodiment of the present application, a process of determining a data segment to be downloaded according to the log file downloading progress in the present application will be described below with reference to fig. 3.
Fig. 3 is a schematic flow chart of determining a data segment to be downloaded according to an embodiment of the present application. The actual progress of the log file in the source database is greater than the progress the processing device is able to perceive the log file. After the log file progress is perceived, a plurality of data blocks corresponding to the log file to be downloaded are determined according to the current log file downloading progress, and the data blocks are the data segments to be downloaded. According to the fact that the actual progress of the log file is continuously increased, the perceived progress of the log file can be continuously updated, so that a new data segment to be downloaded can be determined, and the data segment to be downloaded is divided into a plurality of threads to download a plurality of data blocks.
The data segment to be downloaded may include an identification of the corresponding log file and a data segment offset corresponding to the identification of the log file.
S202, analyzing the plurality of log files to obtain a plurality of pieces of operation information.
And analyzing and processing a plurality of data blocks in any one log file to obtain the operation information corresponding to the data blocks.
The operation information may include an operation statement, and an identification and a global sequence number of a transaction to which the operation statement corresponds.
For example, assuming that there are 2 log files, namely, log file 1 and log file 2, respectively, the log file includes data block 1 and data block 2, the log file 2 includes data block 3, and the operation information shown in table 1 can be obtained after analyzing the 3 data blocks.
TABLE 1
Figure BDA0004100631050000071
The global sequence number may refer to a number of an arrangement order of the plurality of data blocks.
After the data blocks are analyzed by the plurality of analysis threads, the arrangement sequence of the plurality of operation information is different from the arrangement sequence of the plurality of data blocks due to different time when the analysis of the data blocks is completed.
For example, assuming that there are 3 data blocks in total, the order of the data blocks 1-3 in the data block queue is data block 1, data block 2 and data block 3, the global sequence numbers are 1, 2 and 3, respectively, after analysis by a plurality of analysis threads, the data block 1 corresponds to operation information 1, the data block 2 corresponds to operation information 2 and the data block 3 corresponds to operation information 3, and assuming that the order of the analysis completion time of the data blocks 1-3 is data block 1, data block 3 and data block 2, the order of the operation information in the operation information queue is operation information 1, operation information 3 and operation information 2, that is, the order of the operation information after analysis is different from the order of the data blocks.
The plurality of operation information after parsing disorder may be ordered by a global sequence number.
A transaction may refer to a business event that may include multiple operational processes, i.e., the transaction may correspond to multiple operational information. For example, assuming that the transaction 1 is a certain business transaction, the content of the business transaction may include a data increment a of the account 1 and a data decrement a of the account 2, where the data increment a of the account 1 is the operation information 1 and the data decrement a of the account 2 is the operation information 2, that is, the operation information corresponding to the transaction 1 is the operation information 1 and the operation information 2.
Since a plurality of log files are stored in the disk, a plurality of log files can be acquired in the disk, and a plurality of data blocks are determined based on the plurality of log files.
For convenience in describing the embodiments of the present application, a process of acquiring a plurality of data blocks in a disk will be described below with reference to fig. 4.
Fig. 4 is a schematic flow chart of acquiring a plurality of data blocks in a disk according to an embodiment of the present application. The disk comprises a plurality of log files, corresponding data can be acquired according to the identification of the log files, and at least one data block can be determined through the data. After the data corresponding to the identification of the log file is obtained, the corresponding data block can be determined according to the identification of the next log file, and the process is circulated, so that a data block queue formed by a plurality of data blocks can be obtained.
Next, a process of analyzing threads in the present application will be described with reference to fig. 5.
Fig. 5 is a schematic flow chart of a parsing thread according to an embodiment of the present application. The method comprises the steps that a data block can be obtained from a data block queue, operation information can be obtained by analyzing the data block, the operation information comprises operation sentences, and identification and global sequence numbers of transactions corresponding to the operation sentences, the operation information is put into a key information set after a memory lock is added, the memory lock is released, the key information set can be read and cannot be modified by the memory lock, the analysis progress can be updated through the global sequence numbers, and a plurality of analysis threads can conveniently conduct analysis processes at the same time by updating the analysis progress.
S203, sorting the plurality of pieces of operation information according to the global sequence number to obtain sorted pieces of operation information.
After the plurality of analysis threads analyze the plurality of data blocks, the plurality of pieces of operation information can be ordered according to the global sequence number, so that the plurality of pieces of operation information are arranged according to the operation sequence recorded in the log file.
For example, assuming that there are 3 pieces of operation information, namely operation information 1 to 3, respectively, the operation sentence, the transaction identifier, and the global sequence number may be as shown in table 2, and the arrangement order as shown in table 3 may be obtained after the order according to the global sequence number.
TABLE 2
Operation information Operation statement Identification of transactions Global sequence number
Operation information
1 Operation sentence 1 Transaction 1 3
Operation information 2 Operation statement 2 Transaction 2 2
Operation information 3 Operation sentence 3 Transaction 2 1
TABLE 3 Table 3
Operation information Operation statement Identification of transactions Global sequence number
Operation information
3 Operation sentence 3 Transaction 2 1
Operation information 2 Operation statement 2 Transaction 2 2
Operation information 1 Operation sentence 1 Transaction 1 3
S204, determining at least one piece of operation information corresponding to each transaction in the sequenced pieces of operation information.
At least one piece of operation information corresponding to each transaction can be determined according to the identification of the transaction, and the at least one piece of operation information is arranged according to the operation sequence.
For example, assuming that the ordered pieces of operation information are shown in table 3, according to the identification of the transaction, it may be determined that the operation information corresponding to the transaction 2 is operation information 3 and operation information 2, where the operation time of the operation information 3 is earlier than the operation time of the operation information 2; the operation information corresponding to the transaction 1 is operation information 1.
S205, generating a transaction file according to at least one piece of operation information corresponding to each transaction.
At least one operation information corresponding to the transaction can be stored to the disk in the form of a transaction file after being subjected to persistence processing.
Persistence processing may refer to storing temporarily stored data to disk, which may be changed to data stored for a long period of time.
Any one transaction file may include operation information corresponding to at least one transaction.
The memory occupied by the transaction file can be preset according to the requirement, and the preset value is not limited in the embodiment of the application.
For example, assuming that the preset value of the transaction file may be 300M, and operation information and memory corresponding to a plurality of transactions may be as shown in table 4, a transaction file 1 and a transaction file 2 may be generated, where the transaction file 1 includes a transaction 1 and a transaction 2, and the transaction file 2 includes a transaction 3.
TABLE 4 Table 4
Figure BDA0004100631050000101
S206, sending the transaction file to the target device.
For ease of description of the embodiments of the present application, the process of sending a transaction file to a target device is described below in conjunction with fig. 6.
Fig. 6 is a schematic flow chart of sending a transaction file according to an embodiment of the present application. After any transaction file is acquired, the data of the transaction file is read, an analysis result (operation information) is determined according to the data of the transaction file, the analysis result is stored in an analysis result queue, the process is circulated, the analysis result queue can comprise a plurality of operation information, and the analysis result queue is sent to target equipment.
After any transaction file is sent to the target device, the transaction file may be deleted from disk.
According to the data migration method, after the plurality of log files operated by the source database are obtained, the plurality of log files can be analyzed to obtain the plurality of pieces of operation information, the operation information comprises operation sentences, and the identification and the global sequence number of the transaction corresponding to the operation sentences, at least one piece of operation information corresponding to the transaction can be determined according to the identification and the global sequence number of the transaction, the transaction file is generated, and the transaction file is sent to the target device, so that migration efficiency of the log files is improved.
Fig. 7 is a flowchart of another data migration method according to an embodiment of the present application. Referring to fig. 7, the method may include:
s701, acquiring a log file downloading progress.
The execution of S701 may refer to the execution of S201, and will not be described here.
S702, downloading a plurality of data blocks in parallel through a plurality of downloading threads, and storing the plurality of data blocks into a data block queue.
Any one data segment to be downloaded can acquire corresponding data according to the identification of the log file and the offset of the data segment, and the data is determined to be a plurality of data blocks.
Next, a data downloading process by any one of the downloading threads in the present application will be described with reference to fig. 8.
Fig. 8 is a schematic flow chart of data downloading according to an embodiment of the present application. The queue of the data segments to be downloaded comprises a plurality of data segments to be downloaded. The data segment to be downloaded can be obtained from a queue of the data segment to be downloaded, the data corresponding to the data segment to be downloaded is obtained from a source database according to the identification of the log file in the data segment to be downloaded and the data segment offset corresponding to the identification of the log file, a plurality of data blocks are determined by the data, and then the data blocks are put into the data block queue. After the data corresponding to the data segment to be downloaded is obtained in the source database, a new data segment to be downloaded can be obtained again, and the above process is repeated. The data block queue includes a plurality of data blocks.
Because the downloading process is executed by the plurality of downloading threads concurrently, a plurality of data blocks can be obtained at the same time, so that the arrangement sequence of the plurality of data blocks in the obtained data block queue is unordered.
For example, suppose that there are 2 threads concurrently executing the downloading process, namely thread 1 and thread 2 respectively, and there are 2 data segments to be downloaded, namely data segment 1 to be downloaded and data segment 2 to be downloaded respectively; the method comprises the steps that data corresponding to a data segment 1 to be downloaded is downloaded in a source database through a thread 1, a data block 1 is obtained at a moment 1, and a data block 2 is obtained at a moment 2; and downloading data corresponding to the data segment 1 to be downloaded in the source database through the thread 2, obtaining a data block 3 at a time 3, and if the time 1 is earlier than the time 3 and the time 3 is earlier than the time 2, ordering the data blocks in the queue of the data blocks into the data block 1, the data block 3 and the data block 2.
S703, acquiring a plurality of data blocks in a data block queue through an ordering thread.
S704, sorting the plurality of data blocks according to the data block information of the plurality of data blocks to obtain a plurality of ordered data blocks.
The data block information may include an identification of the log file to which the data block corresponds, and a sequence number of the data block in the corresponding log file.
At least one corresponding data block can be determined through the identification of the log file, and then the data blocks in the log file are ordered according to the serial numbers of the data blocks in the corresponding log file.
For example, assuming that there are 3 data blocks in total, the data block information may be shown in table 5, the data block corresponding to the log file 1 is the data block 1, the data block corresponding to the log file 2 is the data block 1 and the data block 2, and then the data blocks are ordered according to the sequence numbers in the log file, the obtained ordered data blocks may be shown in table 6.
TABLE 5
Data block Identification of log files Sequence number in log file
Data block
1 Log file 1 11
Data block 2 Log file 2 22
Data block 3 Log file 2 21
TABLE 6
Data block Identification of log files Sequence number in log file
Data block
1 Log file 1 11
Data block 3 Log file 2 21
Data block 2 Log file 2 22
S705, storing the plurality of ordered data blocks to the disk in the form of log files, so as to obtain a plurality of log files.
After a plurality of ordered data blocks are persisted into a disk, in the data analysis process, the data needs to be acquired again in the disk, so that decoupling is performed between the data downloading process and the data analysis process.
And storing the data blocks into a buffer memory according to the sequence of the data blocks, and storing the data blocks in the buffer memory into a disk until a plurality of log files are obtained when the data quantity in the buffer memory is greater than or equal to a first threshold value.
In order to facilitate description of the embodiments of the present application, a description will be made below of a sorting process of a plurality of data blocks in the present application with reference to fig. 9.
Fig. 9 is a schematic flow chart of data block ordering according to an embodiment of the present application. And acquiring data in a queue of the data blocks, wherein the data comprises a plurality of data blocks, and sequencing the plurality of data blocks through the identification of the log file and the serial numbers of the data blocks in the corresponding log file to obtain a plurality of ordered data blocks, and caching the plurality of ordered data blocks.
If the cache is greater than the first threshold, persisting a plurality of ordered data blocks in the cache to disk; if the buffer is smaller than the first threshold, returning to the process of acquiring data until the buffer is larger than the first threshold, and persisting the plurality of ordered data blocks in the buffer to the disk. In addition, if the last data block corresponding to the identification of any one log file is already in the cache, a plurality of ordered data blocks in the cache are persisted to the disk.
After the ordered data blocks in the cache are persisted to the disk, the cache can be emptied, and the step of obtaining the data is returned, so that the data in the data block queue can be persisted to the disk continuously.
The plurality of data blocks in the disk are stored in the form of log files, and any one log file corresponds to at least one data block.
S706, analyzing the plurality of log files to obtain a plurality of pieces of operation information.
And S707, sorting the plurality of pieces of operation information according to the global sequence number to obtain sorted pieces of operation information.
S708, determining at least one piece of operation information corresponding to each transaction in the sequenced pieces of operation information.
The execution of S706-S708 can be referred to as S202-S204, and will not be described here.
S709, for any one transaction, if the memory occupied by at least one operation information corresponding to the transaction is larger than or equal to a second threshold value, storing the at least one operation information corresponding to the transaction to a disk in a file form so as to obtain a transaction file corresponding to the transaction.
For example, assuming that the second threshold may be 100M, if the operation information 1 and the operation information 2 corresponding to the transaction 1 are stored in the disk, the memory of the operation information 1 is 80M, the memory of the operation information 2 is 50M, and the occupied memory of the operation information corresponding to the transaction 1 is greater than the second threshold, the operation information 1 and the operation information 2 corresponding to the transaction 1 are directly stored in the disk.
If the memory occupied by at least one operation information corresponding to the transaction is greater than or equal to the second threshold, the transaction is a large transaction, and the at least one operation information corresponding to the large transaction can be stored to the disk in the form of a large transaction file.
The large transaction file may also be copied to the transaction file using zero copy techniques to improve data transfer performance.
The transaction file may include at least one operation information corresponding to a plurality of transactions.
S710, if the memory occupied by at least one operation information corresponding to the transaction is smaller than a second threshold, storing the at least one operation information corresponding to the transaction into a cache until the data amount in the cache is greater than or equal to a third threshold, and storing the operation information in the cache into a disk in a file form to obtain a corresponding transaction file.
The third threshold is greater than or equal to the second threshold.
For example, assuming that the second threshold is 100M, the third threshold is 300M, the transaction 1 corresponds to the operation information 1 and the operation information 2, the memory of the operation information 1 is 20M, the memory of the operation information 2 is 40M, the memory occupied by the transaction 1 is smaller than the second threshold, the transaction 1 is stored in the cache, the transaction 2 corresponds to the operation information 3 and the operation information 4, the memory of the operation information 3 is 30M, the memory of the operation information 4 is 25M, the memory occupied by the transaction 2 is smaller than the second threshold, the transaction 1 is stored in the cache, at this time, the memory occupied by the transaction 1 and the transaction 2 is larger than the third threshold, and then the transaction 1 and the transaction 2 can be stored in the form of a transaction file 2.
S711, generating a transaction file according to at least one piece of operation information corresponding to each transaction.
S712, sending the transaction file to the target device.
The execution of S711-S712 may be referred to as S205-206, and will not be described here.
According to the data migration method provided by the embodiment of the application, a plurality of data blocks can be downloaded according to the downloading progress of the source log file, the plurality of data blocks are sequenced and then stored in the disk in the form of a plurality of log files, analysis processing is carried out on the plurality of log files to obtain a plurality of pieces of operation information, after the plurality of pieces of operation information are sequenced, at least one piece of operation information corresponding to each transaction can be determined, and at least one piece of operation information corresponding to the transaction is stored in the disk in the form of a transaction file, so that migration efficiency of the log file is improved.
Fig. 10 is a flowchart of another data migration method according to an embodiment of the present application. Referring to fig. 10, the method may include:
s1001, determining a first transaction to be rolled back.
After determining at least one operation information corresponding to each transaction, a first transaction to be rolled back may be determined according to the transaction.
If at least one operation information corresponding to any one transaction is withdrawn or is wrong, the transaction needs to be rolled back.
S1002, if the first transaction is in the cache, deleting the first transaction in the cache.
Optionally, at least one piece of operation information corresponding to the first transaction, part of the operation information may be in the cache, and the other part of the operation information may be in the disk.
S1003, deleting the first transaction in the disk if the first transaction is located in the disk.
If the first transaction is located in the disk, deleting the first transaction after the operation information corresponding to the first transaction is read from back to front in the disk.
The storage format of the operation information may be in a symmetrical format, for example, { operation information length: operation information: operation information length }.
The transaction file can be read from front to back, the rollback can be from back to front, and the storage format of the operation information adopts a symmetrical format, so that the complete operation information can be read by reading and rollback.
Next, a sorting process of the data migration method provided in the embodiment of the present application is further described with reference to fig. 11.
Fig. 11 is a flow chart of a sorting process according to an embodiment of the present application. The method comprises the steps of determining the minimum progress of a thread, adding a memory lock, obtaining a plurality of operation information which is larger than the minimum progress, determining at least one operation information corresponding to a transaction according to the identification of the transaction, judging whether the transaction is a transaction to be rolled back or not according to the identification of the transaction, if the transaction is the transaction to be rolled back, directly deleting the operation information corresponding to the transaction in a cache, and reading the operation information corresponding to the transaction in a disk and deleting the operation information corresponding to the transaction if the transaction is the transaction to be rolled back.
If the transaction is not to be rolled back, judging whether the transaction is larger than a second threshold value or not, lasting the transaction to a disk, and zero-copying the transaction to a transaction file; and if the transaction is smaller than the second threshold, caching the transaction, judging whether the memory occupied by the cache is larger than a third threshold, if the memory occupied by the cache is smaller than the third threshold, continuing to acquire the next transaction, if the memory occupied by the cache is larger than the third threshold, directly persisting the transaction in the cache to a disk, storing the transaction file with information, and releasing the memory lock after the cache is emptied to acquire the next transaction.
According to the data migration method provided by the embodiment of the application, the first transaction to be rolled back can be determined according to the transaction, and if the first transaction is located in the cache, the first transaction is deleted in the cache; if the first transaction is located in the disk, deleting the first transaction in the disk can delete the operation process of revocation or error in the source database, thereby improving the migration efficiency of the log file.
Fig. 12 is a schematic structural diagram of a data migration apparatus according to an embodiment of the present application. Referring to fig. 12, the data migration apparatus 10 includes an acquisition module 11, a first processing module 12, a second processing module 13, a first determining module 14, a generating module 15, and a sending module 16, where:
The acquisition module 11 is used for acquiring a plurality of log files operating on the source database;
the first processing module 12 is configured to perform parsing processing on the plurality of log files to obtain a plurality of pieces of operation information, where the operation information includes an operation statement, and an identifier and a global sequence number of a transaction corresponding to the operation statement;
the second processing module 13 is configured to perform sorting processing on the pieces of operation information according to the global sequence number, obtain sorted pieces of operation information,
the first determining module 14 is configured to determine at least one operation information corresponding to each transaction from the ordered pieces of operation information;
the generating module 15 is configured to generate a transaction file according to at least one operation information corresponding to each transaction;
the sending module 16 is configured to send the transaction file to the target device.
In one possible implementation, the obtaining module 11 is specifically configured to:
acquiring a log file downloading progress;
and downloading a plurality of data blocks in a source database according to the log file downloading progress, and storing the plurality of data blocks in a disk in the form of log files to obtain a plurality of log files.
In one possible implementation, the obtaining module 11 is specifically configured to:
Downloading a plurality of data blocks in parallel through a plurality of downloading threads, and storing the plurality of data blocks into a data block queue;
a plurality of data blocks are obtained from a data block queue through an ordering thread, and are ordered according to data block information of the plurality of data blocks, so that a plurality of ordered data blocks are obtained, wherein the data block information comprises identifications of log files corresponding to the data blocks and serial numbers of the data blocks in the corresponding log files;
storing the plurality of ordered data blocks to a disk in the form of log files to obtain a plurality of log files.
In one possible implementation, the obtaining module 11 is specifically configured to:
and storing the data blocks into a buffer memory according to the sequence of the data blocks, and storing the data blocks in the buffer memory into a disk until a plurality of log files are obtained when the data quantity in the buffer memory is greater than or equal to a first threshold value.
In one possible implementation, the data migration apparatus 10 further includes a first storage module and a second storage module, wherein:
the first storage module is specifically configured to store, for any one transaction, at least one operation information corresponding to the transaction in a file form to a disk if a memory occupied by at least one operation information corresponding to the transaction is greater than or equal to a second threshold, so as to obtain a transaction file corresponding to the transaction;
The second storage module is specifically configured to store, if the memory occupied by at least one operation information corresponding to the transaction is smaller than a second threshold, at least one operation information corresponding to the transaction to the cache until the data amount in the cache is greater than or equal to a third threshold, and store the operation information in the cache to the disk in a file form to obtain a corresponding transaction file.
In one possible implementation manner, the data migration apparatus 10 further includes a second determining module, a first deleting module, and a second deleting module, where:
the second determining module is specifically configured to determine a first transaction to be rolled back;
the first deleting module is specifically configured to delete the first transaction in the cache if the first transaction is located in the cache;
the second deletion module is specifically configured to delete the first transaction in the disk if the first transaction is located in the disk.
The data migration device provided in the embodiment of the present application may execute the technical solution shown in the foregoing method embodiment, and its implementation principle and beneficial effects are similar, and will not be described herein again.
Fig. 13 is a schematic structural diagram of an electronic device according to an embodiment of the present application, referring to fig. 13, the electronic device 20 may include a processor 21 and a memory 22. The processor 21, the memory 22, and the like are illustratively interconnected by a bus 23.
Memory 22 stores computer-executable instructions;
the processor 21 executes computer-executable instructions stored in the memory 22, causing the processor 21 to perform the data migration method as shown in the method embodiments described above.
Accordingly, embodiments of the present application provide a computer-readable storage medium having stored therein computer-executable instructions for implementing the data migration method of the above-described method embodiments when the computer-executable instructions are executed by a processor.
Accordingly, embodiments of the present application may also provide a computer program product, including a computer program, which, when executed by a processor, may implement the data migration method shown in the foregoing method embodiments.
It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In one typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.
Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.
In the technical scheme of the application, the related information such as financial data or user data is collected, stored, used, processed, transmitted, provided, disclosed and the like, which accords with the regulations of related laws and regulations and does not violate the popular regulations of the public order.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises an element.
The foregoing is merely exemplary of the present application and is not intended to limit the present application. Various modifications and changes may be made to the present application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. which are within the spirit and principles of the present application are intended to be included within the scope of the claims of the present application.

Claims (10)

1. A method of data migration, comprising:
acquiring a plurality of log files for operating a source database;
analyzing the log files to obtain a plurality of pieces of operation information, wherein the operation information comprises operation sentences, and identifiers and global sequence numbers of transactions corresponding to the operation sentences;
sorting the plurality of pieces of operation information according to the global sequence number to obtain sorted pieces of operation information, and determining at least one piece of operation information corresponding to each transaction in the sorted pieces of operation information;
generating a transaction file according to at least one piece of operation information corresponding to each transaction, and sending the transaction file to target equipment.
2. The method of claim 1, wherein obtaining a plurality of log files operating on a source database comprises:
acquiring a log file downloading progress;
and downloading a plurality of data blocks in the source database according to the log file downloading progress, and storing the plurality of data blocks in a disk in the form of log files to obtain the plurality of log files.
3. The method of claim 2, wherein downloading the plurality of data blocks from the source database and storing the plurality of data blocks to disk in the form of log files to obtain the plurality of log files comprises:
Downloading the plurality of data blocks in parallel through a plurality of downloading threads, and storing the plurality of data blocks into a data block queue;
the method comprises the steps that a plurality of data blocks are obtained from a data block queue through an ordering thread, and are subjected to ordering processing according to data block information of the plurality of data blocks, so that a plurality of ordered data blocks are obtained, wherein the data block information comprises identifications of log files corresponding to the data blocks and serial numbers of the data blocks in the corresponding log files;
and storing the plurality of ordered data blocks to a disk in the form of log files to obtain the plurality of log files.
4. A method according to claim 3, wherein storing the plurality of ordered data blocks to disk in the form of log files to obtain the plurality of log files comprises:
storing the data blocks into a cache according to the sequence of the data blocks, and storing the data blocks in the cache into a disk when the data amount in the cache is greater than or equal to a first threshold value until the log files are obtained.
5. A method according to any one of claims 1-3, further comprising, after determining at least one operation information corresponding to each transaction in the ordered plurality of operation information:
For any one transaction, if the memory occupied by at least one piece of operation information corresponding to the transaction is larger than or equal to a second threshold value, storing the at least one piece of operation information corresponding to the transaction to a disk in a file form to obtain a transaction file corresponding to the transaction;
and if the memory occupied by at least one piece of operation information corresponding to the transaction is smaller than the second threshold value, storing the at least one piece of operation information corresponding to the transaction into a cache until the data volume in the cache is larger than or equal to a third threshold value, and storing the operation information in the cache into a disk in a file form to obtain a corresponding transaction file.
6. A method according to any one of claims 1-3, wherein the method further comprises:
determining a first transaction to be rolled back;
deleting the first transaction in the cache if the first transaction is in the cache;
and if the first transaction is located in the disk, deleting the first transaction in the disk.
7. The data migration device is characterized by comprising an acquisition module, a first processing module, a second processing module, a first determining module, a generating module and a sending module, wherein:
The acquisition module is used for acquiring a plurality of log files for operating the source database;
the first processing module is used for analyzing the log files to obtain a plurality of pieces of operation information, wherein the operation information comprises operation sentences, and identifiers and global sequence numbers of transactions corresponding to the operation sentences;
the second processing module is used for sorting the plurality of pieces of operation information according to the global sequence number to obtain sorted pieces of operation information,
the first determining module is used for determining at least one piece of operation information corresponding to each transaction in the sequenced pieces of operation information;
the generating module is used for generating a transaction file according to at least one piece of operation information corresponding to each transaction,
the sending module is used for sending the transaction file to target equipment.
8. An electronic device, comprising: a memory and a processor, wherein the memory is configured to store,
the memory stores computer-executable instructions;
the processor executing computer-executable instructions stored in the memory, causing the processor to perform the data migration method of any one of claims 1 to 6.
9. A computer readable storage medium having stored therein computer executable instructions for implementing the data migration method of any one of claims 1 to 6 when the computer executable instructions are executed by a processor.
10. A computer program product comprising a computer program which, when executed by a processor, implements the data migration method of any one of claims 1 to 6.
CN202310175116.8A 2023-02-27 2023-02-27 Data migration method, device, equipment and readable storage medium Pending CN116166636A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310175116.8A CN116166636A (en) 2023-02-27 2023-02-27 Data migration method, device, equipment and readable storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310175116.8A CN116166636A (en) 2023-02-27 2023-02-27 Data migration method, device, equipment and readable storage medium

Publications (1)

Publication Number Publication Date
CN116166636A true CN116166636A (en) 2023-05-26

Family

ID=86413058

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310175116.8A Pending CN116166636A (en) 2023-02-27 2023-02-27 Data migration method, device, equipment and readable storage medium

Country Status (1)

Country Link
CN (1) CN116166636A (en)

Similar Documents

Publication Publication Date Title
US10983909B2 (en) Trading off cache space and write amplification for Bε-trees
CN110019111B (en) Data processing method, data processing device, storage medium and processor
US20170083537A1 (en) Mapping logical identifiers using multiple identifier spaces
CN111680017A (en) Data synchronization method and device
CN107506466B (en) Small file storage method and system
CN109491834A (en) A kind of data reconstruction method and device applied to database
CN114297204A (en) Data storage and retrieval method and device for heterogeneous data source
CN111680030A (en) Data fusion method and device, and data processing method and device based on meta information
CN112965939A (en) File merging method, device and equipment
CN111026736B (en) Data blood margin management method and device and data blood margin analysis method and device
CN117492670A (en) Log printing sequence determining method and device and electronic equipment
CN116775830A (en) Online technical document searching method, device and medium
CN109542860B (en) Service data management method based on HDFS and terminal equipment
CN116431601A (en) Data table creation method and device
CN117763024A (en) Data fragment extraction method and device
CN116415156A (en) Document similarity calculation method, device and medium
CN116166636A (en) Data migration method, device, equipment and readable storage medium
CN110019295B (en) Database retrieval method, device, system and storage medium
CN113326063B (en) Data processing method, electronic device and storage medium
CN113590566B (en) Method, device, equipment and storage medium for optimizing sequence file storage based on heap structure
US11372832B1 (en) Efficient hashing of data objects
CN108984343B (en) Virtual machine backup and storage management method based on content analysis
CN109376214B (en) Data processing method, device and system, computer equipment and readable medium
CN113391812A (en) Analysis method and device of application program module and analysis tool
CN112579605B (en) Data storage method, device, storage medium and server

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination