CN112597126A - Data migration method and device - Google Patents

Data migration method and device Download PDF

Info

Publication number
CN112597126A
CN112597126A CN202011476755.0A CN202011476755A CN112597126A CN 112597126 A CN112597126 A CN 112597126A CN 202011476755 A CN202011476755 A CN 202011476755A CN 112597126 A CN112597126 A CN 112597126A
Authority
CN
China
Prior art keywords
data
temporary table
migrated
data file
temporary
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202011476755.0A
Other languages
Chinese (zh)
Other versions
CN112597126B (en
Inventor
张向旭
张玉龙
张逸
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
China Construction Bank Corp
Original Assignee
China Construction Bank Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by China Construction Bank Corp filed Critical China Construction Bank Corp
Priority to CN202011476755.0A priority Critical patent/CN112597126B/en
Publication of CN112597126A publication Critical patent/CN112597126A/en
Application granted granted Critical
Publication of CN112597126B publication Critical patent/CN112597126B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21Design, administration or maintenance of databases
    • G06F16/214Database migration support
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22Indexing; Data structures therefor; Storage structures
    • G06F16/2228Indexing structures

Landscapes

  • Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention discloses a data migration method and device, and relates to the technical field of computers. A specific implementation manner of the method comprises the steps of obtaining data to be migrated, determining a target table into which the data to be migrated is imported, and creating a temporary table with a structure consistent with that of the target table; calling a preset segmentation model, dividing the data to be migrated into a plurality of data files, and importing the data files to the temporary table; after each data file is imported to a temporary table, establishing index information of the data file, which is consistent with the target table, for the temporary table; and exchanging the data files in the temporary table to the target table based on a preset exchange model. Therefore, the embodiment of the invention can solve the problem of data updating of the large data table during high concurrent access, and ensures that the service is not interrupted while the data is safely and quickly updated.

Description

Data migration method and device
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a data migration method and apparatus.
Background
The Sql Loader (Sql Loader for short) tool of Oracle (Oracle) can import the data file into the Oracle database, that is, import the data file into the Oracle database through the Sql Loader of Oracle itself, wherein writing the ctl file of Sql Loader can modify the parameters when using Sql Loader.
In the process of implementing the invention, the inventor finds that at least the following problems exist in the prior art:
if data is required to be imported into the database quickly, the direct parameter of sqlldr needs to be set to true, but the index of the database is failed in the derivative process, especially when the data volume is large, the index failure is dangerous, and then the index is rebuilt for a table with large data volume at a great risk. And in this process, access to the table may also be problematic. If the direct parameter is set to false, data can be safely imported without index failure, but the derivative speed is greatly reduced, and the performance is very poor.
Disclosure of Invention
In view of this, embodiments of the present invention provide a data migration method and apparatus, which can solve the security problem and the performance problem caused by importing a large amount of data into a table with an index in the prior art. And simultaneously, the service is ensured not to be interrupted.
In order to achieve the above object, according to an aspect of the embodiments of the present invention, a data migration method is provided, including obtaining data to be migrated, determining a target table into which the data to be migrated is imported, so as to create a temporary table having a structure consistent with that of the target table; calling a preset segmentation model, dividing the data to be migrated into a plurality of data files, and importing the data files to the temporary table; after each data file is imported to a temporary table, establishing index information of the data file, which is consistent with the target table, for the temporary table; and exchanging the data files in the temporary table to the target table based on a preset exchange model.
Optionally, dividing the data to be migrated into a plurality of data files based on a preset segmentation model, including:
determining a service type corresponding to data to be migrated to acquire a partition field of the service type, and dividing the data to be migrated into a plurality of data files according to the partition field.
Optionally, after dividing the data to be migrated into a plurality of data files according to the partition field, the method further includes:
and judging whether the data volume of the data file is larger than a preset number threshold, if so, dividing the data file into a plurality of sub-data files, and if not, processing.
Optionally, importing the data file into the temporary table includes:
importing the data file into the temporary table by an sqlldr multithread method; and setting a direct parameter and a parallel parameter in the ctl file of the sqlldr to true.
Optionally, importing the data file into the temporary table by an sql drr multithreading method includes:
a plurality of threads import the same data file at the same time;
if there is a thread without data file import, the thread state is set to wait.
Optionally, swapping the data file in the temporary table to the target table based on a preset swapping model, including:
and after the importing of one data file is finished, the temporary table and the target table are subjected to partition exchange, and indexes are exchanged while partitions are exchanged.
Optionally, after swapping the data file in the temporary table to the target table, the swapping includes: and deleting the index information in the temporary table.
In addition, the invention also provides a data migration device, which comprises an acquisition module, a data storage module and a data migration module, wherein the acquisition module is used for acquiring data to be migrated, determining a target table into which the data to be migrated is imported, and creating a temporary table with a structure consistent with that of the target table; the processing module is used for calling a preset segmentation model, dividing the data to be migrated into a plurality of data files and then importing the data files to the temporary table; after each data file is imported to a temporary table, establishing index information of the data file, which is consistent with the target table, for the temporary table; and exchanging the data files in the temporary table to the target table based on a preset exchange model.
One embodiment of the above invention has the following advantages or benefits: the invention can use the temporary table to carry out partition exchange, thereby avoiding the problem of index invalidation; moreover, in a high-throughput real-time transaction system, the service is not interrupted, and the batch data updating of the large data table is completed quickly; meanwhile, the invention carries out derivative under the condition of keeping the direct parameter as true, does not cause the problem of index failure, and starts multithreading parallel import to lead the import speed to be faster.
Further effects of the above-mentioned non-conventional alternatives will be described below in connection with the embodiments.
Drawings
The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:
fig. 1 is a schematic diagram of a main flow of a data migration method according to a first embodiment of the present invention;
FIG. 2 is a diagram illustrating an architecture of a data migration method according to a second embodiment of the present invention;
FIG. 3 is a schematic diagram of the main modules of a data migration apparatus according to an embodiment of the present invention;
FIG. 4 is an exemplary system architecture diagram in which embodiments of the present invention may be employed;
fig. 5 is a schematic block diagram of a computer system suitable for use in implementing a terminal device or server of an embodiment of the invention.
Detailed Description
Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, in which various details of embodiments of the invention are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
Fig. 1 is a schematic diagram of a main flow of a data migration method according to a first embodiment of the present invention, as shown in fig. 1, the data migration method includes:
step S101, obtaining data to be migrated, determining a target table into which the data to be migrated is imported, and creating a temporary table with a structure consistent with that of the target table.
In an embodiment, data to be migrated is to be migrated from a data source layer to a target layer, namely a target table, wherein a temporary table with a structure completely consistent with the target table is created as a transit station, and finally the data is migrated to the target layer through a partition exchange technology.
Step S102, calling a preset segmentation model, dividing the data to be migrated into a plurality of data files, and importing the data files into the temporary table; and after each data file is imported to a temporary table, establishing index information of the data file, which is consistent with the target table, for the temporary table.
In some embodiments, the data to be migrated is divided into a plurality of data files based on a preset segmentation model, including determining a service type corresponding to the data to be migrated to obtain a partition field of the service type, and then dividing the data to be migrated into a plurality of data files according to the partition field. For example: for the service type of the complete repayment cash flow plan of the bank, the data can be partitioned according to the branch field.
In a further embodiment, after the data to be migrated is divided into a plurality of data files according to the partition field, whether the data volume of the data files is greater than a preset number threshold value or not may be further determined, if so, the data files are divided into a plurality of sub-data files, and if not, the data files are not processed. For example: the partition field X has 3 values, 3 data files are to be separated, the attribute values of the field X of the t, m, n and 3 data files are different, if the data volume of the data files is larger than a preset number threshold value, the possible files are too large, the data files are continuously divided to obtain t1, t2, t3, t4, m1, m2, n1, n2 and n3, each file contains 50 to 500 ten thousand pieces of data, and the total data volume is required to be referred to.
As another example, importing the data file into the temporary table includes: importing the data file into the temporary table by an sqlldr multithread method; the direct parameter and parallel parameter in the ctl file of the sqlldr are set to true, so that the derivative performance can be improved. It should be noted that sqlldr is a data loading tool of ORACLE. Ctl is the control file for sqlldr. The Direct parameter is to directly fill data into the table, and in the process, if the table has an index, the index will fail. The Parallel parameter is opened and can be opened only when the direct parameter is opened. Both the direct parameter and the parallel parameter will accelerate the derivative speed.
In a further embodiment, the importing the data file into the temporary table by an sql drs multithreading method includes: a plurality of threads import the same data file at the same time; if there is a thread without data file import, the thread state is set to wait. For example: and 3 threads are started, the t data file has 4 sub data files, and when the thread 1 imports the fourth sub data file, the other two threads wait for the thread 1 to finish. That is, each thread imports 1 sub-data file, and if there is only one sub-data file, only 1 thread works.
And step S103, exchanging the data file in the temporary table to the target table based on a preset exchange model.
In some embodiments, swapping the data file in the temporary table to the target table based on a preset swapping model, including after a data file is imported, performing partition swapping on the temporary table and the target table, and swapping indexes while swapping partitions. The index information is an index field, and the index field in the temporary table is completely consistent with the index field in the target table.
In a further embodiment, after the data files in the temporary table are exchanged to the target table, the index information in the temporary table may be deleted, that is, preparation is made for importing other data files, for example: repeating the above operations on the m and n data files until all the data files are imported into the target table.
Fig. 2 is a schematic diagram of a main flow of a data migration method according to a second embodiment of the present invention, the data migration method including:
step S201, acquiring data to be migrated, and determining a target table into which the data to be migrated is imported.
Step S202, a temporary table is created according to the table structure of the target table.
Step S203, determining a service type corresponding to the data to be migrated so as to obtain a partition field of the service type, and dividing the data to be migrated into a plurality of data files according to the partition field.
Step S204, if the data volume of the data file is larger than a preset volume threshold, the data file is divided into a plurality of sub-data files.
Step S205, setting the direct parameter and the parallel parameter in the ctl file of the sqlldr to true.
Step S206, the data file is imported into the temporary table through an sqlldr multithread method.
Step S207, after each data file is imported into the temporary table, establishing index information of the data file consistent with the target table for the temporary table.
And S208, after the importing of one data file is finished, the temporary table and the target table are subjected to partition exchange, and indexes are exchanged while partitions are exchanged.
And step S209, deleting the index information in the temporary table, returning to step S206, and iteratively executing step S206 to step S209 until all data files are stored in the target table.
Fig. 3 is a schematic diagram of main modules of a data migration apparatus according to an embodiment of the present invention, and as shown in fig. 3, the data migration apparatus includes an acquisition module 301 and a processing module 302. The obtaining module 301 obtains data to be migrated, and determines a target table into which the data to be migrated is imported, so as to create a temporary table having a structure consistent with that of the target table; the processing module 302 calls a preset segmentation model, divides the data to be migrated into a plurality of data files, and then imports the data files into the temporary table; after each data file is imported to a temporary table, establishing index information of the data file, which is consistent with the target table, for the temporary table; and exchanging the data files in the temporary table to the target table based on a preset exchange model.
In some embodiments, the processing module 302 divides the data to be migrated into a plurality of data files based on a preset segmentation model, including: determining a service type corresponding to data to be migrated to acquire a partition field of the service type, and dividing the data to be migrated into a plurality of data files according to the partition field.
In some embodiments, after the processing module 302 divides the data to be migrated into a plurality of data files according to the partition field, the method further includes:
and judging whether the data volume of the data file is larger than a preset number threshold, if so, dividing the data file into a plurality of sub-data files, and if not, processing.
In some embodiments, the process module 302 imports the data file into the temporary table, including:
importing the data file into the temporary table by an sqlldr multithread method; and setting a direct parameter and a parallel parameter in the ctl file of the sqlldr to true.
In some embodiments, the processing module 302 imports the data file into the temporary table by an sql drr multithreading method, which includes:
a plurality of threads import the same data file at the same time;
if there is a thread without data file import, the thread state is set to wait.
In some embodiments, the processing module 302 swaps the data file in the temporary table to the target table based on a preset swapping model, including:
and after the importing of one data file is finished, the temporary table and the target table are subjected to partition exchange, and indexes are exchanged while partitions are exchanged.
In some embodiments, after the processing module 302 swaps the data file in the temporary table to the target table, the method includes: and deleting the index information in the temporary table.
It should be noted that the data migration method and the data migration apparatus of the present invention have corresponding relation in the specific implementation content, and therefore the repeated content is not described again.
Fig. 4 illustrates an exemplary system architecture 400 to which the data migration method or the data migration apparatus of the embodiments of the present invention may be applied.
As shown in fig. 4, the system architecture 400 may include terminal devices 401, 402, 403, a network 404, and a server 405. The network 404 serves as a medium for providing communication links between the terminal devices 401, 402, 403 and the server 405. Network 404 may include various types of connections, such as wire, wireless communication links, or fiber optic cables, to name a few.
A user may use terminal devices 401, 402, 403 to interact with a server 405 over a network 404 to receive or send messages or the like. The terminal devices 401, 402, 403 may have installed thereon various communication client applications, such as shopping-like applications, web browser applications, search-like applications, instant messaging tools, mailbox clients, social platform software, etc. (by way of example only).
The terminal devices 401, 402, 403 may be various electronic devices having a data migration screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.
The server 405 may be a server providing various services, such as a background management server (for example only) providing support for shopping websites browsed by users using the terminal devices 401, 402, 403. The backend management server may analyze and perform other processing on the received data such as the product information query request, and feed back a processing result (for example, target push information, product information — just an example) to the terminal device.
It should be noted that the data migration method provided by the embodiment of the present invention is generally executed by the server 405, and accordingly, the computing device is generally disposed in the server 405.
It should be understood that the number of terminal devices, networks, and servers in fig. 4 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Referring now to FIG. 5, shown is a block diagram of a computer system 500 suitable for use with a terminal device implementing an embodiment of the present invention. The terminal device shown in fig. 5 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.
As shown in fig. 5, the computer system 500 includes a Central Processing Unit (CPU)501 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM)502 or a program loaded from a storage section 508 into a Random Access Memory (RAM) 503. In the RAM503, various programs and data necessary for the operation of the computer system 500 are also stored. The CPU401, ROM502, and RAM503 are connected to each other via a bus 504. An input/output (I/O) interface 505 is also connected to bus 504.
The following components are connected to the I/O interface 505: an input portion 506 including a keyboard, a mouse, and the like; an output section 507 including a display such as a Cathode Ray Tube (CRT), a liquid crystal data shifter (LCD), and the like, and a speaker; a storage portion 508 including a hard disk and the like; and a communication section 509 including a network interface card such as a LAN card, a modem, or the like. The communication section 509 performs communication processing via a network such as the internet. The driver 510 is also connected to the I/O interface 505 as necessary. A removable medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 510 as necessary, so that a computer program read out therefrom is mounted into the storage section 508 as necessary.
In particular, according to the embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 509, and/or installed from the removable medium 511. The computer program performs the above-described functions defined in the system of the present invention when executed by the Central Processing Unit (CPU) 501.
It should be noted that the computer readable medium shown in the present invention can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present invention, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The modules described in the embodiments of the present invention may be implemented by software or hardware. The described modules may also be provided in a processor, which may be described as: a processor includes an acquisition module and a processing module. Wherein the names of the modules do not in some cases constitute a limitation of the module itself.
As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be separate and not incorporated into the device. The computer readable medium carries one or more programs, and when the one or more programs are executed by a device, the device comprises a data acquisition unit, a data migration unit and a data migration unit, wherein the data migration unit acquires data to be migrated, determines a target table into which the data to be migrated is imported, and creates a temporary table with a structure consistent with that of the target table; calling a preset segmentation model, dividing the data to be migrated into a plurality of data files, and importing the data files to the temporary table; after each data file is imported to a temporary table, establishing index information of the data file, which is consistent with the target table, for the temporary table; and exchanging the data files in the temporary table to the target table based on a preset exchange model.
According to the technical scheme of the embodiment of the invention, the problem of data updating of the large data table during high concurrent access can be solved, and the service can not be interrupted while the data is safely and quickly updated.
The above-described embodiments should not be construed as limiting the scope of the invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions can occur, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims (10)

1. A method of data migration, comprising:
acquiring data to be migrated, and determining a target table into which the data to be migrated is imported so as to create a temporary table with a structure consistent with that of the target table;
calling a preset segmentation model, dividing the data to be migrated into a plurality of data files, and importing the data files to the temporary table; after each data file is imported to a temporary table, establishing index information of the data file, which is consistent with the target table, for the temporary table;
and exchanging the data files in the temporary table to the target table based on a preset exchange model.
2. The method according to claim 1, wherein the dividing the data to be migrated into a plurality of data files based on a preset segmentation model comprises:
determining a service type corresponding to data to be migrated to acquire a partition field of the service type, and dividing the data to be migrated into a plurality of data files according to the partition field.
3. The method of claim 2, wherein after dividing the data to be migrated into a plurality of data files according to the partition field, further comprising:
and judging whether the data volume of the data file is larger than a preset number threshold, if so, dividing the data file into a plurality of sub-data files, and if not, processing.
4. The method of claim 1, wherein importing the data file into the temporary table comprises:
importing the data file into the temporary table by an sqlldr multithread method; and setting a direct parameter and a parallel parameter in the ctl file of the sqlldr to true.
5. The method of claim 4, wherein importing the data file into the temporary table by an sql drs multithreading method comprises:
a plurality of threads import the same data file at the same time;
if there is a thread without data file import, the thread state is set to wait.
6. The method of claim 1, wherein swapping the data file in the temporary table to the target table based on a preset swapping model comprises:
after a data file is imported, the temporary table and the target table are subjected to partition exchange; the index is swapped while the partitions are swapped.
7. The method according to any of claims 1-6, wherein swapping the data file in the temporary table to the target table comprises:
and deleting the index information in the temporary table.
8. A data migration apparatus, comprising:
the acquisition module is used for acquiring data to be migrated, determining a target table into which the data to be migrated is imported, and creating a temporary table with a structure consistent with that of the target table;
the processing module is used for calling a preset segmentation model, dividing the data to be migrated into a plurality of data files and then importing the data files to the temporary table; after each data file is imported to a temporary table, establishing index information of the data file, which is consistent with the target table, for the temporary table; and exchanging the data files in the temporary table to the target table based on a preset exchange model.
9. An electronic device, comprising:
one or more processors;
a storage device for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-7.
10. A computer-readable medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-7.
CN202011476755.0A 2020-12-14 2020-12-14 Data migration method and device Active CN112597126B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011476755.0A CN112597126B (en) 2020-12-14 2020-12-14 Data migration method and device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011476755.0A CN112597126B (en) 2020-12-14 2020-12-14 Data migration method and device

Publications (2)

Publication Number Publication Date
CN112597126A true CN112597126A (en) 2021-04-02
CN112597126B CN112597126B (en) 2024-09-24

Family

ID=75195672

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011476755.0A Active CN112597126B (en) 2020-12-14 2020-12-14 Data migration method and device

Country Status (1)

Country Link
CN (1) CN112597126B (en)

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113407489A (en) * 2021-06-18 2021-09-17 杭州安恒信息技术股份有限公司 Data import method, device, equipment and computer readable storage medium
CN113568867A (en) * 2021-07-23 2021-10-29 上海淇玥信息技术有限公司 File encryption migration method and device and electronic equipment
CN114064783A (en) * 2021-12-02 2022-02-18 建信金融科技有限责任公司 Method and device for importing data, electronic equipment and storage medium
CN114168542A (en) * 2021-10-27 2022-03-11 中国建设银行股份有限公司 Batch control method, device, electronic device and storage medium for data loading
CN114647657A (en) * 2022-03-28 2022-06-21 重庆长安汽车股份有限公司 Seamless updating method for mass data of vehicle
CN115544027A (en) * 2022-12-05 2022-12-30 北京滴普科技有限公司 Data import method and system for OLAP analysis engine

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103077241A (en) * 2013-01-10 2013-05-01 中国银行股份有限公司 Method for loading data in parallel after splitting files
CN103106271A (en) * 2013-02-05 2013-05-15 广东全通教育股份有限公司 Database backup and recovery method and system based on mass data
CN108287835A (en) * 2017-01-09 2018-07-17 腾讯科技(深圳)有限公司 A kind of data clearing method and device
CN110347651A (en) * 2019-06-11 2019-10-18 平安科技(深圳)有限公司 Method of data synchronization, device, equipment and storage medium based on cloud storage
CN110347673A (en) * 2019-05-30 2019-10-18 平安银行股份有限公司 Data file loading method, device, computer equipment and storage medium
WO2019219010A1 (en) * 2018-05-14 2019-11-21 杭州海康威视数字技术股份有限公司 Data migration method and device and computer readable storage medium
CN110764943A (en) * 2019-10-21 2020-02-07 中国民航信息网络股份有限公司 Data processing method and device for Oracle database
US20200159746A1 (en) * 2018-11-19 2020-05-21 Servicenow, Inc. Concurrent data imports

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103077241A (en) * 2013-01-10 2013-05-01 中国银行股份有限公司 Method for loading data in parallel after splitting files
CN103106271A (en) * 2013-02-05 2013-05-15 广东全通教育股份有限公司 Database backup and recovery method and system based on mass data
CN108287835A (en) * 2017-01-09 2018-07-17 腾讯科技(深圳)有限公司 A kind of data clearing method and device
WO2019219010A1 (en) * 2018-05-14 2019-11-21 杭州海康威视数字技术股份有限公司 Data migration method and device and computer readable storage medium
US20200159746A1 (en) * 2018-11-19 2020-05-21 Servicenow, Inc. Concurrent data imports
CN110347673A (en) * 2019-05-30 2019-10-18 平安银行股份有限公司 Data file loading method, device, computer equipment and storage medium
CN110347651A (en) * 2019-06-11 2019-10-18 平安科技(深圳)有限公司 Method of data synchronization, device, equipment and storage medium based on cloud storage
CN110764943A (en) * 2019-10-21 2020-02-07 中国民航信息网络股份有限公司 Data processing method and device for Oracle database

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
秦峰巍 等: "基于SQL Loader的海量数据装载方案优化", 武汉理工大学学报信息与管理工程版, vol. 32, no. 5, 31 October 2010 (2010-10-31), pages 707 - 716 *

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113407489A (en) * 2021-06-18 2021-09-17 杭州安恒信息技术股份有限公司 Data import method, device, equipment and computer readable storage medium
CN113568867A (en) * 2021-07-23 2021-10-29 上海淇玥信息技术有限公司 File encryption migration method and device and electronic equipment
CN114168542A (en) * 2021-10-27 2022-03-11 中国建设银行股份有限公司 Batch control method, device, electronic device and storage medium for data loading
CN114064783A (en) * 2021-12-02 2022-02-18 建信金融科技有限责任公司 Method and device for importing data, electronic equipment and storage medium
CN114064783B (en) * 2021-12-02 2024-12-20 建信金融科技有限责任公司 Method, device, electronic device and storage medium for importing data
CN114647657A (en) * 2022-03-28 2022-06-21 重庆长安汽车股份有限公司 Seamless updating method for mass data of vehicle
CN115544027A (en) * 2022-12-05 2022-12-30 北京滴普科技有限公司 Data import method and system for OLAP analysis engine

Also Published As

Publication number Publication date
CN112597126B (en) 2024-09-24

Similar Documents

Publication Publication Date Title
CN112597126B (en) Data migration method and device
CN109189835B (en) Method and device for generating data wide table in real time
CN108629029B (en) Data processing method and device applied to data warehouse
US9372880B2 (en) Reclamation of empty pages in database tables
US11421994B2 (en) Method and apparatus for partitioning electronic fence
CN111061680B (en) A method and device for data retrieval
CN107480205B (en) Method and device for partitioning data
CN111125107A (en) Data processing method, device, electronic equipment and medium
CN110795419A (en) Method and device for dynamic database-based routing
CN110851419A (en) Data migration method and device
CN111581930A (en) Online form data processing method and device, electronic equipment and readable medium
CN111984686A (en) Data processing method and device
CN111753019A (en) Data partitioning method and device applied to data warehouse
CN109213815B (en) Method, device, server terminal and readable medium for controlling execution times
CN113704242A (en) Data processing method and device
CN112148705A (en) Data migration method and device
CN113760861B (en) Data migration method and device
CN113760860B (en) Data reading method and device
CN110858199A (en) Document data distributed computing method and device
CN115454971A (en) Data migration method and device, electronic equipment and storage medium
CN114756173A (en) Method, system, device and computer readable medium for file merging
CN108984426B (en) Method and apparatus for processing data
CN114064693A (en) Method, apparatus, electronic device, and computer-readable medium for processing account data
CN113626472A (en) Method and device for processing order data
CN109656519B (en) Method and device for automatically accessing service data

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant