CN117194558A - Full-data synchronization method, device, equipment and medium - Google Patents
Full-data synchronization method, device, equipment and medium Download PDFInfo
- Publication number
- CN117194558A CN117194558A CN202310294028.XA CN202310294028A CN117194558A CN 117194558 A CN117194558 A CN 117194558A CN 202310294028 A CN202310294028 A CN 202310294028A CN 117194558 A CN117194558 A CN 117194558A
- Authority
- CN
- China
- Prior art keywords
- full
- data
- data file
- file
- different
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 38
- 238000012545 processing Methods 0.000 claims abstract description 23
- 230000008569 process Effects 0.000 claims description 7
- 230000005540 biological transmission Effects 0.000 description 8
- 230000001360 synchronised effect Effects 0.000 description 8
- 238000010586 diagram Methods 0.000 description 6
- 230000004048 modification Effects 0.000 description 6
- 238000012986 modification Methods 0.000 description 6
- 230000005012 migration Effects 0.000 description 5
- 238000013508 migration Methods 0.000 description 5
- 238000007792 addition Methods 0.000 description 4
- 230000037430 deletion Effects 0.000 description 4
- 238000012217 deletion Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 3
- 230000008676 import Effects 0.000 description 3
- 230000003203 everyday effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the application discloses a full-data synchronization method, a full-data synchronization device, full-data synchronization equipment and a full-data synchronization medium. The method comprises the steps of obtaining a full-volume data synchronization request sent by a third party system; determining the position of a full-volume data file based on the full-volume data request, and pulling the full-volume data file to a local sftp server; dividing a full-volume data file in a local sftp server into a plurality of data units, and comparing each data unit with a historical full-volume data file to store the full-volume data file into different data files based on a comparison result; and generating different sql sentences for different data files, and carrying out warehousing processing on the different sql sentences to complete full data synchronization. The method solves the problem of data redundancy.
Description
Technical Field
The present application relates to the field of data processing technologies, and in particular, to a full-data synchronization method, apparatus, device, and medium.
Background
The full data refers to the entire data of the database system that currently needs to be migrated. With the popularization and popularity of the internet and big data technologies, data warehouse has become a mainstream. Because the data warehouse has a large storage capacity and can process structured data, the data in the database can be synchronized to the data warehouse for processing.
During the data migration process, the business system needs to synchronize third party system data, such as order data, user data, enterprise data, transaction stream. But the third party system only provides the full amount of data. Because the data volume is too large, accurate migration of a large amount of data is difficult to ensure, and problems such as data redundancy are caused.
Disclosure of Invention
The embodiment of the application provides a full-data synchronization method, a full-data synchronization device, full-data synchronization equipment and a full-data synchronization medium, which are used for solving the following technical problems: because the data volume is too large, accurate migration of a large amount of data is difficult to ensure, and problems such as data redundancy are caused.
The embodiment of the application adopts the following technical scheme:
the embodiment of the application provides a full-data synchronization method. The method comprises the steps of obtaining a full-volume data synchronization request sent by a third party system; determining the position of a full-volume data file based on the full-volume data request, and pulling the full-volume data file to a local sftp server; dividing a full-volume data file in a local sftp server into a plurality of data units, and comparing each data unit with a historical full-volume data file to store the full-volume data file into different data files based on a comparison result; and generating different sql sentences for different data files, and carrying out warehousing processing on the different sql sentences to complete full data synchronization.
The file transmission protocol of the embodiment of the application is suitable for transmitting large problems, and when the full data is synchronous, the file is taken as a carrier, so that the data transmission efficiency can be ensured. Secondly, the embodiment of the application can determine the same or different data with the historical full data file by dividing the full data file in the local sftp server into a plurality of data units and comparing the data units with the historical full data file. The full data file is analyzed into a plurality of data files such as new addition, modification, deletion and the like, and finally converted into sql sentences, and different data are stored differently, so that the problem of full data redundancy is solved.
In one implementation of the present application, after pulling the full data file to the local sftp server, the method further comprises: determining the pushing frequency corresponding to the full data file; setting a local storage catalog of the full-quantity file based on the pushing frequency; and storing the full-volume data files under different local storage directories based on the pushing frequency, and regularly backing up the full-volume data files under the local storage directories.
In one implementation of the present application, before dividing the full volume data file in the local sftp server into the plurality of data units, the method further comprises: determining that the full data file is a newly added data file under the condition that the full data file does not exist in the local sftp server; generating corresponding sql sentences from the newly added data file, and carrying out warehousing processing on different sql sentences.
In one implementation of the present application, a full-volume data file in a local sftp server is divided into a plurality of data units, and each data unit is compared with a historical full-volume data file, so as to save the full-volume data file to different data files based on a comparison result, which specifically includes: dividing the full-size data file into a plurality of data units in a row unit under the condition that the full-size data file exists in a local sftp server, and respectively carrying out unique identification marking on the plurality of data units; and comparing the plurality of data units with the plurality of data units corresponding to the historical full data file to save the full data file to different data files based on the comparison result.
In one implementation of the present application, comparing a plurality of data units corresponding to a historical full-size data file to save the full-size data file to a different data file based on the comparison result, specifically includes: under the condition that the full data file does not exist in the historical full data, storing the data corresponding to the full data file into a newly added data file; storing the modified full data file into a modified data file under the condition that the full data file exists in the historical full data and the full data file is modified; storing data which exists in the historical full data but does not exist in the full data file into the deleted data file; and storing the data existing in both the historical full data and the full data file into the unchanged data file.
In one implementation of the present application, different sql statements are generated for different data files, and the different sql statements are subjected to warehouse entry processing to complete full data synchronization, which specifically includes: generating an insert sql statement from the newly added data file; generating an update sql statement from the modified data file; and generating delete statement from the deleted data file.
In one implementation of the present application, determining a full-size data file location based on a full-size data request, and pulling the full-size data file to a local sftp server, includes: logging in a third party system sftp server through a preset user name and a preset password; determining the position of the full-volume data file in a third-party system sftp server based on the full-volume data request; the full data file is pulled to the local sftp server.
The embodiment of the application provides a full-volume data synchronization device, which comprises: the full-volume data synchronization request acquisition unit acquires a full-volume data synchronization request sent by a third party system; the full-volume data file acquisition unit is used for determining the position of the full-volume data file based on the full-volume data request and pulling the full-volume data file to the local sftp server; the full data comparison unit divides the full data file in the local sftp server into a plurality of data units, and compares each data unit with the historical full data file so as to store the full data file into different data files based on the comparison result; and the sql statement generating unit generates different sql statements for different data files, and performs warehousing processing on the different sql statements so as to complete full data synchronization.
The embodiment of the application provides full-volume data synchronization equipment, which comprises the following components: at least one processor; the method comprises the steps of,
a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executable by the at least one processor to enable the at least one processor to: acquiring a full data synchronization request sent by a third party system; determining the position of a full-volume data file based on the full-volume data request, and pulling the full-volume data file to a local sftp server; dividing a full-volume data file in a local sftp server into a plurality of data units, and comparing each data unit with a historical full-volume data file to store the full-volume data file into different data files based on a comparison result; and generating different sql sentences for different data files, and carrying out warehousing processing on the different sql sentences to complete full data synchronization.
The non-volatile computer storage medium provided by the embodiment of the application stores computer executable instructions, and the computer executable instructions are set as follows: acquiring a full data synchronization request sent by a third party system; determining the position of a full-volume data file based on the full-volume data request, and pulling the full-volume data file to a local sftp server; dividing a full-volume data file in a local sftp server into a plurality of data units, and comparing each data unit with a historical full-volume data file to store the full-volume data file into different data files based on a comparison result; and generating different sql sentences for different data files, and carrying out warehousing processing on the different sql sentences to complete full data synchronization.
The above at least one technical scheme adopted by the embodiment of the application can achieve the following beneficial effects: the file transmission protocol of the embodiment of the application is suitable for transmitting large problems, and when the full data is synchronous, the file is taken as a carrier, so that the data transmission efficiency can be ensured. Secondly, the embodiment of the application can determine the same or different data with the historical full data file by dividing the full data file in the local sftp server into a plurality of data units and comparing the data units with the historical full data file. The full data file is analyzed into a plurality of data files such as new addition, modification, deletion and the like, and finally converted into sql sentences, and different data are stored differently, so that the problem of full data redundancy is solved.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments described in the present application, and other drawings can be obtained according to the drawings without inventive effort for a person skilled in the art. Attached at
In the figure:
FIG. 1 is a flowchart of a full-scale data synchronization method according to an embodiment of the present application;
FIG. 2 is a block diagram of a full-scale data synchronization device according to an embodiment of the present application;
FIG. 3 is a schematic diagram of a full-data synchronization device according to an embodiment of the present application;
fig. 4 is a schematic structural diagram of a full-data synchronization device according to an embodiment of the present application.
Detailed Description
The embodiment of the application provides a full-volume data synchronization method, device, equipment and medium.
In order to make the technical solution of the present application better understood by those skilled in the art, the technical solution of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application, and it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments herein without making any inventive effort, shall fall within the scope of the present application.
The full data refers to the entire data of the database system that currently needs to be migrated. With the popularization and popularity of the internet and big data technologies, data warehouse has become a mainstream. Because the data warehouse has a large storage capacity and can process structured data, the data in the database can be synchronized to the data warehouse for processing.
During the data migration process, the business system needs to synchronize third party system data, such as order data, user data, enterprise data, transaction stream. But the third party system only provides the full amount of data. Because the data volume is too large, accurate migration of a large amount of data is difficult to ensure, and problems such as data redundancy are caused.
In order to solve the above problems, the embodiments of the present application provide a full-data synchronization method, apparatus, device, and medium. The file transmission protocol is suitable for transmitting large problems, and when the whole data are synchronized, the file is taken as a carrier, so that the data transmission efficiency can be ensured. Secondly, the embodiment of the application can determine the same or different data with the historical full data file by dividing the full data file in the local sftp server into a plurality of data units and comparing the data units with the historical full data file. The full data file is analyzed into a plurality of data files such as new addition, modification, deletion and the like, and finally converted into sql sentences, and different data are stored differently, so that the problem of full data redundancy is solved.
The following describes the technical scheme provided by the embodiment of the application in detail through the attached drawings.
Fig. 1 is a flowchart of a full-scale data synchronization method according to an embodiment of the present application, where, as shown in fig. 1, the full-scale data synchronization method includes the following steps:
s101, acquiring a full-volume data synchronization request sent by a third party system.
In one embodiment of the application, the third party system sends a full-volume data synchronization http request to the business system. At this time, the full data is not actually sent to the service system, but the position where the full data is located, that is, the position of the file server where the full data file is located.
Specifically, the service system receives a full-volume data synchronization request sent by the third party system, where the request includes the location of the file of the full-volume data to be synchronized currently. And the service system can determine the position of the current full data according to the received full data synchronization request.
S102, determining the position of the full-volume data file based on the full-volume data request, and pulling the full-volume data file to a local sftp server.
In one embodiment of the application, the third party system sftp server is logged in by a preset user name and a preset password. And determining the position of the full-volume data file in the sftp server of the third party system based on the full-volume data request. The full data file is pulled to the local sftp server.
Specifically, after the third party system sends a full-volume data synchronization request to the service system, the position of the full-volume data file is attached, the service system logs in the third party system sftp server through a user name and a password which are informed in advance, and the full-volume data file is pulled to the local sftp server.
In one embodiment of the application, the push frequency corresponding to the full-volume data file is determined, and the local storage directory of the full-volume file is set based on the push frequency. And storing the full-volume data files under different local storage directories based on the pushing frequency, and regularly backing up the full-volume data files under the local storage directories.
Specifically, the local storage directory of the full-volume file is preset according to the pushing frequency of the full-volume file. For example, if the full data file is pushed every day, the full data file is stored every day by taking the date as a folder, and the files are regularly backed up, so that the historical full data file can be checked.
S103, dividing the full-volume data file in the local sftp server into a plurality of data units, and comparing each data unit with the historical full-volume data file to store the full-volume data file into different data files based on the comparison result.
In one embodiment of the application, the full data file is determined to be a newly added data file in the event that the full data file does not exist in the local sftp server. Generating corresponding sql sentences from the newly added data file, and carrying out warehousing processing on different sql sentences.
Specifically, the third-party full data file does not exist in the local sftp server, which indicates that the full data is not synchronized with the service system, and the full data can be imported into the service system. Firstly, converting the full data file of the third party system into the data file of the service system. Data import files are divided into four categories: new data files, modified data files, deleted data files, and unchanged data files. When the third-party full-volume data import file does not exist in the local sftp server, the converted data import files are newly added data files. Generating an insert sql statement from the newly added data file, and carrying out warehousing processing on the sql statement.
In one embodiment of the application, in the case that the full-volume data file exists in the local sftp server, dividing the full-volume data file into a plurality of data units in a row unit, and respectively carrying out unique identification labeling on the plurality of data units. And comparing the plurality of data units with the plurality of data units corresponding to the historical full data file to save the full data file to different data files based on the comparison result.
In one embodiment of the present application, when the full-size data file does not exist in the historical full-size data, the data corresponding to the full-size data file is stored in the newly added data file. And storing the modified full data file into the modified data file under the condition that the full data file exists in the historical full data and the full data file is modified. And storing the data which exists in the historical full data but does not exist in the full data file into the deleted data file. And storing the data existing in both the historical full data and the full data file into the unchanged data file.
Specifically, when the third-party full-volume data file exists in the local sftp server, the third-party full-volume data file is compared with the historical third-party full-volume data file, and four parts of contents are compared: new part data, delete part data, modify part data, and unchanged part data.
The data comparison has a precondition that the data file is used as a data unit in a row unit and has a unique data identification. The current third-party full data is compared with the historical third-party full data line by line, and under the condition that the full data file does not exist in the historical full data, the data corresponding to the full data file is stored into the newly added data file. And storing the modified full data file into the modified data file under the condition that the full data file exists in the historical full data and the full data file is modified. And storing the data which exists in the historical full data but does not exist in the full data file into the deleted data file. And storing the data existing in both the historical full data and the full data file into the unchanged data file.
S104, generating different sql sentences for different data files, and carrying out warehousing processing on the different sql sentences to complete full data synchronization.
In one embodiment of the application, an insert sql statement is generated from a newly added data file; generating an update sql statement from the modified data file; and generating delete statement from the deleted data file.
Specifically, the newly added data file generates an insert sql statement; modifying the data file to generate an update sql statement; deleting the data file to generate delete statement; the unchanged data file is not processed. And (5) warehousing the generated sql statement, and synchronously finishing the full data.
The file transmission protocol of the embodiment of the application is suitable for transmitting large problems, and when the full data is synchronous, the file is taken as a carrier, so that the data transmission efficiency can be ensured. Secondly, the embodiment of the application can determine the same or different data with the historical full data file by dividing the full data file in the local sftp server into a plurality of data units and comparing the data units with the historical full data file. The full data file is analyzed into a plurality of data files such as new addition, modification, deletion and the like, and finally converted into sql sentences, and different data are stored differently, so that the problem of full data redundancy is solved.
Fig. 2 is a block diagram of a full-data synchronization device according to an embodiment of the present application. As shown in fig. 2, the full-volume data synchronization device includes a third party system, a service system, a database, a third party sftp file server, and a local sftp file server. And the third party system sends an http full-volume data synchronization request to inform the position of the full-volume data file. The third party system stores the full amount of data files on a third party sftp file server. And after receiving the full-volume data synchronization request, the service system pulls the full-volume data file on the third-party sftp file server, and stores the pulled third-party full-volume data file on the local sftp file server. And analyzing the full data on the local sftp file server to determine the newly added, modified and deleted data files, analyzing different files into sql sentences, executing the generated sql sentences to put in storage, and synchronously completing the full data.
Fig. 3 is a schematic structural diagram of a full-scale data synchronization device according to an embodiment of the present application, and as shown in fig. 3, a full-scale data synchronization device 300 includes a full-scale data synchronization request obtaining unit 301, a full-scale data file obtaining unit 302, a full-scale data comparing unit 303, and an sql statement generating unit 304.
A full-volume data synchronization request acquisition unit 301 that acquires a full-volume data synchronization request transmitted by a third party system;
a full-volume data file obtaining unit 302, configured to determine a full-volume data file location based on the full-volume data request, and pull the full-volume data file to a local sftp server;
a full-volume data comparison unit 303, dividing the full-volume data file in the local sftp server into a plurality of data units, and comparing each data unit with the historical full-volume data file to save the full-volume data file to different data files based on the comparison result;
the sql statement generation unit 304 generates different sql statements for the different data files, and performs a warehouse entry process for the different sql statements to complete full data synchronization.
Further, the apparatus further comprises:
a full data file storage unit for determining a push frequency corresponding to the full data file; setting a local storage catalog of the full-quantity file based on the pushing frequency; and storing the full-volume data files under different local storage directories based on the pushing frequency, and regularly backing up the full-volume data files under the local storage directories.
Further, the apparatus further comprises:
a new data file processing unit, configured to determine that the full-size data file is a new data file when the full-size data file does not exist in the local sftp server; generating corresponding sql sentences from the newly added data file, and carrying out warehousing processing on the different sql sentences.
Further, the apparatus further comprises:
a full-volume data file comparison unit, which divides the full-volume data file into a plurality of data units in a row unit under the condition that the full-volume data file exists in the local sftp server, and respectively carries out unique identification marking on the plurality of data units; and comparing the data units with the data units corresponding to the historical full data file to save the full data file to different data files based on the comparison result.
Further, the apparatus further comprises:
a full data file dividing unit for storing data corresponding to the full data file into a new data file when the full data file does not exist in the history full data; storing the modified full data file into a modified data file when the full data file exists in the historical full data and the full data file is modified; storing data which exists in the historical full data but does not exist in the full data file into a deleted data file; and storing the data existing in the historical full data and the full data file into a unchanged data file.
Further, the apparatus further comprises:
an sql statement generation unit which generates an insert sql statement from the newly added data file; generating an update sql statement from the modified data file; and generating delete statement from the deleted data file.
Further, the apparatus further comprises:
the full data file pulling unit logs in a third party system sftp server through a preset user name and a preset password; determining the position of the full-volume data file in the third-party system sftp server based on the full-volume data request; and pulling the full data file to a local sftp server.
Fig. 4 is a schematic structural diagram of a full-data synchronization device according to an embodiment of the present application. As shown in fig. 4, the full-volume data synchronization apparatus includes:
at least one processor; the method comprises the steps of,
a memory communicatively coupled to the at least one processor; wherein,
the memory stores instructions executable by the at least one processor to enable the at least one processor to:
acquiring a full data synchronization request sent by a third party system;
determining the position of a full-volume data file based on the full-volume data request, and pulling the full-volume data file to a local sftp server;
dividing the full data file in the local sftp server into a plurality of data units, and comparing each data unit with the historical full data file to store the full data file into different data files based on a comparison result;
and generating different sql sentences for the different data files, and carrying out warehousing processing on the different sql sentences to complete full data synchronization.
Embodiments of the present application also provide a non-volatile computer storage medium storing computer-executable instructions configured to:
acquiring a full data synchronization request sent by a third party system;
determining the position of a full-volume data file based on the full-volume data request, and pulling the full-volume data file to a local sftp server;
dividing the full data file in the local sftp server into a plurality of data units, and comparing each data unit with the historical full data file to store the full data file into different data files based on a comparison result;
and generating different sql sentences for the different data files, and carrying out warehousing processing on the different sql sentences to complete full data synchronization.
The embodiments of the present application are described in a progressive manner, and the same and similar parts of the embodiments are all referred to each other, and each embodiment is mainly described in the differences from the other embodiments. In particular, for apparatus, devices, non-volatile computer storage medium embodiments, the description is relatively simple, as it is substantially similar to method embodiments, with reference to the section of the method embodiments being relevant.
The foregoing describes certain embodiments of the present application. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims can be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
The foregoing is merely exemplary of the present application and is not intended to limit the present application. Various modifications and changes may be made to the embodiments of the application by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the embodiments of the present application should be included in the scope of the claims of the present application.
Claims (10)
1. A method of full data synchronization, the method comprising:
acquiring a full data synchronization request sent by a third party system;
determining the position of a full-volume data file based on the full-volume data request, and pulling the full-volume data file to a local sftp server;
dividing the full data file in the local sftp server into a plurality of data units, and comparing each data unit with the historical full data file to store the full data file into different data files based on a comparison result;
and generating different sql sentences for the different data files, and carrying out warehousing processing on the different sql sentences to complete full data synchronization.
2. The full-size data synchronization method according to claim 1, wherein after the pulling the full-size data file to the local sftp server, the method further comprises:
determining the pushing frequency corresponding to the full data file;
setting a local storage catalog of the full-quantity file based on the pushing frequency;
and storing the full-volume data files under different local storage directories based on the pushing frequency, and regularly backing up the full-volume data files under the local storage directories.
3. A method of full data synchronization according to claim 1, wherein prior to said dividing said full data file in a local sftp server into a plurality of data units, said method further comprises:
determining that the full data file is a newly added data file under the condition that the full data file does not exist in the local sftp server;
generating corresponding sql sentences from the newly added data file, and carrying out warehousing processing on the different sql sentences.
4. The method for synchronizing full-size data according to claim 1, wherein the dividing the full-size data file in the local sftp server into a plurality of data units, comparing each data unit with a historical full-size data file, and storing the full-size data file to different data files based on the comparison result, specifically comprises:
dividing the full-size data file into a plurality of data units in a row unit under the condition that the full-size data file exists in the local sftp server, and respectively carrying out unique identification marking on the plurality of data units;
and comparing the data units with the data units corresponding to the historical full data file to save the full data file to different data files based on the comparison result.
5. The method for synchronizing full-size data according to claim 4, wherein the comparing the plurality of data units with the plurality of data units corresponding to the historical full-size data file to save the full-size data file to a different data file based on the comparison result comprises:
storing data corresponding to the full data file into a new data file under the condition that the full data file does not exist in the historical full data;
storing the modified full data file into a modified data file when the full data file exists in the historical full data and the full data file is modified;
storing data which exists in the historical full data but does not exist in the full data file into a deleted data file;
and storing the data existing in the historical full data and the full data file into a unchanged data file.
6. The full data synchronization method according to claim 1, wherein the generating different sql statements for the different data files, and performing a warehouse entry process on the different sql statements to complete full data synchronization, specifically includes:
generating an insert sql statement from the newly added data file; and
generating an update sql statement from the modified data file; and
the delete data file is generated into delete statement.
7. The full-scale data synchronization method according to claim 1, wherein determining a full-scale data file location based on the full-scale data request and pulling the full-scale data file to a local sftp server specifically comprises:
logging in a third party system sftp server through a preset user name and a preset password;
determining the position of the full-volume data file in the third-party system sftp server based on the full-volume data request;
and pulling the full data file to a local sftp server.
8. A full-scale data synchronization apparatus, comprising:
the full-volume data synchronization request acquisition unit acquires a full-volume data synchronization request sent by a third party system;
the full-volume data file acquisition unit is used for determining the position of the full-volume data file based on the full-volume data request and pulling the full-volume data file to a local sftp server;
the full data comparison unit divides the full data file in the local sftp server into a plurality of data units, compares each data unit with the historical full data file, and stores the full data file into different data files based on a comparison result;
and the sql statement generating unit generates different sql statements for the different data files, and performs warehousing processing on the different sql statements to complete full data synchronization.
9. A full-volume data synchronization device, comprising:
at least one processor; the method comprises the steps of,
a memory communicatively coupled to the at least one processor; wherein,
the memory stores instructions executable by the at least one processor to enable the at least one processor to:
acquiring a full data synchronization request sent by a third party system;
determining the position of a full-volume data file based on the full-volume data request, and pulling the full-volume data file to a local sftp server;
dividing the full data file in the local sftp server into a plurality of data units, and comparing each data unit with the historical full data file to store the full data file into different data files based on a comparison result;
and generating different sql sentences for the different data files, and carrying out warehousing processing on the different sql sentences to complete full data synchronization.
10. A non-transitory computer storage medium storing computer-executable instructions configured to:
acquiring a full data synchronization request sent by a third party system;
determining the position of a full-volume data file based on the full-volume data request, and pulling the full-volume data file to a local sftp server;
dividing the full data file in the local sftp server into a plurality of data units, and comparing each data unit with the historical full data file to store the full data file into different data files based on a comparison result;
and generating different sql sentences for the different data files, and carrying out warehousing processing on the different sql sentences to complete full data synchronization.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310294028.XA CN117194558A (en) | 2023-03-21 | 2023-03-21 | Full-data synchronization method, device, equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310294028.XA CN117194558A (en) | 2023-03-21 | 2023-03-21 | Full-data synchronization method, device, equipment and medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117194558A true CN117194558A (en) | 2023-12-08 |
Family
ID=89002292
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310294028.XA Pending CN117194558A (en) | 2023-03-21 | 2023-03-21 | Full-data synchronization method, device, equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117194558A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118394849A (en) * | 2024-06-26 | 2024-07-26 | 杭州古珀医疗科技有限公司 | Method and device for comparing difference of full-scale data in medical field |
-
2023
- 2023-03-21 CN CN202310294028.XA patent/CN117194558A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118394849A (en) * | 2024-06-26 | 2024-07-26 | 杭州古珀医疗科技有限公司 | Method and device for comparing difference of full-scale data in medical field |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110442579B (en) | State tree data storage method, synchronization method and equipment and storage medium | |
| US20180285400A1 (en) | Index maintenance based on a comparison of rebuild vs. update | |
| CN104423960B (en) | A kind of method and system of project continuous integrating | |
| CN108920698A (en) | A kind of method of data synchronization, device, system, medium and electronic equipment | |
| CN106503158B (en) | Data synchronization method and device | |
| US9529810B2 (en) | Methods of synchronizing files including synchronized hash map linked lists and related data processing nodes | |
| CN106776783B (en) | Unstructured data memory management method and system | |
| GB2565179A (en) | Component-based synchronization of digital assets | |
| CN109388677B (en) | Method, device and equipment for synchronizing data among clusters and storage medium thereof | |
| CN111221469B (en) | Method, device and system for synchronizing cache data | |
| CN105808653A (en) | User label system-based data processing method and device | |
| CN117194558A (en) | Full-data synchronization method, device, equipment and medium | |
| CN110781197A (en) | Hive offline synchronous verification method and device and electronic equipment | |
| CN111177159A (en) | Data processing system and method and data updating equipment | |
| WO2022078243A1 (en) | Information synchronization method and apparatus | |
| CN115687503A (en) | Method, device and equipment for synchronizing data among databases and storage medium | |
| CN112699118A (en) | Data synchronization method, corresponding device, system and storage medium | |
| CN113742376A (en) | Data synchronization method, first server and data synchronization system | |
| CN112148705A (en) | Data migration method and device | |
| CN108959309B (en) | Method and device for data analysis | |
| CN110196883A (en) | A kind of method of data synchronization and device on business on line without influence | |
| CN110543520B (en) | Data migration method and device | |
| CN113645309B (en) | Processing method and system for multi-client data differentiated secondary cache and synchronization | |
| CN110955722B (en) | Data synchronization method and device | |
| CN115086355A (en) | User information processing method and device based on block chain |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |