CN112380180A - Data synchronization processing method, device, equipment and storage medium - Google Patents
Data synchronization processing method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN112380180A CN112380180A CN202011282969.4A CN202011282969A CN112380180A CN 112380180 A CN112380180 A CN 112380180A CN 202011282969 A CN202011282969 A CN 202011282969A CN 112380180 A CN112380180 A CN 112380180A
- Authority
- CN
- China
- Prior art keywords
- data
- query
- table structure
- distributed file
- file system
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 28
- 238000000547 structure data Methods 0.000 claims abstract description 82
- 238000007418 data mining Methods 0.000 claims abstract description 38
- 238000006243 chemical reaction Methods 0.000 claims abstract description 36
- 238000004140 cleaning Methods 0.000 claims abstract description 34
- 230000001360 synchronised effect Effects 0.000 claims abstract description 9
- 238000012545 processing Methods 0.000 claims description 51
- 238000005065 mining Methods 0.000 claims description 6
- 238000004458 analytical method Methods 0.000 claims description 5
- 238000013507 mapping Methods 0.000 claims description 5
- 238000012216 screening Methods 0.000 claims description 5
- 230000008676 import Effects 0.000 abstract description 18
- 238000005516 engineering process Methods 0.000 abstract description 3
- 238000000034 method Methods 0.000 description 17
- 238000013500 data storage Methods 0.000 description 12
- 238000010586 diagram Methods 0.000 description 6
- 230000008569 process Effects 0.000 description 5
- 230000006870 function Effects 0.000 description 3
- 230000002159 abnormal effect Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 230000002085 persistent effect Effects 0.000 description 1
- 238000005096 rolling process Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001052 transient effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/178—Techniques for file synchronisation in file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/275—Synchronous replication
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to the technical field of big data, and discloses a data synchronous processing method, a device, equipment and a storage medium, which are used for improving data conversion and import efficiency. The data synchronization processing method comprises the following steps: reading a plurality of table structure data in batch from a preset distributed file system according to a data mining request; sequentially cleaning and splicing the data of the plurality of table structure data according to the SQL grammar rule of the structured query language to obtain a plurality of SQL query sentences; connecting a target relational database, and performing data query operation according to a plurality of SQL query sentences to obtain a query result set; performing data cleaning and format conversion on the query result set to obtain a plurality of distributed file system files; and importing the files of the distributed file systems into a preset distributed file system respectively to obtain an updating result, and sending the updating result to a target terminal. In addition, the invention also relates to a block chain technology, and a plurality of table structure data can be stored in the block chain nodes.
Description
Technical Field
The present invention relates to the field of incremental synchronization in big data technology, and in particular, to a data synchronization processing method, apparatus, device, and storage medium.
Background
With the wider application of big data, the data is often extracted from a traditional relational database (such as oracle or mysql), cleaned and assembled, and finally synchronized to a data warehouse of a big data platform.
However, the bottom database system HDFS in the existing common large data platform is greatly different from the traditional relational database, the data storage format and the storage rule are completely different, even if the existing synchronization tool is used, the problem of complicated conversion steps and difficulty in conversion still exists, and meanwhile, when mass data conversion is involved, the problem of low conversion and import efficiency also exists.
Disclosure of Invention
The invention provides a data synchronization processing method, a data synchronization processing device, data synchronization processing equipment and a storage medium, which are used for importing a large amount of data into a preset distributed file system and improving data conversion and importing efficiency.
In order to achieve the above object, a first aspect of the present invention provides a data synchronization processing method, including: receiving a data mining request, acquiring a target script file according to the data mining request, and reading a plurality of table structure data in batch from a preset distributed file system through the target script file, wherein the target script file comprises a plurality of table structure query statements; sequentially performing data cleaning and statement splicing on the plurality of table structure data according to a Structured Query Language (SQL) grammar rule to obtain a plurality of SQL query statements; connecting the target relational database to obtain a connection result, and when the connection result is successful, performing data query operation according to the plurality of SQL query sentences to obtain a query result set; when the query result set is not null, performing data cleaning and format conversion on the query result set to obtain a plurality of distributed file system files, wherein each distributed file system file is used for storing query data corresponding to each SQL query statement; and respectively importing the files of the distributed file systems into the preset distributed file system to obtain an updating result, and sending the updating result to a target terminal.
Optionally, in a first implementation manner of the first aspect of the present invention, the receiving a data mining request, obtaining a target script file according to the data mining request, and reading a plurality of table structure data in batch from a preset distributed file system through the target script file, where the target script file includes a plurality of table structure query statements, includes: receiving a data mining request, and performing parameter analysis on the data mining request to obtain a service identification value; inquiring a preset mapping data table according to the service identification value to obtain script file path information, and reading a target script file according to the script file path information, wherein the target script file comprises a plurality of table structure inquiry sentences; and calling the target script file through a preset standard command, reading a plurality of table structure data in batch from a preset distributed file system, and writing the plurality of table structure data into a target text file.
Optionally, in a second implementation manner of the first aspect of the present invention, the sequentially performing data cleaning and statement splicing on the multiple table structure data according to a structured query language SQL syntax rule, to obtain multiple SQL query statements, includes: executing a preset cleaning statement, screening table field names of a plurality of table structure data in the target text file, and deleting redundant characters to obtain the cleaned table structure data; performing character replacement on all line end symbols except the end statement in the cleaned table structure data to obtain replaced table structure data; dividing and splicing all fields in the replaced table structure data according to commas to obtain a plurality of rows of spliced table structure data, wherein each row of spliced table structure data is used for indicating a plurality of fields in each table structure data; and respectively carrying out data assembly on the multiple rows of spliced table structures according to the SQL grammar rules of the structured query language to obtain multiple SQL query statements.
Optionally, in a third implementation manner of the first aspect of the present invention, the connecting the target relational database to obtain a connection result, and when the connection result is a successful connection, performing data query operation according to the multiple SQL query statements to obtain a query result set, where the method includes: acquiring database connection address information and user name and password information, and connecting a target relational database according to the database connection address information and the user name and password information through a preset database driving module to obtain a connection result; when the connection result is that the connection is successful, creating a database cursor, executing the plurality of SQL query statements according to the database cursor to obtain a query result set, and recording the query result set; closing the database cursor and releasing the memory resource occupied by the database cursor.
Optionally, in a fourth implementation manner of the first aspect of the present invention, when the query result set is not a null value, performing data cleaning and format conversion on the query result set to obtain a plurality of distributed file system files, where each distributed file system file is used to store query data corresponding to each SQL query statement, and the method includes: when the query result set is not null, reserving the names and values of the fields in the query result set; replacing separators among the fields according to the names of the fields, and performing format conversion on the data of the timestamp type to obtain a processed data set; and recording and storing the processed data set based on the redirection command to obtain a plurality of distributed file system files, wherein each distributed file system file is used for storing the query data corresponding to each SQL query statement.
Optionally, in a fifth implementation manner of the first aspect of the present invention, the importing the multiple distributed file system files into the preset distributed file system respectively to obtain an update result, and sending the update result to a target terminal includes: importing the distributed file systems into the preset distributed file system respectively in a transaction mode to obtain an updating result; when the updating result is not a preset value, determining that data synchronization processing fails, generating early warning information, and sending the early warning information to a target terminal so that the target terminal displays the early warning information; and when the updating result is a preset value, determining that the data synchronization processing is successful, and sending the updating result to the target terminal.
Optionally, in a sixth implementation manner of the first aspect of the present invention, after the files of the multiple distributed file systems are respectively imported into the preset distributed file system to obtain an update result, and the update result is sent to a target terminal, the data synchronization processing method further includes: updating the updating result into a log file, wherein the log file records the result of data synchronous processing; and analyzing and counting the data in the log file at regular time to obtain a data synchronization report, and sending the data synchronization report to the target terminal.
A second aspect of the present invention provides a data synchronization processing apparatus, including: the acquisition module is used for receiving a data mining request, acquiring a target script file according to the data mining request, and reading a plurality of table structure data in batch from a preset distributed file system through the target script file, wherein the target script file comprises a plurality of table structure query statements; the mining module is used for respectively and sequentially carrying out data cleaning and statement splicing on the plurality of table structure data according to the SQL grammar rules of the structured query language to obtain a plurality of SQL query statements; the query module is used for connecting the target relational database to obtain a connection result, and when the connection result is successful, performing data query operation according to the plurality of SQL query sentences to obtain a query result set; the conversion module is used for carrying out data cleaning and format conversion on the query result set when the query result set is not a null value to obtain a plurality of distributed file system files, and each distributed file system file is used for storing the query data corresponding to each SQL query statement; and the importing module is used for respectively importing the files of the distributed file systems into the preset distributed file system to obtain an updating result and sending the updating result to the target terminal.
Optionally, in a first implementation manner of the second aspect of the present invention, the obtaining module is specifically configured to: receiving a data mining request, and performing parameter analysis on the data mining request to obtain a service identification value; inquiring a preset mapping data table according to the service identification value to obtain script file path information, and reading a target script file according to the script file path information, wherein the target script file comprises a plurality of table structure inquiry sentences; and calling the target script file through a preset standard command, reading a plurality of table structure data in batch from a preset distributed file system, and writing the plurality of table structure data into a target text file.
Optionally, in a second implementation manner of the second aspect of the present invention, the mining module is specifically configured to: executing a preset cleaning statement, screening table field names of a plurality of table structure data in the target text file, and deleting redundant characters to obtain the cleaned table structure data; performing character replacement on all line end symbols except the end statement in the cleaned table structure data to obtain replaced table structure data; dividing and splicing all fields in the replaced table structure data according to commas to obtain a plurality of rows of spliced table structure data, wherein each row of spliced table structure data is used for indicating a plurality of fields in each table structure data; and respectively carrying out data assembly on the multiple rows of spliced table structures according to the SQL grammar rules of the structured query language to obtain multiple SQL query statements.
Optionally, in a third implementation manner of the second aspect of the present invention, the query module is specifically configured to: acquiring database connection address information and user name and password information, and connecting a target relational database according to the database connection address information and the user name and password information through a preset database driving module to obtain a connection result; when the connection result is that the connection is successful, creating a database cursor, executing the plurality of SQL query statements according to the database cursor to obtain a query result set, and recording the query result set; closing the database cursor and releasing the memory resource occupied by the database cursor.
Optionally, in a fourth implementation manner of the second aspect of the present invention, the conversion module is specifically configured to: when the query result set is not null, reserving the names and values of the fields in the query result set; replacing separators among the fields according to the names of the fields, and performing format conversion on the data of the timestamp type to obtain a processed data set; and recording and storing the processed data set based on the redirection command to obtain a plurality of distributed file system files, wherein each distributed file system file is used for storing the query data corresponding to each SQL query statement.
Optionally, in a fifth implementation manner of the second aspect of the present invention, the import module is specifically configured to: importing the distributed file systems into the preset distributed file system respectively in a transaction mode to obtain an updating result; when the updating result is not a preset value, determining that data synchronization processing fails, generating early warning information, and sending the early warning information to a target terminal so that the target terminal displays the early warning information; and when the updating result is a preset value, determining that the data synchronization processing is successful, and sending the updating result to the target terminal.
Optionally, in a sixth implementation manner of the second aspect of the present invention, the data synchronization processing apparatus further includes: the updating module is used for updating the updating result into a log file, and the log file records the result of data synchronous processing; and the counting module is used for analyzing and counting the data in the log file at regular time to obtain a data synchronization report and sending the data synchronization report to the target terminal.
A third aspect of the present invention provides a data synchronization processing apparatus, including: a memory and at least one processor, the memory having instructions stored therein; the at least one processor calls the instructions in the memory to cause the data synchronization processing device to execute the data synchronization processing method.
A fourth aspect of the present invention provides a computer-readable storage medium having stored therein instructions, which, when run on a computer, cause the computer to execute the above-described data synchronization processing method.
In the technical scheme provided by the invention, a data mining request is received, a target script file is obtained according to the data mining request, and a plurality of table structure data are read in batch from a preset distributed file system through the target script file, wherein the target script file comprises a plurality of table structure query statements; sequentially performing data cleaning and statement splicing on the plurality of table structure data according to a Structured Query Language (SQL) grammar rule to obtain a plurality of SQL query statements; connecting the target relational database to obtain a connection result, and when the connection result is successful, performing data query operation according to the plurality of SQL query sentences to obtain a query result set; when the query result set is not null, performing data cleaning and format conversion on the query result set to obtain a plurality of distributed file system files, wherein each distributed file system file is used for storing query data corresponding to each SQL query statement; and respectively importing the files of the distributed file systems into the preset distributed file system to obtain an updating result, and sending the updating result to a target terminal. In the embodiment of the invention, table structures in a preset distributed file system are read in batch through a target script file, and are preliminarily cleaned to obtain SQL query statements conforming to the syntax of a relational database; and connecting the target relational database, reading and executing SQL query statements, cleaning and assembling query result data to obtain a data storage format conforming to the distributed file system, storing the data storage format in a text file, reading and executing the text file through a shell script, and importing a large amount of data into the preset distributed file system in batches, so that the data conversion and importing efficiency is improved.
Drawings
FIG. 1 is a schematic diagram of an embodiment of a data synchronization processing method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of another embodiment of a data synchronization processing method according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of an embodiment of a data synchronization processing apparatus according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of another embodiment of a data synchronization processing apparatus according to an embodiment of the present invention;
fig. 5 is a schematic diagram of an embodiment of a data synchronization processing device in an embodiment of the present invention.
Detailed Description
The embodiment of the invention provides a data synchronous processing method, a data synchronous processing device, data synchronous processing equipment and a data synchronous processing storage medium, which are used for importing a large amount of data into a preset distributed file system in batches and improving data conversion and importing efficiency.
The terms "first," "second," "third," "fourth," and the like in the description and in the claims, as well as in the drawings, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It will be appreciated that the data so used may be interchanged under appropriate circumstances such that the embodiments described herein may be practiced otherwise than as specifically illustrated or described herein. Furthermore, the terms "comprises," "comprising," or "having," and any variations thereof, are intended to cover non-exclusive inclusions, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
For convenience of understanding, a specific flow of the embodiment of the present invention is described below, and referring to fig. 1, an embodiment of a data synchronization processing method in the embodiment of the present invention includes:
101. the method comprises the steps of receiving a data mining request, obtaining a target script file according to the data mining request, and reading a plurality of table structure data in batches from a preset distributed file system through the target script file, wherein the target script file comprises a plurality of table structure query statements.
The data mining request may correspond to different actual services, for example, the actual service is a claim settlement service or a loan service, or may be other services, which is not limited herein. When the corresponding actual services are different, the data tables corresponding to the data mining requests are also different. Specifically, the server acquires a target script file corresponding to the data mining request from a preset file directory, wherein the preset file directory is used for storing a plurality of preset script files (including the target script file); the method comprises the steps that a server executes a plurality of table structure query statements in a target script file, the server reads a plurality of table structure data in batch from a preset distributed file system (HDFS), and the plurality of table structure query statements are all preset in the target script file according to actual services. Further, the plurality of table structure data are stored in the blockchain database, which is not limited herein.
It is to be understood that the execution subject of the present invention may be a data synchronization processing apparatus, and may also be a terminal or a server, which is not limited herein. The embodiment of the present invention is described by taking a server as an execution subject.
102. And sequentially cleaning data and splicing sentences according to the SQL grammar rule of the structured query language to obtain a plurality of SQL query sentences.
The server performs data replacement, statement splicing and other operations on the multiple table structure data respectively by combining a shell command (for example, a sed command) according to a Structured Query Language (SQL) grammar rule, so as to obtain multiple SQL query statements conforming to the target relational database grammar, and introduces the multiple SQL statements into a target text file, or the server may store the multiple SQL statements into a new file (a newly created file). Structured Query Language (SQL), a database query and programming language, is used to access data and query, update, and manage relational database systems.
103. And connecting the target relational database to obtain a connection result, and when the connection result is successful, performing data query operation according to a plurality of SQL query sentences to obtain a query result set.
The server is connected with the data warehouse hive, specifically, the server acquires and executes a command (for example, ssh user @ ip: port) for database connection to obtain a connection result; the server judges whether the connection result is a preset target value or not, and if the connection result is the preset target value, the server determines that the connection result is successful; if the connection result is not the preset target value, the server determines that the connection result is a connection failure, wherein the preset target value may be 1 or true logic, and the specific details are not limited herein; and when the server is successfully connected with the target relational database, the server executes a plurality of SQL query statements to obtain a query result set of the relational database.
104. And when the query result set is not a null value, performing data cleaning and format conversion on the query result set to obtain a plurality of distributed file system files, wherein each distributed file system file is used for storing the query data corresponding to each SQL query statement.
That is, the name of each distributed file system file may be named according to different names of the data tables, so that the server obtains a plurality of distributed file system files corresponding to the original table in the relational database one by one, and inserts data in the plurality of distributed file system files into corresponding target tables in the data warehouse. The server can also judge whether each distributed file system file has a corresponding data table; and if each distributed file system file has a corresponding data table, the server sets corresponding access authority for each distributed file system file.
105. And importing the files of the distributed file systems into a preset distributed file system respectively to obtain an updating result, and sending the updating result to a target terminal.
Specifically, the server reads and executes a plurality of distributed file system files through the shell script, and imports a large amount of data into a preset distributed file system (HDFS) in batch, wherein instructions in the shell script are as follows:
LOAD DATA LOCAL INPATH'/localPath/file'
OVERWRITE INTO TABLE table_name
PARTITION(creat_day＝'2020-8-7')
the server can directly and quickly import the data in the distributed file system files into the HDFS library through the command in the shell script, so that the conversion and the import (data synchronization processing) from the target relational database to the HDFS are realized. Wherein, the table _ name is consistent with the written data table name. Further, the server obtains the updating result and calls a preset interface to send the updating result to the target terminal, wherein the preset interface is used for transmitting data to the target terminal.
In the embodiment of the invention, table structures in a preset distributed file system are read in batch through a target script file, and are preliminarily cleaned to obtain SQL query statements conforming to the syntax of a relational database; and connecting the target relational database, reading and executing SQL query statements, cleaning and assembling query result data to obtain a data storage format conforming to the distributed file system, storing the data storage format in a text file, reading and executing the text file through a shell script, and importing a large amount of data into the preset distributed file system in batches, so that the data conversion and importing efficiency is improved.
Referring to fig. 2, another embodiment of the data synchronization processing method according to the embodiment of the present invention includes:
201. the method comprises the steps of receiving a data mining request, obtaining a target script file according to the data mining request, and reading a plurality of table structure data in batches from a preset distributed file system through the target script file, wherein the target script file comprises a plurality of table structure query statements.
The data mining request includes a service identification value, the service identification value may be mainly composed of numbers, letters and/or special symbols, the special symbols include underlines, dot symbols and the like, and for example, the server uses service _001 to represent the service identification value.
Optionally, the server receives the data mining request, and performs parameter analysis on the data mining request to obtain a service identification value; the server inquires a preset mapping data table according to the service identification value to obtain script file path information, and the server reads a target script file according to the script file path information, wherein the target script file comprises a plurality of table structure inquiry sentences; and the server calls the target script file through a preset standard command, reads a plurality of table structure data in batch from a preset distributed file system, and writes the plurality of table structure data into the target text file. The preset standard command is a data warehouse hive, for example, the preset standard command is hive-f xxx. The server stores the HQL sentences for inquiring the table structure of the HDFS in a target script file xxx.sql, then the server firstly connects a hive server where the HDFS is located in a shell script, and executes the commands to obtain a target text file out.txt in which all the table structures are stored, wherein ">" is used for expressing output data flow. The target script file and the target text file both have read-write execution authority. It will be appreciated that the name of the target script file and the name of the target text file may be any names that are custom defined.
It should be noted that the target script file is a shell script file, and includes a plurality of preset shell script instructions for connecting to a preset distributed file system and screening related data tables, and each table structure query statement may view the data structure of a single table through a framework Hibernate Query Language (HQL) statement.
202. And sequentially performing data cleaning and statement splicing on the plurality of table structure data according to the Structured Query Language (SQL) grammar rules to obtain a plurality of SQL query statements.
It can be understood that the plurality of table structure data are data read by the server in batch from the preset distributed file system, and the server further needs to convert the plurality of table structure data into a plurality of SQL query statements. Optionally, the server executes a preset cleaning statement, screens table field names for the plurality of table structure data in the target text file, and deletes redundant characters to obtain the cleaned table structure data. So that only table field names are contained in the cleaned-up table structure data and all fields of each table occupy one row of data. The redundant characters may include space characters, blank characters, commas, periods, and other characters, which are not limited herein.
Secondly, the server carries out character replacement on all line end symbols except the end statement in the cleaned list structure data to obtain the replaced list structure data. It should be noted that the server replaces all line end symbols except the end statement in the cleaned table structure data with commas by using a preset positive expression.
And then, the server divides and splices all fields in the replaced table structure data according to the commas to obtain a plurality of lines of spliced table structure data, wherein each line of spliced table structure data is used for indicating a plurality of fields in each table structure data. And finally, the server respectively carries out data assembly on the spliced table structures according to the SQL grammar rule of the structured query language to obtain a plurality of SQL query statements. That is, the server combines the query select statement and the field name for the structured query language SQL syntax rule to obtain a complete traditional database query statement (multiple SQL query statements), and imports the multiple SQL query statements into the target text file.
203. And acquiring database connection address information and user name and password information, and connecting a target relational database through a preset database driving module according to the database connection address information and the user name and password information to obtain a connection result.
Wherein the connection result comprises connection success and connection failure. For example, a statement that the server connects to the target relational database based on the obtained database connection address information and the username and password information is db ═ cx _ Oracle. It should be noted that the server also needs to pre-import a relational database driver package (that is, preset a database driver module), where the target relational database includes Oracle and mysql, or may include other types of databases, and taking the Oracle database and the mysql database as examples, the Oracle database may use a cx _ Oracle database connection module, and the mysql database may use a pymysql database connection module.
204. And when the connection result is that the connection is successful, creating a database cursor, executing a plurality of SQL query statements according to the database cursor to obtain a query result set, and recording the query result set.
Specifically, the server reads each SQL query statement line by line from the target text file, and stores each SQL query statement in a preset variable (assuming that the preset variable is SQL _ str); the server creates a database cursor cur ═ db. The server executes the SQL statement using the database cursor, e.g., the server executes the SQL statement cur. And the server acquires the query result set line by line through the cursors of the database and writes the query result set into the file C.
205. Closing the database cursor and releasing the memory resource occupied by the database cursor.
The server closes the database cursor by db.close (), further, the server releases memory resources occupied by the database cursor, and the data processing efficiency is improved.
206. And when the query result set is not a null value, performing data cleaning and format conversion on the query result set to obtain a plurality of distributed file system files, wherein each distributed file system file is used for storing the query data corresponding to each SQL query statement.
Further, the server determines whether the query result set is a null value, where the null value is null, undefined, 0, or a null string, and may also be a logical false, and is not limited herein. Optionally, when the query result set is not a null value, the server retains the name and the value of each field in the query result set; the method comprises the following steps that a server replaces separators among fields according to names of the fields and performs format conversion on data of a timestamp type to obtain a processed data set, specifically, the server is different according to table structure types, the separators are also different and are defaulted to be spaces, if a file contains data of the timestamp type, a time-minute-second part needs to be removed, only a date part is reserved, and otherwise, processing is abnormal when the server is led into a distributed file system; and the server records and stores the processed data set according to the affiliated data table based on the redirection command to obtain a plurality of distributed file system files, wherein each distributed file system file is used for storing the query data corresponding to each SQL query statement. When the server uses the shell command to operate, the server uses the redirect command ">" to output the processed data set to the distributed file system file, and further, when the server can also use python to perform data writing operation, for example, the server can call a read function read () and a write function write () to perform file reading and writing through a built-in file module. The server may specify a file path (where the file is saved) and a file name (naming method and suffix format) to save a plurality of distributed file system files when writing the processed data set to the file.
207. And importing the files of the distributed file systems into a preset distributed file system respectively to obtain an updating result, and sending the updating result to a target terminal.
Wherein the updating result comprises updating success and updating failure. Optionally, the server separately imports the multiple distributed file system files into a preset distributed file system in a transaction manner, so as to obtain an update result; when the updating result is not a preset value, the server determines that data synchronization processing fails, generates early warning information and sends the early warning information to the target terminal so that the target terminal can display the early warning information; and when the updating result is a preset value, the server determines that the data synchronization processing is successful and sends the updating result to the target terminal. The preset value may be 1, may also be a logic true value, and may also be other values, which are not limited herein.
Further, the server can also import the files of the distributed file systems into the preset distributed file system in a transaction mode to obtain the updating result. Specifically, the server initializes the middleware object; the server calls a file writing interface based on the middleware object to respectively conduct file import operation on the files of the distributed file systems to obtain an updating result, and judges whether the file import operation is successfully imported or not based on the updating result; when the file import operation is successful, submitting transaction write-in operation to a preset distributed file system; when the file import operation is import failure, rolling back the transaction write operation to a preset distributed file system; the server records the transaction submission log or the transaction rollback log, and sends the update result to the target terminal, so that the target terminal generates and displays prompt information according to the update result, and the consistency of data synchronization is improved.
Optionally, the server updates the update result to a log file, where the log file records the result of the data synchronization processing; and the server analyzes and counts the data in the log file at regular time to obtain a data synchronization report, and sends the data synchronization report to the target terminal. It can be understood that the log file is used for recording the data mining and data importing processes of the server, so that the server can trace the source of the abnormal operation of the data synchronization processing.
In the embodiment of the invention, table structures in a preset distributed file system are read in batch through a target script file, and are preliminarily cleaned to obtain SQL query statements conforming to the syntax of a relational database; and connecting the target relational database, reading and executing SQL query statements, cleaning and assembling query result data to obtain a data storage format conforming to the distributed file system, storing the data storage format in a text file, reading and executing the text file through a shell script, and importing a large amount of data into the preset distributed file system in batches, so that the data conversion and importing efficiency is improved.
With reference to fig. 3, the data synchronization processing method in the embodiment of the present invention is described above, and a data synchronization processing apparatus in the embodiment of the present invention is described below, where an embodiment of the data synchronization processing apparatus in the embodiment of the present invention includes:
an obtaining module 301, configured to receive a data mining request, obtain a target script file according to the data mining request, and read a plurality of table structure data in batch from a preset distributed file system through the target script file, where the target script file includes a plurality of table structure query statements;
the mining module 302 is configured to sequentially perform data cleaning and statement splicing on the plurality of table structure data according to a Structured Query Language (SQL) syntax rule to obtain a plurality of SQL query statements;
the query module 303 is configured to connect to the target relational database to obtain a connection result, and when the connection result is a connection success, perform data query operation according to a plurality of SQL query statements to obtain a query result set;
the conversion module 304 is configured to, when the query result set is not a null value, perform data cleaning and format conversion on the query result set to obtain a plurality of distributed file system files, where each distributed file system file is used to store query data corresponding to each SQL query statement;
the importing module 305 is configured to import the multiple distributed file system files into a preset distributed file system, obtain an update result, and send the update result to the target terminal.
Further, the plurality of table structure data are stored in the blockchain database, which is not limited herein.
In the embodiment of the invention, table structures in a preset distributed file system are read in batch through a target script file, and are preliminarily cleaned to obtain SQL query statements conforming to the syntax of a relational database; and connecting the target relational database, reading and executing SQL query statements, cleaning and assembling query result data to obtain a data storage format conforming to the distributed file system, storing the data storage format in a text file, reading and executing the text file through a shell script, and importing a large amount of data into the preset distributed file system in batches, so that the data conversion and importing efficiency is improved.
Referring to fig. 4, another embodiment of a data synchronization processing apparatus according to the embodiment of the present invention includes:
an obtaining module 301, configured to receive a data mining request, obtain a target script file according to the data mining request, and read a plurality of table structure data in batch from a preset distributed file system through the target script file, where the target script file includes a plurality of table structure query statements;
the mining module 302 is configured to sequentially perform data cleaning and statement splicing on the plurality of table structure data according to a Structured Query Language (SQL) syntax rule to obtain a plurality of SQL query statements;
the query module 303 is configured to connect to the target relational database to obtain a connection result, and when the connection result is a connection success, perform data query operation according to a plurality of SQL query statements to obtain a query result set;
the conversion module 304 is configured to, when the query result set is not a null value, perform data cleaning and format conversion on the query result set to obtain a plurality of distributed file system files, where each distributed file system file is used to store query data corresponding to each SQL query statement;
the importing module 305 is configured to import the multiple distributed file system files into a preset distributed file system, obtain an update result, and send the update result to the target terminal.
Optionally, the obtaining module 301 may be further specifically configured to:
receiving a data mining request, and performing parameter analysis on the data mining request to obtain a service identification value;
inquiring a preset mapping data table according to the service identification value to obtain script file path information, and reading a target script file according to the script file path information, wherein the target script file comprises a plurality of table structure inquiry sentences;
and calling the target script file through a preset standard command, reading a plurality of table structure data in batch from a preset distributed file system, and writing the plurality of table structure data into the target text file.
Optionally, the mining module 302 is further specifically configured to:
executing a preset cleaning statement, screening table field names of a plurality of table structure data in a target text file, and deleting redundant characters to obtain the cleaned table structure data;
performing character replacement on all line end symbols except the end statement in the cleaned table structure data to obtain the replaced table structure data;
dividing and splicing all fields in the replaced table structure data according to commas to obtain a plurality of rows of spliced table structure data, wherein each row of spliced table structure data is used for indicating a plurality of fields in each table structure data;
and respectively carrying out data assembly on the spliced table structures according to the SQL grammar rules of the structured query language to obtain a plurality of SQL query statements.
Optionally, the query module 303 is further specifically configured to:
acquiring database connection address information and user name password information, and connecting a target relational database through a preset database driving module according to the database connection address information and the user name password information to obtain a connection result;
when the connection result is that the connection is successful, creating a database cursor, executing a plurality of SQL query statements according to the database cursor to obtain a query result set, and recording the query result set;
closing the database cursor and releasing the memory resource occupied by the database cursor.
Optionally, the conversion module 304 may be further specifically configured to:
when the query result set is not null, the names and the values of the fields in the query result set are reserved;
replacing separators among the fields according to the names of the fields, and performing format conversion on the data of the timestamp type to obtain a processed data set;
and recording and storing the processed data set based on the redirection command to obtain a plurality of distributed file system files, wherein each distributed file system file is used for storing the query data corresponding to each SQL query statement.
Optionally, the import module 305 may be further specifically configured to:
respectively importing a plurality of distributed file system files into a preset distributed file system in a transaction mode to obtain an updating result;
when the updating result is not a preset value, determining that data synchronization processing fails, generating early warning information, and sending the early warning information to a target terminal so that the target terminal displays the early warning information;
and when the updating result is a preset value, determining that the data synchronization processing is successful, and sending the updating result to the target terminal.
Optionally, the data synchronization processing apparatus further includes:
an updating module 306, configured to update an updating result to a log file, where the log file records a result of data synchronization processing;
and the counting module 307 is configured to analyze and count the data in the log file at regular time to obtain a data synchronization report, and send the data synchronization report to the target terminal.
In the embodiment of the invention, table structures in a preset distributed file system are read in batch through a target script file, and are preliminarily cleaned to obtain SQL query statements conforming to the syntax of a relational database; and connecting the target relational database, reading and executing SQL query statements, cleaning and assembling query result data to obtain a data storage format conforming to the distributed file system, storing the data storage format in a text file, reading and executing the text file through a shell script, and importing a large amount of data into the preset distributed file system in batches, so that the data conversion and importing efficiency is improved.
Fig. 3 and fig. 4 describe the data synchronization processing apparatus in the embodiment of the present invention in detail from the perspective of modularization, and the data synchronization processing apparatus in the embodiment of the present invention is described in detail from the perspective of hardware processing.
Fig. 5 is a schematic structural diagram of a data synchronization processing apparatus according to an embodiment of the present invention, where the data synchronization processing apparatus 500 may generate relatively large differences due to different configurations or performances, and may include one or more processors (CPUs) 510 (e.g., one or more processors) and a memory 520, and one or more storage media 530 (e.g., one or more mass storage devices) storing applications 533 or data 532. Memory 520 and storage media 530 may be, among other things, transient or persistent storage. The program stored in the storage medium 530 may include one or more modules (not shown), each of which may include a series of instruction operations for the data synchronization processing apparatus 500. Further, the processor 510 may be configured to communicate with the storage medium 530, and execute a series of instruction operations in the storage medium 530 on the data synchronization processing apparatus 500.
The data synchronization processing apparatus 500 may also include one or more power supplies 540, one or more wired or wireless network interfaces 550, one or more input-output interfaces 560, and/or one or more operating systems 531, such as Windows server, Mac OS X, Unix, Linux, FreeBSD, and the like. Those skilled in the art will appreciate that the data synchronization processing apparatus configuration shown in fig. 5 does not constitute a limitation of the data synchronization processing apparatus, and may include more or less components than those shown, or combine some components, or a different arrangement of components.
The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium, and which may also be a volatile computer-readable storage medium, having stored therein instructions, which, when run on a computer, cause the computer to perform the steps of the data synchronization processing method.
The present invention also provides a data synchronization processing apparatus, which includes a memory and a processor, wherein the memory stores instructions, and the instructions, when executed by the processor, cause the processor to execute the steps of the data synchronization processing method in each of the above embodiments.
Further, the computer-readable storage medium may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function, and the like; the storage data area may store data created according to the use of the blockchain node, and the like.
The block chain is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, a consensus mechanism, an encryption algorithm and the like. A block chain (Blockchain), which is essentially a decentralized database, is a series of data blocks associated by using a cryptographic method, and each data block contains information of a batch of network transactions, so as to verify the validity (anti-counterfeiting) of the information and generate a next block. The blockchain may include a blockchain underlying platform, a platform product service layer, an application service layer, and the like.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to each embodiment of the present invention. And the aforementioned storage medium includes: a U disk, a removable hard disk, a read-only memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and the like.
The above-mentioned embodiments are only used for illustrating the technical solutions of the present invention, and not for limiting the same; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in each of the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of each embodiment of the present invention.
Claims (10)
1. A data synchronization processing method is characterized by comprising the following steps:
receiving a data mining request, acquiring a target script file according to the data mining request, and reading a plurality of table structure data in batch from a preset distributed file system through the target script file, wherein the target script file comprises a plurality of table structure query statements;
sequentially performing data cleaning and statement splicing on the plurality of table structure data according to a Structured Query Language (SQL) grammar rule to obtain a plurality of SQL query statements;
connecting the target relational database to obtain a connection result, and when the connection result is successful, performing data query operation according to the plurality of SQL query sentences to obtain a query result set;
when the query result set is not null, performing data cleaning and format conversion on the query result set to obtain a plurality of distributed file system files, wherein each distributed file system file is used for storing query data corresponding to each SQL query statement;
and respectively importing the files of the distributed file systems into the preset distributed file system to obtain an updating result, and sending the updating result to a target terminal.
2. The data synchronization processing method according to claim 1, wherein the receiving a data mining request, obtaining a target script file according to the data mining request, and reading a plurality of table structure data in batch from a preset distributed file system through the target script file, the target script file including a plurality of table structure query statements, includes:
receiving a data mining request, and performing parameter analysis on the data mining request to obtain a service identification value;
inquiring a preset mapping data table according to the service identification value to obtain script file path information, and reading a target script file according to the script file path information, wherein the target script file comprises a plurality of table structure inquiry sentences;
and calling the target script file through a preset standard command, reading a plurality of table structure data in batch from a preset distributed file system, and writing the plurality of table structure data into a target text file.
3. The data synchronization processing method according to claim 2, wherein the sequentially performing data cleaning and statement splicing on the plurality of table structure data according to Structured Query Language (SQL) syntax rules to obtain a plurality of SQL query statements comprises:
executing a preset cleaning statement, screening table field names of a plurality of table structure data in the target text file, and deleting redundant characters to obtain the cleaned table structure data;
performing character replacement on all line end symbols except the end statement in the cleaned table structure data to obtain replaced table structure data;
dividing and splicing all fields in the replaced table structure data according to commas to obtain a plurality of rows of spliced table structure data, wherein each row of spliced table structure data is used for indicating a plurality of fields in each table structure data;
and respectively carrying out data assembly on the multiple rows of spliced table structures according to the SQL grammar rules of the structured query language to obtain multiple SQL query statements.
4. The data synchronization processing method according to claim 1, wherein the connecting the target relational database to obtain a connection result, and when the connection result is a connection success, performing data query operation according to the plurality of SQL query statements to obtain a query result set, includes:
acquiring database connection address information and user name and password information, and connecting a target relational database according to the database connection address information and the user name and password information through a preset database driving module to obtain a connection result;
when the connection result is that the connection is successful, creating a database cursor, executing the plurality of SQL query statements according to the database cursor to obtain a query result set, and recording the query result set;
closing the database cursor and releasing the memory resource occupied by the database cursor.
5. The data synchronization processing method according to claim 1, wherein when the query result set is not null, performing data cleaning and format conversion on the query result set to obtain a plurality of distributed file system files, each distributed file system file being used for storing query data corresponding to each SQL query statement, includes:
when the query result set is not null, reserving the names and values of the fields in the query result set;
replacing separators among the fields according to the names of the fields, and performing format conversion on the data of the timestamp type to obtain a processed data set;
and recording and storing the processed data set based on the redirection command to obtain a plurality of distributed file system files, wherein each distributed file system file is used for storing the query data corresponding to each SQL query statement.
6. The data synchronization processing method according to claim 1, wherein the importing the plurality of distributed file system files into the preset distributed file system respectively to obtain an update result, and sending the update result to a target terminal includes:
importing the distributed file systems into the preset distributed file system respectively in a transaction mode to obtain an updating result;
when the updating result is not a preset value, determining that data synchronization processing fails, generating early warning information, and sending the early warning information to a target terminal so that the target terminal displays the early warning information;
and when the updating result is a preset value, determining that the data synchronization processing is successful, and sending the updating result to the target terminal.
7. The data synchronization processing method according to any one of claims 1 to 6, wherein after the files of the distributed file systems are respectively imported into the preset distributed file system to obtain an update result, and the update result is sent to a target terminal, the data synchronization processing method further includes:
updating the updating result into a log file, wherein the log file records the result of data synchronous processing;
and analyzing and counting the data in the log file at regular time to obtain a data synchronization report, and sending the data synchronization report to the target terminal.
8. A data synchronization processing apparatus, characterized in that the data synchronization processing apparatus comprises:
the acquisition module is used for receiving a data mining request, acquiring a target script file according to the data mining request, and reading a plurality of table structure data in batch from a preset distributed file system through the target script file, wherein the target script file comprises a plurality of table structure query statements;
the mining module is used for respectively and sequentially carrying out data cleaning and statement splicing on the plurality of table structure data according to the SQL grammar rules of the structured query language to obtain a plurality of SQL query statements;
the query module is used for connecting the target relational database to obtain a connection result, and when the connection result is successful, performing data query operation according to the plurality of SQL query sentences to obtain a query result set;
the conversion module is used for carrying out data cleaning and format conversion on the query result set when the query result set is not a null value to obtain a plurality of distributed file system files, and each distributed file system file is used for storing the query data corresponding to each SQL query statement;
and the importing module is used for respectively importing the files of the distributed file systems into the preset distributed file system to obtain an updating result and sending the updating result to the target terminal.
9. A data synchronization processing apparatus characterized by comprising: a memory and at least one processor, the memory having instructions stored therein;
the at least one processor invokes the instructions in the memory to cause the data synchronization processing apparatus to perform the data synchronization processing method of any one of claims 1-7.
10. A computer-readable storage medium having stored thereon instructions, which when executed by a processor, implement a data synchronization processing method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011282969.4A CN112380180A (en) | 2020-11-17 | 2020-11-17 | Data synchronization processing method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011282969.4A CN112380180A (en) | 2020-11-17 | 2020-11-17 | Data synchronization processing method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112380180A true CN112380180A (en) | 2021-02-19 |
Family
ID=74585614
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011282969.4A Pending CN112380180A (en) | 2020-11-17 | 2020-11-17 | Data synchronization processing method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112380180A (en) |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113010550A (en) * | 2021-02-22 | 2021-06-22 | 建信金融科技有限责任公司 | Batch processing object generation and batch processing method and device for structured data |
| CN113138974A (en) * | 2021-04-23 | 2021-07-20 | 建信金融科技有限责任公司 | Database compliance detection method and device |
| CN113360470A (en) * | 2021-05-25 | 2021-09-07 | 深圳市珍爱捷云信息技术有限公司 | MySQL-based sub-library and sub-table synchronization method and device, electronic equipment and computer-readable storage medium |
| CN113486104A (en) * | 2021-07-15 | 2021-10-08 | 山东齐鲁数通科技有限公司 | Database storage process conversion method and device, terminal equipment and storage medium |
| CN113742316A (en) * | 2021-09-08 | 2021-12-03 | 辽宁振兴银行股份有限公司 | Device and method for synchronizing Mysql database by executing sql sentences in batches |
| CN113806448A (en) * | 2021-09-28 | 2021-12-17 | 中电万维信息技术有限责任公司 | Configuration method for automatic data synchronization |
| CN113885970A (en) * | 2021-09-15 | 2022-01-04 | 浪潮金融信息技术有限公司 | Method, system and medium for generating report data based on script |
| CN114416774A (en) * | 2022-01-05 | 2022-04-29 | 深圳萨摩耶数字科技有限公司 | Cross-platform multi-data-source data fetching method and device, electronic equipment and storage medium |
| CN114860349A (en) * | 2022-07-06 | 2022-08-05 | 深圳华锐分布式技术股份有限公司 | Data loading method, device, equipment and medium |
| CN115470229A (en) * | 2022-08-30 | 2022-12-13 | 昆仑数智科技有限责任公司 | Data table processing method and device, electronic equipment and storage medium |
| CN116028579A (en) * | 2023-02-27 | 2023-04-28 | 南昌国讯信息技术股份有限公司 | Database synchronization method, electronic device and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH10247197A (en) * | 1997-03-04 | 1998-09-14 | Mitsubishi Electric Corp | Data mining device |
| CN104123392A (en) * | 2014-08-11 | 2014-10-29 | 吉林禹硕动漫游戏科技股份有限公司 | Tool and method for transferring relational database to HBase |
| US20150134699A1 (en) * | 2013-11-11 | 2015-05-14 | International Business Machines Corporation | Data movement from a database to a distributed file system |
| CN105243067A (en) * | 2014-07-07 | 2016-01-13 | 北京明略软件系统有限公司 | Method and apparatus for realizing real-time increment synchronization of data |
| CN108388589A (en) * | 2018-01-30 | 2018-08-10 | 福建星瑞格软件有限公司 | A kind of device that database sql query statements automatically generate |
| CN111930862A (en) * | 2020-09-17 | 2020-11-13 | 国网浙江省电力有限公司 | SQL interactive analysis method and system based on big data platform |
-
2020
- 2020-11-17 CN CN202011282969.4A patent/CN112380180A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH10247197A (en) * | 1997-03-04 | 1998-09-14 | Mitsubishi Electric Corp | Data mining device |
| US20150134699A1 (en) * | 2013-11-11 | 2015-05-14 | International Business Machines Corporation | Data movement from a database to a distributed file system |
| CN105243067A (en) * | 2014-07-07 | 2016-01-13 | 北京明略软件系统有限公司 | Method and apparatus for realizing real-time increment synchronization of data |
| CN104123392A (en) * | 2014-08-11 | 2014-10-29 | 吉林禹硕动漫游戏科技股份有限公司 | Tool and method for transferring relational database to HBase |
| CN108388589A (en) * | 2018-01-30 | 2018-08-10 | 福建星瑞格软件有限公司 | A kind of device that database sql query statements automatically generate |
| CN111930862A (en) * | 2020-09-17 | 2020-11-13 | 国网浙江省电力有限公司 | SQL interactive analysis method and system based on big data platform |
Cited By (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113010550B (en) * | 2021-02-22 | 2023-05-05 | 建信金融科技有限责任公司 | Batch object generation and batch processing method and device for structured data |
| CN113010550A (en) * | 2021-02-22 | 2021-06-22 | 建信金融科技有限责任公司 | Batch processing object generation and batch processing method and device for structured data |
| CN113138974A (en) * | 2021-04-23 | 2021-07-20 | 建信金融科技有限责任公司 | Database compliance detection method and device |
| CN113138974B (en) * | 2021-04-23 | 2023-08-22 | 建信金融科技有限责任公司 | Method and device for detecting database compliance |
| CN113360470A (en) * | 2021-05-25 | 2021-09-07 | 深圳市珍爱捷云信息技术有限公司 | MySQL-based sub-library and sub-table synchronization method and device, electronic equipment and computer-readable storage medium |
| CN113486104A (en) * | 2021-07-15 | 2021-10-08 | 山东齐鲁数通科技有限公司 | Database storage process conversion method and device, terminal equipment and storage medium |
| CN113486104B (en) * | 2021-07-15 | 2024-05-31 | 山东齐鲁数通科技有限公司 | Database storage process conversion method and device, terminal equipment and storage medium |
| CN113742316A (en) * | 2021-09-08 | 2021-12-03 | 辽宁振兴银行股份有限公司 | Device and method for synchronizing Mysql database by executing sql sentences in batches |
| CN113885970A (en) * | 2021-09-15 | 2022-01-04 | 浪潮金融信息技术有限公司 | Method, system and medium for generating report data based on script |
| CN113885970B (en) * | 2021-09-15 | 2024-06-11 | 浪潮金融信息技术有限公司 | Method, system and medium for generating report data based on script |
| CN113806448B (en) * | 2021-09-28 | 2023-11-21 | 中电万维信息技术有限责任公司 | Configuration method for automatic synchronization of data |
| CN113806448A (en) * | 2021-09-28 | 2021-12-17 | 中电万维信息技术有限责任公司 | Configuration method for automatic data synchronization |
| CN114416774A (en) * | 2022-01-05 | 2022-04-29 | 深圳萨摩耶数字科技有限公司 | Cross-platform multi-data-source data fetching method and device, electronic equipment and storage medium |
| CN114860349A (en) * | 2022-07-06 | 2022-08-05 | 深圳华锐分布式技术股份有限公司 | Data loading method, device, equipment and medium |
| CN115470229A (en) * | 2022-08-30 | 2022-12-13 | 昆仑数智科技有限责任公司 | Data table processing method and device, electronic equipment and storage medium |
| CN116028579A (en) * | 2023-02-27 | 2023-04-28 | 南昌国讯信息技术股份有限公司 | Database synchronization method, electronic device and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112380180A (en) | Data synchronization processing method, device, equipment and storage medium | |
| US9558207B1 (en) | Versioning of database partition maps | |
| US8417991B2 (en) | Mitigating reduction in availability level during maintenance of nodes in a cluster | |
| US7203929B1 (en) | Design data validation tool for use in enterprise architecture modeling | |
| US20040111728A1 (en) | Method and system for managing metadata | |
| CN113360519B (en) | Data processing method, device, equipment and storage medium | |
| JPH1069410A (en) | Multiple data base access server for application program | |
| CN111324610A (en) | Data synchronization method and device | |
| CN104462269A (en) | Isomerous database data exchange method and system | |
| US20130173544A1 (en) | Materialized query table journaling in a computer database system | |
| CN107025115B (en) | Method for adapting to acquisition of multiple interfaces | |
| CN102355499A (en) | Cloud computing system | |
| CN110427427B (en) | Method for realizing global transaction distributed processing through pin bridging | |
| US11409781B1 (en) | Direct storage loading for adding data to a database | |
| US20030055921A1 (en) | Method and apparatus for reengineering legacy systems for seamless interaction with distributed component systems | |
| US20090100082A1 (en) | Replication and mapping mechanism for recreating memory durations | |
| CN112559525B (en) | Data checking system, method, device and server | |
| CN112131214A (en) | Method, system, equipment and storage medium for data writing and data query | |
| US11188228B1 (en) | Graphing transaction operations for transaction compliance analysis | |
| CN111581227A (en) | Event pushing method and device, computer equipment and storage medium | |
| CN115495440A (en) | Data migration method, device and equipment of heterogeneous database and storage medium | |
| Rooney et al. | Pathfinder: Building the enterprise data map | |
| US11620285B2 (en) | Automatic database query translation | |
| EP3082050A1 (en) | Mass data fusion storage method and system | |
| CN112199426B (en) | Interface call management method, device, server and medium under micro-service architecture |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |