CN107992620A - A kind of method and system device of json data Fast synchronization and record - Google Patents
A kind of method and system device of json data Fast synchronization and record Download PDFInfo
- Publication number
- CN107992620A CN107992620A CN201711402348.3A CN201711402348A CN107992620A CN 107992620 A CN107992620 A CN 107992620A CN 201711402348 A CN201711402348 A CN 201711402348A CN 107992620 A CN107992620 A CN 107992620A
- Authority
- CN
- China
- Prior art keywords
- data
- configuration file
- json
- json data
- database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of json data Fast synchronization and the method and system device of record, its method obtains the synchronisation requirement file of synchrodata for configuration file editor module, creates configuration file；Configuration file parsing module reads the configuration file, and parsing identifies the json data sources of each synchrodata and unique target database, and establishes database connection；The json data sources that json data read modules obtain parsing read json data, according to the SQL statement of the data field mapping relations generation batch synchronization in configuration；Relevant database writing module is connected according to the database, and batch writes json data to target database.The present invention flexibly can rapidly edit, change configuration file, and according to configuration file, the automatic data that carry out are synchronous, control synchronizing process, recording synchronism daily record, the synchronous progress of monitoring and statistics synchronized result.The present invention is convenient, flexible, json data are synchronized in relevant database rapidly and efficiently, greatly improves development efficiency.
Description
Technical field
The present invention relates to computer program development field, more particularly to a kind of data of json forms are synchronized to relationship type number
According to the method for data synchronization and system in storehouse.
Background technology
Json is a kind of data format of lightweight, its a subset based on ECMAScript, its flexible data group
The mode of knitting enjoys developer to favor, therefore is widely used in program development.
The characteristics of relational data library storage data, a typically database included several tables of data, each tables of data
Comprising several data fields, each field represents an attribute of present data entity.
In development process, it is frequently encountered when need json data being synchronized among relevant database.Every time
This exploitation is done, is required for individually writing the code of synchronizing process for actual json data modes, and if multiple classes
As data it is synchronous, it is necessary to write respectively, this significantly increases code redundancy, and reduces development efficiency.Therefore, having must
To be directed to this common exploitation scene and propose a kind of efficient scheme of universal rapid, to improve development efficiency.
The content of the invention
To solve the problems, such as repeated encoding described in background technology, not general and inefficient, the present invention provides one kind
The method of json data Fast synchronization and record, can be rapidly and efficiently json data are synchronized in relevant database.Its
Concrete technical scheme is as follows：
First aspect：A kind of method of json data Fast synchronization and record, the described method includes：
Configuration file editor module obtains the synchronisation requirement file of synchrodata, creates configuration file；
Configuration file parsing module reads the configuration file, parsing identify the json data sources of each synchrodata with it is unique
Target database, and establish database connection；
The json data sources that json data read modules obtain parsing read json data, according to the data field in configuration
Mapping relations generate the SQL statement of batch synchronization；
Relevant database writing module is connected according to the database, and batch writes json data to target database；
Log recording and statistical module record parameter, the index during above-mentioned method of data synchronization in real time.
With reference to first aspect, the first situation in the possible implementation of first aspect is that the json data are
The data of character type, the json data sources include and are not limited to .txt files .json files .js files.
The first situation with reference to first aspect, the second situation in the possible implementation of first aspect are institute
State configuration file and include json data source files information, target database information and field mapping relations.
The second situation with reference to first aspect, the third situation in the possible implementation of first aspect are institute
It is array form to state the json data formats that json data source files include, and each element of array is one json pairs
As each json objects include identical data attribute.
The third situation with reference to first aspect, the 4th kind of situation in the possible implementation of first aspect is institute
It is relevant database to state target database, and data are stored in each tables of data, and tables of data relies on database, tables of data bag
Row and column containing data, data row be field, relational data including but not limited to Oracle, MySql, SQL Server,
MariaDB。
The 4th kind of situation with reference to first aspect, the 5th kind of situation in the possible implementation of first aspect are to go back
Synchronizing process is controlled including data synchronization process control module.
Second aspect, a kind of system of json data Fast synchronization and record, the system comprises：
Configuration file editor module, for, into edlin, modification, after modification, restarting synchronization system to be given birth to configuration file
Effect；
Configuration file parsing module, for reading configuration file automatically, and carries out Exact Solutions analysis, identifies the source of each synchrodata
Data and target database, and establish connection；
Json data read modules, for reading json data sources automatically, give birth to according to the data field mapping relations of configuration file
Synchronous SQL statement in batch；
Relevant database writing module, for being connected according to the database, batch writes json data to target database；
Data synchronization process control module, after starting synchronization program, is controlled synchronizing process；
Log recording and statistical module, for the parameter during real-time recording synchronism, index.
The third aspect, a kind of system and device of json data Fast synchronization and record, the system and device include：
Configuration file editor module, for, into edlin, modification, after modification, restarting synchronization system to be given birth to configuration file
Effect；
Configuration file parsing module, for reading configuration file automatically, and carries out Exact Solutions analysis, identifies the source of each synchrodata
Data and target database, and establish connection；
Json data read modules, for reading json data sources automatically, give birth to according to the data field mapping relations of configuration file
Synchronous SQL statement in batch；
Relevant database writing module, for being connected according to the database, batch writes json data to target database；
Data synchronization process control module, after starting synchronization program, is controlled synchronizing process；
Log recording and statistical module, for the parameter during real-time recording synchronism, index；
The system and device further includes memory, processor, bus, and the memory is with the processor by bus into line number
According to connection, the memory storage has a plurality of operational order, and the processor loads the operational order and performs, and realizes first
Aspect and its first to the method described in the 5th kind of situation.
Beneficial effect：The present invention flexibly can edit rapidly, change configuration file, and according to configuration file, automatically into
Row data are synchronous, control synchronizing process, recording synchronism daily record, the synchronous progress of monitoring and statistics synchronized result.The convenient spirit of the present invention
Living, json data are synchronized in relevant database rapidly and efficiently, greatly improves development efficiency.
Brief description of the drawings
Fig. 1 is the overall procedure schematic diagram of embodiment one;
Fig. 2 is that the configuration file of embodiment two mainly includes content；
Fig. 3 is the system module figure of embodiment three；
Fig. 4 is the system work Organization Chart of example IV.
Embodiment
To make the purpose, technical scheme and advantage of the embodiment of the present invention clearer, below in conjunction with the embodiment of the present invention
In attached drawing, the technical solution in the embodiment of the present invention is clearly and completely described.Obviously, described embodiment is
The section Example of the present invention, instead of all the embodiments.Based on the embodiments of the present invention, those of ordinary skill in the art
All other embodiments obtained without creative efforts, belong to the scope of protection of the invention.
Embodiment one：
As shown in Figure 1, a kind of method of json data Fast synchronization and record, the described method includes：
Step 101, configuration file editor module obtains the synchronisation requirement file of synchrodata, believes comprising json data source files
The configuration file of breath, target database information and field relationship map；
Step 102, configuration file parsing module reads the configuration file, each same according to the configuration file parsing identification being resolved to
The json data sources of step data and unique target database, and establish database connection；
Step 103, json data read modules obtain the json data sources reading json data of parsing, according in configuration
Data field mapping relations generate the SQL statement of batch synchronization；
Step 104, relevant database writing module is connected according to the database, and batch writes json data to target data
Storehouse.
After system starts, the information that is resolved to according to configuration file is quick to read json source data informations, with target data
Connection is established in storehouse, is formed corresponding synchronous SQL statement and is performed, you can it is same to complete landing of the data from file to target database
Step.Log recording and statistical module record parameter, the index during above-mentioned method of data synchronization in real time, and synchronizing process can
To control start and stop at any time, continue.
Embodiment two：
As shown in Fig. 2, the configuration file is as follows comprising content：
1. original json data source files information, such as file type（filetype）, file path（path）, file coding format
（charset）Deng；
2. synchronous target database information, mainly comprising type of database mark dbtype（Mainly have Oracle,
Microsoft SQL Server, MySql etc.）With the host for establishing the target database being synchronized to（host）, port numbers
（port）, user name（username）, password（password）, coded format（charset）, database（database）Deng company
Connect information and write-in tables of data（datatable）.
3. data mapping relations, main each sub- attribute for including json data objects（property）With relational data
Corresponding data sheet field in storehouse（field）The one-to-one mapping established.
This method can be carried out at the same time multiple data with system and work asynchronously.Json data source files, target database letter
Breath and field relationship map are to form the necessary information of a partial data synchronizing process configuration, so if to be carried out at the same time more
A data synchronous working, it is only necessary to multiple such configuration informations, " data A is synchronously configured " as shown in Figure 2 and " data
B is synchronously configured ".
Embodiment three
As shown in figure 3, the present invention provides the system of a kind of json data Fast synchronization and record, the system comprises：
Configuration file editor module, for, into edlin, modification, after modification, restarting synchronization system to be given birth to configuration file
Effect；
Configuration file parsing module, for reading configuration file automatically, and carries out Exact Solutions analysis, identifies the source of each synchrodata
Data and target database, and establish connection；
Json data read modules, for reading json data sources automatically, give birth to according to the data field mapping relations of configuration file
Synchronous SQL statement in batch；
Relevant database writing module, for being connected according to the database, batch writes json data to target database；
Data synchronization process control module, after starting synchronization program, is controlled synchronizing process；
Log recording and statistical module, for the parameter during real-time recording synchronism, index.
It is described present invention also offers a kind of json data Fast synchronization and the system and device of record based on said system
System and device includes：
Configuration file editor module, for, into edlin, modification, after modification, restarting synchronization system to be given birth to configuration file
Effect；
Configuration file parsing module, for reading configuration file automatically, and carries out Exact Solutions analysis, identifies the source of each synchrodata
Data and target database, and establish connection；
Json data read modules, for reading json data sources automatically, give birth to according to the data field mapping relations of configuration file
Synchronous SQL statement in batch；
Relevant database writing module, for being connected according to the database, batch writes json data to target database；
Data synchronization process control module, after starting synchronization program, is controlled synchronizing process；
Log recording and statistical module, for the parameter during real-time recording synchronism, index；
The system and device further includes memory, processor, bus, and the memory is with the processor by bus into line number
According to connection, the memory storage has a plurality of operational order, and the processor loads the operational order and performs, and realizes explanation
Method described in book.
Example IV：
As shown in figure 4, after system starts, configuration file parsing module is according to the configuration file solution of configuration file editor module editor
Information is analysed, json data read modules quickly read json source data informations, establish and connect with target database, are formed corresponding
Synchronous SQL statement is simultaneously performed by relevant database writing module, you can completes landing of the data from file to target database
It is synchronous.The synchronizing process can be controlled start and stop at any time by data synchronization process control module, be continued, and by log recording and system
Module record log is counted with statistical information for query analysis.
For device embodiment, since it is substantially similar to embodiment of the method, so description is fairly simple, it is related
Part illustrates referring to the part of embodiment of the method.
Device embodiment described above is only schematical, can be designed to equipment integrating or group
A set of equipment is synthesized, can also be distributed in multiple network unit.Can select according to the actual needs part therein or
Whole modules realize the purpose of this embodiment scheme.Those of ordinary skill in the art are not paying the situation of performing creative labour
Under, you can to understand and implement.
Through the above description of the embodiments, those skilled in the art can be understood that each embodiment can
Realized by the mode of software and required general hardware platform, naturally it is also possible to pass through hardware.Based on such understanding, on
The part that technical solution substantially in other words contributes to the prior art is stated to embody in the form of software product, should
Computer software product can store in a computer-readable storage medium, such as ROM/RAM, magnetic disc, CD, including some fingers
Order is used so that a computer equipment (can be personal computer, server, or network equipment etc.) performs each implementation
Method described in some parts of example or embodiment.
Finally it should be noted that：The above embodiments are merely illustrative of the technical solutions of the present invention, rather than its limitations；Although
The present invention is described in detail with reference to the foregoing embodiments, those of ordinary skill in the art its still can be to foregoing each
Technical solution described in embodiment is modified, or carries out equivalent substitution to which part technical characteristic；And these are changed
Or replace, the essence of appropriate technical solution is departed from the scope of various embodiments of the present invention technical solution.
Claims (8)
1. a kind of method of json data Fast synchronization and record, it is characterised in that the described method includes：
Configuration file editor module obtains the synchronisation requirement file of synchrodata, creates configuration file；
Configuration file parsing module reads the configuration file, parsing identify the json data sources of each synchrodata with it is unique
Target database, and establish database connection；
The json data sources that json data read modules obtain parsing read json data, according to the data field in configuration
Mapping relations generate the SQL statement of batch synchronization；
Relevant database writing module is connected according to the database, and batch writes json data to target database；
Log recording and statistical module record parameter, the index during above-mentioned method of data synchronization in real time.
2. the method for a kind of json data Fast synchronization according to claim 1 and record, it is characterised in that the json
Data are the data of character type, and the json data sources include and are not limited to .txt files .json files .js files.
3. the method for a kind of json data Fast synchronization according to claim 2 and record, it is characterised in that the configuration
File includes json data source files information, target database information and field mapping relations.
4. the method for a kind of json data Fast synchronization according to claim 3 and record, it is characterised in that the json
The json data formats that data source file includes are array form, and each element of array is a json object, each
Json objects include identical data attribute.
5. the method for a kind of json data Fast synchronization according to claim 4 and record, it is characterised in that the target
Database is relevant database, and data are stored in each tables of data, and tables of data relies on database, and tables of data includes data
Row and column, data row are field, and relational data is including but not limited to Oracle, MySql, SQL Server, MariaDB.
6. the method for a kind of json data Fast synchronization according to claim 5 and record, it is characterised in that further include number
Synchronizing process is controlled according to synchronizing process control module.
A kind of 7. system of json data Fast synchronization and record, it is characterised in that the system comprises：
Configuration file editor module, for, into edlin, modification, after modification, restarting synchronization system to be given birth to configuration file
Effect；
Configuration file parsing module, for reading configuration file automatically, and carries out Exact Solutions analysis, identifies the source of each synchrodata
Data and target database, and establish connection；
Json data read modules, for reading json data sources automatically, give birth to according to the data field mapping relations of configuration file
Synchronous SQL statement in batch；
Relevant database writing module, for being connected according to the database, batch writes json data to target database；
Data synchronization process control module, after starting synchronization program, is controlled synchronizing process；
Log recording and statistical module, for the parameter during real-time recording synchronism, index.
8. the system and device of a kind of json data Fast synchronization and record, it is characterised in that the system and device includes：
Configuration file editor module, for, into edlin, modification, after modification, restarting synchronization system to be given birth to configuration file
Effect；
Configuration file parsing module, for reading configuration file automatically, and carries out Exact Solutions analysis, identifies the source of each synchrodata
Data and target database, and establish connection；
Json data read modules, for reading json data sources automatically, give birth to according to the data field mapping relations of configuration file
Synchronous SQL statement in batch；
Relevant database writing module, for being connected according to the database, batch writes json data to target database；
Data synchronization process control module, after starting synchronization program, is controlled synchronizing process；
Log recording and statistical module, for the parameter during real-time recording synchronism, index；
The system and device further includes memory, processor, bus, and the memory is with the processor by bus into line number
According to connection, the memory storage has a plurality of operational order, and the processor loads the operational order and performs, and realizes right
It is required that the method described in 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711402348.3A CN107992620A (en) | 2017-12-22 | 2017-12-22 | A kind of method and system device of json data Fast synchronization and record |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711402348.3A CN107992620A (en) | 2017-12-22 | 2017-12-22 | A kind of method and system device of json data Fast synchronization and record |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107992620A true CN107992620A (en) | 2018-05-04 |
Family
ID=62039586
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711402348.3A Pending CN107992620A (en) | 2017-12-22 | 2017-12-22 | A kind of method and system device of json data Fast synchronization and record |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107992620A (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111061787A (en) * | 2019-11-21 | 2020-04-24 | 新浪网技术(中国)有限公司 | Method and system for processing object relation in relational database |
| CN111090640A (en) * | 2019-11-13 | 2020-05-01 | 山东中磁视讯股份有限公司 | ETL data cleaning method and system |
| CN111124479A (en) * | 2019-12-18 | 2020-05-08 | 北京像素软件科技股份有限公司 | Configuration file analysis method and system and electronic equipment |
| CN111367975A (en) * | 2018-12-25 | 2020-07-03 | 中国移动通信集团浙江有限公司 | Multi-protocol data conversion processing method and device |
| CN111708750A (en) * | 2019-12-27 | 2020-09-25 | 山东鲁能软件技术有限公司 | Big data platform based storage adaptation method, system, equipment and readable storage medium |
| CN112131291A (en) * | 2020-09-11 | 2020-12-25 | 重庆誉存大数据科技有限公司 | JSON data-based structured analysis method, device, equipment and storage medium |
| CN112765272A (en) * | 2021-01-19 | 2021-05-07 | 城云科技（中国）有限公司 | Multi-source data synchronization method and system and model hub applied by same |
| CN113722394A (en) * | 2021-08-17 | 2021-11-30 | 北京百悟科技有限公司 | Data synchronization method, device and storage medium |
| CN115827777A (en) * | 2022-11-21 | 2023-03-21 | 中国人民财产保险股份有限公司 | Self-adaptive synchronization and difference identification method, device and equipment for multiple data sources |
-
2017
- 2017-12-22 CN CN201711402348.3A patent/CN107992620A/en active Pending
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111367975A (en) * | 2018-12-25 | 2020-07-03 | 中国移动通信集团浙江有限公司 | Multi-protocol data conversion processing method and device |
| CN111367975B (en) * | 2018-12-25 | 2023-11-03 | 中国移动通信集团浙江有限公司 | Multi-protocol data conversion processing method and device |
| CN111090640A (en) * | 2019-11-13 | 2020-05-01 | 山东中磁视讯股份有限公司 | ETL data cleaning method and system |
| CN111061787A (en) * | 2019-11-21 | 2020-04-24 | 新浪网技术(中国)有限公司 | Method and system for processing object relation in relational database |
| CN111124479A (en) * | 2019-12-18 | 2020-05-08 | 北京像素软件科技股份有限公司 | Configuration file analysis method and system and electronic equipment |
| CN111124479B (en) * | 2019-12-18 | 2024-03-22 | 北京像素软件科技股份有限公司 | Method and system for analyzing configuration file and electronic equipment |
| CN111708750A (en) * | 2019-12-27 | 2020-09-25 | 山东鲁能软件技术有限公司 | Big data platform based storage adaptation method, system, equipment and readable storage medium |
| CN112131291A (en) * | 2020-09-11 | 2020-12-25 | 重庆誉存大数据科技有限公司 | JSON data-based structured analysis method, device, equipment and storage medium |
| CN112765272A (en) * | 2021-01-19 | 2021-05-07 | 城云科技（中国）有限公司 | Multi-source data synchronization method and system and model hub applied by same |
| CN113722394A (en) * | 2021-08-17 | 2021-11-30 | 北京百悟科技有限公司 | Data synchronization method, device and storage medium |
| CN115827777A (en) * | 2022-11-21 | 2023-03-21 | 中国人民财产保险股份有限公司 | Self-adaptive synchronization and difference identification method, device and equipment for multiple data sources |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108009282A (en) | A kind of json data are synchronized to the method and system device of relevant database | |
| CN107992620A (en) | A kind of method and system device of json data Fast synchronization and record | |
| CN108121564A (en) | A kind of json data Fast synchronization and the method and system device of control | |
| CN104731896B (en) | A kind of data processing method and system | |
| CN110292775B (en) | Method and device for acquiring difference data | |
| CN110209728B (en) | Distributed heterogeneous database synchronization method, electronic equipment and storage medium | |
| CN111241203B (en) | Hive data warehouse synchronization method, system, equipment and storage medium | |
| US20210232561A1 (en) | Data processing method and apparatus, and computer-readable storage medium | |
| CN112988702A (en) | Heterogeneous data source real-time data transmission method and system, storage medium and terminal | |
| CN111984745B (en) | Database field dynamic expansion method, device, equipment and storage medium | |
| CN114780641A (en) | Multi-library multi-table synchronization method and device, computer equipment and storage medium | |
| CN113407536A (en) | Method and device for associating table data, terminal equipment and medium | |
| CN109271456A (en) | Host data library file deriving method and device | |
| CN114996288A (en) | Data comparison method and device, computer storage medium and electronic equipment | |
| CN114168544B (en) | Clinical trial data processing method, device, computer equipment and storage medium | |
| CN116089535A (en) | Data synchronization method, device, equipment and storage medium | |
| CN105224663A (en) | A kind of data-accessing tasks management method based on multiple data source and device | |
| CN107766519B (en) | Method for visually configuring data structure | |
| KR101252050B1 (en) | System and method for converting cad file, and computer readable recording medium having program to implement the method | |
| JP2006268661A (en) | Data import method and data import device | |
| CN109493918B (en) | Biological data management and phylogenetic analysis process method | |
| CN110941658A (en) | Data export method, device, server and storage medium | |
| US20120317073A1 (en) | Replication Support for Procedures with Arguments of Unsupported Types | |
| CN115952201A (en) | Data query method, device, system and storage medium | |
| CN116089527A (en) | Data verification method, storage medium and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| WD01 | Invention patent application deemed withdrawn after publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20180504 |