CN112765250B - Method and device for supporting json data conversion into relational database - Google Patents
Method and device for supporting json data conversion into relational database Download PDFInfo
- Publication number
- CN112765250B CN112765250B CN202110063490.XA CN202110063490A CN112765250B CN 112765250 B CN112765250 B CN 112765250B CN 202110063490 A CN202110063490 A CN 202110063490A CN 112765250 B CN112765250 B CN 112765250B
- Authority
- CN
- China
- Prior art keywords
- data
- database
- json
- reading
- table structure
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000006243 chemical reaction Methods 0.000 title claims abstract description 25
- 238000000034 method Methods 0.000 title claims abstract description 16
- 238000003780 insertion Methods 0.000 claims description 19
- 230000037431 insertion Effects 0.000 claims description 19
- 238000012795 verification Methods 0.000 claims description 10
- 230000002688 persistence Effects 0.000 claims description 6
- 230000002085 persistent effect Effects 0.000 claims description 5
- 238000004806 packaging method and process Methods 0.000 claims 2
- 230000006870 function Effects 0.000 abstract description 3
- 238000012545 processing Methods 0.000 description 3
- 238000011161 development Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/10—Text processing
- G06F40/12—Use of codes for handling textual entities
- G06F40/151—Transformation
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Computational Linguistics (AREA)
- General Health & Medical Sciences (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a method and a device for supporting json data conversion into relational database data, wherein the method comprises the following steps: the device comprises a database connector module, a data paging device module, a database table structure reading device module and a database table structure conversion object device module. The invention realizes the functions of analyzing json data, comparing with the fields of the current table structure and inputting.
Description
Technical Field
The invention relates to the technical field of computers, in particular to a method and a device for supporting json data conversion into a relational database.
Background
Along with the continuous development of computer technology, developers put forward a lot of convenient and quick implementation methods in terms of codes in development, in the prior art, json data types are various, and (for example int, float, double, string, date … …) people need to convert the types of the json data when the data are inserted into a database, so that errors caused by coding formats are avoided.
Disclosure of Invention
Aiming at the technical problems in the related art, the invention provides a method and a device for supporting the transformation of json data into a relational database, which can overcome the defects of the prior art method.
In order to achieve the technical purpose, the technical scheme of the invention is realized as follows:
a method for supporting the conversion of json data into a relational database, comprising the steps of: before processing data, the table structure of the target to be converted, the related information of the database and the input data set are required to be converted; before formally converting the database object, verifying the correctness of the related information of the database, namely using the input database information to access the database; according to the size of the data, the data is split appropriately, and the data is read in a paging way; after the data reading is completed, a verification step is carried out; after the verification step is finished, converting the json data object into a database object; when a batch of data reaches a threshold value or memory limit or no data exists, performing data insertion operation; and finally returning success.
Further, in the verification database information, if the target database address cannot be accessed, or the user information is wrong, or no relevant authority is required to report errors.
Further, the step of paging read data is: a spatial threshold is set and the read data is traversed.
Further, the checking step is as follows: when the first data input operation is performed, the database structure is read, the data of the database structure is packaged into an object to be loaded into the memory, each parameter of each piece of data is traversed, and the type of each parameter is judged.
Further, after the type judgment is completed, the length of the field is judged, and format conversion is required for the encoding of the field.
Further, the step of converting json data objects into database objects includes the steps of generating a persistence class according to the table structure read from the database when the database table structure is read for the first time, converting json data into persistence objects one by one, adding the persistence objects into an object list to be inserted, and waiting for data insertion.
Further, the data insertion operation steps are: firstly, trying to connect a database; after the connection is successful, converting the batch of data into SQL INSERT sentences for insertion; and disconnecting the database after completion.
According to another aspect of the present invention, there is provided an apparatus for supporting json data conversion into a relational database, comprising a database connector module, a data paging apparatus module, a database table structure reading apparatus module, and a database table structure conversion target apparatus module, wherein,
The database connector module is used for comprehensively verifying the related information of the database before connecting the database, and reporting errors if the target database cannot be accessed.
The data paging device module is used for splitting data according to the size of the data in the database and paging and reading data in a large quantity.
The database table structure reading module is used for reading the database structure, loading the data package in the database structure, and then performing traversal reading.
And the database table structure conversion object device module is used for converting json data into a persistent object after the database table structure is read, and then waiting for data insertion.
The data is converted into relational database data by a conversion device which converts the data by a global parameter management container and a database connection device.
The invention has the beneficial effects that: the method of checking and connecting database information and operating the database realizes the functions of analyzing json data, comparing the json data with the fields of the current table structure and inputting the json data.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are needed in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic diagram of a device structure of a method and a device for supporting json data conversion into relational database data according to an embodiment of the present invention.
Fig. 2 is a schematic diagram of a device conversion flow of a method and a device for supporting json data conversion into relational database data according to an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which are derived by a person skilled in the art based on the embodiments of the invention, fall within the scope of protection of the invention.
As shown in fig. 1-2, a method for supporting json data conversion into relational database data according to an embodiment of the present invention includes the following steps:
Before formally processing data, preparing a table structure of a converted target, related information of a database and json data of an input data set (json array object);
Verifying the correctness of the database information, and before formally converting the database object, verifying the related information of the database, namely using the input database information to access the database for verification;
according to the size of the data, the data is split appropriately, namely the data is read in a paging mode;
checking after the data reading is completed (the threshold value is reached or the established number of the data is reached or no data is readable);
after the verification is finished, the json data object is required to be converted into a database object;
after the operation is finished, when one batch of data reaches a threshold value or memory limit or no data exists, performing data insertion operation;
After all operations are completed, success is returned.
In another aspect, an apparatus for supporting json data conversion into relational database data according to an embodiment of the present invention includes: and the database connector module is used for comprehensively verifying the related information of the database before connecting the database, and reporting errors if the target database cannot be accessed. And the data paging device module is used for splitting data according to the size of the data in the database and paging and reading data in a large quantity. And the database table structure reading module is used for reading the database structure, loading the data package in the database structure, and then performing traversal reading. And the database table structure conversion object device module is used for converting json data into a persistent object after the database table structure is read, and then waiting for data insertion.
In order to facilitate understanding of the above technical solutions of the present invention, the following describes the above technical solutions of the present invention in detail by a specific usage manner.
First, json data of the following data structure is required before formally processing the data:
1. The table structure of the target table and the related information of the database need to be converted;
2. An entered dataset (json array object).
Verifying the correctness of the database information:
Before formally converting the database object, the related information of the database needs to be verified, namely the input database information is used for accessing the database to verify, and if the target database address cannot be accessed, or the user information is wrong, or no related authority needs to report errors.
According to the size of the data, the data is split appropriately, namely the data is read in a paging mode, and the specific steps are as follows:
1. setting a spatial threshold (this threshold is determined by the size of the allocated memory);
2. Traversing the read data, and stopping reading when the data is close to a threshold value or a set number of pieces.
After the data reading is completed (the threshold value is reached or the established number of the data is reached or no data is readable), a verification step is carried out, and the specific steps are as follows:
1. When the operation of inputting data is performed for the first time, the database structure is read, the data of the database structure is packaged into objects and loaded into a memory, each parameter (suitably transferred from hump identification to commonly used underlined field names of the database) of each piece of data is traversed to judge types, such as types of character strings, integer types and the like, the date types are converted according to the input format of json, and the Boolean types are suitably converted into types such as tinyint (1).
2. Judging the length of the field after the type judgment is completed, and if the length of the json entry is larger than the length defined by the database table structure, reporting errors.
3. After the type judgment, the codes of the fields need to be converted into codes corresponding to the database fields.
After the verification step is finished, the json data object is required to be converted into a database object, and the specific steps are as follows:
1. When the table structure of the database is read for the first time, a persistence class is generated according to the table structure read from the database.
2. And converting json data one by one into persistent objects, adding the persistent objects into an object list to be inserted, and waiting for data insertion.
After the above operation is completed, when a batch of data reaches a threshold value or memory limit or no new data is available, the data insertion operation is performed, and the specific operation steps are as follows:
1. Attempting to connect to the database;
2. after the database is successfully connected, converting the batch of data into SQL INSERT sentences for insertion;
3. if one of the data insertion fails, a text file special for acquiring the failed statement acquires the generated sql statement record;
4. After all data are inserted, connecting the sql statement in the failed text with the database, executing again, writing a file if the failure is again, reporting errors if the sql content is still in the failed file after the insertion is finished, and outputting the file;
5. The database connection is disconnected.
After all operations are completed, success is returned.
In summary, by means of the technical scheme, the function of analyzing json data, comparing the json data with the fields of the current table structure and inputting the json data is realized through the method of checking and connecting database information and operating the database.
The foregoing description of the preferred embodiments of the invention is not intended to be limiting, but rather is intended to cover all modifications, equivalents, alternatives, and improvements that fall within the spirit and scope of the invention.
Claims (4)
1. A method for supporting the conversion of json data into relational database data, comprising the steps of:
s1, preparing a table structure of a converted target, related information of a database and json data of an input data set;
S2, verifying the correctness of the related information of the database, namely using the input database information to access the database for verification;
S3, splitting the data according to the size of the data, and performing paging reading data, wherein the steps of paging reading data are as follows: setting a space threshold; traversing and reading data;
S4, after data reading is completed, checking, wherein the checking steps are as follows: when the first data input operation is performed, the database structure is read; packaging data of a database structure into objects and loading the objects into a memory; traversing each parameter of each piece of data, judging the type of each parameter, judging the length of a field after the type judgment is completed, and carrying out format conversion on the encoding of the field;
S5, after the verification is finished, converting the json data object into a database object, wherein the step of converting the json data object into the database object is that when the database table structure is read for the first time, a persistence class is generated according to the table structure read from the database, then json data is converted into persistence objects one by one and added into an object list to be inserted, and data insertion is waited;
S6, when a batch of data reaches a threshold value or memory limit or no data exists, performing data insertion operation, wherein the data insertion operation steps are as follows: firstly, trying to connect a database; after the connection is successful, converting the batch of data into SQL INSERT sentences for insertion, and if one piece of data fails for insertion, obtaining a generated sql sentence record by using a text file specially used for obtaining the failed sentence; after all the data are inserted, connecting the sql statement in the failed text with the database, then executing again, if the failure occurs again, writing a file again, and finally after the insertion is completed, if the sql content is still in the failed file, reporting errors, outputting the file, and disconnecting the connection with the database after the completion;
and S7, finally returning success.
2. The method for supporting json data conversion into relational database data according to claim 1, wherein if the target database address cannot be accessed, or the user information is wrong, or no relevant authority is required to report the mistake in the verification database information.
3. A device for supporting json data conversion into relational database data is characterized by comprising a database connector module, a data paging device module, a database table structure reading device module and a database table structure conversion object device module, wherein,
The database connector module is used for comprehensively verifying the related information of the database before connecting the database, and reporting errors if the target database cannot be accessed;
the data paging device module is used for splitting data according to the size of the data in the database and paging and reading data in a large quantity;
The database table structure reading module is used for reading the database structure, packaging and loading data in the database structure, and then performing traversal reading;
And the database table structure conversion object device module is used for converting json data into a persistent object after the database table structure is read, and then waiting for data insertion.
4. A device for supporting the conversion of json data into relational database data according to claim 3, characterized in that the data is converted into relational database data by conversion means which convert the data by means of a global parameter management container and a database connection means.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110063490.XA CN112765250B (en) | 2021-01-18 | 2021-01-18 | Method and device for supporting json data conversion into relational database |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110063490.XA CN112765250B (en) | 2021-01-18 | 2021-01-18 | Method and device for supporting json data conversion into relational database |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112765250A CN112765250A (en) | 2021-05-07 |
| CN112765250B true CN112765250B (en) | 2024-05-28 |
Family
ID=75702912
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110063490.XA Active CN112765250B (en) | 2021-01-18 | 2021-01-18 | Method and device for supporting json data conversion into relational database |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112765250B (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102541966A (en) * | 2010-12-30 | 2012-07-04 | 国际商业机器公司 | Method and device for verifying correctness of database system |
| CN103116627A (en) * | 2013-01-31 | 2013-05-22 | 汉柏科技有限公司 | Database access method with high concurrency service-oriented architecture (SOA) technology and system |
| CN105045817A (en) * | 2015-06-26 | 2015-11-11 | 司法部司法鉴定科学技术研究所 | SQL Server database evidence obtaining and analyzing system and method based on transaction log |
| CN105528556A (en) * | 2015-12-03 | 2016-04-27 | 中国人民解放军信息工程大学 | Hybrid SQLite3 safety access method |
| CN107423061A (en) * | 2017-07-12 | 2017-12-01 | 江苏速度信息科技股份有限公司 | A kind of list generation method based on ISON data |
| CN108009282A (en) * | 2017-12-22 | 2018-05-08 | 武汉楚鼎信息技术有限公司 | A kind of json data are synchronized to the method and system device of relevant database |
| CN111159264A (en) * | 2019-11-28 | 2020-05-15 | 京东数字科技控股有限公司 | Data processing method and device |
| CN111984739A (en) * | 2019-05-24 | 2020-11-24 | 中移(苏州)软件技术有限公司 | Conversion method, terminal and storage medium for XML and relational database |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10691682B2 (en) * | 2017-10-04 | 2020-06-23 | EMC IP Holding Company LLC | Storing and processing JSON documents in a SQL database table |
-
2021
- 2021-01-18 CN CN202110063490.XA patent/CN112765250B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102541966A (en) * | 2010-12-30 | 2012-07-04 | 国际商业机器公司 | Method and device for verifying correctness of database system |
| CN103116627A (en) * | 2013-01-31 | 2013-05-22 | 汉柏科技有限公司 | Database access method with high concurrency service-oriented architecture (SOA) technology and system |
| CN105045817A (en) * | 2015-06-26 | 2015-11-11 | 司法部司法鉴定科学技术研究所 | SQL Server database evidence obtaining and analyzing system and method based on transaction log |
| CN105528556A (en) * | 2015-12-03 | 2016-04-27 | 中国人民解放军信息工程大学 | Hybrid SQLite3 safety access method |
| CN107423061A (en) * | 2017-07-12 | 2017-12-01 | 江苏速度信息科技股份有限公司 | A kind of list generation method based on ISON data |
| CN108009282A (en) * | 2017-12-22 | 2018-05-08 | 武汉楚鼎信息技术有限公司 | A kind of json data are synchronized to the method and system device of relevant database |
| CN111984739A (en) * | 2019-05-24 | 2020-11-24 | 中移(苏州)软件技术有限公司 | Conversion method, terminal and storage medium for XML and relational database |
| CN111159264A (en) * | 2019-11-28 | 2020-05-15 | 京东数字科技控股有限公司 | Data processing method and device |
Non-Patent Citations (1)
| Title |
|---|
| 基于JSON 的异构数据源数据交换技术研究;王华志;《中国优秀硕士学位论文全文数据库 信息科技辑》;20180115(第1期);I138-898 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112765250A (en) | 2021-05-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110347598B (en) | Test script generation method and device, server and storage medium | |
| CN110941593B (en) | File warehousing system and method | |
| CN109558525B (en) | Test data set generation method, device, equipment and storage medium | |
| CN109165209B (en) | Data verification method, device, equipment and medium for object types in database | |
| CN112231407B (en) | DDL synchronization method, device, equipment and medium of PostgreSQL database | |
| CN110990276A (en) | Automatic testing method and device for interface field and storage medium | |
| CN113761879B (en) | Message format verification method, device and storage medium | |
| CN106445815A (en) | Automated testing method and device | |
| CN106843878A (en) | A kind of model generating method and system | |
| CN116226170A (en) | Database statement conversion method and device, electronic equipment and storage medium | |
| CN116860949A (en) | Question-answering processing method, device, system, computing equipment and computer storage medium | |
| CN117785723A (en) | Dynamic interface parameter association method and device and electronic equipment | |
| CN111767213B (en) | Database check point testing method and device, electronic equipment and storage medium | |
| CN112765250B (en) | Method and device for supporting json data conversion into relational database | |
| CN110688823A (en) | XML file verification method and device | |
| CN114461193B (en) | Method, device, equipment and storage medium for generating communication protocol code | |
| CN113672491B (en) | Method for detecting legality of SQL statement, storage medium and terminal equipment | |
| CN114780420A (en) | Method, device, equipment and storage medium for automatic test based on test case | |
| CN113238940A (en) | Interface test result comparison method, device, equipment and storage medium | |
| CN112733199A (en) | Data processing method and device, electronic equipment and readable storage medium | |
| CN111399825A (en) | API (application programming interface) development management method and system based on API document | |
| CN117390055B (en) | JOOQ continuous list sentence generation method, device and medium | |
| CN115576959B (en) | Real-time quality inspection method and device, electronic equipment and storage medium | |
| CN114676049A (en) | Case testing method and device, electronic equipment and storage medium | |
| CN110109774B (en) | Method and system for generating Sercedit recognizable INF file through SCM (Single chip computer) knowledge base |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |