CN108710667A - A kind of character types conversion method based on big data - Google Patents
A kind of character types conversion method based on big data Download PDFInfo
- Publication number
- CN108710667A CN108710667A CN201810463979.4A CN201810463979A CN108710667A CN 108710667 A CN108710667 A CN 108710667A CN 201810463979 A CN201810463979 A CN 201810463979A CN 108710667 A CN108710667 A CN 108710667A
- Authority
- CN
- China
- Prior art keywords
- database
- data
- type
- field
- big data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Abstract
The present invention discloses a kind of database field type conversion method, it is imported into big data platform for traditional database, big data platform only preserves field information of the data information without preserving data, the problem of loss so as to cause field information in traditional database table；The present invention creates field transition matrix table and database field information table by the built-in Hive databases in big data platform, Hive databases；Data import when, in Hive databases newdata library table and keep database table type with import being consistent property of database table；When data export, newdata library table and the consistency of database table type in database table type and Hive is kept in export database；By intermediate Hive database tables, the consistency for importing field type in database, export database ensure that.
Description
Technical field
The invention belongs to big data field, more particularly to a kind of character types switch technology based on big data.
Background technology
The rise of big data technology so that in large-scale data set, data cleansing, data export become a reality.But work as
In preceding data set, in order to reduce the complexity of data importing, it is artificially lost the field attribute of data and only retention data
Information, so as to cause the loss of data field information.
Currently, large-scale data exporting mainly faces following difficulty：
1. the type of database diversification imported, different types of having differences property of database field definition.
2. the diversity of database version, between same type database different editions, having differences property of field definition.
Type of database diversification, different types of having differences property of database field definition derived from 3..
Invention content
The present invention proposes a kind of database field type conversion method, by intermediate Hive database tables, ensure that
Import the consistency of field type in database, export database.
The technical solution adopted by the present invention is：A kind of database field type conversion method, including：
Built-in Hive databases in big data platform, Hive databases create field transition matrix table and database field information
Table；Field transition matrix table preserves Common database field type definition information；Database field information table at least records importing
Each field type and importing database address in tables of data；
The newdata table in new database, the newly-built data table types and data table types one in Hive databases
It causes.
Further, when exporting data from big data platform, according to export type of database, pass through field transition matrix
Table generates new data conversion type.
Further, big data platform export data carry out data conversion according to new data conversion type.
Further, the tables of data of same type is generated in new database according to new data conversion type.
Further, transformed data are imported into the same type tables of data generated in new database.
Beneficial effects of the present invention：A kind of character types conversion method based on big data of the present invention, first from importing
The field information of specified database table in database generates the field information after conversion according to character conversion table, thus by palpus
Derived data carry out data conversion according to the requirement of export database and export in new database；Pass through field type
Conversion, to ensure that the consistency of field format.
Description of the drawings
Fig. 1 is the solution of the present invention principle schematic.
Specific implementation mode
For ease of those skilled in the art understand that the present invention technology contents, below in conjunction with the accompanying drawings to the content of present invention into one
Step is illustrated.
A kind of character types conversion method based on big data of the present invention, by Hive data built in big data platform
Library table ensure that the consistency for importing field type in database, export database.As shown in Figure 1, being specially：
Built-in Hive databases in big data platform, Hive databases create field transition matrix table and database field information
Table；Field transition matrix table preserves the Common databases field type definition information such as Oracle, Mysql, DB2.
When traditional database imports big data platform, record is with importing each field type in tables of data, importing database
Location is simultaneously stored in Database field information table.
When big data platform exports to traditional database, according to information, the export data in Database field information table
Library and field transition matrix export correct format and are saved in derived database.
The method of the present invention includes following flow：
S1. from extraction database table information in database is imported, field information is extracted from database table information；
S2. the field information that will be extracted is stored in the Database field information table created in big data platform；
S3. the Database field conversion table in big data platform generates new data conversion type according to export type of database；
When S4. exporting data, the data conversion type according to step S3 carries out data conversion and exports in new database.
In step sl, the field information of tables of data is obtained first from traditional database.Herein with Mysql databases
It is illustrated for YY_Table tables of data in YY_DataBase.In the present embodiment, the field information exp_data_ of table
export` (
`id` int(10) NOT NULL DEFAULT '0',
`_ID_` varchar(255) DEFAULT NULL,
`idcard` varchar(255) DEFAULT NULL,
`name` varchar(255) DEFAULT NULL,
`emailNumber` varchar(255) DEFAULT NULL,
`phone` varchar(255) DEFAULT NULL,
`role` varchar(255) DEFAULT NULL,
`QQNumber` int(10) NOT NULL DEFAULT '0'
) ENGINE=InnoDB DEFAULT CHARSET=latin1。
In step s 2, big data platform safeguards Database field information table.Database field information table includes to import number
According to source IP address, port, database name, database table name, storage database address, field name and type information.In this example
In, newly-increased single record information is as follows after importing database：
10.8.250.10,3306, YY_DataBase, YY_Table ,/Hive/YY_DataBase/YY_Table,
{id, int(10)}, {_ID_, varchar}, {idcard, varchar}, {name, varchar},
{emailNumber, varchar}, {phone, varchar}, {role, varchar}, {QQNumber, int
(10)}}
It is recorded by this, clearly obtains and import database, storage database and field information.
In step s3, for the Database field conversion table in big data platform according to export type of database, it is suitable to generate
Data conversion type.In database conversion table, data are imported and exported including common, this example with Mysql, Oracle this two
For kind type of database, table 1 is the representation of same field type disparate databases.
The representation of 1 same field type disparate databases of table
| Database | Int（1） | Int（2） | Int（4） | INT(8) | VARchar | DATE |
| Mysql | TINYINT | SMALLINT | INT | BIGINT | VARchar | DATE |
| Oracle | NUMBER(1) | NUMBER(2) | NUMBER(4) | NUMBER(8) | VARchar2 | DATE |
According to storage address/Hive/YY_DataBase/YY_Table in big data platform, by Database field information table
Field information, matched with Database field conversion table, to obtain such as following table information
{{ id, NUMBER(4)}, {_ID_, varchar2}, {idcard, varchar2}, {name,
varchar2}, {emailNumber, varchar2}, {phone, varchar2}, {role, varchar2},
{QQNumber, NUMBRE(4)}}
In step s 4, it for derived data, carries out data conversion according to new data conversion type and exports to new number
According in library.According to the data sheet field type information generated in step s3, same type is generated in new database Oracle
Tables of data, type is as follows：
Create table YY_Table_Oracle ( id NUMBER(4)), (idcard varchar2), (name
varchar2), (emailNumber varchar2), (phone varchar2), (role varchar2),
(QQNumber NUMBRE(4))
Export operation is executed, general/Hive/YY_DataBase/YY_Table data export to oracle database YY_
In DataBase_Oracle/YY_Table_Oracle tables.
Those of ordinary skill in the art will understand that the embodiments described herein, which is to help reader, understands this hair
Bright principle, it should be understood that protection scope of the present invention is not limited to such specific embodiments and embodiments.For ability
For the technical staff in domain, the invention may be variously modified and varied.All within the spirits and principles of the present invention, made by
Any modification, equivalent substitution, improvement and etc. should be included within scope of the presently claimed invention.
Claims (5)
1. a kind of database field type conversion method, which is characterized in that including：
Built-in Hive databases in big data platform, Hive databases create field transition matrix table and database field information
Table；Field transition matrix table preserves Common database field type definition information；Database field information table at least records importing
Each field type and importing database address in tables of data；
The newdata table in new database, the newly-built data table types and data table types one in Hive databases
It causes.
2. a kind of database field type conversion method according to claim 1, which is characterized in that when from big data platform
When exporting data, according to export type of database, new data conversion type is generated by field transition matrix table.
3. a kind of database field type conversion method according to claim 2, which is characterized in that big data platform exports
Data carry out data conversion according to new data conversion type.
4. a kind of database field type conversion method according to claim 3, which is characterized in that turned according to new data
Change the tables of data that type generates same type in new database.
5. a kind of database field type conversion method according to claim 4, which is characterized in that by transformed data
It imported into the same type tables of data generated in new database.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810463979.4A CN108710667A (en) | 2018-05-15 | 2018-05-15 | A kind of character types conversion method based on big data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810463979.4A CN108710667A (en) | 2018-05-15 | 2018-05-15 | A kind of character types conversion method based on big data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108710667A true CN108710667A (en) | 2018-10-26 |
Family
ID=63867987
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810463979.4A Pending CN108710667A (en) | 2018-05-15 | 2018-05-15 | A kind of character types conversion method based on big data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108710667A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109783555A (en) * | 2018-12-13 | 2019-05-21 | 东软集团股份有限公司 | Form templat storage method, device, storage medium and electronic equipment |
| CN112527792A (en) * | 2019-09-18 | 2021-03-19 | 北京国双科技有限公司 | Data storage method, device, equipment and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8041679B1 (en) * | 2008-06-04 | 2011-10-18 | Symantec Operating Corporation | Synthetic differential backups creation for a database using binary log conversion |
| CN102841784A (en) * | 2011-06-24 | 2012-12-26 | 镇江华扬信息科技有限公司 | Method for dynamically importing Excel data into database |
| CN103530327A (en) * | 2013-09-25 | 2014-01-22 | 清华大学深圳研究生院 | Method for migrating data from non-relational database to relational database |
| CN105069033A (en) * | 2015-07-22 | 2015-11-18 | 北京京东尚科信息技术有限公司 | Method and device for creating database table model |
| CN107766376A (en) * | 2016-08-19 | 2018-03-06 | 华为技术有限公司 | Data alignment method and device |
-
2018
- 2018-05-15 CN CN201810463979.4A patent/CN108710667A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8041679B1 (en) * | 2008-06-04 | 2011-10-18 | Symantec Operating Corporation | Synthetic differential backups creation for a database using binary log conversion |
| CN102841784A (en) * | 2011-06-24 | 2012-12-26 | 镇江华扬信息科技有限公司 | Method for dynamically importing Excel data into database |
| CN103530327A (en) * | 2013-09-25 | 2014-01-22 | 清华大学深圳研究生院 | Method for migrating data from non-relational database to relational database |
| CN105069033A (en) * | 2015-07-22 | 2015-11-18 | 北京京东尚科信息技术有限公司 | Method and device for creating database table model |
| CN107766376A (en) * | 2016-08-19 | 2018-03-06 | 华为技术有限公司 | Data alignment method and device |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109783555A (en) * | 2018-12-13 | 2019-05-21 | 东软集团股份有限公司 | Form templat storage method, device, storage medium and electronic equipment |
| CN109783555B (en) * | 2018-12-13 | 2021-07-30 | 东软集团股份有限公司 | Form template storage method and device, storage medium and electronic equipment |
| CN112527792A (en) * | 2019-09-18 | 2021-03-19 | 北京国双科技有限公司 | Data storage method, device, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103761318B (en) | A kind of method and system of relationship type synchronization of data in heterogeneous database | |
| CN104915450B (en) | A kind of big data storage and retrieval method and system based on HBase | |
| Beaulieu | Learning SQL: master SQL fundamentals | |
| Li | Transforming relational database into HBase: A case study | |
| CN105868204B (en) | A kind of method and device for converting Oracle scripting language SQL | |
| US7805341B2 (en) | Extraction, transformation and loading designer module of a computerized financial system | |
| US8666960B2 (en) | Schema-based data transfer between a data-based application and a document application | |
| CN105069109B (en) | A kind of method and system of distributed data base dilatation | |
| CN103593456B (en) | Worksheet self-defining method for designing and device | |
| CN107391306A (en) | A kind of isomeric data library backup file access pattern method | |
| EP0912948A1 (en) | Database apparatus | |
| US20130086091A1 (en) | Methods and apparatus for organizing data in a database | |
| CN103425762A (en) | Telecom operator mass data processing method based on Hadoop platform | |
| ZA200100187B (en) | Value-instance-connectivity computer-implemented database. | |
| US7877417B2 (en) | Method and apparatus for exchanging data with a database | |
| Leno et al. | Automated discovery of data transformations for robotic process automation | |
| CA2379930A1 (en) | Multi-model access to data | |
| CN108710667A (en) | A kind of character types conversion method based on big data | |
| CN106503214A (en) | A kind of complex rule matching process based on Redis memory databases | |
| CN109522332A (en) | Customer profile data merging method, device, equipment and readable storage medium storing program for executing | |
| US20070150447A1 (en) | Techniques for generic data extraction | |
| CN107103035A (en) | This earth's surface data-updating method and device | |
| CN107346317A (en) | A kind of data query method and apparatus | |
| CN115221143A (en) | Cross-type migration operator-based multi-source big data processing method | |
| CN101789023A (en) | Food material information release management system based on Web and release management method thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20181026 |