CN106951536A - Data method for transformation and system - Google Patents
Data method for transformation and system Download PDFInfo
- Publication number
- CN106951536A CN106951536A CN201710175759.7A CN201710175759A CN106951536A CN 106951536 A CN106951536 A CN 106951536A CN 201710175759 A CN201710175759 A CN 201710175759A CN 106951536 A CN106951536 A CN 106951536A
- Authority
- CN
- China
- Prior art keywords
- data
- database
- conversion
- write
- instrument
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Computing Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of data conversion system, including：Request receiving module, for receiving data conversion request；Instrument determining module, determines to read the reading instrument of data from the source database for the type according to source database and data form, is determined to write the write-in instrument of data in the purpose database according to the type of purpose database and data form；Data acquisition module, for reading data to be transformed from the source database using the reading instrument；Data conversion module, for the data to be transformed to be converted into data corresponding with the data form of the purpose database；Data after the conversion are write the purpose database by Data write. module for obtaining the data after conversion, and using said write instrument.The conversion of data between database can be more flexibly carried out using the present invention.
Description
Technical field
The present invention relates to database technical field, more particularly to a kind of data method for transformation and system.
Background technology
At present, with the development of computer software technology, large-scale IT system (such as occurrences in human life system used in enterprises
System, financial system) it is more and more, different IT systems often generate substantial amounts of data in normal work.The data one
As be stored in corresponding database.Generally, often there is more close dependence between the large-scale IT system of enterprises,
One IT system in normal operation may be frequently necessary to that data are imported or exported from one or more other IT systems,
Or, often by the data syn-chronization of the IT system into one or more other IT systems.
However, the custom of different IT system developers at the beginning of exploitation is different, the type of service corresponding to different IT systems
There is also difference so that the database that most IT system is pre-configured with is often different.For example, Finance System
The MySQL database that may be used stores financial data, and the Oracle numbers that the occurrences in human life data of enterprise personnel system are used
Stored according to storehouse.This causes between different IT systems when carrying out data importing, data export and data syn-chronization task,
Generally require to need longer time, have a strong impact on the operating efficiency of IT system.
The content of the invention
In view of this, the present invention proposes a kind of data method for transformation and system, can more quickly and efficiently carry out enterprise
The transformation of database data between internal IT system.
First, to achieve the above object, the present invention proposes a kind of data conversion system, and the system includes：
Request receiving module, for receiving data conversion request, the data conversion request includes the type of source database
With data form, data to be transformed, the type and data form of purpose database；
Instrument determining module, determines to read from the source database for the type according to the source database and data form
The reading instrument of data, determines to write data in the purpose database according to the type of the purpose database and data form
Write-in instrument；
Data acquisition module, for reading the data to be transformed from the source database using the reading instrument；
Data conversion module, it is corresponding with the data form of the purpose database for the data to be transformed to be converted into
Data；
Data write. module, for obtaining the data after conversion, and utilizes said write instrument by the number after the conversion
According to the write-in purpose database.
Alternatively, the data conversion module, specifically for：
The data to be transformed read are received from the reading instrument, and the data to be transformed are converted into preset format
Character；
The character of the preset format is parsed, the character of the preset format is converted into the data of the purpose database
The data of form.
Alternatively, the purpose database include it is multiple, wherein：
The instrument determining module, be additionally operable to according to the type and data form of each purpose database determine with it is each
The corresponding write-in instrument of individual purpose database；
The data conversion module, is additionally operable to the data to be transformed being separately converted to and each purpose database
The corresponding data of data form；
Data write. module, for obtaining the data after the conversion corresponding with each purpose database, and profit respectively
With the write-in instrument corresponding with each purpose database, corresponding turn is respectively written into each purpose database
Data after change.
Alternatively, the character of the preset format includes the character of Json forms.
Alternatively, data conversion request includes data and imports request, data export request and data syn-chronization asking
Ask；The source database or the purpose database include MySQL, Oracle, Hive, Hdfs.
Compared to prior art, data conversion system proposed by the invention, first according to the type sum of source database
According to form, the type and data form of purpose data, reading instrument and the write-in data institute read needed for data is determined respectively
The write-in instrument needed；Then the data to be transformed that user specifies are read from source database using identified Reading and writing instrument, and
By the data conversion to be transformed into predetermined format character；Then character is medium in a predetermined format, and data to be transformed are converted
As the data for meeting purpose database data call format, finally the data after conversion are write in purpose data.So, lead to
The media format of lightweight is crossed, quickly introducing, exporting and synchronous for data between multiple databases is realized, improves data and turn
The speed of change and the operating efficiency of IT system.Meanwhile, the present invention is when there is the database of format, it is only necessary to added in the system
Corresponding Reading and writing instrument be can be achieved the database data and legacy data storehouse between data convert, without overlapping development,
Multiple data conversion instruments are safeguarded, possess stronger expansibility.
In addition, to achieve the above object, the present invention also provides a kind of data method for transformation, and methods described includes：
Methods described includes：
Receive data conversion request, data conversion request includes the type and data form, to be transformed of source database
Data, the type and data form of purpose database；
Determined to read the reading instrument of data, root from the source database according to the type of the source database and data form
Determine to write the write-in instrument of data in the purpose database according to the type and data form of the purpose database；
The data to be transformed are read from the source database using the reading instrument；
The data to be transformed are converted into data corresponding with the data form of the purpose database；And
The data after conversion are obtained, and the data after the conversion are write into the purpose data using said write instrument
Storehouse.
Alternatively, it is described that the data to be transformed are converted into data corresponding with the data form of the purpose database
The step of, specifically include：
The data to be transformed read are received from the reading instrument, and the data to be transformed are converted into preset format
Character；And
The character of the preset format is parsed, the character of the preset format is converted into the data of the purpose database
The data of form.
Alternatively, when the purpose database includes multiple, the type and data according to the purpose database
The step of form determines to write the write-in instrument of data in the purpose database, also includes：
Write-in corresponding with each purpose database is determined according to the type of each purpose database and data form
Instrument；
It is described the step of the data to be transformed are converted into the corresponding data with the data form of the purpose database,
Also include：
The data to be transformed are separately converted to the data corresponding with the data form of each purpose database；
The data obtained after conversion, and the data after the conversion are write into the purpose using said write instrument
The step of database, also include：
The data after the conversion corresponding with each purpose database are obtained respectively, and are utilized and each purpose data
The corresponding write-in instrument in storehouse, the data after being respectively written into corresponding conversion in each purpose database.
Alternatively, the character of the preset format includes the character of Json forms.
Alternatively, data conversion request includes data and imports request, data export request and data syn-chronization asking
Ask；The source database or the purpose database include MySQL, Oracle, Hive, Hdfs.
Compared to prior art, data method for transformation proposed by the invention, first according to the type sum of source database
According to form, the type and data form of purpose data, reading instrument and the write-in data institute read needed for data is determined respectively
The write-in instrument needed；Then the data to be transformed that user specifies are read from source database using identified Reading and writing instrument, and
By the data conversion to be transformed into predetermined format character；Then character is medium in a predetermined format, and data to be transformed are converted
As the data for meeting purpose database data call format, finally the data after conversion are write in purpose data.So, lead to
The media format of lightweight is crossed, quickly introducing, exporting and synchronous for data between multiple databases is realized, improves data and turn
The speed of change and the operating efficiency of IT system.Meanwhile, the present invention is when there is the database of format, it is only necessary to added in the system
Corresponding Reading and writing instrument be can be achieved the database data and legacy data storehouse between data convert, without overlapping development,
Multiple data conversion instruments are safeguarded, possess stronger expansibility.
Brief description of the drawings
Fig. 1 is the optional applied environment schematic diagram of each embodiment one of the invention；
Fig. 2 is the schematic diagram of the optional hardware structure of electronic installation one in Fig. 1；
Fig. 3 is the functional block diagram of the embodiment of data conversion system first, second of the present invention；
Fig. 4 A-4B are the schematic diagrames of Framework+plugin frameworks in the present invention；
Fig. 5 be in the present invention purpose database for it is multiple when data flow schematic diagram；
Fig. 6 is the schematic diagram of multithread mode progress data conversion in the present invention；
Fig. 7 is the implementation process diagram of data method for transformation first embodiment of the present invention；
Fig. 8 is the implementation process diagram of data method for transformation second embodiment of the present invention.
Reference：
The realization, functional characteristics and advantage of the object of the invention will be described further referring to the drawings in conjunction with the embodiments
Embodiment
In order to make the purpose , technical scheme and advantage of the present invention be clearer, it is right below in conjunction with drawings and Examples
The present invention is further elaborated.It should be appreciated that specific embodiment described herein is only to explain the present invention, not
For limiting the present invention.Based on the embodiment in the present invention, those of ordinary skill in the art are not before creative work is made
The every other embodiment obtained is put, the scope of protection of the invention is belonged to.
It should be noted that the description for being related to " first ", " second " etc. in the present invention is only used for describing purpose, and can not
It is interpreted as indicating or implies its relative importance or the implicit quantity for indicating indicated technical characteristic.Thus, define " the
One ", at least one this feature can be expressed or be implicitly included to the feature of " second ".In addition, the skill between each embodiment
Art scheme can be combined with each other, but must can be implemented as basis with those of ordinary skill in the art, when technical scheme
With reference to occur it is conflicting or when can not realize it will be understood that the combination of this technical scheme is not present, also not in application claims
Protection domain within.
Refering to shown in Fig. 1, being the optional applied environment schematic diagram of each embodiment one of the invention.
In the present embodiment, data conversion system provided by the present invention and method can be applied to include but not limited to,
In electronic installation 1, server 2, the applied environment of network 3.Wherein, the electronic installation 1 can be mobile phone, intelligence electricity
Words, notebook computer, digit broadcasting receiver, PDA (personal digital assistant), PAD (panel computer), PMP (portable multimedias
Player), the mobile terminal of guider etc., and such as numeral TV, desktop computer, notebook, server etc.
Fixed terminal.The server 2 can be rack-mount server, blade server, tower server or Cabinet-type server
Deng computing device, the server 2 can be the server set that independent server 2 or multiple servers 2 are constituted
Group.The network 3 can be intranet (Intranet), internet (Internet), global system for mobile communications
(Global System of Mobile communication, GSM), WCDMA (Wideband Code
Division Multiple Access, WCDMA), 4G networks, 5G networks, bluetooth (Bluetooth), Wi-Fi etc. is wireless or has
Gauze network.
Wherein, data conversion system 200 of the present invention and one or more enterprises are installed in the electronic installation 1
IT system, the Enterprise IT System includes Finance System, enterprise personnel system, attendance system for company etc..The electronics dress
1 is put to communicate to connect with server one or more described 2 (only showing 2 in figure) respectively by the network 3.Described one or
The database corresponding with one or more of Enterprise IT Systems is deployed with 2 in many services, the database may include
MySQL, Oracle, Hive etc., are respectively used to store corresponding Enterprise IT System operationally produced various data,
Personnel information, enterprise financial report data, staff attendance record such as enterprise staff.It is pointed out that with described one
The corresponding database of individual or multiple Enterprise IT Systems can be deployed in a server 2 (such as with business finance system simultaneously
The corresponding database of system, enterprise personnel system, attendance system for company is deployed in server A), it can also be respectively deployed in
(the data Kuku of such as Finance System is deployed in server A, and corresponding with enterprise personnel system in different servers 2
Database be then deployed in server B).
In the present embodiment, when the electronic installation 1 runs described data conversion system 200, the electronic installation 1 is real-time
Receive user triggering data conversion request, and according to the data convert request out of server described in one 2 deployment it is a certain
Read in database (i.e. source database) and treat default conversion data, and then the data to be transformed are converted, this is treated
Conversion data is converted into the character of preset format.Eventually through using the character of the preset format as medium, wait to turn by described
Change the data that data are converted into the data form that another or multiple databases (i.e. purpose database) are supported, and after converting
Data be written in another or multiple databases.So as to it is quick, neatly realize different IT systems database it
Between data conversion.
Refering to the schematic diagram for shown in Fig. 2, being the optional hardware structure of electronic installation 1 one in Fig. 1.It is described in the present embodiment
Data conversion system 200 install and run in the electronic installation 1, the electronic installation 1 may include, but be not limited only to, can
Connection memory 11, processor 12, display 13 and network interface 14 are in communication with each other by system bus.It is pointed out that
Fig. 2 illustrate only the electronic installation 1 with component 11-14, it should be understood that being not required for implementing all groups shown
Part, what can be substituted implements more or less components.
Wherein, the memory 11 at least includes a type of storage medium, and the storage medium includes flash memory, hard
Disk, multimedia card, card-type memory (for example, SD or DX memories etc.), random access storage device (RAM), static random-access
Memory (SRAM), read-only storage (ROM), Electrically Erasable Read Only Memory (EEPROM), programmable read-only storage
Device (PROM), magnetic storage, disk, CD etc..In certain embodiments, the memory 11 can be the electronic installation
The hard disk or internal memory of 1 internal storage unit, such as electronic installation 1.In further embodiments, the memory 11 also may be used
To be the External memory equipment of the electronic installation 1, such as the plug-in type hard disk being equipped with the electronic installation 1, intelligent memory card
(Smart Media Card, SMC), secure digital (Secure Digital, SD) card, flash card (Flash Card) etc..When
So, the memory 11 can also both include the internal storage unit of the electronic installation 1 or including its External memory equipment.This
In embodiment, the memory 11 is generally used for storing the operating system and types of applications software that are installed on the electronic installation 1,
Such as the program code of the data conversion system 200.In addition, the memory 11 can be also used for temporarily storing
Output or the Various types of data that will be exported.
The processor 12 can be in certain embodiments central processing unit (Central Processing Unit,
CPU), controller, microcontroller, microprocessor or other data processing chips.The processor 12 is generally used for controlling the electricity
The overall operation of sub-device 1, for example, perform the control and processing related to the server 2 progress data interaction or communication
Deng.In the present embodiment, the processor 12 is used to run the program code stored in the memory 11 or processing data, example
Data conversion system 200 as described in running.
The display 13 is include but not limited to, and liquid crystal display (LCD), thin film transistor (TFT) LCD (TFT-LCD), is had
At least one in machine light emitting diode (OLED) display, flexible display, three-dimensional (3D) display etc..Further,
Some in these displays may be constructed such that transparence to allow user to be watched from outside, and this is properly termed as Transparence Display
Device, typical transparent display can be, for example, transparent organic light emitting diode (TOLED) display etc..The display 13 leads to
It is usually used in being shown in the information handled in the electronic installation 1, such as user interface (UI) or graphic user interface (GUI), figure
UI or GUI of picture, video and correlation function etc..In the present embodiment, the display 13 is mainly used in display and converted with data
The related information of process, such as Database Lists, data table name, data conversion tools menu.
The network interface 14 may include radio network interface or wired network interface, and the network interface 14 is generally used for
Communication connection is set up between the electronic installation 1 and other electronic equipments.In the present embodiment, the network interface 14 is mainly used in
The electronic installation 1 is connected with one or more servers 2 by the network 3, the electronic installation 1 with it is described
Data transmission channel and communication connection are set up between server 2.
So far, oneself is through describing the applied environment of each embodiment of the invention and the hardware configuration of relevant device and work(in detail
Energy.Below, above-mentioned applied environment and relevant device will be based on, each embodiment of the present invention is proposed.
First, the present invention proposes a kind of data conversion system 200.
Refering to the functional block diagram for shown in Fig. 3, being the first embodiment of data conversion system 200 of the present invention.In the present embodiment
In, described data conversion system 200 can be divided into one or more modules, and one or more of modules are stored
In the memory 11, and it is performed by one or more processors (being the processor 12 in the present embodiment), to complete
The present invention.For example, in figure 3, described data conversion system 200 can be divided into request receiving module 201, instrument and determine
Module 202, data acquisition module 203, data conversion module 204 and Data write. module 205.Function alleged by the present invention
Module is the series of computation machine programmed instruction section for referring to complete specific function, is turned than program more suitable for describing the data
Implementation procedure of the change system 200 in the electronic installation 1.The function of putting up with each functional module 201-205 below is carried out in detail
Description.
The request receiving module 201, for receiving data conversion request, the data conversion request includes source database
Type and data form, data to be transformed, the type and data form of purpose database.
Usually, often there is more closely contact, a traffic issues are generally needed between the IT system of enterprises
Wanting two or more IT systems to cooperate could perfectly solve.For example, the attendance record to obtain enterprise staff in real time,
Generally require first to read enterprise staff information from personnel system, then obtain corresponding attendance record ability from attendance checking system again
Complete.Need to lead the data of an IT system from the IT system thus, it will be seen that current enterprise customer often suffers from
Go out to another IT system or the data of another IT system are imported into current IT system or by one from the IT system
The data syn-chronization of individual IT system is into other one or more IT systems, the business during a certain traffic issues that could be solved
Scape.Briefly, i.e., the mutual transition problem of data is frequently referred between current enterprise IT system, how to be quickly completed not
With IT system, the importing of data between corresponding multiple databases, export and synchronous, directly affect the work effect of Enterprise IT System
Rate.
Therefore, in the present embodiment, the detecting real-time of request receiving module 201 is installed on all in the electronic installation 1
The pending business that IT system is operationally produced, and when the pending business includes data conversion request, obtain institute
State the type and data form, data to be transformed, the type and data of purpose database of the source database in data conversion request
Form.Wherein described source database is the database where the data to be transformed, and the purpose database described will be waited to turn
Change the database that data are imported or are synchronized in own database.The data to be transformed may include, but be not limited only to, data
Table, data field, data file etc..
In certain embodiments, it is described to ask that receiver module 201 directly provide the user data conversion interface, the number
According to conversion interface display in all IT in the display 13 of the electronic installation 1, and including being installed in the electronic installation 1
The title of system, and the type of database corresponding with all IT systems and data form, database address etc..So, when
User need carry out data conversion when, only need to the data conversion interface in choose source database (such as MySQL) and its
Data form, purpose database (such as Oracle) and data form and corresponding data to be transformed are (such as according to data
The tables of data a) stored under a certain root address that table name is searched, you can the triggering data conversion request.
The instrument determining module 202, is determined from the source number for the type according to the source database and data form
The reading instrument of data is read according to storehouse, is determined according to the type of the purpose database and data form in the purpose database
Write the write-in instrument of data.
Usually, needed when data are read out of different database by different reading instruments.For example, will be from
When reading data in MySQL database, generally require to use MysqlReader instruments；And in oracle database receive data
According to when, then usually require OracleReader instruments.Correspondingly, when needing to write corresponding data in MySQL database,
Generally require and use MysqlWriter instruments；And when needing to write corresponding data in oracle database, then usually use
OracleWriter instruments.Therefore, in the present embodiment, the determination instrument determining module 202 can be respectively according to the source number
According to storehouse and the respective type of purpose database and data form each is supported, corresponding reading instrument and write-in work is determined
Tool.For example, when the source database is MySQL, it is MysqlReader instruments to determine the reading instrument, in the source number
During according to storehouse for Oracle, it is OracleReader instruments etc. to determine the reading instrument.
It should be noted that in the present embodiment, the instrument determining module 202 is with card cage (Framework+
Plugin form) loads the reading instrument and said write instrument.Specifically, as shown in Figure 4 A, the reading instrument and
Said write instrument is connected to Framework two ends, and the Framework then as the reading instrument and described writes
Enter the data transmission channel of instrument.Wherein, the reading instrument is used for the data of gathered data source (i.e. source database), then will
Institute's gathered data is sent to Framework.Said write instrument is then used to constantly obtain data from Framework, and then by number
According to being written to destination (i.e. purpose database).For example, Fig. 4 B show an example, source database described in the example
For MySQL, the purpose database is Hdfs, and the reading instrument is MysqlReader, and said write instrument is
HdfsWriter.The data read in the example from MySQL are finally write via the transmission of Framework+plugin frameworks
Enter into Hdfs.
The data acquisition module 203, for waiting described in reading from the source database using the reading instrument to turn
Change data.
It will be recalled from above that utilizing the reading instrument corresponding from the type and data form of different databases, Ke Yicong
Data are read in different databases.For example, using the Mysqlreader instruments, can be read out of MySQL database
Data, using the Oraclereader instruments, can read data etc. out of oracle database.Therefore, in the present embodiment,
The data acquisition module 203 utilizes the reading instrument corresponding with the source database, and the band conversion pre-set
The information of data, can read the data to be transformed, such as according to the data to be transformed from the source database
Address or title, read described data to be transformed etc. from the source database.
The data conversion module 204, for the data to be transformed to be converted into the data with the purpose database
The corresponding data of form.
It will be recalled from above that can be read by reading instrument corresponding with source database from the source database described
Data to be transformed.However, the data form of the data to be transformed read out remains the data form of the source database,
The data form can not be written directly to entirely different with the source database type and data form with conversion data
In purpose database.Therefore, in the present embodiment, reading instrument is gone to read out the data to be transformed from the source database when described
Afterwards, the data conversion module 204 changes into the data to be transformed the character of preset format, then described pre- by parsing
If the character of form, then the character of the preset format is converted into the data of the data form of the purpose data.
Wherein, the character of the preset format may include the character of json forms.The json forms are a kind of lightweights
Data interchange format, employ the text formatting independently of language, similar XML, but simpler than XML possesses readability, easily compiled
Write and be easy to parse and generate, transmit the advantages of shared network bandwidth is less.Transmitted and turned using the character of json forms
It is a kind of more ripe prior art to change database data, nearly all existing database (such as MySQL, Oracle,
Hive, Hdfs) it can very easily be converted into the character of json forms.Therefore, the data conversion module 204 can profit
The conversion of the data to be transformed is completed with the json forms.
Specifically, the data conversion module 204 can receive the reading instrument from described by Framework first
The data to be transformed that source data is read；Then the data to be transformed are parsed, it is json to convert the data to be transformed
The character of form and preservation；Then according to the data form of the purpose database, the data to be transformed are transformed
The characters of the json forms parsed, the character of the json forms is converted into the data lattice of the purpose database
The data of formula.
The Data write. module 205, for obtaining the data after conversion, and utilizes said write instrument by the conversion
Data afterwards write the purpose database.
In the present embodiment, when the data to be transformed are via the character of the preset format as transformation medium, it is converted into
After the data for the data form for meeting the purpose database, the Data write. module 205 can call fixed and institute
The type of the purpose database write-in instrument corresponding with data form is stated, the data after the conversion are written to corresponding mesh
Database in.
Pass through above-mentioned functions module 201-205, data conversion system 200 proposed by the invention, by previously according to source
The different characteristic of database and purpose database, determine respectively read data needed for reading instrument and write-in data needed for
Write-in instrument；Then the data to be transformed that user specifies are read from source database using identified Reading and writing instrument, and are passed through
The json forms of lightweight are as medium by the data conversion to be transformed of source database data form into meeting purpose database
The data of data format requirement, enable to be written directly into purpose database.So, the medium lattice of lightweight are passed through
Formula, realizes quickly introducing, exporting and synchronous for data between multiple databases, improves the speed and IT system of data conversion
Operating efficiency.
Further, the above-mentioned first embodiment based on data conversion system 200 of the present invention, proposes that the second of the present invention is real
Apply example.
As shown in figure 5, be in the present invention purpose database for it is multiple when data flow schematic diagram, wherein：
The instrument determining module 202, be additionally operable to according to the type and data form of each purpose database determine with
The corresponding write-in instrument of each purpose database.
In the present embodiment, if the purpose database is multiple, mean described from source data place reading
Data to be transformed need to be directed to the different conversions of different purpose databases progress.Namely the data to be transformed are being converted
, it is necessary to be converted into the data of different-format, to meet the data format requirement of different purpose databases after the character of preset format,
Different purpose databases could be write.Therefore, in the present embodiment, the instrument determining module 202 needs to be directed to different
The type and data form of purpose database, it is determined that the write-in instrument corresponding with each purpose database, in order to follow-up
The data after conversion are write in corresponding purpose database using different write-in instruments.
The data conversion module 204, is additionally operable to the data to be transformed being separately converted to and each purpose data
The corresponding data of the data form in storehouse.
In the present embodiment, when the purpose database is multiple, the data conversion module 204 is read from described first
Instrument obtains the data to be transformed for having been converted into the preset format character, then according to the number of each purpose database
The character of the preset format is parsed respectively according to form, the character of the preset format is separately converted to meet
The data of the data format requirement of each purpose database, in order to the above-mentioned fixed write-in instrument of later use, by institute
Data are stated to be respectively written into corresponding purpose database.
You need to add is that, in the present embodiment, the data conversion module 204 completes above-mentioned incite somebody to action using multithread mode
The preset format character is separately converted to the work of the data corresponding from different purpose databases.Specifically, such as Fig. 6 institutes
Show, the data conversion module 204 is created according to each purpose database information and the preset format character information first
One pending business job；Then, cutting is carried out to the job according to different purpose databases, by the job cuttings
For multiple small task task, the data conversion task that each of which task is responsible for being directed to a purpose database (will
The preset format character changes into the task of data corresponding with a purpose database data form)；Then, will be described
Task tasks are grouped by default concurrent data, generate task task groups；Finally, distinguished simultaneously in the form of task task groups
Hair performs all task tasks, completes the data conversion task.
The Data write. module 205, for obtaining the number after the conversion corresponding with each purpose database respectively
According to, and the write-in instrument corresponding with each purpose database is utilized, it is respectively written into therewith in each purpose database
Data after corresponding conversion.
Pass through above-mentioned functions module 202,204,205, data conversion system 200 proposed by the invention, in purpose data
When storehouse has multiple, the different data format requirement of each purpose database is directed to respectively, the institute to being converted into media format
State data to be transformed and carry out different processing, so that the data to be transformed turn to be changed into respectively to meet different purpose numbers
According to the data of database data call format, and it can be written into different purpose databases.
In addition, the present invention also proposes a kind of data method for transformation.
Refering to the implementation process diagram for shown in Fig. 7, being data method for transformation first embodiment of the present invention.In the present embodiment
In, according to different demands, the execution sequence of the step in flow chart shown in Fig. 7 can change, and some steps can be omitted.
Step S301, receives data conversion request, and the data conversion request includes the type and data lattice of source database
Formula, data to be transformed, the type and data form of purpose database.
Usually, often there is more closely contact, a traffic issues are generally needed between the IT system of enterprises
Wanting two or more IT systems to cooperate could perfectly solve.For example, the attendance record to obtain enterprise staff in real time,
Generally require first to read enterprise staff information from personnel system, then obtain corresponding attendance record ability from attendance checking system again
Complete.Need to lead the data of an IT system from the IT system thus, it will be seen that current enterprise customer often suffers from
Go out to another IT system or the data of another IT system are imported into current IT system or by one from the IT system
The data syn-chronization of individual IT system is into other one or more IT systems, the business during a certain traffic issues that could be solved
Scape.Briefly, i.e., the mutual transition problem of data is frequently referred between current enterprise IT system, how to be quickly completed not
With IT system, the importing of data between corresponding multiple databases, export and synchronous, directly affect the work effect of Enterprise IT System
Rate.
Therefore, in the present embodiment, all IT systems that can be installed on detecting real-time in the electronic installation 1 are operationally produced
Raw pending business, and when the pending business includes data conversion request, obtain in the data conversion request
Source database type and data form, data to be transformed, the type and data form of purpose database.Wherein described source number
Be the database where the data to be transformed according to storehouse, the purpose database will the data to be transformed import or be synchronized to
Database in own database.The data to be transformed may include, but be not limited only to, tables of data, data field, data file
Deng.
In certain embodiments, data conversion interface, data conversion interface display can also directly be provided the user
In in the display 13 of the electronic installation 1, and including the title for all IT systems being installed in the electronic installation 1, with
And the type of database corresponding with all IT systems and data form, database address etc..So, when user is needed into line number
During according to conversion, only source database (such as MySQL) and its data form, purpose number need to be chosen in data conversion interface
According to storehouse (such as Oracle) and data form and corresponding data to be transformed (such as certain searched according to data table name
The tables of data a) stored under one root address, you can the triggering data conversion request.
Step S302, determines to read the reading of data from the source database according to the type of the source database and data form
Instrument is taken, is determined to write the write-in work of data in the purpose database according to the type of the purpose database and data form
Tool.
Usually, needed when data are read out of different database by different reading instruments.For example, will be from
When reading data in MySQL database, generally require to use MysqlReader instruments；And in oracle database receive data
According to when, then usually require OracleReader instruments.Correspondingly, when needing to write corresponding data in MySQL database,
Generally require and use MysqlWriter instruments；And when needing to write corresponding data in oracle database, then usually use
OracleWriter instruments.Therefore, the present embodiment can be respective according to the source database and the purpose database respectively
Type and data form each is supported, determine corresponding reading instrument and write-in instrument.For example, being in the source database
During MySQL, it is MysqlReader instruments to determine the reading instrument, when the source database is Oracle, determines the reading
It is OracleReader instruments etc. to take instrument.
It should be noted that the present embodiment loads the reading work in the form of card cage (Framework+plugin)
Tool and said write instrument.Specifically, as shown in Figure 4 A, the reading instrument and said write instrument are connected to
Framework two ends, the Framework is then as the reading instrument and the data transmission channel of said write instrument.
Wherein, the reading instrument is used for the data of gathered data source (i.e. source database), is then sent to institute's gathered data
Framework.Said write instrument is then used to constantly obtain data from Framework, and then writes data into destination (i.e.
Purpose database).For example, Fig. 4 B show an example, and source database described in the example is MySQL, the purpose
Database is Hdfs, and the reading instrument is MysqlReader, and said write instrument is HdfsWriter.In the example from
The data that MySQL is read are eventually written into Hdfs via the transmission of Framework+plugin frameworks.
Step S303, the data to be transformed are read using the reading instrument from the source database.
It will be recalled from above that utilizing the reading instrument corresponding from the type and data form of different databases, Ke Yicong
Data are read in different databases.For example, using the Mysqlreader instruments, can be read out of MySQL database
Data, using the Oraclereader instruments, can read data etc. out of oracle database.Therefore, the present embodiment profit
, can be from described with the reading instrument corresponding with the source database, and the information with conversion data pre-set
The data to be transformed are read in source database, such as according to the address of the data to be transformed or title, from the source number
According to reading described data to be transformed etc. in storehouse.
The data to be transformed are converted into data corresponding with the data form of the purpose database by step S304.
It will be recalled from above that can be read by reading instrument corresponding with source database from the source database described
Data to be transformed.However, the data form of the data to be transformed read out remains the data form of the source database,
The data form can not be written directly to entirely different with the source database type and data form with conversion data
In purpose database.Therefore, in the present embodiment, reading instrument is gone to read out the data to be transformed from the source database when described
Afterwards, it is necessary to which the data to be transformed to be changed into the character of preset format, then by parsing the character of the preset format, then
The character of the preset format is converted into the data of the data form of the purpose data.
Wherein, the character of the preset format may include the character of json forms.The json forms are a kind of lightweights
Data interchange format, employ the text formatting independently of language, similar XML, but simpler than XML possesses readability, easily compiled
Write and be easy to parse and generate, transmit the advantages of shared network bandwidth is less.Transmitted and turned using the character of json forms
It is a kind of more ripe prior art to change database data, nearly all existing database (such as MySQL, Oracle,
Hive, Hdfs) it can very easily be converted into the character of json forms.Therefore, the present embodiment can utilize the json
Form completes the conversion of the data to be transformed.
Specifically, the present embodiment can receive the reading instrument by Framework first and be read from the source data
The data to be transformed taken；Then the data to be transformed are parsed, the character that the data to be transformed are json forms is converted
And preserve；Then according to the data form of the purpose database, the json that the data to be transformed are transformed
The character of form is parsed, and the character of the json forms is converted into the data of the data form of the purpose database.
Step S305, obtains the data after conversion, and the data after the conversion are write into institute using said write instrument
State purpose database.
When character of the data to be transformed via the preset format as transformation medium, it is converted into and meets the purpose
After the data of the data form of database, the present embodiment can call fixed and type and data of the purpose database
The corresponding write-in instrument of form, the data after the conversion are written in corresponding purpose database.
Pass through above-mentioned steps S301-305, data method for transformation proposed by the invention, by previously according to source database
With the different characteristic of purpose database, determine respectively read data needed for reading instrument and write-in data needed for write-in work
Tool；Then the data to be transformed that user specifies are read from source database using identified Reading and writing instrument, and pass through lightweight
Json forms as medium by the data conversion to be transformed of source database data form into meeting purpose database data lattice
The data of formula requirement, enable to be written directly into purpose database.So, by the media format of lightweight, realize
Data are quickly introduced, exported and synchronous between multiple databases, improve the speed of data conversion and the work of IT system
Efficiency.
Further, the above-mentioned first embodiment based on data method for transformation of the present invention, proposes data conversion side of the present invention
The second embodiment of method.
As shown in figure 8, being the implementation process diagram of data method for transformation second embodiment of the present invention.In the present embodiment,
When the purpose database includes multiple：
Above-mentioned steps S302 also includes：
S401, is determined corresponding with each purpose database according to the type of each purpose database and data form
Write-in instrument.
In the present embodiment, if the purpose database is multiple, mean described from source data place reading
Data to be transformed need to be directed to the different conversions of different purpose databases progress.Namely the data to be transformed are being converted
, it is necessary to be converted into the data of different-format, to meet the data format requirement of different purpose databases after the character of preset format,
Different purpose databases could be write.Therefore, current embodiment require that being directed to the type and data of different purpose databases
Form, it is determined that the write-in instrument corresponding with each purpose database, in order to which the different write-in instrument of later use will turn
Data after change are write in corresponding purpose database.
Above-mentioned steps S304 also includes：
The data to be transformed are separately converted to the number corresponding with the data form of each purpose database by S402
According to；
In the present embodiment, when the purpose database is multiple, is obtained from the reading instrument have been converted into institute first
The data to be transformed of preset format character are stated, then according to the data form of each purpose database respectively to described pre-
If the character of form is parsed, the character of the preset format is separately converted to meet each purpose database
The data of data format requirement, in order to the above-mentioned fixed write-in instrument of later use, the data are respectively written into therewith
In corresponding purpose database.
You need to add is that, the present embodiment completes above-mentioned to turn the preset format character respectively using multithread mode
Turn to the work of the data corresponding from different purpose databases.Specifically, as shown in fig. 6, the data conversion module 204
A pending business job is created according to each purpose database information and the preset format character information first；So
Afterwards, cutting is carried out to the job according to different purpose databases, is multiple small task task by the job cuttings, wherein
Each task be responsible for being directed to a purpose database data conversion task (will the preset format character change into
The task of the corresponding data of one purpose database data form)；Then, the task tasks are entered by default concurrent data
Row packet, generates task task groups；Finally, all task tasks are concurrently performed respectively in the form of task task groups, are completed
The data convert task.
Above-mentioned steps S305 also includes：
S403, obtains the data after the conversion corresponding with each purpose database, and utilize and each mesh respectively
The corresponding write-in instrument of database, the data after being respectively written into corresponding conversion in each purpose database.
By above-mentioned steps S401-S403, data method for transformation proposed by the invention, have multiple in purpose database
When, the different data format requirement of each purpose database is directed to respectively, to being converted into the described to be transformed of media format
Data carry out different processing, so that the data to be transformed turn to be changed into respectively to meet different purpose database datas
The data of call format, and can be written into different purpose databases.
The embodiments of the present invention are for illustration only, and the quality of embodiment is not represented.
Through the above description of the embodiments, those skilled in the art can be understood that above-described embodiment side
Method can add the mode of required general hardware platform to realize by software, naturally it is also possible to by hardware, but in many cases
The former is more preferably embodiment.Understood based on such, technical scheme is substantially done to prior art in other words
Going out the part of contribution can be embodied in the form of software product, and the computer software product is stored in a storage medium
In (such as ROM/RAM, magnetic disc, CD), including some instructions are make it that a station terminal equipment (can be mobile phone, computer, clothes
It is engaged in device, air-conditioner, or network equipment etc.) perform method described in each embodiment of the invention.
The preferred embodiments of the present invention are these are only, are not intended to limit the scope of the invention, it is every to utilize this hair
Equivalent structure or equivalent flow conversion that bright specification and accompanying drawing content are made, or directly or indirectly it is used in other related skills
Art field, is included within the scope of the present invention.
Claims (10)
1. a kind of data conversion system, it is characterised in that the system includes：
Request receiving module, for receiving data conversion request, the data conversion request includes the type sum of source database
According to form, data to be transformed, the type and data form of purpose database；
Instrument determining module, determines to read data from the source database for the type according to the source database and data form
Reading instrument, determined to write writing for data in the purpose database according to the type of the purpose database and data form
Enter instrument；
Data acquisition module, for reading the data to be transformed from the source database using the reading instrument；
Data conversion module, for the data to be transformed to be converted into number corresponding with the data form of the purpose database
According to；
Data write. module, for obtaining the data after conversion, and is write the data after the conversion using said write instrument
Enter the purpose database.
2. data conversion system as claimed in claim 1, it is characterised in that the data conversion module, specifically for：
The data to be transformed that are read are received from the reading instrument, and the data to be transformed are converted into the word of preset format
Symbol；
The character of the preset format is parsed, the character of the preset format is converted into the data form of the purpose database
Data.
3. data conversion system as claimed in claim 2, it is characterised in that the purpose database include it is multiple, wherein：
The instrument determining module, is additionally operable to be determined and each mesh according to the type and data form of each purpose database
The corresponding write-in instrument of database；
The data conversion module, is additionally operable to being separately converted to the data to be transformed into the data with each purpose database
The corresponding data of form；
Data write. module, for obtaining the data after the conversion corresponding with each purpose database respectively, and using with
The corresponding write-in instrument of each purpose database, after being respectively written into corresponding conversion in each purpose database
Data.
4. the data conversion system as described in claim any one of 2-3, it is characterised in that the character of the preset format includes
The character of Json forms.
5. data conversion system as claimed in claim 4, it is characterised in that the data conversion request includes data importing please
Ask, data export is asked and data synchronization request；The source database or the purpose database include MySQL,
Oracle、Hive、Hdfs。
6. a kind of data method for transformation, it is characterised in that methods described includes：
Receive data conversion request, data conversion request include the type and data form of source database, data to be transformed,
The type and data form of purpose database；
Determined to read the reading instrument of data from the source database according to the type of the source database and data form, according to institute
The type and data form for stating purpose database determine to write the write-in instrument of data in the purpose database；
The data to be transformed are read from the source database using the reading instrument；
The data to be transformed are converted into data corresponding with the data form of the purpose database；And
The data after conversion are obtained, and the data after the conversion are write into the purpose database using said write instrument.
7. data method for transformation as claimed in claim 6, it is characterised in that described to be converted into the data to be transformed and institute
The step of stating the data form corresponding data of purpose database, specifically includes：
The data to be transformed that are read are received from the reading instrument, and the data to be transformed are converted into the word of preset format
Symbol；And
The character of the preset format is parsed, the character of the preset format is converted into the data form of the purpose database
Data.
8. data method for transformation as claimed in claim 7, it is characterised in that when the purpose database includes multiple, institute
State and determined to write the write-in instrument of data in the purpose database according to the type and data form of the purpose database
Step, also includes：
Write-in instrument corresponding with each purpose database is determined according to the type of each purpose database and data form；
It is described the step of the data to be transformed are converted into the corresponding data with the data form of the purpose database, also wrap
Include：
The data to be transformed are separately converted to the data corresponding with the data form of each purpose database；
The data obtained after conversion, and the data after the conversion are write into the purpose data using said write instrument
The step of storehouse, also include：
The data after the conversion corresponding with each purpose database are obtained respectively, and are utilized and each purpose database phase
Corresponding write-in instrument, the data after being respectively written into corresponding conversion in each purpose database.
9. the data method for transformation as described in claim any one of 7-8, it is characterised in that the character of the preset format includes
The character of Json forms.
10. data method for transformation as claimed in claim 9, it is characterised in that the data conversion request includes data importing
Request, data export request and data synchronization request；The source database or the purpose database include MySQL,
Oracle、Hive、Hdfs。
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710175759.7A CN106951536A (en) | 2017-03-22 | 2017-03-22 | Data method for transformation and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710175759.7A CN106951536A (en) | 2017-03-22 | 2017-03-22 | Data method for transformation and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN106951536A true CN106951536A (en) | 2017-07-14 |
Family
ID=59472293
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710175759.7A Pending CN106951536A (en) | 2017-03-22 | 2017-03-22 | Data method for transformation and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106951536A (en) |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107862084A (en) * | 2017-11-28 | 2018-03-30 | 湖北三新文化传媒有限公司 | Data managing method, device and storage medium based on web services |
| CN108052681A (en) * | 2018-01-12 | 2018-05-18 | 毛彬 | The synchronous method and system of structural data between a kind of relevant database |
| CN108920659A (en) * | 2018-07-03 | 2018-11-30 | 广州唯品会信息科技有限公司 | Data processing system and its data processing method, computer readable storage medium |
| CN108933830A (en) * | 2018-07-16 | 2018-12-04 | 广东亿迅科技有限公司 | Data assemblage method and device based on nb-iot |
| CN109218661A (en) * | 2017-10-27 | 2019-01-15 | 广州恒强信息科技有限公司 | A kind of enterprise's Active Eyes |
| CN109344193A (en) * | 2018-10-24 | 2019-02-15 | 中国银行股份有限公司 | A kind of method and system converting structural data to unstructured data |
| CN109582720A (en) * | 2018-11-27 | 2019-04-05 | 泰康保险集团股份有限公司 | Date storage method, device and storage medium |
| CN110471970A (en) * | 2019-08-08 | 2019-11-19 | 紫光云技术有限公司 | A kind of regulation engine data forwarding method that can flexibly match |
| WO2021013057A1 (en) * | 2019-07-19 | 2021-01-28 | 深圳前海微众银行股份有限公司 | Data management method and apparatus, and device and computer-readable storage medium |
| CN112632176A (en) * | 2020-12-31 | 2021-04-09 | 中国农业银行股份有限公司 | Interaction method and device for supervising submission database |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101706809A (en) * | 2009-11-17 | 2010-05-12 | 北京灵图软件技术有限公司 | Method, device and system for processing multi-source map data |
| US20100218184A1 (en) * | 2009-02-24 | 2010-08-26 | Kabushiki Kaisha Toshiba | Information processing device, method, and computer program product |
| CN104462269A (en) * | 2014-11-24 | 2015-03-25 | 中国联合网络通信集团有限公司 | Isomerous database data exchange method and system |
-
2017
- 2017-03-22 CN CN201710175759.7A patent/CN106951536A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100218184A1 (en) * | 2009-02-24 | 2010-08-26 | Kabushiki Kaisha Toshiba | Information processing device, method, and computer program product |
| CN101706809A (en) * | 2009-11-17 | 2010-05-12 | 北京灵图软件技术有限公司 | Method, device and system for processing multi-source map data |
| CN104462269A (en) * | 2014-11-24 | 2015-03-25 | 中国联合网络通信集团有限公司 | Isomerous database data exchange method and system |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109218661A (en) * | 2017-10-27 | 2019-01-15 | 广州恒强信息科技有限公司 | A kind of enterprise's Active Eyes |
| CN107862084A (en) * | 2017-11-28 | 2018-03-30 | 湖北三新文化传媒有限公司 | Data managing method, device and storage medium based on web services |
| CN108052681A (en) * | 2018-01-12 | 2018-05-18 | 毛彬 | The synchronous method and system of structural data between a kind of relevant database |
| CN108920659A (en) * | 2018-07-03 | 2018-11-30 | 广州唯品会信息科技有限公司 | Data processing system and its data processing method, computer readable storage medium |
| CN108933830A (en) * | 2018-07-16 | 2018-12-04 | 广东亿迅科技有限公司 | Data assemblage method and device based on nb-iot |
| CN109344193A (en) * | 2018-10-24 | 2019-02-15 | 中国银行股份有限公司 | A kind of method and system converting structural data to unstructured data |
| CN109582720A (en) * | 2018-11-27 | 2019-04-05 | 泰康保险集团股份有限公司 | Date storage method, device and storage medium |
| CN109582720B (en) * | 2018-11-27 | 2021-12-07 | 泰康保险集团股份有限公司 | Data storage method, device and storage medium |
| WO2021013057A1 (en) * | 2019-07-19 | 2021-01-28 | 深圳前海微众银行股份有限公司 | Data management method and apparatus, and device and computer-readable storage medium |
| CN110471970A (en) * | 2019-08-08 | 2019-11-19 | 紫光云技术有限公司 | A kind of regulation engine data forwarding method that can flexibly match |
| CN112632176A (en) * | 2020-12-31 | 2021-04-09 | 中国农业银行股份有限公司 | Interaction method and device for supervising submission database |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106951536A (en) | Data method for transformation and system | |
| CN106980669B (en) | Data storage and acquisition method and device | |
| CN107967316A (en) | A kind of method of data synchronization, equipment and computer-readable recording medium | |
| CN108428104A (en) | Wages computational methods, application server and computer readable storage medium | |
| CN109766082A (en) | The method and apparatus that the application program page jumps | |
| CN110764798A (en) | Microcode upgrading method, device, computer equipment and storage medium | |
| US9049164B2 (en) | Dynamic message retrieval by subdividing a message queue into sub-queues | |
| CN110427182A (en) | A kind of template type construction APP method and device | |
| CN109697537A (en) | The method and apparatus of data audit | |
| CN113254445B (en) | Real-time data storage method, device, computer equipment and storage medium | |
| CN107015851A (en) | A kind of serviced component management method and system | |
| CN111552463A (en) | Page jump method and device, computer equipment and storage medium | |
| CN113010542A (en) | Service data processing method and device, computer equipment and storage medium | |
| CN113204425A (en) | Method and device for process management internal thread, electronic equipment and storage medium | |
| CN115712422A (en) | Form page generation method and device, computer equipment and storage medium | |
| US20210160143A1 (en) | Information technology (it) toplogy solutions according to operational goals | |
| CN107491700A (en) | A kind of data access method and equipment | |
| CN112308590A (en) | Parameter processing method and device, computing equipment and storage medium | |
| CN113157523B (en) | Service monitoring method and device, computer equipment and storage medium | |
| CN106888264A (en) | A kind of method for interchanging data and device | |
| CN106856441A (en) | VIM systems of selection and device in NFVO | |
| CN110765610B (en) | PDM integration method, PDM integration device, computer equipment and storage medium | |
| CN116450723A (en) | Data extraction method, device, computer equipment and storage medium | |
| US9135095B2 (en) | Methods and systems for inter-application communication | |
| US9098880B2 (en) | Method for optimizing power consumption in planned projects |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20170714 |
|
| RJ01 | Rejection of invention patent application after publication |