CN107798069A - Method, apparatus and computer-readable medium for data loading - Google Patents
Method, apparatus and computer-readable medium for data loading Download PDFInfo
- Publication number
- CN107798069A CN107798069A CN201710882919.1A CN201710882919A CN107798069A CN 107798069 A CN107798069 A CN 107798069A CN 201710882919 A CN201710882919 A CN 201710882919A CN 107798069 A CN107798069 A CN 107798069A
- Authority
- CN
- China
- Prior art keywords
- data
- mapping relations
- content
- instruction
- data pick
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
A kind of method, apparatus and computer-readable medium for data loading of disclosure.It is related to database information process field, this method includes：Mapping relations are generated by source database content and target database content；Create the table structure of the target database；Pass through the mapping relations, generation data pick-up instruction；Data pick-up instruction is deployed in data loading tool；And perform the data pick-up instruction and complete data loading.Method, apparatus and computer-readable medium disclosed in the present application for data loading, new demand can quickly be tackled, the different database of slitless connection and different ETL instruments, data warehouse core logic is allowed accurately can comprehensively to be expressed by design documentation.
Description
Technical field
The present invention relates to database information process field, in particular to a kind of method, apparatus for data loading
And computer-readable medium.
Background technology
Data load (ETL, Extract-Transform-Load), are for describing data from source terminal by extracting
(extract) (transform), the process of loading (load) to destination, are changed.Data are as extracted into warp from a place
The process that conversion is loaded into another place is crossed, ETL processes are to build the core content of data warehouse；ETL is realized in general
Process have two kinds of main modes；The first is write code wrap by way of coding, by developer and extracts logic, configuration
Scheduler task, extracted；Another kind is by external tool, is developed using ETL instruments, is related to conversion and workflow
Design create；Both the above mode all be unable to do without the processing of Mapping and Converting logic, wherein very important a part of content is
Source and the field mapping relations of target and the working process of field, either which kind of mode, are required for informing skill by business personnel
Art personnel's mapping relations, technical staff form document by design, then realize ETL processes by exploitation, of the prior art
This implementation causes the maintenance of mapping relations and realization can not form a unified entirety.
For traditional data warehouse project when tackling both above demand, generalized flowsheet is first to design to develop afterwards, i.e., by opening
Hair personnel confirm service logic with related side, design table structure, form design documentation；Transferred to again by the design documentation after confirmation
Developer, developer are shone document content and developed；Over time, the adjustment of demand may be directly in program
Upper realization, the effect of document diminish further so that design and development departs from.
Therefore, it is necessary to a kind of new method, apparatus and computer-readable medium that are used for data loading.
Above- mentioned information is only used for strengthening the understanding of the background to the present invention, therefore it disclosed in the background section
It can include not forming the information to prior art known to persons of ordinary skill in the art.
The content of the invention
In view of this, the present invention provides a kind of method, apparatus and computer-readable medium for data loading, can be fast
Speed tackles new demand, the different database of slitless connection and different ETL instruments, and data warehouse core logic can be led to
Design documentation is crossed accurately comprehensively to express.
Other characteristics and advantage of the present invention will be apparent from by following detailed description, or partially by the present invention
Practice and acquistion.
According to an aspect of the invention, it is proposed that a kind of method for data loading, this method includes：Pass through source database
Content and target database content generation mapping relations；Create the table structure of the target database；By the mapping relations,
Generate data pick-up instruction；Data pick-up instruction is deployed in data loading tool；And perform the data pick-up
Data loading is completed in instruction.
In a kind of exemplary embodiment of the disclosure, in addition to：The source database content is subjected to initial treatment, it is raw
Into predetermined data object describing mode.
It is described that the source database content is subjected to initial treatment in a kind of exemplary embodiment of the disclosure, including：
The source database content is subjected to initial treatment by changing dictionary.
It is described to pass through source database content and the target database content in a kind of exemplary embodiment of the disclosure
Mapping relations are generated, including：By the field definition of the field definition of table in source database and the table in the target database,
Generate mapping relations；And generate mapping relations document by the mapping relations.
In a kind of exemplary embodiment of the disclosure, the table structure for creating the target database, including：Pass through
Script creates the table structure of the target database.
It is described to be created by script described in the target database in a kind of exemplary embodiment of the disclosure
Table structure, including：Table structurally interrelated information in the target database is stored in VBA variables；Read the VBA variables
Content, generate the script；And the script is performed to generate the table structure in the target database.
It is described to pass through the mapping relations, generation data pick-up instruction, bag in a kind of exemplary embodiment of the disclosure
Include：Judge the type of the data loading tool；And the different type according to the data loading tool, generation is different respectively
Data pick-up instruction.
It is described by the mapping relations in a kind of exemplary embodiment of the disclosure, generation data pick-up instruction, go back
Including：Judge by the data pick-up mode of the source database to the target database；And according to different data pick-ups
Mode, different data pick-up instructions is generated respectively.
It is described by the mapping relations in a kind of exemplary embodiment of the disclosure, generation data pick-up instruction, go back
Including：Predetermined information is write in tentation data extraction instruction form, generates assignment template.
It is described by the mapping relations in a kind of exemplary embodiment of the disclosure, generation data pick-up instruction, go back
Including：Assignment information assignment will be treated to the assignment template, to generate the first formatted file；And pass through first form text
Part generates the data pick-up instruction.
According to an aspect of the invention, it is proposed that a kind of device for data loading, the device includes：Mapping block, use
In passing through source database content mapping relations are generated with target database content；Construction module, for creating the target data
The table structure in storehouse；Instruction module, for passing through the mapping relations, generation data pick-up instruction；Deployment module, for by described in
Data pick-up instruction is deployed in data loading tool；And execution module, complete number for performing the data pick-up instruction
According to loading.
In a kind of exemplary embodiment of the disclosure, in addition to：Processing module, for the source database content to be entered
Row initial treatment, generate predetermined data object describing mode.
According to an aspect of the invention, it is proposed that a kind of computer-readable medium, is stored thereon with computer program, its feature
It is, method as mentioned in the above is realized when program is executed by processor.
Method, apparatus and the computer-readable medium that data load are used for according to the present invention, can quickly be tackled new
Demand, the different database of slitless connection and different ETL instruments.
It should be appreciated that the general description and following detailed description of the above are only exemplary, this can not be limited
Invention.
Brief description of the drawings
Its example embodiment is described in detail by referring to accompanying drawing, above and other target of the invention, feature and advantage will
Become more fully apparent.Drawings discussed below is only some embodiments of the present invention, for the ordinary skill of this area
For personnel, on the premise of not paying creative work, other accompanying drawings can also be obtained according to these accompanying drawings.
Fig. 1 is a kind of flow chart of method for data loading according to an exemplary embodiment.
Fig. 2 is the schematic diagram of the mapping relations according to an exemplary embodiment.
Fig. 3 is the output data schematic diagram according to an exemplary embodiment.
Fig. 4 is the schematic diagram of the source table content according to an exemplary embodiment.
Fig. 5 is the schematic diagram of the conversion dictionary conversion according to an exemplary embodiment.
Fig. 6 is the schematic diagram of the object table content according to an exemplary embodiment.
Fig. 7 is a kind of system flow chart of method for data loading according to an exemplary embodiment.
Fig. 8 is a kind of schematic diagram of method for data loading according to another exemplary embodiment.
Fig. 9 is a kind of schematic diagram of method for data loading according to an exemplary embodiment.
Figure 10 is the data structure schematic diagram according to an exemplary embodiment.
Figure 11 is the data structure schematic diagram according to another exemplary embodiment.
Figure 12 is a kind of block diagram of device for data loading according to an exemplary embodiment.
Figure 13 is a kind of block diagram of device for data loading according to another exemplary embodiment.
Figure 14 is according to a kind of computer-readable medium schematic diagram shown in an exemplary embodiment.
Specific embodiment
Example embodiment is described more fully with referring now to accompanying drawing.However, example embodiment can be real in a variety of forms
Apply, and be not understood as limited to embodiment set forth herein；On the contrary, these embodiments are provided so that the present invention will be comprehensively and complete
It is whole, and the design of example embodiment is comprehensively communicated to those skilled in the art.Identical reference represents in figure
Same or similar part, thus repetition thereof will be omitted.
In addition, described feature, structure or characteristic can be incorporated in one or more implementations in any suitable manner
In example.In the following description, there is provided many details fully understand so as to provide to embodiments of the invention.However,
It will be appreciated by persons skilled in the art that technical scheme can be put into practice without one or more in specific detail,
Or other methods, constituent element, device, step etc. can be used.In other cases, side known in being not shown in detail or describe
Method, device, realization are operated to avoid fuzzy each aspect of the present invention.
Block diagram shown in accompanying drawing is only functional entity, not necessarily must be corresponding with physically separate entity.
I.e., it is possible to realize these functional entitys using software form, or realized in one or more hardware modules or integrated circuit
These functional entitys, or these functional entitys are realized in heterogeneous networks and/or processor device and/or microcontroller device.
Flow chart shown in accompanying drawing is merely illustrative, it is not necessary to including all contents and operation/step,
It is not required to perform by described order.For example, some operation/steps can also decompose, and some operation/steps can close
And or partly merging, therefore the order actually performed is possible to be changed according to actual conditions.
It should be understood that although herein various assemblies may be described using term first, second, third, etc., these groups
Part should not be limited by these terms.These terms are to distinguish a component and another component.Therefore, first group be discussed herein below
Part can be described as teaching of second component without departing from disclosure concept.As used herein, term " and/or " include it is associated
All combinations for listing any one and one or more in project.
It will be understood by those skilled in the art that accompanying drawing is the schematic diagram of example embodiment, module or flow in accompanying drawing
Necessary to not necessarily implementing the present invention, therefore it cannot be used for limiting the scope of the invention.
Disclosure example embodiment is described in detail below in conjunction with the accompanying drawings.
Fig. 1 is a kind of flow chart of method for data loading according to an exemplary embodiment.
As shown in figure 1, in S102, mapping relations are generated by source database content and target database content.Pass through
The field definition of the field definition of table and the table in the target database in source database, generates mapping relations；And pass through
The mapping relations generate mapping relations document.Can be for example, obtaining the field definition and mesh of data pending in source database
The field definition of object table in database is marked, analyzes above-mentioned field definition, forms source database content and target database content
Mapping relations, can be described in such as Fig. 2.Can also be for example, generating mapping relations document by above-mentioned mapping relations.
In S104, the table structure of the target database is created., it is necessary to first create target database during ETL
Corresponding table could be by ETL instruments toward inserting data in the table of target database.Can be for example, creating institute by script
State the table structure of target database.Including：Table structurally interrelated information in the target database is stored in VBA variables
In；The VBA variant contents are read, generate the script；And the script is held in the target database
Go to generate the table structure.Can be for example, being classified by the keyword of agreement to the format information of object table, according to classification
Content in information, is successively read object table content from top to bottom, the content of reading include target table name, field, index, point
The particular content in area etc..Can also be for example, the form of data format corresponding to multiple target databases set in advance and agreement turns
Rule is changed, for example for oracle database, it requires that field type is Number numeric types；For example safeguarded for index type
IDX then will be considered that it is simple index etc..And then the content read is combined to the content that is converted to database and can identify,
The conversion of conversion, index type including field type, conversion of subregion classifying type etc..Can also be for example, object table by reading
Content carries out the tissue of different structure, forms the script that can be identified by target database.
Can also be for example, obtaining the format information and its related content of object table；The format information of the object table is divided into four
Block, can such as target table name, field contents, index content, partition content, these contents are first stored in VBA variables temporarily
In；Content can be changed while reading, the rule of conversion is that form making an appointment or being stored in VBA modules turns
Definition is changed, the conversion of predominantly field type is changed in Fig. 3 transfers, and VC can be converted to Varchar2, and N is converted to Number, D conversions
For Date etc.；After conversion, it can judge that the factor of influence table structure SQL statement is deposited and be not present, the factor influenceed in upper figure is subregion
Whether content fills out；Judge that partition content is not safeguarded, then can enter the content that back transfer is finished changing according to the structure of routine
Row tissue；Form the script that can be identified by target database.
In S106, pass through the mapping relations, generation data pick-up instruction.In this application, data pick-up instruction can
For example,：Core ETL codes.Including：Judge the type of the data loading tool；And according to the data loading tool
Different type, different data pick-up instructions is generated respectively.Also include：Judge by the source database to the target database
Data pick-up mode；And according to different data pick-up modes, different data pick-up instructions is generated respectively.Also include：
Predetermined information is write in tentation data extraction instruction form, generates assignment template.Assignment information assignment will be treated to the assignment
Template, to generate the first formatted file；And the data pick-up is generated by first formatted file and instructed.
In S108, data pick-up instruction is deployed in data loading tool.Data pick-up instructs (core ETL
Code) performed in ETL instruments, data pick-up instruction is deployed in data loading tool.
In S110, perform the data pick-up instruction and complete data loading.ETL instruments read the core ETL generations of generation
Code come generate mapping or workflow, so as to complete most ETL developments.
According to the method for being used for data and loading of the present invention, by the way that data pick-up instruction is deployed in ETL instruments, aid in
The mode of data pick-up, new demand, the different database of slitless connection and different ETL instruments can be quickly tackled, is allowed
Data warehouse core logic accurately can comprehensively be expressed by design documentation.
It will be clearly understood that the present disclosure describe how formation and using particular example, but the principle of the present invention is not limited to
Any details of these examples.On the contrary, the teaching based on present disclosure, these principles can be applied to many other
Embodiment.
In a kind of exemplary embodiment of the disclosure, in addition to：The source database content is subjected to initial treatment, it is raw
Into predetermined data object describing mode.Can for example including：The source database content is subjected to initial place by changing dictionary
Reason.Can for example, delineation needs the source table content that gathers, by source table content according to certain form maintenance on document, Mei Geyuan
Table, which needs to gather, extracts key message as shown in Figure 4.
Can also be for example, carrying out the data acquisition of source database as follows：According to source database configuration information and source
Database is attached；In connection after source database, the table structure content of delineation source table is read from source database, in database
System table storage table structure content；After source table table structure is read, content is written to the position that document specifies, is divided into table
Name, field contents, index content etc..
Fig. 5 is the schematic diagram of the conversion dictionary conversion according to an exemplary embodiment.As shown in figure 5, can also for example,
The data object of not genbank database is uniformly converted to the data object describing mode of standard, turned by self-defining data object
Dictionary is changed to realize.Wherein, object table can be for example as shown in Figure 6.
Fig. 7 is a kind of system flow chart of method for data loading according to another exemplary embodiment.
As shown in fig. 7, the source table content for needing to gather by the delineation of source database 010, by source table content according to certain
Form maintenance generates source table structure 011 on document, and each source table, which needs to gather, extracts key message as shown in Figure 4.Pass through
Self-defining data object conversion dictionary 021 is uniformly converted to the source table structure 011 of not genbank database the object table of standard
Structure 012.The field definition of object table structure 012 and the field definition of source table structure 011 in target database are analyzed, forms source
The mapping relations of table and object table, and the mapping relations between source table and target literary name section are safeguarded in a document, generation mapping
Relational document 022.
Fig. 8 is a kind of schematic diagram of method for data loading according to another exemplary embodiment.Fig. 8 is to figure
Exemplary description has been carried out by VBA023 structure database scripts 013 in 7.
Can be for example including following processing procedure：
(1) format information and its related content of object table are obtained.
(2) format information of object table is classified by the keyword of agreement, extracts content-location information.
(3) content-location information is relied on, is successively read object table content from top to bottom, the content of reading includes target table name
Title, field, index, the particular content etc. of subregion.
(4) according to the form transformation rule for prestoring data format corresponding to multiple target databases and agreement, such as
For oracle database, it requires that field type is Number numeric types；For example maintain IDX for index type and can then recognize
It is simple index etc. for it.According to the content of format storage unit, the content read combination is converted into database to know
Other content, including the conversion of field type, the conversion of index type, conversion of subregion classifying type etc..
(5) judge that the factor that can influence table structure SQL statement whether there is.According to judged result, by the object table of reading
Content carries out the tissue of different structure, forms the script that can be identified by target database.Will after structure is organized
Content exports line by line, that is, is written to text.
The object table of generation is built into table script (database script 013) to perform in target database 020, so as in number of targets
According to generation object table structure in storehouse 020.
The good mapping relations of above design maintenance are generated into ETL core codes 014 by writing the VBA024 of particular module.
Fig. 9 is a kind of schematic diagram of method for data loading according to an exemplary embodiment.Can be for example, logical
Cross step generation ETL core codes 014 as shown in Figure 9：
(1) content of maintenance is classified by keyword, is broadly divided into source table table name, object table table name, extraction side
Formula, mapping content etc., by the positional information interim storage of these contents in a document, positional information is the same as the position in recording documents
Information.
(2) content-location information of previous step is relied on, reads document content corresponding to these positional informations, predominantly source table
Table name, target table name and field mapping relations between the two, these contents are temporarily stored in VBA variables.
(3) judge that data are drawn into the mode of object table, such as the difference such as date increment, full dose, conversion increment from source table
Extraction mode；Judge the type of target ETL instruments, storage and identification of the different ETL instruments to ETL contents are different, are formed
ETL core code forms it is also different, judgement herein provides precondition for the calling of form below.
(4) setting carried out for different ETL instruments or mode and to form, can be for example, pre-define target ETL
The form that instrument can identify, while define which crucial ETL information needs to carry out assignment；For different ETL instruments, in advance
First format unit conventional defined in VBA modules, is called according to the result of judgement.
For example form is<Source name=" "></source>, this is the expression source table table name that certain ETL instrument is fixed
Form, wherein behind name be need carry out assignment place, that is to say, that body matter has been formed, it is only necessary to next
Imparting will be worth by walking assignment unit, you can form the core ETL codes 014 identified by target ETL instruments.
(5) can be for example, core ETL information be written in specific ETL forms, such as the source table table name that previous step is mentioned
Form, then the source table table name XXX assignment that will be read from document is needed to be formed to behind above-mentioned name<source name
=" source table table name "></source>Such content, so as to inform ETL instruments, there are Zhang Yuan's table, the entitled XXX of source table；
Form is similar to a template, and template definition has got well the overall structure of ETL codes, but the value of the inside key content is waited to fill out；
Assignment is the process that the key content that will be read from document is filled into template.
(6) by key content assignment, into ETL forms, rear content is output to the text of specific format, output file
Path, title etc..
According to the present invention be used for data load method, by by design data model, design ETL process with document
For carrier, based on the document format content of specification, by writing specific VBA modules, realization can be directed to different target number
Table structure is generated according to storehouse and the function of ETL core codes is generated for different ETL instruments.
Its form that can identify of different ETL instruments is different, target ETL instrument classifications is first judged, then according to difference
ETL instruments call different format unit.Figure 10,11 be the data structure schematic diagram according to an exemplary embodiment.
The ETL forms that two kinds of ETL instruments can identify are illustrated, by taking the definition of object table structure as an example, ETL instruments lead to corresponding to Figure 10
XML format is crossed to define, i.e., the position of mark 2, code are<Target name=" object tables "><Targetfield name
=" aiming field "></Target>, wherein object table and aiming field is the content read from document, and other are fixation
Form, the form be inform the ETL tool target tables title and comprising field contents.ETL instruments lead to corresponding to Figure 11
The record of table structure is crossed to represent, be i.e. object definition in figure, key message is inserted into table corresponding to the ETL instruments, the SQL
Sentence is equally fixed form, and VBA assignment unit can be carried out wherein crucial field value with the document content read
Replace, so as to inform the definition of ETL tool target table structures；After by core ETL information assignment into form, it can generate such as
The lower two kinds core ETL codes for being directed to different ETL instruments.
It will be appreciated by those skilled in the art that realize that all or part of step of above-described embodiment is implemented as being performed by CPU
Computer program.When the computer program is performed by CPU, the above-mentioned work(that the above method provided by the invention is limited is performed
Energy.Described program can be stored in a kind of computer-readable recording medium, and the storage medium can be read-only storage, magnetic
Disk or CD etc..
Further, it should be noted that above-mentioned accompanying drawing is only the place included by method according to an exemplary embodiment of the present invention
Reason schematically illustrates, rather than limitation purpose.It can be readily appreciated that above-mentioned processing shown in the drawings is not intended that or limited at these
The time sequencing of reason.In addition, being also easy to understand, these processing for example can be performed either synchronously or asynchronously in multiple modules.
Following is apparatus of the present invention embodiment, can be used for performing the inventive method embodiment.It is real for apparatus of the present invention
The details not disclosed in example is applied, refer to the inventive method embodiment.
Figure 12 is a kind of block diagram of device for data loading according to an exemplary embodiment.
Mapping block 1202 is used to generate mapping relations by source database content and target database content.
Construction module 1204 is used for the table structure for creating the target database.
Instruction module 1206 is used for by the mapping relations, generation data pick-up instruction.
Deployment module 1208 is used to data pick-up instruction being deployed in data loading tool.
Execution module 1210 is used to perform the data pick-up instruction completion data loading.
In a kind of exemplary embodiment of the disclosure, in addition to：Processing module (not shown) is used for the source
Data-base content carries out initial treatment, generates predetermined data object describing mode.
According to the device for being used for data and loading of the present invention, by the way that data pick-up instruction is deployed in ETL instruments, aid in
The mode of data pick-up, new demand, the different database of slitless connection and different ETL instruments can be quickly tackled, is allowed
Data warehouse core logic accurately can comprehensively be expressed by design documentation.
Figure 13 is a kind of block diagram of device for data loading according to another exemplary embodiment.Shown in Figure 13
The step of generate ETL core codes process.
Taxon 1302 is classified the content of maintenance by keyword.
The content-location information that reading unit 1304 obtains by previous step taxon 1302, reads these positional informations
Corresponding document content, these contents are temporarily stored in VBA variables.
Judging unit 1306 is used to judge that data are drawn into the mode of object table from source table.
Format unit 1308 is used for the setting carried out for different ETL instruments or mode to form.
Core ETL information is written in specific ETL forms by assignment unit 1310, and assignment is will to be read from document
Key content be filled into process in template.
By previous step, i.e. by key content assignment, into ETL forms, rear content is output to specific format to output unit 1312
Text.
Figure 14 is according to a kind of computer-readable medium schematic diagram shown in an exemplary embodiment.
With reference to shown in figure 14, the program product for being used to realize the above method according to the embodiment of the present invention is described
400, it can use portable compact disc read only memory (CD-ROM) and including program code, and can in terminal device,
Such as run on PC.However, the program product not limited to this of the present invention, in this document, readable storage medium storing program for executing can be with
Be it is any include or the tangible medium of storage program, the program can be commanded execution system, device either device use or
It is in connection.
Described program product can use any combination of one or more computer-readable recording mediums.Computer-readable recording medium can be readable letter
Number medium or readable storage medium storing program for executing.Readable storage medium storing program for executing for example can be but be not limited to electricity, magnetic, optical, electromagnetic, infrared ray or
System, device or the device of semiconductor, or any combination above.The more specifically example of readable storage medium storing program for executing is (non exhaustive
List) include：It is electrical connection, portable disc, hard disk, random access memory (RAM) with one or more wires, read-only
Memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, portable compact disc read only memory
(CD-ROM), light storage device, magnetic memory device or above-mentioned any appropriate combination.
The computer-readable recording medium can include believing in a base band or as the data that a carrier wave part is propagated
Number, wherein carrying readable program code.The data-signal of this propagation can take various forms, including but not limited to electromagnetism
Signal, optical signal or above-mentioned any appropriate combination.Readable storage medium storing program for executing can also be any beyond readable storage medium storing program for executing
Computer-readable recording medium, the computer-readable recording medium can send, propagate either transmit for being used by instruction execution system, device or device or
Person's program in connection.The program code included on readable storage medium storing program for executing can be transmitted with any appropriate medium, bag
Include but be not limited to wireless, wired, optical cable, RF etc., or above-mentioned any appropriate combination.
Can being combined to write the program operated for performing the present invention with one or more programming languages
Code, described program design language include object oriented program language-Java, C++ etc., include routine
Procedural programming language-such as " C " language or similar programming language.Program code can be fully in user
Perform on computing device, partly perform on a user device, the software kit independent as one performs, is partly calculated in user
Its upper side point is performed or performed completely in remote computing device or server on a remote computing.It is remote being related to
In the situation of journey computing device, remote computing device can pass through the network of any kind, including LAN (LAN) or wide area network
(WAN) user calculating equipment, is connected to, or, it may be connected to external computing device (such as utilize ISP
To pass through Internet connection).
Above computer computer-readable recording medium carries one or more program, when said one or multiple programs are by one
When the equipment performs so that the computer-readable medium implements function such as：By in source database content and target database
Hold generation mapping relations；Create the table structure of the target database；Pass through the mapping relations, generation data pick-up instruction；
Data pick-up instruction is deployed in data loading tool；And perform the data pick-up instruction and complete data loading.
It will be appreciated by those skilled in the art that above-mentioned each module can be distributed in device according to the description of embodiment, also may be used
To carry out respective change uniquely different from one or more devices of the present embodiment.The module of above-described embodiment can be merged into
One module, can also be further split into multiple submodule.
The description of embodiment more than, those skilled in the art is it can be readily appreciated that example embodiment described herein
It can be realized, can also be realized by way of software combines necessary hardware by software.Therefore, implemented according to the present invention
The technical scheme of example can be embodied in the form of software product, and the software product can be stored in a non-volatile memories
In medium (can be CD-ROM, USB flash disk, mobile hard disk etc.) or on network, including some instructions are make it that a computing device (can
To be personal computer, server, mobile terminal or network equipment etc.) perform method according to embodiments of the present invention.
Detailed description more than, those skilled in the art is it can be readily appreciated that according to embodiments of the present invention is used to count
Had one or more of the following advantages according to the method, apparatus and computer-readable medium of loading.
According to some embodiments, the method for being used for data loading of the invention, by the way that data pick-up instruction is deployed into ETL
In instrument, mode that assistance data extracts can quickly tackle new demand, the different database of slitless connection and different
ETL instruments, data warehouse core logic is allowed accurately can comprehensively to be expressed by design documentation.
According to other embodiments, the of the invention method for being used for data loading, by by design data model, design
ETL process is using document as carrier, and based on the document format content of specification, by writing specific VBA modules, realization can
Table structure is generated for different target database and the function of ETL core codes is generated for different ETL instruments.
The exemplary embodiment of the present invention is particularly shown and described above.It should be appreciated that the invention is not restricted to
Detailed construction, set-up mode or implementation method described herein；On the contrary, it is intended to cover included in appended claims
Various modifications and equivalence setting in spirit and scope.
In addition, structure, ratio, size shown by this specification Figure of description etc., only coordinating specification institute
Disclosure, for skilled in the art realises that with reading, being not limited to the enforceable qualifications of the disclosure, therefore
Do not have technical essential meaning, the modification of any structure, the change of proportionate relationship or the adjustment of size, do not influenceing the disclosure
Under the technique effect that can be generated and achieved purpose, it all should still fall and obtain and can cover in the technology contents disclosed in the disclosure
In the range of.Meanwhile in this specification it is cited such as " on ", " first ", the term of " second " and " one ", be also only and be easy to
Narration understands, and is not used to limit the enforceable scope of the disclosure, and its relativeness is altered or modified, without substantive change
Under technology contents, when being also considered as the enforceable category of the present invention.
Claims (13)
- A kind of 1. method for data loading, it is characterised in that including：Mapping relations are generated by source database content and target database content；Create the table structure of the target database；Pass through the mapping relations, generation data pick-up instruction；Data pick-up instruction is deployed in data loading tool；AndPerform the data pick-up instruction and complete data loading.
- 2. the method as described in claim 1, it is characterised in that also include：The source database content is subjected to initial treatment, generates predetermined data object describing mode.
- 3. the method as described in claim 1, it is characterised in that described that the source database content is subjected to initial treatment, bag Include：The source database content is subjected to initial treatment by changing dictionary.
- 4. the method as described in claim 1, it is characterised in that described by source database content and the target database Hold generation mapping relations, including：By the field definition of the field definition and the table in the target database of table in source database, mapping relations are generated； AndMapping relations document is generated by the mapping relations.
- 5. the method as described in claim 1, it is characterised in that the table structure for creating the target database, including：The table structure of the target database is created by script.
- 6. method as claimed in claim 5, it is characterised in that the institute that the target database is created by script Table structure is stated, including：Table structurally interrelated information in the target database is stored in VBA variables；The VBA variant contents are read, generate the script；AndThe script is performed to generate the table structure in the target database.
- 7. the method as described in claim 1, it is characterised in that it is described to be instructed by the mapping relations, generation data pick-up, Including：Judge the type of the data loading tool；AndAccording to the different type of the data loading tool, different data pick-up instructions is generated respectively.
- 8. the method as described in claim 1, it is characterised in that it is described to be instructed by the mapping relations, generation data pick-up, Also include：Judge by the data pick-up mode of the source database to the target database；AndAccording to different data pick-up modes, different data pick-up instructions is generated respectively.
- 9. the method as described in claim 1, it is characterised in that it is described to be instructed by the mapping relations, generation data pick-up, Also include：Predetermined information is write in tentation data extraction instruction form, generates assignment template.
- 10. method as claimed in claim 9, it is characterised in that described to be referred to by the mapping relations, generation data pick-up Order, in addition to：Assignment information assignment will be treated to the assignment template, to generate the first formatted file；AndThe data pick-up is generated by first formatted file to instruct.
- A kind of 11. device for data loading, it is characterised in that including：Mapping block, for generating mapping relations by source database content and target database content；Construction module, for creating the table structure of the target database；Instruction module, for passing through the mapping relations, generation data pick-up instruction；Deployment module, for data pick-up instruction to be deployed in data loading tool；And execution module, for performing Data loading is completed in the data pick-up instruction.
- 12. device as claimed in claim 11, it is characterised in that also include：Processing module, for the source database content to be carried out into initial treatment, generate predetermined data object describing mode.
- 13. a kind of computer-readable medium, is stored thereon with computer program, it is characterised in that described program is held by processor The method as described in any in claim 1-10 is realized during row.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710882919.1A CN107798069A (en) | 2017-09-26 | 2017-09-26 | Method, apparatus and computer-readable medium for data loading |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710882919.1A CN107798069A (en) | 2017-09-26 | 2017-09-26 | Method, apparatus and computer-readable medium for data loading |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107798069A true CN107798069A (en) | 2018-03-13 |
Family
ID=61531577
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710882919.1A Pending CN107798069A (en) | 2017-09-26 | 2017-09-26 | Method, apparatus and computer-readable medium for data loading |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107798069A (en) |
Cited By (19)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109271431A (en) * | 2018-08-14 | 2019-01-25 | 中国平安人寿保险股份有限公司 | Data pick-up method, apparatus, computer equipment and storage medium |
| CN109508355A (en) * | 2018-10-19 | 2019-03-22 | 平安科技（深圳）有限公司 | A kind of data pick-up method, system and terminal device |
| CN109992621A (en) * | 2019-04-11 | 2019-07-09 | 郭承湘 | Foods supervision information resources increment ETL system and method |
| CN110442634A (en) * | 2019-08-15 | 2019-11-12 | 中国银行股份有限公司 | The method and device of data storage |
| CN110516081A (en) * | 2019-09-02 | 2019-11-29 | 北京明略软件系统有限公司 | The display methods and device of tables of data mapping relations |
| CN110515995A (en) * | 2019-08-22 | 2019-11-29 | 深圳前海环融联易信息科技服务有限公司 | Quickly generate the ETL operational method and device of big data platform |
| CN110825735A (en) * | 2019-10-10 | 2020-02-21 | 中国平安财产保险股份有限公司 | Data table processing method and related equipment |
| CN111177247A (en) * | 2019-12-30 | 2020-05-19 | 腾讯科技（深圳）有限公司 | Data conversion method, device and storage medium |
| CN111324647A (en) * | 2020-01-21 | 2020-06-23 | 北京东方金信科技有限公司 | Method and device for generating ETL code |
| CN111459943A (en) * | 2020-04-03 | 2020-07-28 | 中国建设银行股份有限公司 | Data processing method, device, system, equipment and storage medium |
| CN111712809A (en) * | 2018-04-16 | 2020-09-25 | 甲骨文国际公司 | Learning ETL rules by example |
| CN112000643A (en) * | 2020-08-07 | 2020-11-27 | 杭州数梦工场科技有限公司 | Method and device for processing data model |
| CN112346780A (en) * | 2020-11-05 | 2021-02-09 | 海光信息技术股份有限公司 | Information processing method, device and storage medium |
| CN112699131A (en) * | 2021-01-18 | 2021-04-23 | 中国电子系统技术有限公司 | Mapping connection interaction method and device |
| CN112783900A (en) * | 2021-02-25 | 2021-05-11 | 百果园技术(新加坡)有限公司 | Database management method, device, equipment and storage medium |
| CN113535844A (en) * | 2021-09-15 | 2021-10-22 | 山东耕元数据科技有限公司 | Data aggregation method and system |
| CN113760891A (en) * | 2020-12-02 | 2021-12-07 | 北京沃东天骏信息技术有限公司 | Data table generation method, device, equipment and storage medium |
| US11238016B1 (en) * | 2020-08-24 | 2022-02-01 | Cigna Intellectual Property, Inc. | Systems and methods for automated data integration |
| CN115408473A (en) * | 2022-11-01 | 2022-11-29 | 神州数码融信云技术服务有限公司 | Data synchronization method and device, computer equipment and computer readable storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120271865A1 (en) * | 2007-01-09 | 2012-10-25 | International Business Machines Corporation | Method and apparatus for modelling data exchange in a data flow of an extract, transform, and load (etl) process |
| CN105069142A (en) * | 2015-08-18 | 2015-11-18 | 山大地纬软件股份有限公司 | System and method for extraction, transformation and distribution of data increments |
| CN106126753A (en) * | 2016-08-23 | 2016-11-16 | 易联众信息技术股份有限公司 | The method of increment extractions based on big data |
| CN107038177A (en) * | 2016-02-03 | 2017-08-11 | 维布络有限公司 | The method and apparatus for automatically generating extraction-conversion-loading code |
-
2017
- 2017-09-26 CN CN201710882919.1A patent/CN107798069A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120271865A1 (en) * | 2007-01-09 | 2012-10-25 | International Business Machines Corporation | Method and apparatus for modelling data exchange in a data flow of an extract, transform, and load (etl) process |
| CN105069142A (en) * | 2015-08-18 | 2015-11-18 | 山大地纬软件股份有限公司 | System and method for extraction, transformation and distribution of data increments |
| CN107038177A (en) * | 2016-02-03 | 2017-08-11 | 维布络有限公司 | The method and apparatus for automatically generating extraction-conversion-loading code |
| CN106126753A (en) * | 2016-08-23 | 2016-11-16 | 易联众信息技术股份有限公司 | The method of increment extractions based on big data |
Non-Patent Citations (1)
| Title |
|---|
| 刘如九 等: "一种通用的多数据库间数据抽取方法及应用", 《北京交通大学学报》 * |
Cited By (26)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111712809A (en) * | 2018-04-16 | 2020-09-25 | 甲骨文国际公司 | Learning ETL rules by example |
| CN109271431B (en) * | 2018-08-14 | 2023-11-24 | 中国平安人寿保险股份有限公司 | Data extraction method, device, computer equipment and storage medium |
| CN109271431A (en) * | 2018-08-14 | 2019-01-25 | 中国平安人寿保险股份有限公司 | Data pick-up method, apparatus, computer equipment and storage medium |
| CN109508355A (en) * | 2018-10-19 | 2019-03-22 | 平安科技（深圳）有限公司 | A kind of data pick-up method, system and terminal device |
| CN109992621A (en) * | 2019-04-11 | 2019-07-09 | 郭承湘 | Foods supervision information resources increment ETL system and method |
| CN110442634A (en) * | 2019-08-15 | 2019-11-12 | 中国银行股份有限公司 | The method and device of data storage |
| CN110442634B (en) * | 2019-08-15 | 2022-05-27 | 中国银行股份有限公司 | Data storage method and device |
| CN110515995A (en) * | 2019-08-22 | 2019-11-29 | 深圳前海环融联易信息科技服务有限公司 | Quickly generate the ETL operational method and device of big data platform |
| CN110516081A (en) * | 2019-09-02 | 2019-11-29 | 北京明略软件系统有限公司 | The display methods and device of tables of data mapping relations |
| CN110825735A (en) * | 2019-10-10 | 2020-02-21 | 中国平安财产保险股份有限公司 | Data table processing method and related equipment |
| CN110825735B (en) * | 2019-10-10 | 2023-08-22 | 中国平安财产保险股份有限公司 | Data table processing method and related equipment |
| CN111177247A (en) * | 2019-12-30 | 2020-05-19 | 腾讯科技（深圳）有限公司 | Data conversion method, device and storage medium |
| CN111177247B (en) * | 2019-12-30 | 2023-10-20 | 腾讯科技（深圳）有限公司 | Data conversion method, device and storage medium |
| CN111324647A (en) * | 2020-01-21 | 2020-06-23 | 北京东方金信科技有限公司 | Method and device for generating ETL code |
| CN111459943A (en) * | 2020-04-03 | 2020-07-28 | 中国建设银行股份有限公司 | Data processing method, device, system, equipment and storage medium |
| CN112000643A (en) * | 2020-08-07 | 2020-11-27 | 杭州数梦工场科技有限公司 | Method and device for processing data model |
| US11238016B1 (en) * | 2020-08-24 | 2022-02-01 | Cigna Intellectual Property, Inc. | Systems and methods for automated data integration |
| CN112346780A (en) * | 2020-11-05 | 2021-02-09 | 海光信息技术股份有限公司 | Information processing method, device and storage medium |
| CN112346780B (en) * | 2020-11-05 | 2022-11-15 | 海光信息技术股份有限公司 | Information processing method, device and storage medium |
| CN113760891A (en) * | 2020-12-02 | 2021-12-07 | 北京沃东天骏信息技术有限公司 | Data table generation method, device, equipment and storage medium |
| CN112699131B (en) * | 2021-01-18 | 2021-11-30 | 中国电子系统技术有限公司 | Mapping connection interaction method and device |
| CN112699131A (en) * | 2021-01-18 | 2021-04-23 | 中国电子系统技术有限公司 | Mapping connection interaction method and device |
| CN112783900A (en) * | 2021-02-25 | 2021-05-11 | 百果园技术(新加坡)有限公司 | Database management method, device, equipment and storage medium |
| CN113535844B (en) * | 2021-09-15 | 2021-12-07 | 山东耕元数据科技有限公司 | Data aggregation method and system |
| CN113535844A (en) * | 2021-09-15 | 2021-10-22 | 山东耕元数据科技有限公司 | Data aggregation method and system |
| CN115408473A (en) * | 2022-11-01 | 2022-11-29 | 神州数码融信云技术服务有限公司 | Data synchronization method and device, computer equipment and computer readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107798069A (en) | Method, apparatus and computer-readable medium for data loading | |
| CN111339774B (en) | Text entity relation extraction method and model training method | |
| CN111026842B (en) | Natural language processing method, natural language processing device and intelligent question-answering system | |
| CN105051721B (en) | Natural language description is converted to because of the program of the language different and different because of domain for electrical form | |
| CN110110041A (en) | Wrong word correcting method, device, computer installation and storage medium | |
| CN103930868B (en) | Customization optimization/conversion/process tactful system and method | |
| CN110377913B (en) | Emotion analysis method and device, electronic equipment and storage medium | |
| CN108389606A (en) | A kind of the data quality control system and its control method of electronic medical record homepage | |
| CN108694214A (en) | Generation method, generating means, readable medium and the electronic equipment of data sheet | |
| CN104050223A (en) | Pivot facets for text mining and search | |
| EP3846034B1 (en) | Systems and methods for automated testing using artificial intelligence techniques | |
| CN110083623A (en) | A kind of business rule generation method and device | |
| CN109815486A (en) | Spatial term method, apparatus, equipment and readable storage medium storing program for executing | |
| WO2024022354A1 (en) | Object recommendation method and apparatus for implementing ia in view of rpa and ai, and storage medium | |
| CN109657251A (en) | Method and apparatus for translating sentence | |
| CN109271403A (en) | A kind of operating method of data query, device, medium and electronic equipment | |
| CN108255837A (en) | A kind of SQL resolvers and method | |
| CN108829884A (en) | data mapping method and device | |
| CN102023859B (en) | Digital development environment-oriented software integration method with reliability, maintainability and supportability | |
| CN110262784A (en) | A kind of cloud notes implementation method and device | |
| CN102402505A (en) | Delta language translation method and system | |
| CN109298954A (en) | A kind of intelligence calls the method and device of Dubbo remote service | |
| CN109241104A (en) | The resolver and its implementation of AISQL in decision type distributed data base system | |
| CN107679141A (en) | Data storage method, device, equipment and computer-readable recording medium | |
| CN110245349A (en) | A kind of syntax dependency parsing method, apparatus and a kind of electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180313 |
|
| RJ01 | Rejection of invention patent application after publication |