CN110209662A - A kind of method and apparatus of automation load data - Google Patents
A kind of method and apparatus of automation load data Download PDFInfo
- Publication number
- CN110209662A CN110209662A CN201810149117.4A CN201810149117A CN110209662A CN 110209662 A CN110209662 A CN 110209662A CN 201810149117 A CN201810149117 A CN 201810149117A CN 110209662 A CN110209662 A CN 110209662A
- Authority
- CN
- China
- Prior art keywords
- data
- object table
- interim
- database
- indication field
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 44
- 238000005192 partition Methods 0.000 claims abstract description 29
- 230000008676 import Effects 0.000 claims abstract description 26
- 238000004590 computer program Methods 0.000 claims description 6
- 238000005516 engineering process Methods 0.000 abstract description 3
- 230000008569 process Effects 0.000 description 10
- 238000010586 diagram Methods 0.000 description 9
- 230000006870 function Effects 0.000 description 9
- 238000012545 processing Methods 0.000 description 9
- 230000006854 communication Effects 0.000 description 6
- 238000004891 communication Methods 0.000 description 5
- 238000004364 calculation method Methods 0.000 description 4
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000005856 abnormality Effects 0.000 description 2
- 230000005291 magnetic effect Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 241001269238 Data Species 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000005611 electricity Effects 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 210000003127 knee Anatomy 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 241000894007 species Species 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses the method and apparatus of automation load data, are related to field of computer technology.One specific embodiment of this method includes: acquisition data source, to import the interim table in database；The interim table is read, if there is partition data to be treated in preset time, then reads preset indication field in partition data corresponding data source；It, will be in the data write-in object table in the interim table according to the indication field.The embodiment is able to solve load data age in the prior art and not can guarantee, the problem of data accuracy and safety difference.
Description
Technical field
The present invention relates to field of computer technology more particularly to a kind of method and apparatus of automation load data.
Background technique
Currently, there are many business datums that can not directly take from system in data on flows parsing, need to handle manually
Manually it is added in file system again afterwards.Specifically, it at hive (is based on Hadoop that new data on flows, which needs Manual arranging,
The data file of structuring can be mapped as a database table by one Tool for Data Warehouse, and provide complete sql inquiry
Sql sentence can be converted to MapReduce task and run by function.) tabular data set, then in backup warehouse
Table data, and need to delete (the distributed text of Hadoop Distributed File System of hdfs in backup warehouse manually
Part system) under data, and need manually under the path latest data put to hdfs.
In realizing process of the present invention, at least there are the following problems in the prior art for inventor's discovery:
It due to use is all artificial treatment during loading business datum, so not can guarantee timeliness many times
Property, also influence data accuracy and safety.
Summary of the invention
In view of this, the embodiment of the present invention provides a kind of method and apparatus of automation load data, can solve existing
Business datum timeliness is loaded in technology not can guarantee, the problem of data accuracy and safety difference.
To achieve the above object, according to an aspect of an embodiment of the present invention, it provides a kind of automate and loads data
Method, comprising: data source is obtained, to import the interim table in database；The interim table is read, if there is in preset time
Partition data to be treated then reads preset indication field in partition data corresponding data source；According to the mark
Field, will be in the data write-in object table in the interim table.
Optionally, the acquisition data source, to import the interim table in database, comprising: reading external data source, to not
Data source with format is parsed；Data in different formats after parsing is processed into the format that database can be read, to import
Interim table in database.
Optionally, object table in the database for executing corresponding operating is verified, to judge whether data load is normal.
Optionally, in the reading partition data corresponding data source before preset indication field, further includes: caching
The object table file size of current database；
Object table verifies in the database of described pair of execution corresponding operating, comprising: compares object table and executes corresponding behaviour
Make the file size of front and back, if executing the object table file size after corresponding operating is greater than the object table before executing corresponding operating
Then data load is normal for file size.
It optionally, will be in the data write-in object table in the interim table, comprising: if institute according to the indication field
Whether stating indication field is newly-increased operation, then obtains the major key of every data in interim table, judge to have in object table and temporarily
The data of the identical major key of table；
If the data of major key not identical as interim table, by interim table data and object table in data close
And then rewrite back object table.
It optionally, will be in the data write-in object table in the interim table, comprising: if institute according to the indication field
Indication field is stated to update operation, then obtains the major key of every data in interim table, there is identical major key in delete target table
Data；In interim table data and object table in data merge, then rewrite back object table.
In addition, according to an aspect of an embodiment of the present invention, providing a kind of device of automation load data, including lead
Enter module, for obtaining data source, to import the interim table in database；Read module, for reading the interim table, if
There are partition data to be treated in preset time, then preset mark word in partition data corresponding data source is read
Section；Execution module, for according to the indication field, the data in the interim table to be written in object table.
Optionally, the import modul obtains data source, when importing the interim table in database, comprising: reads external
Data source parses the data source of different-format；Data in different formats after parsing is processed into what database can be read
Format, to import the interim table in database.
Optionally, the execution module, is also used to: object table in the database for executing corresponding operating verified, with
Judge whether data load is normal.
Optionally, the read module is read in partition data corresponding data source before preset indication field, also
It include: the object table file size for caching current database；
The execution module verifies object table in the database for executing corresponding operating, comprising: compares object table and holds
File size before and after row corresponding operating, if before executing the object table file size after corresponding operating greater than corresponding operating is executed
Object table file size then data load is normal.
Optionally, the data in the interim table are written in object table according to the indication field for the execution module,
If including: the indication field is newly-increased operation, obtain the major key of every data in interim table, judge be in object table
The no data for having major key identical as interim table；If the data of major key not identical as interim table, by the data in interim table
It is merged with data in object table, then rewrites back object table.
Optionally, the data in the interim table are written in object table according to the indication field for the execution module,
If including: the indication field to update operation, the major key of every data in interim table is obtained, is had in delete target table
The data of identical major key；In interim table data and object table in data merge, then rewrite back object table.
Other side according to an embodiment of the present invention, additionally provides a kind of electronic equipment, comprising:
One or more processors；
Storage device, for storing one or more programs,
When one or more of programs are executed by one or more of processors, so that one or more of processing
Device realizes method described in the embodiment of any of the above-described automation load data.
Other side according to an embodiment of the present invention additionally provides a kind of computer-readable medium, is stored thereon with meter
Calculation machine program realizes method described in the embodiment of any of the above-described automation load data when described program is executed by processor.
One embodiment in foregoing invention have the following advantages that or the utility model has the advantages that because use obtain data source, with
Import the interim table in database；The interim table is read, if there is partition data to be treated in preset time, is then read
Take preset indication field in partition data corresponding data source；According to the indication field, by the number in the interim table
According to the technological means in write-in object table, so avoiding the exposure of the data in business datum loading procedure, and realize certainly
The load of dynamicization business datum.
Further effect possessed by above-mentioned non-usual optional way adds hereinafter in conjunction with specific embodiment
With explanation.
Detailed description of the invention
Attached drawing for a better understanding of the present invention, does not constitute an undue limitation on the present invention.Wherein:
Fig. 1 is the schematic diagram of the main flow of the method for automation load data according to an embodiment of the present invention；
Fig. 2 is the schematic diagram that can refer to the main flow of the method for automation load data of embodiment according to the present invention；
Fig. 3 is the schematic diagram of the main modular of the device of automation load data according to an embodiment of the present invention；
Fig. 4 is that the embodiment of the present invention can be applied to exemplary system architecture figure therein；
Fig. 5 is adapted for the structural representation of the computer system for the terminal device or server of realizing the embodiment of the present invention
Figure.
Specific embodiment
Below in conjunction with attached drawing, an exemplary embodiment of the present invention will be described, including the various of the embodiment of the present invention
Details should think them only exemplary to help understanding.Therefore, those of ordinary skill in the art should recognize
It arrives, it can be with various changes and modifications are made to the embodiments described herein, without departing from scope and spirit of the present invention.Together
Sample, for clarity and conciseness, descriptions of well-known functions and structures are omitted from the following description.
Fig. 1 is the method for automation load data according to an embodiment of the present invention, as shown in Figure 1, the automation loads
The method of data includes:
Step S101 obtains data source, to import the interim table in database.
In embodiment, the data source of different-format can be parsed with reading external data source.Then after parsing
Data in different formats be processed into the format that database can be read, to import the interim table in database.Preferably, reading external
Data parsing can be carried out by format of the different data interface to different data sources after data source, data are processed into data
Format can be read in library.
Step S102 reads the interim table if there is partition data to be treated in preset time and then reads institute
State preset indication field in partition data corresponding data source.
Preferably, can be set as the same day in preset time, naturally it is also possible to be other times section.
In addition, every data is all preset with indication field in the data source of acquisition, the indication field records this number
According to the operation species of progress.
Step S103 will be in the data write-in object table in the interim table according to the indication field.It is specific to implement
Process includes:
If the indication field is newly-increased operation, the major key of every data in interim table is obtained, is judged in object table
In whether have the data of major key identical as interim table；If the data of major key not identical as interim table, will be in interim table
Data merge in data and object table, then rewrite back object table.
If the indication field is to update operation, the major key of every data in interim table is obtained, in delete target table
Data with identical major key；In interim table data and object table in data merge, then rewrite back object table.
It, can also be to the number for executing corresponding operating after executing the step S103 as another embodiment of the invention
It is verified according to object table in library, to judge whether data load is normal.Specifically implementation process includes:
It is cached before preset indication field in executing step S102 reading partition data corresponding data source current
The object table file size of database, and after executing the step S103, compare the file before and after object table execution corresponding operating
Size counts if executing the object table file size that the object table file size after corresponding operating is greater than before executing corresponding operating
It is normal according to load.
According to various embodiments above, it can be seen that the method for the automation load data, the not direct face of data
To technical staff, and will not external disclosure, exposure, ensure that the safety of data, at the same realize automation data upload,
It updates, ensure that its accuracy, in addition to more precisely, also being verified to the data after load.
Fig. 2 is the schematic diagram that can refer to the main flow of the method for automation load data of embodiment according to the present invention,
For being applied to HDFS file system (it is of course not solely limited to HDFS file system, can be applied in various frameworks for example
Storm, MapReduce, Spark etc. are merely illustrative herein), the method for the automation load data may include:
Step S201 obtains external data source, to import in the interim table of database.
In embodiment, the data source of different-format can be carried out by reading data device reading external data source
The data of different-format can be processed into the format that database can be read by parsing.Preferably, can after reading external data source
To carry out data parsing by format of the different data interface to different data sources, data, which are processed into database, can be read lattice
Formula.
It should be noted that providing multiple types: CSV (Comma-Separated Values, comma point for data source
Every value file format), TXT (being a kind of text document), XML (be a kind of structural for marking electronic document to make it have
Markup language), SQL (Structured Query Language, structured query language) Sentence format, different-format plus
Load mode is not also identical, i.e., the data of different-format is processed into the format that database can be read and specifically includes:
A) CSV and TXT format is then introduced into the interim table of database by decollator of tab key.
B) XML format is first parsed, and the XML file after then reading parsing is spliced into the text using tab key as decollator
Part, then Load is imported in the interim table of database.
C) for SQL statement, then directly by sql engine execute SQL statement be inserted into data to database interim table
In.
Step S202 reads temp (interim) table, and judging whether there is in HDFS file system needs to locate in preset time
The partition data of reason.Step S203 is then carried out if it exists, otherwise carries out step S204.
Preferably, can be set as the same day in preset time, naturally it is also possible to be other times section.
Preferably, in judging HDFS file system with the presence or absence of can be with when partition data to be treated in preset time
Judged by program return value Flag, partition data to be treated is not present if program return value Flag is 0, if journey
Sequence return value Flag is 1, and there are partition datas to be treated.
Then step S203, caching of target table current file size execute step S205.
Step S204 exits the process, and record exits reason in log.
Step S205 reads preset indication field in the corresponding data source of the partition data.
In embodiment, be defined in data source the last one field (isUpdate) be used to identify data it is newly-increased or
Data update.
Step S206 will be in the data write-in object table in the interim table according to the return value of the indication field.
Preferably, read indication field by program, return value is 1 to represent the newly-increased operation of data, and return value is 0 to represent number
It is operated according to updating.
Further, if the return value of the last one field (isUpdate) is 1, the newly-increased operation of data, tool are represented
The implementation process of body includes:
Step 1: the major key of every data in temp table is read.
Step 2: judge the data for whether having major key identical as temp table in object table.
Step 3: problematic if there is just representing reported data according to judging result, i.e., data report repetition, then enter
To the abnormality processing stage, (preferably, abnormality processing, which can be, sends mail relevant information notice business side.Under the logical finger in business side
Data consumer is swum, can be analyzed by data, out report, or does product support etc. with the data).
If illustrating that reported data is correct there is no major key repetition, in temp table data and object table in number
According to merging, object table is then rewritten back.
Another further carries out example, if the return value of the last one field (isUpdate) is 0, represents data
Operation is updated, specific implementation process includes:
Step 1: data in temp table are read, the major key of every data is obtained.
Step 2: with the data of identical major key in delete target table.
Step 3: in temp table data and object table in data merge, then rewrite back object table.
Step S207 verifies the object table for having executed operation.
As embodiment, data file size before and after object table is compared, if target list file is less than before executing operation
The object table file size of caching then enters exception processing module, sends out mail notification business side；If it is greater than be then it is normal, can
The Safe withdrawing process.
In addition, can refer to the specific implementation content of the method for the data of automation load described in embodiment in the present invention,
It has been described in detail in the method for automation load data described above, therefore has no longer illustrated in this duplicate contents.
Fig. 3 is the device of automation load data according to an embodiment of the present invention, as shown in figure 3, the automation loads
The device 300 of data includes import modul 301, read module 302 and execution module 303.Wherein, import modul 301 obtains
Data source, to import the interim table in database.Then read module 302 reads the interim table, if there is preset time
Interior partition data to be treated then reads preset indication field in partition data corresponding data source.Execution module 303
It, will be in the data write-in object table in the interim table according to the indication field.
Preferably, import modul 301 can parse the data source of different-format with reading external data source.Then
Data in different formats after parsing is processed into the format that database can be read, to import the interim table in database.Further
Ground, reading external data source can carry out data parsing by format of the different data interface to different data sources later, number
Format can be read according to database is processed into.
In another embodiment, read module 302 writes the data in the interim table according to the indication field
When entering in object table, if the indication field is newly-increased operation, the major key of every data in interim table is obtained, is judged in mesh
Whether the data of as interim table identical major key are had in mark table；If the data of major key not identical as interim table, by interim table
In data and object table in data merge, then rewrite back object table.
If the indication field is to update operation, the major key of every data in interim table is obtained, in delete target table
Data with identical major key；In interim table data and object table in data merge, then rewrite back object table.
As another embodiment of the invention, execution module 303 can also be to mesh in the database for executing corresponding operating
Mark table is verified, to judge whether data load is normal.Specifically implementation process includes:
Read module 302, which is read, caches current data before preset indication field in partition data corresponding data source
The object table file size in library, and execution module 303 compares object table after executing corresponding operating to the object table in database
The file size before and after corresponding operating is executed, executes corresponding operating if executing the object table file size after corresponding operating and being greater than
Then data load is normal for preceding object table file size.
It should be noted that the specific implementation content of the device in automation load data of the present invention, in institute above
It states and has been described in detail in the method for automation load data, therefore no longer illustrate in this duplicate contents.
Fig. 4, which is shown, can load data using the method or automation of the automation load data of the embodiment of the present invention
The exemplary system architecture 400 of device.Or Fig. 4 shows the side that data can be loaded using the automation of the embodiment of the present invention
The exemplary system architecture 400 of the device of method or automation load data.
As shown in figure 4, system architecture 400 may include terminal device 401,402,403, network 404 and server 405.
Network 404 between terminal device 401,402,403 and server 405 to provide the medium of communication link.Network 404 can be with
Including various connection types, such as wired, wireless communication link or fiber optic cables etc..
User can be used terminal device 401,402,403 and be interacted by network 404 with server 405, to receive or send out
Send message etc..Various telecommunication customer end applications, such as the application of shopping class, net can be installed on terminal device 401,402,403
(merely illustrative) such as the application of page browsing device, searching class application, instant messaging tools, mailbox client, social platform softwares.
Terminal device 401,402,403 can be the various electronic equipments with display screen and supported web page browsing, packet
Include but be not limited to smart phone, tablet computer, pocket computer on knee and desktop computer etc..
Server 405 can be to provide the server of various services, such as utilize terminal device 401,402,403 to user
The shopping class website browsed provides the back-stage management server (merely illustrative) supported.Back-stage management server can be to reception
To the data such as information query request analyze etc. processing, and by processing result (such as target push information, product letter
Breath -- merely illustrative) feed back to terminal device.
It should be noted that the method for automation load data is generally by server 405 provided by the embodiment of the present invention
It executes, correspondingly, the device of automation load data is generally positioned in server 405.
It should be understood that the number of terminal device, network and server in Fig. 4 is only schematical.According to realization need
It wants, can have any number of terminal device, network and server.
Below with reference to Fig. 5, it illustrates the computer systems 600 for the terminal device for being suitable for being used to realize the embodiment of the present invention
Structural schematic diagram.Terminal device shown in Fig. 5 is only an example, function to the embodiment of the present invention and should not use model
Shroud carrys out any restrictions.
As shown in figure 5, computer system 500 includes central processing unit (CPU) 501, it can be read-only according to being stored in
Program in memory (ROM) 502 or be loaded into the program in random access storage device (RAM) 503 from storage section 508 and
Execute various movements appropriate and processing.In RAM 503, also it is stored with system 500 and operates required various programs and data.
CPU 501, ROM 502 and RAM 503 are connected with each other by bus 504.Input/output (I/O) interface 505 is also connected to always
Line 504.
I/O interface 505 is connected to lower component: the importation 506 including keyboard, mouse etc.；It is penetrated including such as cathode
The output par, c 507 of spool (CRT), liquid crystal display (LCD) etc. and loudspeaker etc.；Storage section 508 including hard disk etc.；
And the communications portion 509 of the network interface card including LAN card, modem etc..Communications portion 509 via such as because
The network of spy's net executes communication process.Driver 510 is also connected to I/O interface 505 as needed.Detachable media 511, such as
Disk, CD, magneto-optic disk, semiconductor memory etc. are mounted on as needed on driver 510, in order to read from thereon
Computer program be mounted into storage section 508 as needed.
Particularly, disclosed embodiment, the process described above with reference to flow chart may be implemented as counting according to the present invention
Calculation machine software program.For example, embodiment disclosed by the invention includes a kind of computer program product comprising be carried on computer
Computer program on readable medium, the computer program include the program code for method shown in execution flow chart.?
In such embodiment, which can be downloaded and installed from network by communications portion 509, and/or from can
Medium 511 is dismantled to be mounted.When the computer program is executed by central processing unit (CPU) 501, system of the invention is executed
The above-mentioned function of middle restriction.
It should be noted that computer-readable medium shown in the present invention can be computer-readable signal media or meter
Calculation machine readable storage medium storing program for executing either the two any combination.Computer readable storage medium for example can be --- but not
Be limited to --- electricity, magnetic, optical, electromagnetic, infrared ray or semiconductor system, device or device, or any above combination.Meter
The more specific example of calculation machine readable storage medium storing program for executing can include but is not limited to: have the electrical connection, just of one or more conducting wires
Taking formula computer disk, hard disk, random access storage device (RAM), read-only memory (ROM), erasable type may be programmed read-only storage
Device (EPROM or flash memory), optical fiber, portable compact disc read-only memory (CD-ROM), light storage device, magnetic memory device,
Or above-mentioned any appropriate combination.In the present invention, computer readable storage medium can be it is any include or storage journey
The tangible medium of sequence, the program can be commanded execution system, device or device use or in connection.And at this
In invention, computer-readable signal media may include in a base band or as carrier wave a part propagate data-signal,
Wherein carry computer-readable program code.The data-signal of this propagation can take various forms, including but unlimited
In electromagnetic signal, optical signal or above-mentioned any appropriate combination.Computer-readable signal media can also be that computer can
Any computer-readable medium other than storage medium is read, which can send, propagates or transmit and be used for
By the use of instruction execution system, device or device or program in connection.Include on computer-readable medium
Program code can transmit with any suitable medium, including but not limited to: wireless, electric wire, optical cable, RF etc. are above-mentioned
Any appropriate combination.
Flow chart and block diagram in attached drawing are illustrated according to the system of various embodiments of the invention, method and computer journey
The architecture, function and operation in the cards of sequence product.In this regard, each box in flowchart or block diagram can generation
A part of one module, program segment or code of table, a part of above-mentioned module, program segment or code include one or more
Executable instruction for implementing the specified logical function.It should also be noted that in some implementations as replacements, institute in box
The function of mark can also occur in a different order than that indicated in the drawings.For example, two boxes succeedingly indicated are practical
On can be basically executed in parallel, they can also be executed in the opposite order sometimes, and this depends on the function involved.Also it wants
It is noted that the combination of each box in block diagram or flow chart and the box in block diagram or flow chart, can use and execute rule
The dedicated hardware based systems of fixed functions or operations is realized, or can use the group of specialized hardware and computer instruction
It closes to realize.
Being described in module involved in the embodiment of the present invention can be realized by way of software, can also be by hard
The mode of part is realized.Described module also can be set in the processor, for example, can be described as: a kind of processor packet
Include import modul, read module and execution module.Wherein, the title of these modules is not constituted under certain conditions to the mould
The restriction of block itself.
As on the other hand, the present invention also provides a kind of computer-readable medium, which be can be
Included in equipment described in above-described embodiment；It is also possible to individualism, and without in the supplying equipment.Above-mentioned calculating
Machine readable medium carries one or more program, when said one or multiple programs are executed by the equipment, makes
Obtaining the equipment includes: acquisition data source, to import the interim table in database；Read the interim table, if there is it is default when
Interior partition data to be treated then reads preset indication field in partition data corresponding data source；According to described
Indication field loads the technological means in the data write-in object table in the interim table so avoiding in business datum
The exposure of data in the process, and realize the load of automation business datum.
Above-mentioned specific embodiment, does not constitute a limitation on the scope of protection of the present invention.Those skilled in the art should be bright
It is white, design requirement and other factors are depended on, various modifications, combination, sub-portfolio and substitution can occur.It is any
Made modifications, equivalent substitutions and improvements etc. within the spirit and principles in the present invention, should be included in the scope of the present invention
Within.
Claims (14)
1. a kind of method of automation load data characterized by comprising
Data source is obtained, to import the interim table in database；
The interim table is read, if there is partition data to be treated in preset time, then reads the partition data pair
Answer preset indication field in data source；
It, will be in the data write-in object table in the interim table according to the indication field.
2. the method according to claim 1, wherein the acquisition data source, interim in database to import
Table, comprising:
Reading external data source parses the data source of different-format；
Data in different formats after parsing is processed into the format that database can be read, to import the interim table in database.
3. the method according to claim 1, wherein further include:
Object table in the database for executing corresponding operating is verified, to judge whether data load is normal.
4. according to the method described in claim 3, it is characterized in that, being preset in the reading partition data corresponding data source
Indication field before, further includes:
Cache the object table file size of current database；
Object table verifies in the database of described pair of execution corresponding operating, comprising:
The file size before and after object table execution corresponding operating is compared, if the object table file size executed after corresponding operating is big
Then data load is normal for object table file size before executing corresponding operating.
5. the method according to claim 1, wherein according to the indication field, by the number in the interim table
According in write-in object table, comprising:
If the indication field is newly-increased operation, obtain the major key of every data in interim table, judge be in object table
The no data for having major key identical as interim table；
If the data of major key not identical as interim table, by interim table data and object table in data merge,
Then object table is rewritten back.
6. the method according to claim 1, wherein according to the indication field, by the number in the interim table
According in write-in object table, comprising:
If the indication field is to update operation, the major key of every data in interim table is obtained, is had in delete target table
The data of identical major key；
In interim table data and object table in data merge, then rewrite back object table.
7. a kind of device of automation load data characterized by comprising
Import modul, for obtaining data source, to import the interim table in database；
Read module if there is partition data to be treated in preset time, then reads institute for reading the interim table
State preset indication field in partition data corresponding data source；
Execution module, for according to the indication field, the data in the interim table to be written in object table.
8. device according to claim 7, which is characterized in that the import modul obtains data source, to import database
In interim table when, comprising:
Reading external data source parses the data source of different-format；
Data in different formats after parsing is processed into the format that database can be read, to import the interim table in database.
9. device according to claim 7, which is characterized in that the execution module is also used to:
Object table in the database for executing corresponding operating is verified, to judge whether data load is normal.
10. device according to claim 9, which is characterized in that the read module reads the partition data and corresponds to number
Before indication field preset in source, further includes:
Cache the object table file size of current database；
The execution module verifies object table in the database for executing corresponding operating, comprising:
The file size before and after object table execution corresponding operating is compared, if the object table file size executed after corresponding operating is big
Then data load is normal for object table file size before executing corresponding operating.
11. device according to claim 7, which is characterized in that the execution module, will be described according to the indication field
In data write-in object table in interim table, comprising:
If the indication field is newly-increased operation, obtain the major key of every data in interim table, judge be in object table
The no data for having major key identical as interim table；
If the data of major key not identical as interim table, by interim table data and object table in data merge,
Then object table is rewritten back.
12. device according to claim 7, which is characterized in that the execution module, will be described according to the indication field
In data write-in object table in interim table, comprising:
If the indication field is to update operation, the major key of every data in interim table is obtained, is had in delete target table
The data of identical major key；
In interim table data and object table in data merge, then rewrite back object table.
13. a kind of electronic equipment characterized by comprising
One or more processors；
Storage device, for storing one or more programs,
When one or more of programs are executed by one or more of processors, so that one or more of processors are real
Now such as method as claimed in any one of claims 1 to 6.
14. a kind of computer-readable medium, is stored thereon with computer program, which is characterized in that described program is held by processor
Such as method as claimed in any one of claims 1 to 6 is realized when row.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810149117.4A CN110209662A (en) | 2018-02-13 | 2018-02-13 | A kind of method and apparatus of automation load data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810149117.4A CN110209662A (en) | 2018-02-13 | 2018-02-13 | A kind of method and apparatus of automation load data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110209662A true CN110209662A (en) | 2019-09-06 |
Family
ID=67778583
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810149117.4A Pending CN110209662A (en) | 2018-02-13 | 2018-02-13 | A kind of method and apparatus of automation load data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110209662A (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111104527A (en) * | 2019-12-18 | 2020-05-05 | 写逸网络科技(上海)有限公司 | Rich media file parsing method |
| CN111367987A (en) * | 2020-03-31 | 2020-07-03 | 西安震有信通科技有限公司 | Data import method, system and device suitable for multiple database types |
| CN111930349A (en) * | 2020-08-04 | 2020-11-13 | 中国建设银行股份有限公司 | Program package generation method and device, electronic equipment and storage medium |
| CN112256775A (en) * | 2020-09-27 | 2021-01-22 | 建信金融科技有限责任公司 | Method and device for timed data loading of Oracle database |
| CN112579673A (en) * | 2020-12-25 | 2021-03-30 | 中国建设银行股份有限公司 | Multi-source data processing method and device |
| CN113807054A (en) * | 2021-09-16 | 2021-12-17 | 北京沃东天骏信息技术有限公司 | Method, device, equipment and computer readable medium for processing data |
| CN113849548A (en) * | 2021-09-22 | 2021-12-28 | 广东电网有限责任公司 | Data extraction method, device, equipment and medium |
| CN115544027A (en) * | 2022-12-05 | 2022-12-30 | 北京滴普科技有限公司 | Data import method and system for OLAP analysis engine |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080120304A1 (en) * | 2006-11-21 | 2008-05-22 | Calio Robert J | Method and system for providing high performance data modification of relational database tables |
| CN101504664A (en) * | 2009-03-18 | 2009-08-12 | 中国工商银行股份有限公司 | Apparatus and method for extracting, converting and loading total source data |
| CN102841897A (en) * | 2011-06-23 | 2012-12-26 | 阿里巴巴集团控股有限公司 | Incremental data extracting method, device and system |
| CN105094852A (en) * | 2015-09-07 | 2015-11-25 | 浪潮软件集团有限公司 | Tool for importing and exporting data EXCEL |
| CN105260485A (en) * | 2015-11-20 | 2016-01-20 | 杭州数梦工场科技有限公司 | Method and device for loading data |
| CN106325933A (en) * | 2016-08-24 | 2017-01-11 | 明算科技（北京）股份有限公司 | Method and device for synchronizing batch data |
| CN106339274A (en) * | 2015-07-14 | 2017-01-18 | 阿里巴巴集团控股有限公司 | Method and system for obtaining data snapshot |
| CN107301214A (en) * | 2017-06-09 | 2017-10-27 | 广州虎牙信息科技有限公司 | Data migration method, device and terminal device in HIVE |
-
2018
- 2018-02-13 CN CN201810149117.4A patent/CN110209662A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080120304A1 (en) * | 2006-11-21 | 2008-05-22 | Calio Robert J | Method and system for providing high performance data modification of relational database tables |
| CN101504664A (en) * | 2009-03-18 | 2009-08-12 | 中国工商银行股份有限公司 | Apparatus and method for extracting, converting and loading total source data |
| CN102841897A (en) * | 2011-06-23 | 2012-12-26 | 阿里巴巴集团控股有限公司 | Incremental data extracting method, device and system |
| CN106339274A (en) * | 2015-07-14 | 2017-01-18 | 阿里巴巴集团控股有限公司 | Method and system for obtaining data snapshot |
| CN105094852A (en) * | 2015-09-07 | 2015-11-25 | 浪潮软件集团有限公司 | Tool for importing and exporting data EXCEL |
| CN105260485A (en) * | 2015-11-20 | 2016-01-20 | 杭州数梦工场科技有限公司 | Method and device for loading data |
| CN106325933A (en) * | 2016-08-24 | 2017-01-11 | 明算科技（北京）股份有限公司 | Method and device for synchronizing batch data |
| CN107301214A (en) * | 2017-06-09 | 2017-10-27 | 广州虎牙信息科技有限公司 | Data migration method, device and terminal device in HIVE |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111104527A (en) * | 2019-12-18 | 2020-05-05 | 写逸网络科技(上海)有限公司 | Rich media file parsing method |
| CN111104527B (en) * | 2019-12-18 | 2023-06-23 | 写逸网络科技(上海)有限公司 | Rich media file analysis method |
| CN111367987A (en) * | 2020-03-31 | 2020-07-03 | 西安震有信通科技有限公司 | Data import method, system and device suitable for multiple database types |
| CN111930349A (en) * | 2020-08-04 | 2020-11-13 | 中国建设银行股份有限公司 | Program package generation method and device, electronic equipment and storage medium |
| CN112256775A (en) * | 2020-09-27 | 2021-01-22 | 建信金融科技有限责任公司 | Method and device for timed data loading of Oracle database |
| CN112579673A (en) * | 2020-12-25 | 2021-03-30 | 中国建设银行股份有限公司 | Multi-source data processing method and device |
| CN113807054A (en) * | 2021-09-16 | 2021-12-17 | 北京沃东天骏信息技术有限公司 | Method, device, equipment and computer readable medium for processing data |
| CN113849548A (en) * | 2021-09-22 | 2021-12-28 | 广东电网有限责任公司 | Data extraction method, device, equipment and medium |
| CN115544027A (en) * | 2022-12-05 | 2022-12-30 | 北京滴普科技有限公司 | Data import method and system for OLAP analysis engine |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110209662A (en) | A kind of method and apparatus of automation load data | |
| CN110807067B (en) | Data synchronization method, device and equipment for relational database and data warehouse | |
| CN110472207A (en) | List generation method and device | |
| CN109542445A (en) | A kind of method and apparatus that Android plug-in unit melts hair | |
| US20200286014A1 (en) | Information updating method and device | |
| CN109522341A (en) | Realize method, apparatus, the equipment of the stream data processing engine based on SQL | |
| CN110389762A (en) | The method and apparatus of AAR packet are handled in Android engineering | |
| CN110427304A (en) | O&M method, apparatus, electronic equipment and medium for banking system | |
| CN113760722A (en) | Test system and test method | |
| CN111125064A (en) | Method and device for generating database mode definition statement | |
| CN112988583A (en) | Method and device for testing syntax compatibility of database | |
| CN109960212A (en) | Task sending method and device | |
| CN112765102B (en) | File system management method and device | |
| CN108694172B (en) | Information output method and device | |
| CN113900944A (en) | Logic verification method and device applied to Flink SQL | |
| CN113760969A (en) | Data query method and device based on elastic search | |
| CN110109912A (en) | A kind of identifier generation method and device | |
| CN109901934B (en) | Method and device for generating interface help document | |
| CN111858621B (en) | Method, apparatus, device and computer readable medium for monitoring business process | |
| CN110110032B (en) | Method and device for updating index file | |
| CN110399187A (en) | A kind for the treatment of method and apparatus of language resource | |
| CN115794637A (en) | Method, device, equipment and computer readable medium for automatic test | |
| CN110688355A (en) | Method and device for changing container state | |
| CN115658127A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN113312053A (en) | Data processing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |