CN108984549A - Table data pick-up method and apparatus are divided in an a kind of point library based on dynamic configuration data library - Google Patents

Table data pick-up method and apparatus are divided in an a kind of point library based on dynamic configuration data library Download PDF

Info

Publication number
CN108984549A
CN108984549A CN201710400242.3A CN201710400242A CN108984549A CN 108984549 A CN108984549 A CN 108984549A CN 201710400242 A CN201710400242 A CN 201710400242A CN 108984549 A CN108984549 A CN 108984549A
Authority
CN
China
Prior art keywords
data
task
extraction
physical table
configuration file
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201710400242.3A
Other languages
Chinese (zh)
Inventor
王军涛
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Jingdong Century Trading Co Ltd
Beijing Jingdong Shangke Information Technology Co Ltd
Original Assignee
Beijing Jingdong Century Trading Co Ltd
Beijing Jingdong Shangke Information Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Jingdong Century Trading Co Ltd, Beijing Jingdong Shangke Information Technology Co Ltd filed Critical Beijing Jingdong Century Trading Co Ltd
Priority to CN201710400242.3A priority Critical patent/CN108984549A/en
Publication of CN108984549A publication Critical patent/CN108984549A/en
Pending legal-status Critical Current

Links

Landscapes

  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The present invention discloses an a kind of point library based on dynamic configuration data library and divides table data pick-up method and apparatus, and table rule can be divided a point library to carry out configurationization management, convenient for exploitation and maintenance, improves extraction efficiency, can be realized tasks interrupt continued access.The described method includes: generating configuration file, corresponding relationship and data extraction condition including physical table each in database and logical table;Configuration file is parsed, so that being that each physical table generates an extraction task, and extraction task is put into task pool according to the data pick-up condition;Each extraction task in task pool is executed, corresponding extraction data are generated after each extraction task execution;Merge and extract data, so that the data extracted from the physical table of corresponding identity logic table merge.

Description

Table data pick-up method and apparatus are divided in an a kind of point library based on dynamic configuration data library
Technical field
The present invention relates to field of computer technology more particularly to an a kind of point library based on dynamic configuration data library to divide table data Abstracting method, device, electronic equipment and computer-readable medium.
Background technique
As Internet user's amount is increasing, the database design of original list library list table is no longer satisfied high concurrent It is required that.So the database of most operation system has all been changed to a point library and has divided table technology.Divide in table technology in a point library, single table Data volume reduce, concurrency increase, but thereupon can also lead to the problem of it is new, database carry out data pick-up when, originally only Need to extract the data of a physical table, it is now desired to the data of several hundred even upper thousand sheets physical tables are extracted, but to purpose For end or a table.This just increases difficulty to extraction process, not only to extract mass data, but also to carry out data Merge, most Zhongdao destination is a table.
In realizing process of the present invention, at least there are the following problems in the prior art for inventor's discovery:
Point library divides table data pick-up mainly to realize at present by two methods, and one is java code is write, sequence is extracted The data of every table, are then merged again, and this method is realized more complicated, it is not easy to which flexible configuration, processing data volume are big Task when, easily cause memory to overflow.
Another kind is to pass through Oozie (the workflow tune of Hadoop by Hadoop (a kind of distributed system infrastructure) Spend engine) Sqoop (between Hadoop and traditional database carry out data transmitting) configuration, big data can be prevented Memory overflows, but a Sqoop node can only configure and extract a physical table, extracts a point of library and divides the data needs of table several hundred A node implements quite cumbersome.
Summary of the invention
In view of this, the embodiment of the present invention, which provides an a kind of point library based on dynamic configuration data library, divides table data pick-up side Method, device, electronic equipment and computer-readable medium, a point library can be divided table rule carry out configurationization management, convenient for exploitation and Maintenance;The data of physical table are extracted by multi-threaded parallel, improve extraction efficiency;Since each extracts the corresponding object of task Table is managed, node can be extracted to the physics list notation extracted, certain failure for extracting task will not influence whole extraction task and hold Row facilitates the continued access of task if generating interruption.
To achieve the above object, according to an aspect of an embodiment of the present invention, it provides a kind of based on dynamic configuration data Table data pick-up method is divided in point library in library characterized by comprising
Configuration file is generated, the configuration file includes the corresponding relationship of each physical table and logical table in the database With data extraction condition;
The configuration file is parsed, is each physical table one extraction task of generation according to the data pick-up condition, and And the extraction task is put into task pool;
Each extraction task in the task pool is executed, corresponding extraction data are generated after each extraction task execution;
Merge the extraction data, so that the data extracted from the physical table of corresponding identity logic table merge.
Optionally, the configuration file further include:
Database where database user name password, physical table where the type of database, physical table where physical table A point library for IP address list and port information, customized physical table divides table regular.
Optionally, the content of the configuration file is configured with label.
Optionally, the extraction task has multithreading interface.
Optionally, the extraction task in the task pool is concurrently executed according to scheduled concurrency.
It optionally,, can be to being taken out so when carrying out extraction task since each extracts the corresponding physical table of task Node is extracted in the physics list notation taken, described to extract the number that node includes the time for extracting task execution or the physical table According to major key.
To achieve the above object, according to another aspect of the present invention, it provides a kind of based on dynamic configuration data library Table data pick-up device is divided in point library characterized by comprising
Configuration module, for generating configuration file, the configuration file include in the database each physical table with patrol Collect the corresponding relationship and data extraction condition of table;
Parsing module is that each physical table generates one according to the data pick-up condition for parsing the configuration file A extraction task, and the extraction task is put into task pool;
Execution module generates phase after each extraction task execution for executing each extraction task in the task pool The extraction data answered;
Merging module, for merging the extraction data, so that the number extracted from the physical table of corresponding identity logic table According to merging.
Optionally, the configuration file further include:
Database where database user name password, physical table where the type of database, physical table where physical table A point library for IP address list and port information, customized physical table divides table regular.
Optionally, the content of the configuration file is configured with label.
Optionally, the extraction task has multithreading interface.
Optionally, the extraction task in the task pool is concurrently executed according to scheduled concurrency.
It optionally,, can be to being taken out so when carrying out extraction task since each extracts the corresponding physical table of task Node is extracted in the physics list notation taken, described to extract the number that node includes the time for extracting task execution or the physical table According to major key.
To achieve the above object, according to a further aspect of the invention, it provides a kind of based on dynamic configuration data library The electronic device terminal of table data pick-up is divided in point library characterized by comprising
One or more processors;
Storage device, for storing one or more programs,
When one or more of programs are executed by one or more of processors, so that one or more of processing Device realizes that any method in table data pick-up method is divided in point library based on dynamic configuration data library.
To achieve the above object, according to a further aspect of the invention, it provides a kind of based on dynamic configuration data library The computer-readable medium of table data pick-up is divided in point library, is stored thereon with computer program, which is characterized in that described program is located Reason device realizes that any method in table data pick-up method is divided in point library based on dynamic configuration data library when executing.
One embodiment in foregoing invention have the following advantages that or the utility model has the advantages that because using to a point library divide table rule into Row configurationization management, the data of physical table are extracted by multi-threaded parallel, each extracts the corresponding physical table of task, can be right The technological means of node is extracted in the physics list notation extracted, so overcoming conventional method realizes that more complex, memory easily overflows The technical issues of, and then reach and divide table database convenient for exploitation and maintenance in a point library, extraction efficiency is improved, certain extracts task It unsuccessfully will not influence whole extraction task execution and facilitate the technical effect of task continued access if generating interruption.
Further effect possessed by above-mentioned non-usual optional way adds hereinafter in conjunction with specific embodiment With explanation.
Detailed description of the invention
Attached drawing for a better understanding of the present invention, does not constitute an undue limitation on the present invention.Wherein:
Fig. 1 is that table data pick-up method is divided in an a kind of point library based on dynamic configuration data library according to an embodiment of the present invention The schematic diagram of main flow;
Fig. 2 is that an a kind of point library based on dynamic configuration data library according to an embodiment of the present invention is divided in table data pick-up method Mapreduce model schematic.
Fig. 3 is that table data pick-up device is divided in an a kind of point library based on dynamic configuration data library according to an embodiment of the present invention The schematic diagram of major part;
Fig. 4 is that the embodiment of the present invention can be applied to exemplary system architecture figure therein;
Fig. 5 is adapted for the structural representation of the computer system for the terminal device or server of realizing the embodiment of the present invention Figure.
Specific embodiment
Below in conjunction with attached drawing, an exemplary embodiment of the present invention will be described, including the various of the embodiment of the present invention Details should think them only exemplary to help understanding.Therefore, those of ordinary skill in the art should recognize It arrives, it can be with various changes and modifications are made to the embodiments described herein, without departing from scope and spirit of the present invention.Together Sample, for clarity and conciseness, descriptions of well-known functions and structures are omitted from the following description.
Fig. 1 is that table data pick-up method is divided in an a kind of point library based on dynamic configuration data library according to an embodiment of the present invention The schematic diagram of main flow, as shown in Figure 1:
Step S101 indicates generation configuration file, the corresponding relationship sum number of each physical table and logical table in the database According to extraction condition.Wherein, the corresponding one or more physical tables of a logical table.The configuration file includes: number where physical table According to the IP address list of database where the type in library, database user name password, physical table where physical table and port information, A point library for customized physical table divides table regular.Such as by configuration file with xml (extensible markup language, standard generalized markup language The subset of speech is a kind of for marking electronic document to make it have structural markup language.) document form preservation, pass through use The rule of table is divided in a kind of bean label (component write as with java language) configuration point library, and server label configures each database IP, the port information of server, schema label configure the information of each database, and table label configures the information of every table, Establish logical table.
Step S102 indicates to parse the configuration file, is that each physical table generates one according to the data pick-up condition Extraction task, and the extraction task is put into task pool.The extraction task has as realized runnable interface (one kind Multithreading interface) multithreading interface, which includes: mission number, server number, database name, physics table name, data (sqoop is the tool of a open source, is mainly used in Hadoop (Hive) and tradition by library user name, database password, sqoop Database between carry out data transmitting) inquiry sql (structured query language (Structured Query Language) letter Claim SQL), sqoop output directory.
Command Line Parsing code is as follows:
// // // // // // // // // replacement IP address, database name
Int seq=getArgsValueSeq (" -- connect ", args);
If (seq==-1)
throw new Exception("sqoop config'--connect'is not exist!");
}
Args [seq]=schemaConfig.getJdbcUrl (dbName);
// // // // // // // // ///replacement user name, password.
Seq=getArgsValueSeq (" -- username ", args);
If (seq==-1)
throw new Exception("sqoop config'--username'is not exist!");
}
Args [seq]=schemaConfig.getUserName ();
Seq=getArgsValueSeq (" -- password ", args);
If (seq==-1)
throw new Exception("sqoop config'--password'is not exist!");
}
Args [seq]=schemaConfig.getPassword ();
// // // // // // // // // replacement table name
Seq=getArgsValueSeq (" -- query ", args);
If (seq==-1)
throw new Exception("sqoop config'--query'is not exist!");
}
Args [seq]=args [seq] .replace (" { sqoop:table } ", tableName);
String tableKey=getTableKey ();
String incrmentId=map.get (tableKey)==null? initialTime:
map.get(tableKey);
Args [seq]=String.format (args [seq], new Object [] tableKey,
incrmentId});
System.out.println("whereSql:"+args[seq]);
// // // // // // // ///interim the output directory of setting
Seq=getArgsValueSeq (" -- target-dir ", args);
If (seq==-1)
throw new Exception("sqoop config'--target-dir'is not exist!");
}
if(!args[seq].endsWith("/")){
Args [seq]=args [seq]+"/";
}
Args [seq]=args [seq]+totalSeq;
String [] newArgs=addSeftArgs (args);
Int flag=Sqoop.runTool (newArgs, conf);
if(flag!=0)
throw new Exception("sqoop run error!");
}
Step S103 is indicated to execute each extraction task in the task pool, be generated after each extraction task execution corresponding Extraction data.Wherein the extraction task in the task pool is concurrently executed according to scheduled concurrency.Such as concurrency setting It is 30, then it represents that while thering are the data of 30 physical tables being extracted, pass through countdownlatch (a kind of coincidence counter) Concurrent lock control, when the wheel concurrent tasks all after the completion of, restart in next step operate;If certain task execution is interrupted, can Node is extracted to the physics list notation extracted, the node that extracts includes the time for extracting task execution or the physics The data major key of table.
Scheduled concurrency is provided that
CountDownLatch downLatch=new CountDownLatch (tasks.size ());
for(SqoopTask task:tasks){
task.setCountDownLatch(downLatch);
task.setConf(conf);
ifNeedWait();
executor.execute(task);
try{
Thread.sleep(submitFrequency);// 500 milliseconds of suspend mode of default, prevents from frequently submitting
}catch(InterruptedException e){
}
}
downLatch.await();
executor.shutdown();
Step S104 merges the extraction data, so that the data extracted from the physical table of corresponding identity logic table merge Together.Such as after parsing configuration file, need to collect 1024 physical tables, then 1024 numbers can be generated after the completion of task execution According to.
1024 outputs generated after the completion of task execution are as follows:
/user/${user.name}/input_data/tableOutputPath_temp/0
/user/${user.name}/input_data/tableOutputPath_temp/1
/user/${user.name}/input_data/tableOutputPath_temp/2
……
/user/${user.name}/input_data/tableOutputPath_temp/1023
Final output data are as follows:
/user/${user.name}/input_data/tableOutputPath
Using Mapreduce, (MapReduce is a kind of programming model, for the parallel of large-scale dataset (being greater than 1TB) Operation) method, as shown in Fig. 2, with/user/ $ { user.name }/input_data/tableOutputPath_temp/0 etc. 1024 catalogues are as input, using :/user/ $ { user.name }/input_data/tableOutputPath catalogue as defeated Out.
It is 1 that specification quantity, which is arranged, using the physical table name where data as data major key after mapping, such phase jljl The data of reason table can be sent in the same specification function.Following operation is done in specification:
1. obtaining the major key of the maximum data of every physical table
In mapreduce programming model, it is assumed that using physical table name as data major key, then the data of same physical table It can be put into a reduce function.Then it is sorted, can thus be obtained each according to data major key in each specification method The maximum ID of physical table.When next increment extraction can the ID do increment extraction.
2. outputting data to the hdfs of final result, (HDFS is the letter of Hadoop Distribute File System Claim, that is, a distributed file system of Hadoop) catalogue
3. saving the maximum data major key of every physical table to hdfs catalogue
Fig. 3 is that table data pick-up device is divided in an a kind of point library based on dynamic configuration data library according to an embodiment of the present invention The schematic diagram of 300 major part, wherein
Configuration module 301, for realizing the S101 step in Fig. 1, including for generating configuration file, the configuration file Corresponding relationship and data extraction condition including each physical table in the database and logical table, wherein a logical table pair Answer one or more physical tables;
Parsing module 302, for realizing the S102 step in Fig. 1, including for parsing the configuration file, so that according to The data pick-up condition is that each physical table generates an extraction task, and the extraction task is put into task pool;
Execution module 303, for realizing the S103 step in Fig. 1, including for executing each pumping in the task pool Task is taken, corresponding extraction data are generated after each extraction task execution;
Merging module 304, for realizing the S104 step in Fig. 1, including for merging the extraction data, so that from right The data extracted in the physical table of identity logic table are answered to merge.
Fig. 4, which is shown, can execute the method for push task using a kind of selectivity of the embodiment of the present invention or selectivity is held The exemplary system architecture 400 of the device of row push task.
As shown in figure 4, system architecture 400 may include terminal device 401,402,403, network 404 and server 405. Network 404 between terminal device 401,402,403 and server 405 to provide the medium of communication link.Network 404 can be with Including various connection types, such as wired, wireless communication link or fiber optic cables etc..
User can be used terminal device 401,402,403 and be interacted by network 404 with server 405, to receive or send out Send message etc..Various telecommunication customer end applications, such as the application of shopping class, net can be installed on terminal device 401,402,403 (merely illustrative) such as the application of page browsing device, searching class application, instant messaging tools, mailbox client, social platform softwares.
Terminal device 401,402,403 can be the various electronic equipments with display screen and supported web page browsing, packet Include but be not limited to smart phone, tablet computer, pocket computer on knee and desktop computer etc..
Server 405 can be to provide the server of various services, such as utilize terminal device 401,402,403 to user The shopping class website browsed provides the back-stage management server (merely illustrative) supported.Back-stage management server can be to reception To the data such as information query request analyze etc. processing, and by processing result (such as target push information, product letter Breath -- merely illustrative) feed back to terminal device.
It should be noted that a kind of selectively execute provided by the embodiment of the present invention pushes multitask method generally by servicing Device 405 executes, and correspondingly, the device that selectivity executes push task is generally positioned in server 405.
It should be understood that the number of terminal device, network and server in Fig. 4 is only schematical.According to realization need It wants, can have any number of terminal device, network and server.
Fig. 5 shows the structural representation for being suitable for the computer system 500 for the terminal device for being used to realize the embodiment of the present invention Figure.Terminal device shown in fig. 5 is only an example, should not function to the embodiment of the present invention and use scope bring it is any Limitation.
As shown in figure 5, computer system 500 includes central processing unit (CPU) 501, it can be read-only according to being stored in Program in memory (ROM) 502 or be loaded into the program in random access storage device (RAM) 503 from storage section 508 and Execute various movements appropriate and processing.In RAM 503, also it is stored with system 500 and operates required various programs and data. CPU 501, ROM 502 and RAM 503 are connected with each other by bus 504.Input/output (I/O) interface 505 is also connected to always Line 504.
I/O interface 505 is connected to lower component: the importation 506 including keyboard, mouse etc.;It is penetrated including such as cathode The output par, c 507 of spool (CRT), liquid crystal display (LCD) etc. and loudspeaker etc.;Storage section 508 including hard disk etc.; And the communications portion 509 of the network interface card including LAN card, modem etc..Communications portion 509 via such as because The network of spy's net executes communication process.Driver 410 is also connected to I/O interface 505 as needed.Detachable media 511, such as Disk, CD, magneto-optic disk, semiconductor memory etc. are mounted on as needed on driver 510, in order to read from thereon Computer program be mounted into storage section 508 as needed.
Particularly, disclosed embodiment according to the present invention may be implemented as computer above with reference to the process of Fig. 1 description Software program.For example, embodiment disclosed by the invention includes a kind of computer program product comprising be carried on computer-readable Computer program on medium, the computer program include the program code for executing method shown in FIG. 1.In such reality It applies in example, which can be downloaded and installed from network by communications portion 509, and/or from detachable media 511 are mounted.When the computer program is executed by central processing unit (CPU) 501, executes and limited in system of the invention Above-mentioned function.
It should be noted that computer-readable medium shown in the present invention includes computer-readable signal media or computer Readable storage medium storing program for executing or the two any combination.Computer readable storage medium include but is not limited to electricity, magnetic, light, Electromagnetism, infrared ray, the system of semiconductor, any combination of device or device or above content.Computer readable storage medium Be specifically including but not limited to: electrical connection, portable computer diskette with one or more conducting wires, hard disk, random access are deposited Reservoir (RAM), read-only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, Portable, compact Disk read-only memory (CD-ROM), light storage device, magnetic memory device or above content any combination.In the present invention In, computer readable storage medium, which includes any, includes or the tangible medium of storage program, which can be commanded and execute system System, device or device use or in connection;Computer-readable signal media includes in a base band or conduct The data-signal that carrier wave a part is propagated, wherein carrying computer-readable program code, the data-signal of this propagation can To take various forms, including but not limited to electromagnetic signal, any combination of optical signal or above-mentioned signal.Computer-readable letter Number medium can also be any computer-readable medium other than computer readable storage medium, which can be with It sends, propagate or transmits for by the use of instruction execution system, device or device or program in connection.Meter The program code for including on calculation machine readable medium can transmit with any suitable medium, including but not limited to: wireless, electric wire, Any combination of optical cable, RF etc. or above-mentioned medium.
Fig. 1 in attached drawing, illustrate the system according to various embodiments of the invention, method and computer program product can The architecture, function and operation being able to achieve, each box in Fig. 1 can represent the one of a module, program segment or code Part, a part of above-mentioned module, program segment or code, which includes that one or more is for implementing the specified logical function, to be held Row instruction.It should be noted that in some implementations as replacements, function marked in the box can also be to be different from attached drawing The sequence marked occurs.For example, two boxes succeedingly indicated can actually be performed in parallel, it sometimes can also be by phase Anti- sequence executes, and execution sequence depends on the functions involved.It is also noted that each box in Fig. 1 and its Combination, can the dedicated hardware based systems of the functions or operations as defined in executing realize, or can with it is dedicated firmly The combination of part and computer instruction is realized.
Being described in module involved in the embodiment of the present invention can be realized by way of software, can also be by hard The mode of part is realized.Described module also can be set in the processor, for example, can be described as: a kind of processor packet Include configuration module, parsing module, execution module and merging module.Wherein, the title of these modules not structure under certain conditions The restriction of the pairs of module itself, for example, merging module be also described as " for merging the extraction data so that from The module that the data extracted in the physical table of corresponding identity logic table merge ".
On the other hand, the embodiment of the invention also provides a kind of computer-readable medium, which can be with It is included in equipment described in above-described embodiment;It is also possible to individualism, and without in the supplying equipment.Above-mentioned meter Calculation machine readable medium carries one or more program, when said one or multiple programs are executed by the equipment, So that the equipment includes:
Configuration file is obtained, the configuration file includes the corresponding relationship of each physical table and logical table in the database With data extraction condition, wherein the corresponding one or more physical tables of a logical table;
The configuration file is parsed, so that being that each physical table generates an extraction times according to the data pick-up condition Business, and the extraction task is put into task pool;
Each extraction task in the task pool is executed, corresponding extraction data are generated after each extraction task execution;
Merge the extraction data, so that the data extracted from the physical table of corresponding identity logic table merge.
The technical solution of the embodiment of the present invention, because dividing table rule to carry out configurationization management using to a point library, by multi-thread The data of journey paralleling abstracting physical table, are then combined with, each extracts the corresponding physical table of task, can be to the physics extracted The technological means of node is extracted in list notation, so overcoming conventional method realizes the technical issues of more complex, memory easily overflows, into And reach and divide table database to improve extraction efficiency convenient for exploitation and maintenance in a point library, certain failure for extracting task will not influence It is whole to extract task execution, if generating interruption, facilitate the technical effect of task continued access.
Above-mentioned specific embodiment, does not constitute a limitation on the scope of protection of the present invention.Those skilled in the art should be bright It is white, design requirement and other factors are depended on, various modifications, combination, sub-portfolio and substitution can occur.It is any Made modifications, equivalent substitutions and improvements etc. within the spirit and principles in the present invention, should be included in the scope of the present invention Within.

Claims (14)

1. table data pick-up method is divided in an a kind of point library based on dynamic configuration data library characterized by comprising
Configuration file is generated, the configuration file includes the corresponding relationship sum number of each physical table and logical table in the database According to extraction condition;;
The configuration file is parsed, is that each physical table generates an extraction task, and handle according to the data pick-up condition The extraction task is put into task pool;
Each extraction task in the task pool is executed, corresponding extraction data are generated after each extraction task execution;
Merge the extraction data, so that the data extracted from the physical table of corresponding identity logic table merge.
2. the method according to claim 1, wherein the configuration file further include:
The IP of database where database user name password, physical table where the type of database, physical table where physical table A point library for location list and port information, customized physical table divides table regular.
3. according to the method described in claim 2, it is characterized in that, configuring the content of the configuration file with label.
4. the method according to claim 1, wherein the extraction task has multithreading interface.
5. the method according to claim 1, wherein the extraction task in the task pool is according to scheduled concurrent Degree concurrently executes.
6. the method according to claim 1, wherein since each extracts the corresponding physical table of task, institute When carrying out extraction task, can extract node to the physics list notation extracted, the extraction node includes that described extract is appointed The data major key of time or the physical table that business executes.
7. table data pick-up device is divided in an a kind of point library based on dynamic configuration data library characterized by comprising
Configuration module, for generating configuration file, the configuration file includes each physical table and logical table in the database Corresponding relationship and data extraction condition;
Parsing module is that each physical table generates a pumping according to the data pick-up condition for parsing the configuration file Task is taken, and the extraction task is put into task pool;
Execution module generates corresponding for executing each extraction task in the task pool after each extraction task execution Extract data;
Merging module, for merging the extraction data, so that the data extracted from the physical table of corresponding identity logic table are closed And together.
8. device according to claim 7, which is characterized in that the configuration file further include:
The IP of database where database user name password, physical table where the type of database, physical table where physical table A point library for location list and port information, customized physical table divides table regular.
9. device according to claim 8, which is characterized in that configure the content of the configuration file with label.
10. device according to claim 7, which is characterized in that the extraction task has multithreading interface.
11. device according to claim 7, which is characterized in that extraction task in the task pool according to it is scheduled simultaneously Hair degree concurrently executes.
12. device according to claim 7, which is characterized in that since each extracts the corresponding physical table of task, institute When carrying out extraction task, can extract node to the physics list notation extracted, the extraction node includes that described extract is appointed The data major key of time or the physical table that business executes.
13. a kind of electronic device terminal for dividing table data pick-up based on point library in dynamic configuration data library, which is characterized in that packet It includes:
One or more processors;
Storage device, for storing one or more programs,
When one or more of programs are executed by one or more of processors, so that one or more of processors are real Now such as method as claimed in any one of claims 1 to 6.
14. a kind of computer-readable medium, is stored thereon with computer program, which is characterized in that described program is held by processor Such as method as claimed in any one of claims 1 to 6 is realized when row.
CN201710400242.3A 2017-05-31 2017-05-31 Table data pick-up method and apparatus are divided in an a kind of point library based on dynamic configuration data library Pending CN108984549A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201710400242.3A CN108984549A (en) 2017-05-31 2017-05-31 Table data pick-up method and apparatus are divided in an a kind of point library based on dynamic configuration data library

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201710400242.3A CN108984549A (en) 2017-05-31 2017-05-31 Table data pick-up method and apparatus are divided in an a kind of point library based on dynamic configuration data library

Publications (1)

Publication Number Publication Date
CN108984549A true CN108984549A (en) 2018-12-11

Family

ID=64502470

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201710400242.3A Pending CN108984549A (en) 2017-05-31 2017-05-31 Table data pick-up method and apparatus are divided in an a kind of point library based on dynamic configuration data library

Country Status (1)

Country Link
CN (1) CN108984549A (en)

Cited By (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110275913A (en) * 2019-04-25 2019-09-24 深圳壹账通智能科技有限公司 Data furnishing method, device and storage medium and electronic device
CN110780946A (en) * 2019-10-30 2020-02-11 京东数字科技控股有限公司 Method and apparatus for processing information
CN112632191A (en) * 2020-12-29 2021-04-09 中国农业银行股份有限公司 Data processing method and system
CN112632125A (en) * 2020-12-23 2021-04-09 福信富通科技股份有限公司 Sub-table and sub-area query method for MySQL
CN112685485A (en) * 2021-01-04 2021-04-20 拉卡拉支付股份有限公司 Data acquisition method, data acquisition device, electronic equipment, storage medium and program product
CN112765184A (en) * 2021-04-07 2021-05-07 四川新网银行股份有限公司 Real-time acquisition method based on Mysql database and table division
CN112860776A (en) * 2021-01-20 2021-05-28 山东众阳健康科技集团有限公司 Method and system for extracting and scheduling various data
CN113190603A (en) * 2021-04-28 2021-07-30 中国邮政储蓄银行股份有限公司 Data processing method, data processing device, computer readable storage medium and processor
CN113360470A (en) * 2021-05-25 2021-09-07 深圳市珍爱捷云信息技术有限公司 MySQL-based sub-library and sub-table synchronization method and device, electronic equipment and computer-readable storage medium

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103927331A (en) * 2014-03-21 2014-07-16 珠海多玩信息技术有限公司 Data querying method, data querying device and data querying system
CN104317960A (en) * 2014-11-14 2015-01-28 中国建设银行股份有限公司 Method and device for processing batch job
CN105183901A (en) * 2015-09-30 2015-12-23 北京京东尚科信息技术有限公司 Method and device for reading database table through data query engine
CN106095878A (en) * 2016-06-07 2016-11-09 中国建设银行股份有限公司 The database manipulation device and method of table is divided based on point storehouse
US20160334998A1 (en) * 2015-05-15 2016-11-17 Cisco Technology, Inc. Tenant-level sharding of disks with tenant-specific storage modules to enable policies per tenant in a distributed storage system
CN106294565A (en) * 2016-07-27 2017-01-04 中国农业银行股份有限公司 A kind of data bank access method and system
US20170103094A1 (en) * 2015-10-07 2017-04-13 Oracle International Corporation Request routing and query processing in a sharded database

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103927331A (en) * 2014-03-21 2014-07-16 珠海多玩信息技术有限公司 Data querying method, data querying device and data querying system
CN104317960A (en) * 2014-11-14 2015-01-28 中国建设银行股份有限公司 Method and device for processing batch job
US20160334998A1 (en) * 2015-05-15 2016-11-17 Cisco Technology, Inc. Tenant-level sharding of disks with tenant-specific storage modules to enable policies per tenant in a distributed storage system
CN105183901A (en) * 2015-09-30 2015-12-23 北京京东尚科信息技术有限公司 Method and device for reading database table through data query engine
US20170103094A1 (en) * 2015-10-07 2017-04-13 Oracle International Corporation Request routing and query processing in a sharded database
CN106095878A (en) * 2016-06-07 2016-11-09 中国建设银行股份有限公司 The database manipulation device and method of table is divided based on point storehouse
CN106294565A (en) * 2016-07-27 2017-01-04 中国农业银行股份有限公司 A kind of data bank access method and system

Cited By (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110275913A (en) * 2019-04-25 2019-09-24 深圳壹账通智能科技有限公司 Data furnishing method, device and storage medium and electronic device
CN110780946A (en) * 2019-10-30 2020-02-11 京东数字科技控股有限公司 Method and apparatus for processing information
CN110780946B (en) * 2019-10-30 2021-05-25 京东数字科技控股有限公司 Method and apparatus for processing information
CN112632125A (en) * 2020-12-23 2021-04-09 福信富通科技股份有限公司 Sub-table and sub-area query method for MySQL
CN112632191A (en) * 2020-12-29 2021-04-09 中国农业银行股份有限公司 Data processing method and system
CN112632191B (en) * 2020-12-29 2024-06-11 中国农业银行股份有限公司 Data processing method and system
CN112685485A (en) * 2021-01-04 2021-04-20 拉卡拉支付股份有限公司 Data acquisition method, data acquisition device, electronic equipment, storage medium and program product
CN112860776A (en) * 2021-01-20 2021-05-28 山东众阳健康科技集团有限公司 Method and system for extracting and scheduling various data
CN112860776B (en) * 2021-01-20 2022-12-06 众阳健康科技集团有限公司 Method and system for extracting and scheduling various data
CN112765184A (en) * 2021-04-07 2021-05-07 四川新网银行股份有限公司 Real-time acquisition method based on Mysql database and table division
CN113190603A (en) * 2021-04-28 2021-07-30 中国邮政储蓄银行股份有限公司 Data processing method, data processing device, computer readable storage medium and processor
CN113190603B (en) * 2021-04-28 2024-09-27 中国邮政储蓄银行股份有限公司 Data processing method, device, computer readable storage medium and processor
CN113360470A (en) * 2021-05-25 2021-09-07 深圳市珍爱捷云信息技术有限公司 MySQL-based sub-library and sub-table synchronization method and device, electronic equipment and computer-readable storage medium

Similar Documents

Publication Publication Date Title
CN108984549A (en) Table data pick-up method and apparatus are divided in an a kind of point library based on dynamic configuration data library
US20220327125A1 (en) Query scheduling based on a query-resource allocation and resource availability
US11580107B2 (en) Bucket data distribution for exporting data to worker nodes
US11321321B2 (en) Record expansion and reduction based on a processing task in a data intake and query system
US11586627B2 (en) Partitioning and reducing records at ingest of a worker node
US11593377B2 (en) Assigning processing tasks in a data intake and query system
US11599541B2 (en) Determining records generated by a processing task of a query
US20200364223A1 (en) Search time estimate in a data intake and query system
US20190258632A1 (en) Determining a Record Generation Estimate of a Processing Task
US20200050612A1 (en) Supporting additional query languages through distributed execution of query engines
US20200065303A1 (en) Addressing memory limits for partition tracking among worker nodes
WO2020027867A1 (en) Generating a subquery for a distinct data intake and query system
CN111108479A (en) Autonomic multi-tenant database cloud service framework
CN107491382B (en) Log output method and device
CN107506218A (en) The management method and management system of a kind of configuration file
CN110765187A (en) Data source route management method and device
CN109241033A (en) The method and apparatus for creating real-time data warehouse
CN109522341A (en) Realize method, apparatus, the equipment of the stream data processing engine based on SQL
CN108958744A (en) Dispositions method, device, medium and the electronic equipment of big data distributed type assemblies
CN110019539A (en) A kind of method and apparatus that the data of data warehouse are synchronous
CN110019044A (en) Big data cluster quasi real time Yarn Mission Monitor analysis method
CN110764769B (en) Method and device for processing user request
CN112818026A (en) Data integration method and device
CN115309767A (en) Data fusion method and device
CN109960212A (en) Task sending method and device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication
RJ01 Rejection of invention patent application after publication

Application publication date: 20181211