CN108984549A - Table data pick-up method and apparatus are divided in an a kind of point library based on dynamic configuration data library - Google Patents
Table data pick-up method and apparatus are divided in an a kind of point library based on dynamic configuration data library Download PDFInfo
- Publication number
- CN108984549A CN108984549A CN201710400242.3A CN201710400242A CN108984549A CN 108984549 A CN108984549 A CN 108984549A CN 201710400242 A CN201710400242 A CN 201710400242A CN 108984549 A CN108984549 A CN 108984549A
- Authority
- CN
- China
- Prior art keywords
- data
- task
- extraction
- physical table
- configuration file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention discloses an a kind of point library based on dynamic configuration data library and divides table data pick-up method and apparatus, and table rule can be divided a point library to carry out configurationization management, convenient for exploitation and maintenance, improves extraction efficiency, can be realized tasks interrupt continued access.The described method includes: generating configuration file, corresponding relationship and data extraction condition including physical table each in database and logical table；Configuration file is parsed, so that being that each physical table generates an extraction task, and extraction task is put into task pool according to the data pick-up condition；Each extraction task in task pool is executed, corresponding extraction data are generated after each extraction task execution；Merge and extract data, so that the data extracted from the physical table of corresponding identity logic table merge.
Description
Technical field
The present invention relates to field of computer technology more particularly to an a kind of point library based on dynamic configuration data library to divide table data
Abstracting method, device, electronic equipment and computer-readable medium.
Background technique
As Internet user's amount is increasing, the database design of original list library list table is no longer satisfied high concurrent
It is required that.So the database of most operation system has all been changed to a point library and has divided table technology.Divide in table technology in a point library, single table
Data volume reduce, concurrency increase, but thereupon can also lead to the problem of it is new, database carry out data pick-up when, originally only
Need to extract the data of a physical table, it is now desired to the data of several hundred even upper thousand sheets physical tables are extracted, but to purpose
For end or a table.This just increases difficulty to extraction process, not only to extract mass data, but also to carry out data
Merge, most Zhongdao destination is a table.
In realizing process of the present invention, at least there are the following problems in the prior art for inventor's discovery:
Point library divides table data pick-up mainly to realize at present by two methods, and one is java code is write, sequence is extracted
The data of every table, are then merged again, and this method is realized more complicated, it is not easy to which flexible configuration, processing data volume are big
Task when, easily cause memory to overflow.
Another kind is to pass through Oozie (the workflow tune of Hadoop by Hadoop (a kind of distributed system infrastructure)
Spend engine) Sqoop (between Hadoop and traditional database carry out data transmitting) configuration, big data can be prevented
Memory overflows, but a Sqoop node can only configure and extract a physical table, extracts a point of library and divides the data needs of table several hundred
A node implements quite cumbersome.
Summary of the invention
In view of this, the embodiment of the present invention, which provides an a kind of point library based on dynamic configuration data library, divides table data pick-up side
Method, device, electronic equipment and computer-readable medium, a point library can be divided table rule carry out configurationization management, convenient for exploitation and
Maintenance；The data of physical table are extracted by multi-threaded parallel, improve extraction efficiency；Since each extracts the corresponding object of task
Table is managed, node can be extracted to the physics list notation extracted, certain failure for extracting task will not influence whole extraction task and hold
Row facilitates the continued access of task if generating interruption.
To achieve the above object, according to an aspect of an embodiment of the present invention, it provides a kind of based on dynamic configuration data
Table data pick-up method is divided in point library in library characterized by comprising
Configuration file is generated, the configuration file includes the corresponding relationship of each physical table and logical table in the database
With data extraction condition；
The configuration file is parsed, is each physical table one extraction task of generation according to the data pick-up condition, and
And the extraction task is put into task pool；
Each extraction task in the task pool is executed, corresponding extraction data are generated after each extraction task execution；
Merge the extraction data, so that the data extracted from the physical table of corresponding identity logic table merge.
Optionally, the configuration file further include:
Database where database user name password, physical table where the type of database, physical table where physical table
A point library for IP address list and port information, customized physical table divides table regular.
Optionally, the content of the configuration file is configured with label.
Optionally, the extraction task has multithreading interface.
Optionally, the extraction task in the task pool is concurrently executed according to scheduled concurrency.
It optionally,, can be to being taken out so when carrying out extraction task since each extracts the corresponding physical table of task
Node is extracted in the physics list notation taken, described to extract the number that node includes the time for extracting task execution or the physical table
According to major key.
To achieve the above object, according to another aspect of the present invention, it provides a kind of based on dynamic configuration data library
Table data pick-up device is divided in point library characterized by comprising
Configuration module, for generating configuration file, the configuration file include in the database each physical table with patrol
Collect the corresponding relationship and data extraction condition of table；
Parsing module is that each physical table generates one according to the data pick-up condition for parsing the configuration file
A extraction task, and the extraction task is put into task pool；
Execution module generates phase after each extraction task execution for executing each extraction task in the task pool
The extraction data answered；
Merging module, for merging the extraction data, so that the number extracted from the physical table of corresponding identity logic table
According to merging.
Optionally, the configuration file further include:
Database where database user name password, physical table where the type of database, physical table where physical table
A point library for IP address list and port information, customized physical table divides table regular.
Optionally, the content of the configuration file is configured with label.
Optionally, the extraction task has multithreading interface.
Optionally, the extraction task in the task pool is concurrently executed according to scheduled concurrency.
It optionally,, can be to being taken out so when carrying out extraction task since each extracts the corresponding physical table of task
Node is extracted in the physics list notation taken, described to extract the number that node includes the time for extracting task execution or the physical table
According to major key.
To achieve the above object, according to a further aspect of the invention, it provides a kind of based on dynamic configuration data library
The electronic device terminal of table data pick-up is divided in point library characterized by comprising
One or more processors；
Storage device, for storing one or more programs,
When one or more of programs are executed by one or more of processors, so that one or more of processing
Device realizes that any method in table data pick-up method is divided in point library based on dynamic configuration data library.
To achieve the above object, according to a further aspect of the invention, it provides a kind of based on dynamic configuration data library
The computer-readable medium of table data pick-up is divided in point library, is stored thereon with computer program, which is characterized in that described program is located
Reason device realizes that any method in table data pick-up method is divided in point library based on dynamic configuration data library when executing.
One embodiment in foregoing invention have the following advantages that or the utility model has the advantages that because using to a point library divide table rule into
Row configurationization management, the data of physical table are extracted by multi-threaded parallel, each extracts the corresponding physical table of task, can be right
The technological means of node is extracted in the physics list notation extracted, so overcoming conventional method realizes that more complex, memory easily overflows
The technical issues of, and then reach and divide table database convenient for exploitation and maintenance in a point library, extraction efficiency is improved, certain extracts task
It unsuccessfully will not influence whole extraction task execution and facilitate the technical effect of task continued access if generating interruption.
Further effect possessed by above-mentioned non-usual optional way adds hereinafter in conjunction with specific embodiment
With explanation.
Detailed description of the invention
Attached drawing for a better understanding of the present invention, does not constitute an undue limitation on the present invention.Wherein:
Fig. 1 is that table data pick-up method is divided in an a kind of point library based on dynamic configuration data library according to an embodiment of the present invention
The schematic diagram of main flow；
Fig. 2 is that an a kind of point library based on dynamic configuration data library according to an embodiment of the present invention is divided in table data pick-up method
Mapreduce model schematic.
Fig. 3 is that table data pick-up device is divided in an a kind of point library based on dynamic configuration data library according to an embodiment of the present invention
The schematic diagram of major part；
Fig. 4 is that the embodiment of the present invention can be applied to exemplary system architecture figure therein；
Fig. 5 is adapted for the structural representation of the computer system for the terminal device or server of realizing the embodiment of the present invention
Figure.
Specific embodiment
Below in conjunction with attached drawing, an exemplary embodiment of the present invention will be described, including the various of the embodiment of the present invention
Details should think them only exemplary to help understanding.Therefore, those of ordinary skill in the art should recognize
It arrives, it can be with various changes and modifications are made to the embodiments described herein, without departing from scope and spirit of the present invention.Together
Sample, for clarity and conciseness, descriptions of well-known functions and structures are omitted from the following description.
Fig. 1 is that table data pick-up method is divided in an a kind of point library based on dynamic configuration data library according to an embodiment of the present invention
The schematic diagram of main flow, as shown in Figure 1:
Step S101 indicates generation configuration file, the corresponding relationship sum number of each physical table and logical table in the database
According to extraction condition.Wherein, the corresponding one or more physical tables of a logical table.The configuration file includes: number where physical table
According to the IP address list of database where the type in library, database user name password, physical table where physical table and port information,
A point library for customized physical table divides table regular.Such as by configuration file with xml (extensible markup language, standard generalized markup language
The subset of speech is a kind of for marking electronic document to make it have structural markup language.) document form preservation, pass through use
The rule of table is divided in a kind of bean label (component write as with java language) configuration point library, and server label configures each database
IP, the port information of server, schema label configure the information of each database, and table label configures the information of every table,
Establish logical table.
Step S102 indicates to parse the configuration file, is that each physical table generates one according to the data pick-up condition
Extraction task, and the extraction task is put into task pool.The extraction task has as realized runnable interface (one kind
Multithreading interface) multithreading interface, which includes: mission number, server number, database name, physics table name, data
(sqoop is the tool of a open source, is mainly used in Hadoop (Hive) and tradition by library user name, database password, sqoop
Database between carry out data transmitting) inquiry sql (structured query language (Structured Query Language) letter
Claim SQL), sqoop output directory.
Command Line Parsing code is as follows:
// // // // // // // // // replacement IP address, database name
Int seq=getArgsValueSeq (" -- connect ", args)；
If (seq==-1)
throw new Exception("sqoop config'--connect'is not exist！")；
}
Args [seq]=schemaConfig.getJdbcUrl (dbName)；
// // // // // // // // ///replacement user name, password.
Seq=getArgsValueSeq (" -- username ", args)；
If (seq==-1)
throw new Exception("sqoop config'--username'is not exist！")；
}
Args [seq]=schemaConfig.getUserName ()；
Seq=getArgsValueSeq (" -- password ", args)；
If (seq==-1)
throw new Exception("sqoop config'--password'is not exist！")；
}
Args [seq]=schemaConfig.getPassword ()；
// // // // // // // // // replacement table name
Seq=getArgsValueSeq (" -- query ", args)；
If (seq==-1)
throw new Exception("sqoop config'--query'is not exist！")；
}
Args [seq]=args [seq] .replace (" { sqoop:table } ", tableName)；
String tableKey=getTableKey ()；
String incrmentId=map.get (tableKey)==null? initialTime:
map.get(tableKey)；
Args [seq]=String.format (args [seq], new Object [] tableKey,
incrmentId})；
System.out.println("whereSql:"+args[seq])；
// // // // // // // ///interim the output directory of setting
Seq=getArgsValueSeq (" -- target-dir ", args)；
If (seq==-1)
throw new Exception("sqoop config'--target-dir'is not exist！")；
}
if(！args[seq].endsWith("/")){
Args [seq]=args [seq]+"/"；
}
Args [seq]=args [seq]+totalSeq；
String [] newArgs=addSeftArgs (args)；
Int flag=Sqoop.runTool (newArgs, conf)；
if(flag！=0)
throw new Exception("sqoop run error！")；
}
Step S103 is indicated to execute each extraction task in the task pool, be generated after each extraction task execution corresponding
Extraction data.Wherein the extraction task in the task pool is concurrently executed according to scheduled concurrency.Such as concurrency setting
It is 30, then it represents that while thering are the data of 30 physical tables being extracted, pass through countdownlatch (a kind of coincidence counter)
Concurrent lock control, when the wheel concurrent tasks all after the completion of, restart in next step operate；If certain task execution is interrupted, can
Node is extracted to the physics list notation extracted, the node that extracts includes the time for extracting task execution or the physics
The data major key of table.
Scheduled concurrency is provided that
CountDownLatch downLatch=new CountDownLatch (tasks.size ())；
for(SqoopTask task:tasks){
task.setCountDownLatch(downLatch)；
task.setConf(conf)；
ifNeedWait()；
executor.execute(task)；
try{
Thread.sleep(submitFrequency)；// 500 milliseconds of suspend mode of default, prevents from frequently submitting
}catch(InterruptedException e){
}
}
downLatch.await()；
executor.shutdown()；
Step S104 merges the extraction data, so that the data extracted from the physical table of corresponding identity logic table merge
Together.Such as after parsing configuration file, need to collect 1024 physical tables, then 1024 numbers can be generated after the completion of task execution
According to.
1024 outputs generated after the completion of task execution are as follows:
/user/${user.name}/input_data/tableOutputPath_temp/0
/user/${user.name}/input_data/tableOutputPath_temp/1
/user/${user.name}/input_data/tableOutputPath_temp/2
……
/user/${user.name}/input_data/tableOutputPath_temp/1023
Final output data are as follows:
/user/${user.name}/input_data/tableOutputPath
Using Mapreduce, (MapReduce is a kind of programming model, for the parallel of large-scale dataset (being greater than 1TB)
Operation) method, as shown in Fig. 2, with/user/ $ { user.name }/input_data/tableOutputPath_temp/0 etc.
1024 catalogues are as input, using :/user/ $ { user.name }/input_data/tableOutputPath catalogue as defeated
Out.
It is 1 that specification quantity, which is arranged, using the physical table name where data as data major key after mapping, such phase jljl
The data of reason table can be sent in the same specification function.Following operation is done in specification:
1. obtaining the major key of the maximum data of every physical table
In mapreduce programming model, it is assumed that using physical table name as data major key, then the data of same physical table
It can be put into a reduce function.Then it is sorted, can thus be obtained each according to data major key in each specification method
The maximum ID of physical table.When next increment extraction can the ID do increment extraction.
2. outputting data to the hdfs of final result, (HDFS is the letter of Hadoop Distribute File System
Claim, that is, a distributed file system of Hadoop) catalogue
3. saving the maximum data major key of every physical table to hdfs catalogue
Fig. 3 is that table data pick-up device is divided in an a kind of point library based on dynamic configuration data library according to an embodiment of the present invention
The schematic diagram of 300 major part, wherein
Configuration module 301, for realizing the S101 step in Fig. 1, including for generating configuration file, the configuration file
Corresponding relationship and data extraction condition including each physical table in the database and logical table, wherein a logical table pair
Answer one or more physical tables；
Parsing module 302, for realizing the S102 step in Fig. 1, including for parsing the configuration file, so that according to
The data pick-up condition is that each physical table generates an extraction task, and the extraction task is put into task pool；
Execution module 303, for realizing the S103 step in Fig. 1, including for executing each pumping in the task pool
Task is taken, corresponding extraction data are generated after each extraction task execution；
Merging module 304, for realizing the S104 step in Fig. 1, including for merging the extraction data, so that from right
The data extracted in the physical table of identity logic table are answered to merge.
Fig. 4, which is shown, can execute the method for push task using a kind of selectivity of the embodiment of the present invention or selectivity is held
The exemplary system architecture 400 of the device of row push task.
As shown in figure 4, system architecture 400 may include terminal device 401,402,403, network 404 and server 405.
Network 404 between terminal device 401,402,403 and server 405 to provide the medium of communication link.Network 404 can be with
Including various connection types, such as wired, wireless communication link or fiber optic cables etc..
User can be used terminal device 401,402,403 and be interacted by network 404 with server 405, to receive or send out
Send message etc..Various telecommunication customer end applications, such as the application of shopping class, net can be installed on terminal device 401,402,403
(merely illustrative) such as the application of page browsing device, searching class application, instant messaging tools, mailbox client, social platform softwares.
Terminal device 401,402,403 can be the various electronic equipments with display screen and supported web page browsing, packet
Include but be not limited to smart phone, tablet computer, pocket computer on knee and desktop computer etc..
Server 405 can be to provide the server of various services, such as utilize terminal device 401,402,403 to user
The shopping class website browsed provides the back-stage management server (merely illustrative) supported.Back-stage management server can be to reception
To the data such as information query request analyze etc. processing, and by processing result (such as target push information, product letter
Breath -- merely illustrative) feed back to terminal device.
It should be noted that a kind of selectively execute provided by the embodiment of the present invention pushes multitask method generally by servicing
Device 405 executes, and correspondingly, the device that selectivity executes push task is generally positioned in server 405.
It should be understood that the number of terminal device, network and server in Fig. 4 is only schematical.According to realization need
It wants, can have any number of terminal device, network and server.
Fig. 5 shows the structural representation for being suitable for the computer system 500 for the terminal device for being used to realize the embodiment of the present invention
Figure.Terminal device shown in fig. 5 is only an example, should not function to the embodiment of the present invention and use scope bring it is any
Limitation.
As shown in figure 5, computer system 500 includes central processing unit (CPU) 501, it can be read-only according to being stored in
Program in memory (ROM) 502 or be loaded into the program in random access storage device (RAM) 503 from storage section 508 and
Execute various movements appropriate and processing.In RAM 503, also it is stored with system 500 and operates required various programs and data.
CPU 501, ROM 502 and RAM 503 are connected with each other by bus 504.Input/output (I/O) interface 505 is also connected to always
Line 504.
I/O interface 505 is connected to lower component: the importation 506 including keyboard, mouse etc.；It is penetrated including such as cathode
The output par, c 507 of spool (CRT), liquid crystal display (LCD) etc. and loudspeaker etc.；Storage section 508 including hard disk etc.；
And the communications portion 509 of the network interface card including LAN card, modem etc..Communications portion 509 via such as because
The network of spy's net executes communication process.Driver 410 is also connected to I/O interface 505 as needed.Detachable media 511, such as
Disk, CD, magneto-optic disk, semiconductor memory etc. are mounted on as needed on driver 510, in order to read from thereon
Computer program be mounted into storage section 508 as needed.
Particularly, disclosed embodiment according to the present invention may be implemented as computer above with reference to the process of Fig. 1 description
Software program.For example, embodiment disclosed by the invention includes a kind of computer program product comprising be carried on computer-readable
Computer program on medium, the computer program include the program code for executing method shown in FIG. 1.In such reality
It applies in example, which can be downloaded and installed from network by communications portion 509, and/or from detachable media
511 are mounted.When the computer program is executed by central processing unit (CPU) 501, executes and limited in system of the invention
Above-mentioned function.
It should be noted that computer-readable medium shown in the present invention includes computer-readable signal media or computer
Readable storage medium storing program for executing or the two any combination.Computer readable storage medium include but is not limited to electricity, magnetic, light,
Electromagnetism, infrared ray, the system of semiconductor, any combination of device or device or above content.Computer readable storage medium
Be specifically including but not limited to: electrical connection, portable computer diskette with one or more conducting wires, hard disk, random access are deposited
Reservoir (RAM), read-only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, Portable, compact
Disk read-only memory (CD-ROM), light storage device, magnetic memory device or above content any combination.In the present invention
In, computer readable storage medium, which includes any, includes or the tangible medium of storage program, which can be commanded and execute system
System, device or device use or in connection；Computer-readable signal media includes in a base band or conduct
The data-signal that carrier wave a part is propagated, wherein carrying computer-readable program code, the data-signal of this propagation can
To take various forms, including but not limited to electromagnetic signal, any combination of optical signal or above-mentioned signal.Computer-readable letter
Number medium can also be any computer-readable medium other than computer readable storage medium, which can be with
It sends, propagate or transmits for by the use of instruction execution system, device or device or program in connection.Meter
The program code for including on calculation machine readable medium can transmit with any suitable medium, including but not limited to: wireless, electric wire,
Any combination of optical cable, RF etc. or above-mentioned medium.
Fig. 1 in attached drawing, illustrate the system according to various embodiments of the invention, method and computer program product can
The architecture, function and operation being able to achieve, each box in Fig. 1 can represent the one of a module, program segment or code
Part, a part of above-mentioned module, program segment or code, which includes that one or more is for implementing the specified logical function, to be held
Row instruction.It should be noted that in some implementations as replacements, function marked in the box can also be to be different from attached drawing
The sequence marked occurs.For example, two boxes succeedingly indicated can actually be performed in parallel, it sometimes can also be by phase
Anti- sequence executes, and execution sequence depends on the functions involved.It is also noted that each box in Fig. 1 and its
Combination, can the dedicated hardware based systems of the functions or operations as defined in executing realize, or can with it is dedicated firmly
The combination of part and computer instruction is realized.
Being described in module involved in the embodiment of the present invention can be realized by way of software, can also be by hard
The mode of part is realized.Described module also can be set in the processor, for example, can be described as: a kind of processor packet
Include configuration module, parsing module, execution module and merging module.Wherein, the title of these modules not structure under certain conditions
The restriction of the pairs of module itself, for example, merging module be also described as " for merging the extraction data so that from
The module that the data extracted in the physical table of corresponding identity logic table merge ".
On the other hand, the embodiment of the invention also provides a kind of computer-readable medium, which can be with
It is included in equipment described in above-described embodiment；It is also possible to individualism, and without in the supplying equipment.Above-mentioned meter
Calculation machine readable medium carries one or more program, when said one or multiple programs are executed by the equipment,
So that the equipment includes:
Configuration file is obtained, the configuration file includes the corresponding relationship of each physical table and logical table in the database
With data extraction condition, wherein the corresponding one or more physical tables of a logical table；
The configuration file is parsed, so that being that each physical table generates an extraction times according to the data pick-up condition
Business, and the extraction task is put into task pool；
Each extraction task in the task pool is executed, corresponding extraction data are generated after each extraction task execution；
Merge the extraction data, so that the data extracted from the physical table of corresponding identity logic table merge.
The technical solution of the embodiment of the present invention, because dividing table rule to carry out configurationization management using to a point library, by multi-thread
The data of journey paralleling abstracting physical table, are then combined with, each extracts the corresponding physical table of task, can be to the physics extracted
The technological means of node is extracted in list notation, so overcoming conventional method realizes the technical issues of more complex, memory easily overflows, into
And reach and divide table database to improve extraction efficiency convenient for exploitation and maintenance in a point library, certain failure for extracting task will not influence
It is whole to extract task execution, if generating interruption, facilitate the technical effect of task continued access.
Above-mentioned specific embodiment, does not constitute a limitation on the scope of protection of the present invention.Those skilled in the art should be bright
It is white, design requirement and other factors are depended on, various modifications, combination, sub-portfolio and substitution can occur.It is any
Made modifications, equivalent substitutions and improvements etc. within the spirit and principles in the present invention, should be included in the scope of the present invention
Within.
Claims (14)
1. table data pick-up method is divided in an a kind of point library based on dynamic configuration data library characterized by comprising
Configuration file is generated, the configuration file includes the corresponding relationship sum number of each physical table and logical table in the database
According to extraction condition；；
The configuration file is parsed, is that each physical table generates an extraction task, and handle according to the data pick-up condition
The extraction task is put into task pool；
Each extraction task in the task pool is executed, corresponding extraction data are generated after each extraction task execution；
Merge the extraction data, so that the data extracted from the physical table of corresponding identity logic table merge.
2. the method according to claim 1, wherein the configuration file further include:
The IP of database where database user name password, physical table where the type of database, physical table where physical table
A point library for location list and port information, customized physical table divides table regular.
3. according to the method described in claim 2, it is characterized in that, configuring the content of the configuration file with label.
4. the method according to claim 1, wherein the extraction task has multithreading interface.
5. the method according to claim 1, wherein the extraction task in the task pool is according to scheduled concurrent
Degree concurrently executes.
6. the method according to claim 1, wherein since each extracts the corresponding physical table of task, institute
When carrying out extraction task, can extract node to the physics list notation extracted, the extraction node includes that described extract is appointed
The data major key of time or the physical table that business executes.
7. table data pick-up device is divided in an a kind of point library based on dynamic configuration data library characterized by comprising
Configuration module, for generating configuration file, the configuration file includes each physical table and logical table in the database
Corresponding relationship and data extraction condition；
Parsing module is that each physical table generates a pumping according to the data pick-up condition for parsing the configuration file
Task is taken, and the extraction task is put into task pool；
Execution module generates corresponding for executing each extraction task in the task pool after each extraction task execution
Extract data；
Merging module, for merging the extraction data, so that the data extracted from the physical table of corresponding identity logic table are closed
And together.
8. device according to claim 7, which is characterized in that the configuration file further include:
The IP of database where database user name password, physical table where the type of database, physical table where physical table
A point library for location list and port information, customized physical table divides table regular.
9. device according to claim 8, which is characterized in that configure the content of the configuration file with label.
10. device according to claim 7, which is characterized in that the extraction task has multithreading interface.
11. device according to claim 7, which is characterized in that extraction task in the task pool according to it is scheduled simultaneously
Hair degree concurrently executes.
12. device according to claim 7, which is characterized in that since each extracts the corresponding physical table of task, institute
When carrying out extraction task, can extract node to the physics list notation extracted, the extraction node includes that described extract is appointed
The data major key of time or the physical table that business executes.
13. a kind of electronic device terminal for dividing table data pick-up based on point library in dynamic configuration data library, which is characterized in that packet
It includes:
One or more processors；
Storage device, for storing one or more programs,
When one or more of programs are executed by one or more of processors, so that one or more of processors are real
Now such as method as claimed in any one of claims 1 to 6.
14. a kind of computer-readable medium, is stored thereon with computer program, which is characterized in that described program is held by processor
Such as method as claimed in any one of claims 1 to 6 is realized when row.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710400242.3A CN108984549A (en) | 2017-05-31 | 2017-05-31 | Table data pick-up method and apparatus are divided in an a kind of point library based on dynamic configuration data library |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710400242.3A CN108984549A (en) | 2017-05-31 | 2017-05-31 | Table data pick-up method and apparatus are divided in an a kind of point library based on dynamic configuration data library |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108984549A true CN108984549A (en) | 2018-12-11 |
Family
ID=64502470
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710400242.3A Pending CN108984549A (en) | 2017-05-31 | 2017-05-31 | Table data pick-up method and apparatus are divided in an a kind of point library based on dynamic configuration data library |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108984549A (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110275913A (en) * | 2019-04-25 | 2019-09-24 | 深圳壹账通智能科技有限公司 | Data furnishing method, device and storage medium and electronic device |
| CN110780946A (en) * | 2019-10-30 | 2020-02-11 | 京东数字科技控股有限公司 | Method and apparatus for processing information |
| CN112632191A (en) * | 2020-12-29 | 2021-04-09 | 中国农业银行股份有限公司 | Data processing method and system |
| CN112632125A (en) * | 2020-12-23 | 2021-04-09 | 福信富通科技股份有限公司 | Sub-table and sub-area query method for MySQL |
| CN112685485A (en) * | 2021-01-04 | 2021-04-20 | 拉卡拉支付股份有限公司 | Data acquisition method, data acquisition device, electronic equipment, storage medium and program product |
| CN112765184A (en) * | 2021-04-07 | 2021-05-07 | 四川新网银行股份有限公司 | Real-time acquisition method based on Mysql database and table division |
| CN112860776A (en) * | 2021-01-20 | 2021-05-28 | 山东众阳健康科技集团有限公司 | Method and system for extracting and scheduling various data |
| CN113190603A (en) * | 2021-04-28 | 2021-07-30 | 中国邮政储蓄银行股份有限公司 | Data processing method, data processing device, computer readable storage medium and processor |
| CN113360470A (en) * | 2021-05-25 | 2021-09-07 | 深圳市珍爱捷云信息技术有限公司 | MySQL-based sub-library and sub-table synchronization method and device, electronic equipment and computer-readable storage medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103927331A (en) * | 2014-03-21 | 2014-07-16 | 珠海多玩信息技术有限公司 | Data querying method, data querying device and data querying system |
| CN104317960A (en) * | 2014-11-14 | 2015-01-28 | 中国建设银行股份有限公司 | Method and device for processing batch job |
| CN105183901A (en) * | 2015-09-30 | 2015-12-23 | 北京京东尚科信息技术有限公司 | Method and device for reading database table through data query engine |
| CN106095878A (en) * | 2016-06-07 | 2016-11-09 | 中国建设银行股份有限公司 | The database manipulation device and method of table is divided based on point storehouse |
| US20160334998A1 (en) * | 2015-05-15 | 2016-11-17 | Cisco Technology, Inc. | Tenant-level sharding of disks with tenant-specific storage modules to enable policies per tenant in a distributed storage system |
| CN106294565A (en) * | 2016-07-27 | 2017-01-04 | 中国农业银行股份有限公司 | A kind of data bank access method and system |
| US20170103094A1 (en) * | 2015-10-07 | 2017-04-13 | Oracle International Corporation | Request routing and query processing in a sharded database |
-
2017
- 2017-05-31 CN CN201710400242.3A patent/CN108984549A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103927331A (en) * | 2014-03-21 | 2014-07-16 | 珠海多玩信息技术有限公司 | Data querying method, data querying device and data querying system |
| CN104317960A (en) * | 2014-11-14 | 2015-01-28 | 中国建设银行股份有限公司 | Method and device for processing batch job |
| US20160334998A1 (en) * | 2015-05-15 | 2016-11-17 | Cisco Technology, Inc. | Tenant-level sharding of disks with tenant-specific storage modules to enable policies per tenant in a distributed storage system |
| CN105183901A (en) * | 2015-09-30 | 2015-12-23 | 北京京东尚科信息技术有限公司 | Method and device for reading database table through data query engine |
| US20170103094A1 (en) * | 2015-10-07 | 2017-04-13 | Oracle International Corporation | Request routing and query processing in a sharded database |
| CN106095878A (en) * | 2016-06-07 | 2016-11-09 | 中国建设银行股份有限公司 | The database manipulation device and method of table is divided based on point storehouse |
| CN106294565A (en) * | 2016-07-27 | 2017-01-04 | 中国农业银行股份有限公司 | A kind of data bank access method and system |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110275913A (en) * | 2019-04-25 | 2019-09-24 | 深圳壹账通智能科技有限公司 | Data furnishing method, device and storage medium and electronic device |
| CN110780946A (en) * | 2019-10-30 | 2020-02-11 | 京东数字科技控股有限公司 | Method and apparatus for processing information |
| CN110780946B (en) * | 2019-10-30 | 2021-05-25 | 京东数字科技控股有限公司 | Method and apparatus for processing information |
| CN112632125A (en) * | 2020-12-23 | 2021-04-09 | 福信富通科技股份有限公司 | Sub-table and sub-area query method for MySQL |
| CN112632191A (en) * | 2020-12-29 | 2021-04-09 | 中国农业银行股份有限公司 | Data processing method and system |
| CN112632191B (en) * | 2020-12-29 | 2024-06-11 | 中国农业银行股份有限公司 | Data processing method and system |
| CN112685485A (en) * | 2021-01-04 | 2021-04-20 | 拉卡拉支付股份有限公司 | Data acquisition method, data acquisition device, electronic equipment, storage medium and program product |
| CN112860776A (en) * | 2021-01-20 | 2021-05-28 | 山东众阳健康科技集团有限公司 | Method and system for extracting and scheduling various data |
| CN112860776B (en) * | 2021-01-20 | 2022-12-06 | 众阳健康科技集团有限公司 | Method and system for extracting and scheduling various data |
| CN112765184A (en) * | 2021-04-07 | 2021-05-07 | 四川新网银行股份有限公司 | Real-time acquisition method based on Mysql database and table division |
| CN113190603A (en) * | 2021-04-28 | 2021-07-30 | 中国邮政储蓄银行股份有限公司 | Data processing method, data processing device, computer readable storage medium and processor |
| CN113190603B (en) * | 2021-04-28 | 2024-09-27 | 中国邮政储蓄银行股份有限公司 | Data processing method, device, computer readable storage medium and processor |
| CN113360470A (en) * | 2021-05-25 | 2021-09-07 | 深圳市珍爱捷云信息技术有限公司 | MySQL-based sub-library and sub-table synchronization method and device, electronic equipment and computer-readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108984549A (en) | Table data pick-up method and apparatus are divided in an a kind of point library based on dynamic configuration data library | |
| US20220327125A1 (en) | Query scheduling based on a query-resource allocation and resource availability | |
| US11580107B2 (en) | Bucket data distribution for exporting data to worker nodes | |
| US11321321B2 (en) | Record expansion and reduction based on a processing task in a data intake and query system | |
| US11586627B2 (en) | Partitioning and reducing records at ingest of a worker node | |
| US11593377B2 (en) | Assigning processing tasks in a data intake and query system | |
| US11599541B2 (en) | Determining records generated by a processing task of a query | |
| US20200364223A1 (en) | Search time estimate in a data intake and query system | |
| US20190258632A1 (en) | Determining a Record Generation Estimate of a Processing Task | |
| US20200050612A1 (en) | Supporting additional query languages through distributed execution of query engines | |
| US20200065303A1 (en) | Addressing memory limits for partition tracking among worker nodes | |
| WO2020027867A1 (en) | Generating a subquery for a distinct data intake and query system | |
| CN111108479A (en) | Autonomic multi-tenant database cloud service framework | |
| CN107491382B (en) | Log output method and device | |
| CN107506218A (en) | The management method and management system of a kind of configuration file | |
| CN110765187A (en) | Data source route management method and device | |
| CN109241033A (en) | The method and apparatus for creating real-time data warehouse | |
| CN109522341A (en) | Realize method, apparatus, the equipment of the stream data processing engine based on SQL | |
| CN108958744A (en) | Dispositions method, device, medium and the electronic equipment of big data distributed type assemblies | |
| CN110019539A (en) | A kind of method and apparatus that the data of data warehouse are synchronous | |
| CN110019044A (en) | Big data cluster quasi real time Yarn Mission Monitor analysis method | |
| CN110764769B (en) | Method and device for processing user request | |
| CN112818026A (en) | Data integration method and device | |
| CN115309767A (en) | Data fusion method and device | |
| CN109960212A (en) | Task sending method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20181211 |