CN108694084A - Method and apparatus for calling data - Google Patents
Method and apparatus for calling data Download PDFInfo
- Publication number
- CN108694084A CN108694084A CN201710233433.5A CN201710233433A CN108694084A CN 108694084 A CN108694084 A CN 108694084A CN 201710233433 A CN201710233433 A CN 201710233433A CN 108694084 A CN108694084 A CN 108694084A
- Authority
- CN
- China
- Prior art keywords
- task
- data
- library
- lock
- database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
This application discloses the method and apparatus for calling data.Method includes:Obtain data call task;Obtain the lock for the database that data call task is called;Obtain the quantity for the performer that each point of library is run in database;By point library of the minimum number of the performer run, it is determined as point library of execution task;Data call task is added in the task queue in point library of execution task.The method achieve multiple pending data call task cuttings to multiple logic units are carried out parallel processing, it can be by increasing the hardware device of client come task processing of raising speed, the hardware capabilities of equalization data library server end, improve the response speed for business demand.
Description
Technical field
This application involves field of computer technology, and in particular to technical field of the computer network more particularly, to calls
The method and apparatus of data.
Background technology
In today of data scale rapid expansion, application system is required for the data of processing magnanimity daily.For magnanimity knot
The processing of structure data, typically now there are two types of schemes:One is the route fields by data routing engine by record, will
Pending operation is routed to specific library table and is handled, also will point library table handled as logic list library list table it is (following
Abbreviation routing engine scheme), another scheme is the operation log by analytical database, and daily record importing distributed post is ordered
It reads message system (Kafka) and then carries out subsequent processing, or use group as distributed data base synchronization system (Otter)
Part is handled (hereinafter referred to as logging schemes).
However, routing engine scheme for processing mass data in single or minimum scale data be feasible, but
It is when encountering to need to handle mass data (for example data cleansing, data synchronize), since the topological structure to backstage library is complete
It is transparent, all databases can not be effectively utilized, batch operation can not be efficiently performed, performance much cannot be satisfied business and need
It asks.Database journal scheme can only single treatment full dose daily record, cannot achieve to the processing of the selectivity of data, handle repeatedly,
Lack the ability of the retractility demand of reply business.
In addition, since the processing of mass data generally requires to expend few then a few hours, up to a couple of days, under such circumstances,
Either routing engine scheme or database journal scheme, the monitoring power of progress and execution efficiency for current task processing
Degree it is weaker and can not adjustment algorithm task, the speed for responding business demand is slower.
Invention content
The purpose of the application is to propose a kind of improved method and apparatus for calling data, to solve background above
The technical issues of technology segment is mentioned.
In a first aspect, the embodiment of the present application provides a kind of method for calling data, method includes:Obtain data tune
Use task;Obtain the lock for the database that data call task is called;Obtain the task execution that each point of library is run in database
The quantity of person;By point library of the minimum number of the performer run, it is determined as point library of execution task;By data call
Task is added in the task queue in point library of execution task.
In some embodiments, method further includes:It, will if the task that point library of the task of execution executes in task queue finishes
Point library of execution task is discharged from local database.
In some embodiments, method further includes:Next task queue is executed not in the empty task queue for dividing library
Task.
In some embodiments, obtaining the lock of database that data call task is called includes:In correspondence database
The first transient node is generated under predefined paths;The first transient node is deleted when discharging lock.
In some embodiments, method further includes:In dividing in the execution task queue of library for the task of the task of execution, obtain
The lock of the task table to be called;If the data in task call table finish, the lock of free list.
In some embodiments, the lock of the acquisition task table to be called includes:It is generated under the preset path of corresponding table
Second transient node;The second transient node is deleted in the lock of free list.
In some embodiments, method further includes:Using distributed caching recording responses in execute in task queue appoint
The data record for carried out batch processed of being engaged in.
In some embodiments, method further includes:It is abnormal in response to executing the task in task queue, it is different in generation
After normal suspension of task, the last task recorded data record in calling task queue, recovery tasks operating status;
And/or in response to receiving the adjust instruction to data record, data record is adjusted, appointing for the data record after corresponding adjustment is executed
Business.
Second aspect, the embodiment of the present application provide a kind of device for calling data, and device includes:Task obtains single
Member, for obtaining data call task;Database lock acquiring unit, for obtaining database that data call task is called
Lock;Number obtainment unit, the quantity for obtaining the performer that each point of library is run in database;Divide library determination unit,
For by point library of the minimum number of the performer run, being determined as point library of execution task;Task adding device is used
In the task queue in point library that data call task is added to execution task.
In some embodiments, device further includes:Database releasing unit, if for execute task divide library execute task
Task in queue finishes, and point library for the task that executes is discharged from local database.
In some embodiments, device further includes:Task executing units are empty for executing next task queue not
Divide the task in the task queue in library.
In some embodiments, Database lock acquiring unit is further used for:It is raw under the predefined paths of correspondence database
At the first transient node;The first transient node is deleted when discharging lock.
In some embodiments, device further includes:The lock acquiring unit of table, for dividing library to execute task in the task of execution
When task in queue, the lock of the acquisition task table to be called;The lock releasing unit of table, if for the number in task call table
According to finishing, the lock of free list.
In some embodiments, the lock acquiring unit of table is further used for:Second is generated under the preset path of corresponding table
Transient node;The second transient node is deleted in the lock of free list.
In some embodiments, device further includes:Data record unit, for using distributed caching recording responses in holding
The data record for the batch processed that task in row task queue is carried out.
In some embodiments, device further includes:Call unit is recorded, in response to executing the task in task queue
It is abnormal, after the suspension of task being abnormal, the last task recorded data record in calling task queue is extensive
Multiple task run state;And/or task executing units, in response to receiving the adjust instruction to data record, adjusting data
Record executes the task of the data record after corresponding adjustment.
The third aspect, the embodiment of the present application provide a kind of equipment, including:One or more processors;Storage device is used
In the one or more programs of storage;When one or more programs are executed by one or more processors so that at one or more
It manages device and realizes as above any method for calling data.
Fourth aspect, the embodiment of the present application provide a kind of computer-readable storage media, store computer journey thereon
Sequence realizes as above any method for calling data when the program is executed by processor.
Method and apparatus provided by the embodiments of the present application for calling data obtain data call task first;Later,
Obtain the lock for the database that data call task is called;Later, the performer that each point of library is run in database is obtained
Quantity;Later, by point library of the minimum number of the performer run, it is determined as point library of execution task;Later, will
Data call task is added in the task queue in point library of execution task.The embodiment is by by multiple pending data tune
It, can be by increasing the hardware device of client come at the task of raising speed with task cutting to multiple logic units come parallel processing
Reason, the hardware capabilities of equalization data library server end improve the response speed for business demand.
Description of the drawings
By reading a detailed description of non-restrictive embodiments in the light of the attached drawings below, the application's is other
Feature, objects and advantages will become more apparent upon:
Fig. 1 is that this application can be applied to exemplary system architecture figures therein;
Fig. 2 is the schematic flow chart according to one embodiment of the method for calling data of the application;
Fig. 3 is the schematic flow chart according to another embodiment of the method for calling data of the application;
Fig. 4 is the schematic flow chart according to the third embodiment of the method for calling data of the application;
Fig. 5 a show the exemplary block diagram of the database according to the embodiment of the present application;
Figure 5b shows that the sides for calling data based on database shown in Fig. 5 a according to the embodiment of the present application
One schematic flow chart of method;
Fig. 6 is the exemplary block diagram according to one embodiment of the device for calling data of the application;
Fig. 7 is adapted for the structural representation of the computer system for the terminal device or server of realizing the embodiment of the present application
Figure.
Specific implementation mode
The application is described in further detail with reference to the accompanying drawings and examples.It is understood that this place is retouched
The specific embodiment stated is used only for explaining related invention, rather than the restriction to the invention.It also should be noted that in order to
Convenient for description, is illustrated only in attached drawing and invent relevant part with related.
It should be noted that in the absence of conflict, the features in the embodiments and the embodiments of the present application can phase
Mutually combination.The application is described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.
Fig. 1 shows the implementation of the method for calling data or the device for calling data that can apply the application
The exemplary system architecture 100 of example.
As shown in Figure 1, system architecture 100 may include terminal device 101,102,103, network 104 and server 105,
106.Network 104 between terminal device 101,102,103 and server 105,106 provide communication link medium.Net
Network 104 may include various connection types, such as wired, wireless communication link or fiber optic cables etc..
User 110 can be interacted by network 104 with server 105,106 with using terminal equipment 101,102,103, to connect
Receive or send message etc..Various client applications can be installed on terminal device 101,102,103, such as video playback class is answered
With the application of, search engine class, the application of shopping class, instant messaging tools, mailbox client, social platform software etc..
Terminal device 101,102,103 can be the various electronic equipments for having display screen and supporting interactive function, packet
Include but be not limited to smart mobile phone, tablet computer, E-book reader, MP3 player (Moving Picture Experts
Group Audio Layer III, dynamic image expert's compression standard audio level 3), MP4 (Moving Picture
Experts Group Audio Layer IV, dynamic image expert's compression standard audio level 4) it is player, on knee portable
Computer and desktop computer etc..
Server 105,106 can be to provide the server of various services, such as provide terminal device 101,102,103
The background server of support.Background server can carry out the processing such as analyzing based on the data collected from terminal, and processing is tied
Fruit (such as data of calling) feeds back to terminal device.
It should be noted that in the application embodiment provided for call the methods of data generally by server 105,
106 execute, correspondingly, for calling the device of data to be generally positioned in server 105,106.
It should be understood that the number of the terminal device, network and server in Fig. 1 is only schematical.According to realization need
It wants, can have any number of terminal device, network and server.
With further reference to Fig. 2, Fig. 2 shows a schematic flows according to the method for calling data of the application
Figure.
As shown in Fig. 2, this is used to call the method 200 of data include:
In step 210, data call task is obtained.
In the present embodiment, data call task refers to the task for calling data, run the application for calling
The electronic equipment (server as shown in Figure 1) of the method for data, can obtain data call task input by user or obtain
The data call task for taking application program to input.
In a step 220, the lock for the database that data call task is called is obtained.
In the present embodiment, the database that data call task is called is abbreviated as DB Group, represents a logic
The abstract concept of database, this logical data base are the set of multiple points of libraries (DB), and it includes multiple tables each to divide library again
(Table).Inside each DB Group, the business processing logic that is executed between different point library tables is consistent repeatable
's.Herein, can be between multiple physical servers, multiple threads, the application of the harmonious lock to database.
In some optional realization methods of the present embodiment, the lock packet for the database that data call task is called is obtained
It includes:The first transient node is generated under the predefined paths of correspondence database;The first transient node is deleted when discharging lock.
In this realization method, illustratively, in a distributed manner for application program coordination service (zookeeper), for
The bright lock for how obtaining the database that data call task is called.When demand is locked, in the specified circuit of zookeeper servers
Under diameter, can generate transient node can delete the transient node when release is locked.When holder's (thread) of lock is abnormal
When, zookeeper ensure that transient node can discharge automatically after session is expired.
The format of transient node can need to be arranged according to user, for example, the format of a typical transient node can be with
For:The service framework root path of project database divide library service conversation identification code+client thread name, namely:/
{ base_path }/{ DB group }/{ DB }/{ worker node }, wherein { base_path } is the zookeeper roots of project
Path is the path of DB group, DB below.Performer node (Worker node) is needed in the more of distributed environment
Keep globally unique in multiple threads of platform server, we have selected { Zookeeper session id }+{ Thread
Name } performer (worker) uniquely is identified, { zookeeper session id } is zookeeper connections
The unique identifier of client is distributed to after success, { Thread Name } is the thread name of active client, in existing customer
Internal end is also unique.
It should be appreciated that the first transient node here, only represents the transient node occurred for the first time in this application, and
The transient node is the transient node generated under the predefined paths of correspondence database, does not represent the special limit to transient node
It is fixed.
In step 230, the quantity for the performer that each point of library is run in database is obtained.
In the present embodiment, the DB started in task chooses the stage, and after locking DBGroup, algorithm passes through DB's
The length (namely quantity of performer) of worker queues carrys out the approximate load for characterizing DB.Since a worker is represented
One Table thread, this point of library table algorithm ensure that all Table are approximate equalizations, to realize one it is feasible
It is approximate.
In step 240, by point library of the minimum number of the performer run, it is determined as point of execution task
Library.
In the present embodiment, point library of the minimum number of the performer run, namely load it is minimum divide library, because
This divides the load in library for equilibrium, can divide point library of load minimum to library as execution task.
In step 250, data call task is added in the task queue in point library of execution task.
In the present embodiment, by the way that data call task to be added in the task queue in point library of execution task, to
Subsequent processing is waited for execute task according to the sequence of task queue.
The method for calling data that above-described embodiment of the application provides balanced can divide the load in library, to lead to
Cross increase client hardware device come raise speed task processing, the hardware capabilities of equalization data library server end, improve for
The response speed of business demand.
Another embodiment of the method for calling data according to the application is shown with further reference to Fig. 3, Fig. 3
Schematic flow chart.
As shown in figure 3, for calling the method 300 of data to include:
In the step 310, data call task is obtained.
In step 320, the lock for the database that data call task is called is obtained.
In a step 330, the quantity for the performer that each point of library is run in database is obtained.
In step 340, by point library of the minimum number of the performer run, it is determined as point of execution task
Library.
In step 350, data call task is added in the task queue in point library of execution task.
It should be appreciated that the step 310 in Fig. 3 is corresponding to step 250 with the step 210 in Fig. 2 respectively to step 350,
Therefore, in Fig. 2 step 310 is equally applicable to for step 210 to the described operation of step 250 and feature to step 350,
This is repeated no more.Unlike Fig. 2, in figure 3, for calling the method 300 of data further comprising the steps of:
In step 360, if the task that point library of the task of execution executes in task queue finishes, by point library for the task that executes
It is discharged from local database.
In the present embodiment, after end of operation, point library for discharging execution task is needed, avoids resource from being depleted, leads
Cause system crash.
In step 370, it is the task in the empty task queue for dividing library to execute next task queue not.
In the present embodiment, for calling the methods of data in having executed a task queue not and be empty point library of the task
After queue, it is the queue in the task queue in empty point library that can execute next task queue not.
In step 380, in dividing in the execution task queue of library for the task of the task of execution, acquisition task to be called
The lock of table;If the data in task call table finish, the lock of free list.
In the present embodiment, by the way that in dividing in the execution task queue of library for the task of the task of execution, acquisition task is wanted
The lock of the table of calling, and if the data in task call table finish, the lock of free list, can be minimized lock holds the time,
The granularity for reducing lock, reduces the competition of lock.
In some optional realization methods of the present embodiment, the lock of the acquisition task table to be called includes:In corresponding table
Preset path under generate the second transient node;The second transient node is deleted in the lock of free list.
In this realization method, the setting of transient node equally can need to be arranged according to user, and details are not described herein.
Here the second transient node only represents two different transient nodes compared with the first above-mentioned transient node, wherein the
One transient node only represents the transient node generated under the predefined paths of correspondence database, and the second transient node only represents
The transient node generated under the preset path of corresponding table.It will be appreciated by those skilled in the art that the first or second therein is not
Constitute the particular determination to transient node.
The method for calling data that above-described embodiment of the application provides, the method phase with the calling data in Fig. 2
Than, increase point library for the task that executes is discharged from local database when executing task queue and finishing, execute it is next
Task queue is not the task in the empty task queue for dividing library, and divides appointing in the execution task queue of library in the task of execution
When business, the lock of the acquisition task table to be called, if the data in task call table finish, these steps of the lock of free list, from
And can access global variable simultaneously to avoid multiple threads causes data to be destroyed, and release in time divides library, minimizes number
Hold the time according to the lock of the lock and table in library, minimizes the competition of locking.
Another embodiment of the method for calling data according to the application is shown with further reference to Fig. 4, Fig. 4
Schematic flow chart.
As shown in figure 4, for calling the method 400 of data to include:
In step 410, data call task is obtained.
At step 420, the lock for the database that data call task is called is obtained.
In step 430, the quantity for the performer that each point of library is run in database is obtained.
In step 440, by point library of the minimum number of the performer run, it is determined as point of execution task
Library.
In step 450, data call task is added in the task queue in point library of execution task.
It should be appreciated that the step 410 in Fig. 4 is corresponding to step 250 with the step 210 in Fig. 2 respectively to step 450,
Therefore, in Fig. 2 step 410 is equally applicable to for step 210 to the described operation of step 250 and feature to step 450,
This is repeated no more.Unlike Fig. 2, in Fig. 4, for calling the method 400 of data further comprising the steps of:
In step 460, the batch carried out in executing in task queue for task using distributed caching recording responses
The data record of processing.
In the present embodiment, the format of data record is:Modification time+major key.In the inside of single task, data record
It is the progress batch processed in the form of batch, a plurality of record of every batch of processing can maximize I/O in this way from the angle of I/O
Performance in this way can be with every batch of logger task state from the angle of task scheduling, can be in the promptness and minimum of state
Balance is obtained between (recording status) added cost.
In step 470, it is abnormal in response to executing the task in task queue, in the suspension of task being abnormal
Afterwards, the last task recorded data record in calling task queue, recovery tasks operating status.
In the present embodiment, by means of distributed caching technology (such as Redis to realize state recording), it can go and cause disaster
Standby scheme can be from the fast quick-recovery of batch point being abnormal after task abort.
In step 480, in response to receiving the adjust instruction to data record, the data record is adjusted, correspondence is executed
The task of data record after adjustment.
In the present embodiment, by means of distributed caching technology (such as Redis to realize state recording), due to state point
It is recorded in distributed caching, therefore can be needed to manually adjust this record point according to business, to realize inter-related task again
Scheduling skips part record etc. target, significantly increases the flexibility of task scheduling automatically.
It should be appreciated that embodiment, the only exemplary embodiment of the application shown in Fig. 4, do not represent to the application
Restriction.For example, in Fig. 4 for calling the methods 400 of data, can include not only step 470 and step 480 simultaneously,
It can only include the step 470 in step 470 and step 480, or only include the step 480 in step 470 and step 480.
The method for calling data that above-described embodiment of the application provides, the method phase with the calling data in Fig. 2
Than increasing the data note of the batch processed carried out in executing in task queue for task using distributed caching recording responses
Record is abnormal in response to executing the task in task queue, after the suspension of task being abnormal, in calling task queue
Last task recorded data record, recovery tasks operating status;And/or the adjustment of data record is referred in response to receiving
It enables, adjusts the data record, the task of data record these steps after corresponding adjustment are executed, so as to easily track
The state of task processing, is quickly responded and is adjusted to task disposition.
Below in conjunction with Fig. 5 a and Fig. 5 b, it is used to call one of the method for data to show in description according to an embodiment of the present application
Example property application scenarios.
In this application scene, Fig. 5 a show the exemplary block diagram of database used in embodiments herein.
As shown in Figure 5 a, database 500 used in embodiments herein includes:N divides library, divide library marked as
510 to 5n0;Divide in library at each, including m table, and each task handled in point library can be recorded as one
Performer.Such as the first point of library 510, including table 511 to table 51m;For n-th point of library 5n0, wherein wrapping
Table 5n0 is included to table 5nm;For database 500, one co-exists in w performer, label be respectively performer 530 to
Performer 53w.
Here database represents the abstract concept of a logical data base, the set in multiple points of libraries is represented, every
The inside of a database is different to be divided between the table of library, and the business processing logic of execution is consistent repeatable.Point library represents
The business processing logic of finer grain, from the angle of hardware resource, a point of library represents the database server of a physics
And its physical hardware resources represented.From the angle for making full use of application resource, all business processings should be rationally dispersed in
All divides in library, and parallel handle.It should be appreciated that this point of library is a logical concept, it is task scheduling entity.One
As in the case of, point library of a database can be mapped as a point of library object, but under individual cases, can also be by an object
Reason Database Mapping is multiple logical data bases, carrys out the scheduling in support logic, for example, since the algorithm in point library determines individually
Physics divides library light load, which can be at this time mapped to multiple points of libraries, in this way, dispatching algorithm can incite somebody to action
Multiple scheduler task thread loads are on an actual physical hardware.The actual treatment minimum scheduling unit of business is represented,
Actual treatment thread completes subtask in batches by one table of single treatment, in order to realize minimum scheduling cost, a table
Thread should be pasted and be handled on a point of library as far as possible, when a point library hardware resource is deposited during idle time, can pass through increase
Table Thread Count improves task treatment effeciency, until the hardware resource of database service becomes bottleneck.
Figure 5b shows that the sides for calling data based on database shown in Fig. 5 a according to the embodiment of the present application
One schematic flow chart of method.
As shown in Figure 5 b, the method for calling data based on database shown in Fig. 5 a includes the following steps:
In step 540, start to call data task, later, execute step 541;
In step 541, the corresponding database entity for calling data task is initialized, later, executes step 542;
In step 542, the lock of the corresponding database for calling data task is obtained, later, executes step 543;
In step 543, the quantity for dividing execution task person in library, namely selection database most idle in database is chosen
Minimum divides library, later, executes step 544;
In step 544, judge whether to choose successfully, if success, execute step 545, if unsuccessful, executes step 546;
In step 545, data task will be called to be registered to task queue, later, executes step 542;
In step 546, release lock executes step 547 later;
In step 547, judge to obtain whether point library succeeds, if success, thens follow the steps 548, if unsuccessful, execute
Step 550;
In step 548, processing divides the task in the task queue in library, later, executes step 549;
In step 549, after the task in the task queue in point library is disposed, release divides library.
In step 550, terminate to call data task.
It should be appreciated that the method shown in Fig. 5 a and Fig. 5 b for calling data is only the exemplary of the embodiment of the present application
Description, does not represent the restriction for the application.For example, the method for calling data shown in Fig. 5 b can also include such as Fig. 4
Middle step 460 is to the content described in step 480.
The method for calling data in this application scene, can discharge a point library, minimize the lock of database in time
And the lock of table holds the time, minimizes the competition of locking, and can be improved at task by increase table Thread Count
Manage efficiency.
With further reference to Fig. 6, as an implementation of the above method, the embodiment of the present application provides a kind of for calling data
Device one embodiment, this be used to call the embodiment of the device of data with shown in Fig. 1, Fig. 2, Fig. 3, Fig. 4 and Fig. 5
For calling the embodiment of the method for data corresponding, as a result, above with respect in Fig. 1, Fig. 2, Fig. 3, Fig. 4 and Fig. 5 for calling
The operation of the method description of data and feature, the step of being equally applicable to the device 600 for calling data and wherein include,
This is repeated no more.
As shown in fig. 6, the device 600 for being configured to calling data includes:Task acquiring unit 610, for obtaining number
According to calling task;Database lock acquiring unit 620, the lock for obtaining the database that data call task is called;Quantity obtains
Unit 630 is taken, the quantity for obtaining the performer that each point of library is run in database;Divide library determination unit 640, is used for
By point library of the minimum number of the performer run, it is determined as point library of execution task;Task adding device 650 is used
In the task queue in point library that data call task is added to execution task.
In some optional realization methods of the present embodiment, device further includes:Database releasing unit, if appointing for executing
The task that point library of business executes in task queue finishes, and point library for the task that executes is discharged from local database.
In some optional realization methods of the present embodiment, device further includes:Task executing units, it is next for executing
Task queue is not the task in the empty task queue for dividing library.
In some optional realization methods of the present embodiment, Database lock acquiring unit is further used for:In corresponding data
The first transient node is generated under the predefined paths in library;The first transient node is deleted when discharging lock.
In some optional realization methods of the present embodiment, device further includes:The lock acquiring unit of table, for appointing in execution
When library being divided to execute the task in task queue of business, the lock of the acquisition task table to be called;The lock releasing unit of table, if for
Data in task call table finish, the lock of free list.
In some optional realization methods of the present embodiment, the lock acquiring unit of table is further used for:In the pre- of corresponding table
If generating the second transient node under path;The second transient node is deleted in the lock of free list.
In some optional realization methods of the present embodiment, device further includes:Data record unit, for using distributed
The data record for the batch processed that caching record is carried out in response to the task in execution task queue.
In some optional realization methods of the present embodiment, device further includes:Call unit is recorded, in response to executing
Task in task queue is abnormal, after the suspension of task being abnormal, the last task institute in calling task queue
The data record of record, recovery tasks operating status;And/or task executing units, in response to receiving to data record
Adjust instruction adjusts data record, executes the task of the data record after corresponding adjustment.
Present invention also provides a kind of embodiments of equipment, including:One or more processors;Storage device, for storing up
Deposit one or more programs;When one or more programs are executed by one or more processors so that one or more processors
Realize as above any method for calling data.
Present invention also provides a kind of embodiments of computer-readable storage media, store computer program thereon, should
As above any method for calling data is realized when program is executed by processor.
Below with reference to Fig. 7, it illustrates the calculating suitable for terminal device or server for realizing the embodiment of the present application
The structural schematic diagram of machine system 700.Terminal device shown in Fig. 7 is only an example, should not be to the work(of the embodiment of the present application
Any restrictions can be brought with use scope.
As shown in fig. 7, computer system 700 includes central processing unit (CPU) 701, it can be read-only according to being stored in
Program in reservoir (ROM) 702 or be loaded into the program in random access reservoir (RAM) 703 from storage part 708 and
Execute various actions appropriate and processing.In RAM 703, also stores system 700 and operate required various programs and data.
CPU 701, ROM 702 and RAM 703 are connected with each other by bus 704.Input/output (I/O) interface 705 is also connected to always
Line 704.
It is connected to I/O interfaces 705 with lower component:Importation 706 including keyboard, mouse etc.;It is penetrated including such as cathode
The output par, c 707 of spool (CRT), liquid crystal display (LCD) etc. and loud speaker etc.;Storage part 708 including hard disk etc.;
And the communications portion 709 of the network interface card including LAN card, modem etc..Communications portion 709 via such as because
The network of spy's net executes communication process.Driver 710 is also according to needing to be connected to I/O interfaces 705.Detachable media 711, such as
Disk, CD, magneto-optic disk, semiconductor memory etc. are mounted on driver 710, as needed in order to be read from thereon
Computer program be mounted into as needed storage part 708.
Particularly, in accordance with an embodiment of the present disclosure, it may be implemented as computer above with reference to the process of flow chart description
Software program.For example, embodiment of the disclosure includes a kind of computer program product comprising be carried on computer-readable medium
On computer program, which includes the program code for method shown in execution flow chart.In such reality
It applies in example, which can be downloaded and installed by communications portion 709 from network, and/or from detachable media
711 are mounted.When the computer program is executed by central processing unit (CPU) 701, limited in execution the present processes
Above-mentioned function.
It should be noted that computer-readable medium described herein can be computer-readable signal media or meter
The readable storage medium of calculation machine either the two arbitrarily combines.Computer-readable storage media for example can be --- but not
Be limited to --- electricity, magnetic, optical, electromagnetic, infrared ray or semiconductor system, device or device, or arbitrary above combination.Meter
The more specific example of the readable storage medium of calculation machine can include but is not limited to:Electrical connection with one or more conducting wires, just
It takes formula computer disk, hard disk, random access reservoir (RAM), read-only memory (ROM), erasable type and may be programmed read-only storage
Device (EPROM or flash memory), optical fiber, portable compact disc read-only memory (CD-ROM), optical memory part, magnetic memory device,
Or above-mentioned any appropriate combination.In this application, can be any include computer-readable storage media or storage journey
The tangible medium of sequence, the program can be commanded the either device use or in connection of execution system, device.And at this
In application, computer-readable signal media may include in a base band or as the data-signal that a carrier wave part is propagated,
Wherein carry computer-readable program code.Diversified forms may be used in the data-signal of this propagation, including but unlimited
In electromagnetic signal, optical signal or above-mentioned any appropriate combination.Computer-readable signal media can also be that computer can
Any computer-readable medium other than storage medium is read, which can send, propagates or transmit and be used for
By instruction execution system, device either device use or program in connection.Include on computer-readable medium
Program code can transmit with any suitable medium, including but not limited to:Wirelessly, electric wire, optical cable, RF etc. or above-mentioned
Any appropriate combination.
Flow chart in attached drawing and block diagram, it is illustrated that according to the system of the various embodiments of the application, method and computer journey
The architecture, function and operation in the cards of sequence product.In this regard, each box in flowchart or block diagram can generation
A part for the part of one unit of table, program segment or code, the unit, program segment or code includes one or more uses
The executable instruction of the logic function as defined in realization.It should also be noted that in some implementations as replacements, being marked in box
The function of note can also occur in a different order than that indicated in the drawings.For example, two boxes succeedingly indicated are actually
It can be basically executed in parallel, they can also be executed in the opposite order sometimes, this is depended on the functions involved.Also it to note
Meaning, the combination of each box in block diagram and or flow chart and the box in block diagram and or flow chart can be with holding
The dedicated hardware based system of functions or operations as defined in row is realized, or can use specialized hardware and computer instruction
Combination realize.
Being described in unit involved in the embodiment of the present application can be realized by way of software, can also be by hard
The mode of part is realized.Described unit can also be arranged in the processor, for example, can be described as:A kind of processor packet
Include task acquiring unit, Database lock acquiring unit, number obtainment unit, point library determination unit and task adding device.Wherein,
The title of these units does not constitute the restriction to the unit itself under certain conditions, for example, task acquiring unit can be with
It is described as " obtaining the unit of data call task ".
Computer-readable medium, the computer-readable medium can be included in the device described in above-described embodiment
's;Can also be individualism, and without be incorporated the device in.Above computer readable medium carries one or more journey
Sequence, when said one or multiple programs are executed by the device so that the device:Obtain data call task;Obtain data
The lock for the database that calling task is called;Obtain the quantity for the performer that each point of library is run in database;It will be transported
The minimum number of capable performer divides library, is determined as point library of execution task;Data call task is added to execution
In the task queue for dividing library of task.
Above description is only the preferred embodiment of the application and the explanation to institute's application technology principle.People in the art
Member should be appreciated that invention scope involved in the application, however it is not limited to technology made of the specific combination of above-mentioned technical characteristic
Scheme, while should also cover in the case where not departing from foregoing invention design, it is carried out by above-mentioned technical characteristic or its equivalent feature
Other technical solutions of arbitrary combination and formation.Such as features described above has similar work(with (but not limited to) disclosed herein
Can technical characteristic replaced mutually and the technical solution that is formed.
Claims (18)
1. a kind of method for calling data, which is characterized in that the method includes:
Obtain data call task;
Obtain the lock for the database that the data call task is called;
Obtain the quantity for the performer that each point of library is run in the database;
By point library of the minimum number of the performer run, it is determined as point library of execution task;
The data call task is added in the task queue in point library of the execution task.
2. according to the method described in claim 1, it is characterized in that, the method further includes:
If the task that point library of the execution task executes in the task queue finishes, divide library from originally the execution task
It is discharged in the database on ground.
3. according to the method described in claim 2, it is characterized in that, the method further includes:
It is the task in the empty task queue for dividing library to execute next task queue not.
4. according to the method described in claims 1 to 3 any one, which is characterized in that described to obtain the data call task
The lock of the database called includes:
The first transient node is generated under the predefined paths of the correspondence database;
First transient node is deleted when discharging lock.
5. according to the method described in claim 1, it is characterized in that, the method further includes:
When point library of the execution task executes the task in the task queue, the table that the task to be called is obtained
Lock;
If the data in table described in the task call finish, the lock of the table is discharged.
6. according to the method described in claim 5, it is characterized in that, the lock packet for obtaining the task table to be called
It includes:
The second transient node is generated under the preset path of the correspondence table;
Second transient node is deleted when discharging the lock of the table.
7. according to the method described in claim 1-6 any one, which is characterized in that the method further includes:
Using the data note for the batch processed that distributed caching recording responses are carried out in executing in the task queue for task
Record.
8. the method according to the description of claim 7 is characterized in that the method further includes:
It is abnormal in response to executing the task in the task queue, after the suspension of task being abnormal, calls described appoint
The data record that last task in business queue is recorded, recovery tasks operating status;And/or
In response to receiving the adjust instruction to the data record, the data record is adjusted, is executed described after corresponding adjustment
The task of data record.The format of the data record is:Modification time+major key.
9. a kind of for calling the devices of data, which is characterized in that described device includes:
Task acquiring unit, for obtaining data call task;
Database lock acquiring unit, the lock for obtaining the database that the data call task is called;
Number obtainment unit, the quantity for obtaining the performer that each point of library is run in the database;
Divide library determination unit, for by point library of the minimum number of the performer run, being determined as point of execution task
Library;
Task adding device, in the task queue in point library for the data call task to be added to the execution task.
10. device according to claim 9, which is characterized in that described device further includes:
Database releasing unit finishes if point library for the execution task executes the task in the task queue, by institute
Point library for stating execution task is discharged from local database.
11. device according to claim 10, which is characterized in that described device further includes:
Task executing units are not the task in the task queue in empty point library for executing next task queue.
12. according to the device described in claim 9 to 11 any one, which is characterized in that the Database lock acquiring unit into
One step is used for:
The first transient node is generated under the predefined paths of the correspondence database;
First transient node is deleted when discharging lock.
13. device according to claim 9, which is characterized in that described device further includes:
The lock acquiring unit of table, for when point library of the execution task executes the task in the task queue, obtaining institute
The lock of the task of the stating table to be called;
The lock releasing unit of table discharges the lock of the table if the data in table described in the task call finish.
14. device according to claim 13, which is characterized in that the lock acquiring unit of the table is further used for:
The second transient node is generated under the preset path of the correspondence table;
Second transient node is deleted when discharging the lock of the table.
15. according to the device described in claim 9-14 any one, which is characterized in that described device further includes:
Data record unit, for what is carried out using task of the distributed caching recording responses in the execution task queue
The data record of batch processed.
16. device according to claim 15, which is characterized in that described device further includes:
Call unit is recorded, for being abnormal in response to executing the task in the task queue, in being abnormal for task
After suspension, the data record for calling the last task in the task queue to be recorded, recovery tasks operating status;
And/or
Task executing units, in response to receiving the adjust instruction to the data record, adjusting the data record, executing
The task of the data record after corresponding adjustment.
17. a kind of equipment, which is characterized in that including:
One or more processors;
Storage device, for storing one or more programs;
When one or more of programs are executed by one or more of processors so that one or more of processors are real
Now such as the method according to any one of claims 1-8 for calling data.
18. a kind of computer-readable storage media, stores computer program thereon, which is characterized in that the program is by processor
Such as the method according to any one of claims 1-8 for calling data is realized when execution.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710233433.5A CN108694084A (en) | 2017-04-11 | 2017-04-11 | Method and apparatus for calling data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710233433.5A CN108694084A (en) | 2017-04-11 | 2017-04-11 | Method and apparatus for calling data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108694084A true CN108694084A (en) | 2018-10-23 |
Family
ID=63843305
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710233433.5A Pending CN108694084A (en) | 2017-04-11 | 2017-04-11 | Method and apparatus for calling data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108694084A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111813559A (en) * | 2020-07-23 | 2020-10-23 | 济南大陆机电股份有限公司 | Industrial Internet distributed identification coding method and system |
| CN117032832A (en) * | 2023-08-25 | 2023-11-10 | 重庆邮电大学 | Minimum task calling cost unloading method based on mobile edge calculation |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140189695A1 (en) * | 2012-12-28 | 2014-07-03 | Futurewei Technologies, Inc. | Methods for Packet Scheduling with Order in Software-Based Parallel Processing |
| CN103997498A (en) * | 2014-05-27 | 2014-08-20 | 北京京东尚科信息技术有限公司 | Distributed lock service implementation method and component |
| CN105100259A (en) * | 2015-08-18 | 2015-11-25 | 北京京东尚科信息技术有限公司 | Distributed timed task execution method and system |
| CN105955807A (en) * | 2016-04-20 | 2016-09-21 | 上海瀚银信息技术有限公司 | System and method for processing task |
| CN106406983A (en) * | 2015-07-27 | 2017-02-15 | 阿里巴巴集团控股有限公司 | Task scheduling method and device in cluster |
-
2017
- 2017-04-11 CN CN201710233433.5A patent/CN108694084A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140189695A1 (en) * | 2012-12-28 | 2014-07-03 | Futurewei Technologies, Inc. | Methods for Packet Scheduling with Order in Software-Based Parallel Processing |
| CN103997498A (en) * | 2014-05-27 | 2014-08-20 | 北京京东尚科信息技术有限公司 | Distributed lock service implementation method and component |
| CN106406983A (en) * | 2015-07-27 | 2017-02-15 | 阿里巴巴集团控股有限公司 | Task scheduling method and device in cluster |
| CN105100259A (en) * | 2015-08-18 | 2015-11-25 | 北京京东尚科信息技术有限公司 | Distributed timed task execution method and system |
| CN105955807A (en) * | 2016-04-20 | 2016-09-21 | 上海瀚银信息技术有限公司 | System and method for processing task |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111813559A (en) * | 2020-07-23 | 2020-10-23 | 济南大陆机电股份有限公司 | Industrial Internet distributed identification coding method and system |
| CN117032832A (en) * | 2023-08-25 | 2023-11-10 | 重庆邮电大学 | Minimum task calling cost unloading method based on mobile edge calculation |
| CN117032832B (en) * | 2023-08-25 | 2024-03-08 | 重庆邮电大学 | Minimum task calling cost unloading method based on mobile edge calculation |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108182111A (en) | Task scheduling system, method and apparatus | |
| US11210131B2 (en) | Method and apparatus for assigning computing task | |
| CN108933822B (en) | Method and apparatus for handling information | |
| CN110262807A (en) | Cluster creates Progress Log acquisition system, method and apparatus | |
| CN109388626A (en) | Method and apparatus for being numbered to traffic assignments | |
| CN110377416A (en) | Distributed subregion method for scheduling task and device | |
| CN110427304A (en) | O&M method, apparatus, electronic equipment and medium for banking system | |
| CN110135925A (en) | Order processing system, method and apparatus | |
| CN110166507A (en) | More resource regulating methods and device | |
| CN109873863A (en) | The asynchronous invoking method and device of service | |
| CN111861298A (en) | Order canceling method and device | |
| CN113760488B (en) | Method, apparatus, device and computer readable medium for scheduling tasks | |
| CN110007936A (en) | Data processing method and device | |
| CN113537893A (en) | Order processing method, device, equipment and computer readable medium | |
| CN108694084A (en) | Method and apparatus for calling data | |
| CN109005208A (en) | Method and apparatus for pushed information | |
| CN111161072A (en) | Block chain-based random number generation method, equipment and storage medium | |
| CN108667872A (en) | Archiving method and device for dispatch server | |
| CN106850707A (en) | Method, apparatus and system that material batch is uploaded | |
| CN110109912A (en) | A kind of identifier generation method and device | |
| CN107045452A (en) | Dispatching method of virtual machine and device | |
| CN109614603A (en) | Method and apparatus for generating information | |
| CN110213333A (en) | Task processing method, device, electronic equipment and computer-readable medium | |
| CN111626802A (en) | Method and apparatus for processing information | |
| CN115170026A (en) | Task processing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20181023 |