CN107608773A - task concurrent processing method, device and computing device - Google Patents
task concurrent processing method, device and computing device Download PDFInfo
- Publication number
- CN107608773A CN107608773A CN201710734931.8A CN201710734931A CN107608773A CN 107608773 A CN107608773 A CN 107608773A CN 201710734931 A CN201710734931 A CN 201710734931A CN 107608773 A CN107608773 A CN 107608773A
- Authority
- CN
- China
- Prior art keywords
- task
- tasks
- data
- processing
- data block
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Abstract
This specification provides a kind of task concurrent processing method, device and computing device, task can be made a distinction by dimension of data block, the task of corresponding different pieces of information block is distributed into set of tasks, the task in same set of tasks is set to correspond to different data blocks, in concurrent processing, task in same set of tasks is subjected to concurrent processing, so as to during a concurrent processing, each task is handled both for different data blocks, therefore can prevent transaction slot be occupied full caused by row lock conflict, the situation of reduction task processing failure, raising task treatment effeciency.
Description
Technical field
This specification is related to field of computer technology, more particularly to task concurrent processing method, device and computing device.
Background technology
Data block is the least unit of data base management system read-write data or most basic unit.When a task needs
When updating the data the data in block, the task needs to take the transaction slot in data block, and by the relevant information of current task
Write in transaction slot.In the case where concurrently performing the scene of multiple tasks, data block may face the processing of multiple concurrent tasks, data
Block is provided with multiple transaction slots, will occupy the transaction slot of the data block for multiple concurrent tasks of same data block, afterwards
The plurality of concurrent tasks are performed according to the sequencing occupied.If multiple transaction slots in data block are all occupied, other are needed more
The task of new data will be rejected processing.
The content of the invention
To overcome problem present in correlation technique, present description provides task concurrent processing method, device and calculating
Equipment.
A kind of task concurrent processing method, methods described include：
The multiple tasks of pending concurrent processing are determined, the task is corresponding with number to be dealt with when the task is performed
According to block；
The multiple task is distributed into one or more set of tasks, wherein, the task pair in same set of tasks
The data block answered is different；
Task in same set of tasks is subjected to concurrent processing.
Optionally, it is described to distribute the multiple task into one or more set of tasks, wherein, same set of tasks
In task corresponding to data block it is different, including：
The data block identifier of data block corresponding to the task is determined, using the data block identifier by the multiple task
Distribute into one or more set of tasks, wherein, being corresponding with identical data block identification of the task is distributed to different task-sets
In conjunction.
Optionally, the data block identifier for determining data block corresponding to the task, including：
The row address that column catalogue area stores in the data block is read, with the data object numbering in row address, data text
Part is numbered and data block number determines the data block identifier.
Optionally, the processing state recording of the multiple tasks for treating concurrent processing is in a status data table；
Methods described also includes：
After the task in same set of tasks is carried out into concurrent processing, the processing state of the task is obtained；
Delete processing state of the task before concurrent processing is carried out recorded in the status data table；
Increase processing state of the task after concurrent processing is carried out newly in the status data table.
Optionally, the data block includes the data block in oracle database management system.
A kind of task concurrent processing method, methods described include：
Pending one or more processing tasks are fished for, the processing task is corresponding with the data that need to carry out data processing
Table；
The processing task is split as multiple subtasks, determines multiple subtasks of pending concurrent processing, the son
Task is corresponding with the data block in tables of data to be dealt with when the subtask performs；
The multiple subtask is distributed into one or more set of tasks, wherein, the son in same set of tasks is appointed
Data block corresponding to business is different；
One or more of set of tasks are committed in thread pool, the subtask in same set of tasks is carried out simultaneously
Hair processing.
A kind of task concurrent processing device, methods described include：
Task determining module, is used for：The multiple tasks of pending concurrent processing are determined, the task is corresponding with the task quilt
Data block to be dealt with during execution；
Task allocating module, it is used for：The multiple task is distributed into one or more set of tasks, wherein, it is same
Data block corresponding to task in set of tasks is different；
Concurrent processing module, is used for：Task in same set of tasks is subjected to concurrent processing.
Optionally, the task allocating module, is additionally operable to：
The data block identifier of data block corresponding to the task is determined, using the data block identifier by the multiple task
Distribute into one or more set of tasks, wherein, being corresponding with identical data block identification of the task is distributed to different task-sets
In conjunction.
Optionally, the task allocating module, is additionally operable to：
The row address that column catalogue area stores in the data block is read, with the data object numbering in row address, data text
Part is numbered and data block number determines the data block identifier.
Optionally, the processing state recording of the multiple tasks for treating concurrent processing is in a status data table；
Described device also includes status data table update module, is used for：
After the task in same set of tasks is carried out into concurrent processing, the processing state of the task is obtained；
Delete processing state of the task before concurrent processing is carried out recorded in the status data table；
Increase processing state of the task after concurrent processing is carried out newly in the status data table.
Optionally, the data block includes the data block in oracle database management system.
A kind of task concurrent processing device, described device include：
Task fishes for module, is used for：Pending one or more processing tasks are fished for, the processing task, which is corresponding with, to be needed
Carry out the tables of data of data processing；
Task splits module, is used for：The processing task is split as multiple subtasks, determines pending concurrent processing
Multiple subtasks, the subtask are corresponding with the data block in tables of data to be dealt with when the subtask performs；
Task allocating module, it is used for：The multiple subtask is distributed into one or more set of tasks, wherein, together
Data block corresponding to subtask in one set of tasks is different；
Concurrent processing module, is used for：One or more of set of tasks are committed in thread pool, by same task
Subtask in set carries out concurrent processing.
A kind of computing device, including：
Processor；
For storing the memory of processor-executable instruction；
Wherein, the processor is configured as：
The multiple tasks of pending concurrent processing are determined, the task is corresponding with number to be dealt with when the task is performed
According to block；
The multiple task is distributed into one or more set of tasks, wherein, the task pair in same set of tasks
The data block answered is different；
Task in same set of tasks is subjected to concurrent processing.
A kind of computing device, including：
Processor；
For storing the memory of processor-executable instruction；
Wherein, the processor is configured as：
Pending one or more processing tasks are fished for, the processing task is corresponding with the data that need to carry out data processing
Table；
The processing task is split as multiple subtasks, determines multiple subtasks of pending concurrent processing, the son
Task is corresponding with the data block in tables of data to be dealt with when the subtask performs；
The multiple subtask is distributed into one or more set of tasks, wherein, the son in same set of tasks is appointed
Data block corresponding to business is different；
One or more of set of tasks are committed in thread pool, the subtask in same set of tasks is carried out simultaneously
Hair processing.
The technical scheme that the embodiment of this specification provides can include the following benefits：
In this specification embodiment, task can be made a distinction by dimension of data block, by corresponding different pieces of information block
Task is distributed into set of tasks, the task in same set of tasks is corresponded to different data blocks, will be same in concurrent processing
Task carries out concurrent processing in one set of tasks, so as to which during a concurrent processing, each task is both for different
Data block is handled, thus can prevent transaction slot be occupied full caused by row lock conflict, reduce task processing failure feelings
Condition, improve task treatment effeciency.
It should be appreciated that the general description and following detailed description of the above are only exemplary and explanatory, not
The application can be limited.
Brief description of the drawings
Accompanying drawing herein is merged in specification and forms the part of this specification, shows the reality for meeting this specification
Apply example, and be used for together with specification the principle for explaining this specification.
Fig. 1 is a kind of schematic diagram of data block of this specification according to an exemplary embodiment.
Fig. 2A is a kind of flow chart of task concurrent processing method of this specification according to an exemplary embodiment.
Fig. 2 B are the application scenario diagrams of task concurrent processing method of this specification according to an exemplary embodiment.
Fig. 3 A are a kind of flow charts of task concurrent processing method of this specification according to an exemplary embodiment.
Fig. 3 B are the flow charts of another task concurrent processing method of this specification according to an exemplary embodiment.
Fig. 4 is a kind of hardware structure diagram of computing device where this specification task concurrent processing device.
Fig. 5 is a kind of block diagram of task concurrent processing device of this specification according to an exemplary embodiment.
Fig. 6 is a kind of block diagram of task concurrent processing device of this specification according to an exemplary embodiment.
Embodiment
Here exemplary embodiment will be illustrated in detail, its example is illustrated in the accompanying drawings.Following description is related to
During accompanying drawing, unless otherwise indicated, the same numbers in different accompanying drawings represent same or analogous key element.Following exemplary embodiment
Described in embodiment do not represent all embodiments consistent with this specification.On the contrary, they are only and such as institute
The example of the consistent apparatus and method of some aspects be described in detail in attached claims, this specification.
It is only merely for the purpose of description specific embodiment in the term that this specification uses, and is not intended to be limiting this explanation
Book." one kind " of used singulative, " described " and "the" are also intended to bag in this specification and in the appended claims
Most forms are included, unless context clearly shows that other implications.It is also understood that term "and/or" used herein is
Refer to and any or all may be combined comprising the associated list items purpose of one or more.
It will be appreciated that though various information may be described using term first, second, third, etc. in this specification, but
These information should not necessarily be limited by these terms.These terms are only used for same type of information being distinguished from each other out.For example, do not taking off
In the case of this specification scope, the first information can also be referred to as the second information, and similarly, the second information can also be claimed
For the first information.Depending on linguistic context, word as used in this " if " can be construed to " ... when " or
" when ... " or " in response to determining ".
Some involved terms in this specification embodiment are explained first.
Data block (Data Blocks), it is the minimum storage list of data base management system (for example, oracle database)
Position, data are stored in " block ".One data block takes certain disk space.The each request data of block management data system
When, all it is in units of data block.That is, the data that block management data system is asked every time are the integral multiples of block.Such as
The data volume of fruit block management data system request is less than the data volume of a data block, and block management data system can still be read entirely
Block.It that is to say, " data block " is the least unit of block management data system read-write data or most basic unit.
The data of tables of data and the data of index are deposited in data block, no matter deposit which type of data, data block
Form (Data Block Format) is all identical, as shown in figure 1, being a kind of schematic diagram of data block, data block includes block
Head (header/Common and Variable), table directory area (table directory), column catalogue area (row
Directory), free space area (free space), row data field (row data).
Build (header/Common and Variable)：Deposit the essential information of block, including standard content and variable
Content, such as：The physical address of data block, the type (being data segment or index segment) of the section belonging to data block.
Table directory area (Table Directory)：The information of tables of data is deposited, i.e.,：If data are stored in tables of data
In this data block, then the relevant information of the table will be stored in " entry record ".
Column catalogue area (Row Directory)：If it will be remembered with the presence of row data, the information of these rows in data block
In column catalogue, these information include address of row etc. for record.
Row data field (Row Data)：The place of real stored table data and index data, this segment space be by
The space that data row takes.
Free space area (Free Space)：Free space is untapped region in a data block, and this panel region is used
In the insertion and the renewal of existing row of newline.
Wherein, one or more ITL (Interested Transaction List) affairs can be included in data block head
Groove, ITL are a parts inside data block, and for recording the affairs of all generations of the data block, an ITL can be right
A task (alternatively referred to as affairs) record is answered, if this affairs is submitted, then this ITL position can is anti-
It is multiple to use, if a task is never submitted, then this affairs will take an ITL grooves position always.If data block
Interior transaction slot is all occupied, and data block can also use the Free Space structure ITL in data block to be used for task, such as
Fruit Free Space, which also take, to be over, and follow-up task requests wait for.That is, it is assumed that there is a waiting task to need
It is related to the processing to data block, the pending affairs need to occupy an ITL of data block, only complete in the waiting task
Cheng Hou, the ITL transaction slots can be just released, and can just be occupied by other waiting tasks.It is often necessary to take appointing for transaction slot
Business includes the renewal to data in data block, and transaction slot can be not take up by deleting data and newly-increased data.
Row lock：The lock mechanism of the databases such as Oracle is a kind of locking mechanism of lightweight, and the mechanism is not to pass through structure
List is locked to carry out the lock management of data, but will directly lock the attribute as data block, is stored in the build of data block.Often
When an affairs will update the data the data in block, it is necessary to an ITL groove is first obtained, then by Current transaction ID, office
Data block address, No. SCN, Current transaction the information such as whether submit to write in ITL grooves.The affairs of ITL grooves are submitted or rollback
Afterwards, can just be covered by other affairs, can the new ITL grooves of dynamic creation one when new affairs find inadequate ITL grooves.
In the case where concurrently performing the scene of multiple tasks, data block may face the processing of multiple concurrent tasks, and data block carries
Be provided with multiple transaction slots, the transaction slot of the data block will be occupied for multiple concurrent tasks of same data block, afterwards according to
The sequencing occupied performs the plurality of concurrent tasks.If multiple transaction slots in data block are all occupied, other need to update number
Processing will be rejected according to, need to the occupying transaction slot of the task.On the other hand, due to occupying the task of transaction slot after processing is completed
The transaction slot is just released, and then other new processing tasks can take up the transaction slot, frequently right when task concurrency is huge
Data block adds row lock, discharges transaction slot, occupies the operation such as transaction slot, will be very easy to cause ITL grooves full and cause row lock conflict.
Illustrated so that processing one includes the tables of data task of multiple user profile as an example, user's letter in the tables of data
Breath reaches ten million rank, and some task is related to the modification of some attribute to the table partial user.Because the modification task relates to
And data volume it is huge, concurrent processing can be carried out by the way of the task to be split as to multiple subtasks, to improve data
The efficiency of processing.The process of concurrent processing can be：According to concurrent processing dimension (can be with user account, business paper number etc.)
Different subtasks is split as, these subtasks split will be imported into thread pool, when processing, from thread pool
Fish for multiple subtask concurrent processing.For the task in thread pool, a task data sheet can also be generated, the task data sheet
Record the processing state of each pending subtask.During each concurrent processing, subtask number can be first locked in a manner of Pessimistic Locking
According to table, then judge that subtask handles state, if untreated, then handle the task, after processing is completed, update subtask
State is to have completed.
When concurrent intensity is larger, multiple subtask data split are handled for same data block, concurrently mistake
ITL grooves may be caused to take greatly to increase, and ITL grooves have little time to discharge, and finally cause row lock conflict.In practical application, it can adopt
Increase the transaction slot in data block with adjustment ITL parameters, expand affairs tankage, while optimize application, reduce transaction slot and account for
With the mode of time.
And in this specification embodiment, task can be made a distinction using data block as dimension, by corresponding different pieces of information block
Task distribute into set of tasks, the task in same set of tasks is corresponded to different data blocks, in concurrent processing, will
Task carries out concurrent processing in same set of tasks, so as to which during a concurrent processing, each task is both for difference
Data block handled, therefore can prevent ITL grooves be occupied full caused by row lock conflict, reduce task processing failure feelings
Condition, improve task treatment effeciency.
As shown in Figure 2 A, it is the flow of task concurrent processing method of this specification according to an exemplary embodiment
Figure, methods described include：
In step 202, the multiple tasks of pending concurrent processing are determined, when the task is corresponding with the task and is performed
Data block to be dealt with.
In step 204, the multiple task is distributed into one or more set of tasks, wherein, same task-set
Data block corresponding to task in conjunction is different.
In step 206, the task in same set of tasks is subjected to concurrent processing.
As shown in Figure 2 B, it is the applied field of task concurrent processing method of this specification according to an exemplary embodiment
Jing Tu, Fig. 2 B include the computing cluster and database that multiple tasks provider includes multiple computing devices composition, this explanation
The scheme of book embodiment may be applicable in the computing cluster of carry out task concurrent processing.Task provides direction computing cluster and carried
Hand over the processing task that is operated of some tables of data in database that is directed to, computing cluster according to the process object of processing task, from
Data in the targeted tables of data of processing task are determined in database.
In some instances, computing device may face the processing task of magnanimity, and some tasks may need operation one
Or in multiple tables of data flood tide data, therefore computing cluster may concurrently perform multiple processing tasks, it is also possible to by certain
A little huge Task-decomposings for the treatment of capacity concurrently perform subtask into multiple subtasks.Selectable, computing cluster can configure
There are the computing device for being exclusively used in acquisition task, configuration management to treat that the thread pool of concurrent processing task, configuration are exclusively used in processing task
Computing device etc., it is necessary to which the task of concurrent processing is placed in thread pool, computing cluster can be according to the processing energy of reality
Power, batch fishes for multiple tasks and carries out concurrent processing from thread pool.
For the task of pending concurrent processing in this specification embodiment, the relevant information (example of task can be got
Such as task identification, the object for producing the task, the process object of task generation time or task).From described above,
The object of task processing is related to data in database, and data block is the base unit of block management data system read-write data, this
Embodiment can according to the relevant information of task, it is determined that data block to be processed corresponding to each task, afterwards using data block as
Dimension distributes task into one or more set of tasks, wherein, task corresponds to different data in same set of tasks
Block.In concurrent processing, the task in same set of tasks can be subjected to concurrent processing, it is each during due to each concurrent processing
Task corresponds to different data blocks, therefore can prevent row lock conflict caused by ITL transaction slots are stained with data block, prevents from appointing
Business processing failure.
Wherein, the multiple task is distributed into one or more set of tasks, wherein, the task in same set of tasks
Corresponding data block is different, including：
The data block identifier of data block corresponding to the task is determined, using the data block identifier by the multiple task
Distribute into one or more set of tasks, wherein, being corresponding with identical data block identification of the task is distributed to different task-sets
In conjunction.
In the present embodiment, data block identifier can be obtained, the specific acquisition modes of data block identifier can be according to actual field
Data base management system used in scape and flexibly determine.Can read the data block by taking Oralce databases as an example
The row address of middle column catalogue area storage, determined with the data object numbering in row address, data file numbering and data block number
The data block identifier.
In Oralce databases, to have rowid, rowid be in Oralce databases to record in column catalogue area in data block
Capable globally unique address, for every a line in data, the address of rowid puppet row return rows.Rowid values mainly include the row
The data block in data file where the object number of data, the row, the number in the row where the position of data block, data row
According to file.
18 are generally included in rowid, with rowid：Exemplified by OOOOOOFFFBBBBBBRRR, wherein：
OOOOOO：Represent data object numbering (occupying 6)
FFF：Represent associated data files numbering (occupying 3)
BBBBBB：Represent data block numbering (occupying 6)
RRR：Represent line number (occupying 3) in data block
Therefore, can be seen that each rowid from rowid form first 15 can be used for distinguishing different data blocks, because
This can determine the data block identifier with the data object numbering in row address, data file numbering and data block number, from
And accurately data block can be made a distinction.
In practical application, computing cluster can configure a status data table, and the multiple tasks of concurrent processing are treated for recording
Processing state, processing state includes that untreated, processing is completed or processing failure etc..Computing cluster can be according to status number
According to the processing state of each task recorded in table, if there is untreated task, fish for a part of task and out concurrently located
Reason, in processing procedure, status data table will be locked in a manner of Pessimistic Locking, after this concurrent processing terminates, according to
The processing state whether each task successfully completes, the processing state to task in status data table are updated.
It is appreciated that when being updated to status data table in the processing state of task, will also be related to status data
The processing operation of table, due to relate to update to the data in tables of data, from Such analysis, when being updated to data block
Need to occupy the ITL transaction slots of data block, the present embodiment in order to reduce may generation row lock conflict, can be by same task
After task in set carries out concurrent processing, the processing state of the task is obtained, is deleted recorded in the status data table
Processing state of the task before concurrent processing is carried out, increased newly in the status data table task carry out it is concurrent
Processing state after processing.Due to increasing the mode of a data newly in tables of data, it is not necessary to occupy ITL affairs in data block
Groove, and newly-increased data will be inserted into other data blocks, the ITL transaction slots of the data block of original logger task processing state then may be used
To be cleared, so that other tasks use.
Next this specification embodiment is illustrated again with reference to Fig. 3 A and Fig. 3 B, Fig. 3 A and Fig. 3 B are this explanations
Another task concurrent processing method of the book according to an exemplary embodiment.In practical application, it is understood that there may be following scene：
Service side maintains the user data of even hundred million grades of millions, and service side may be needed in some periods to partly or entirely using
Some data at family are updated.In such a scene, user data is stored in some tables of data, for this more new task,
Because the data volume being related to is huge, this subtask can be split as multiple subtasks by service side, to realize that rapidly data are more
Newly.Wherein, service side can be configured with computing cluster as shown in Figure 2 B, and method shown in application drawing 3A, including：
In step 302, pending one or more processing tasks are fished for, the processing task, which is corresponding with, need to enter line number
According to the tables of data of processing；
In step 304, the processing task is split as multiple subtasks, determines more height of pending concurrent processing
Task, the subtask are corresponding with the data block in tables of data to be dealt with when the subtask performs；
Within step 306, the multiple subtask is distributed into one or more set of tasks, wherein, same task
Data block corresponding to subtask in set is different；
In step 308, one or more of set of tasks are committed in thread pool, by same set of tasks
Subtask carry out concurrent processing.
In the present embodiment, for the huge processing task of data volume, process performance, data volume that can be according to physical device
The needs of size or related application scene etc. factor and processing task is split as multiple subtasks.For example, can incite somebody to action
The processing task for being related to a general-purpose family is split as 100 subtasks, and each subtask is related to the data renewal of 100 users,
After fractionation, the data block in tables of data to be processed needed for each subtask can be accordingly determined.
For all subtasks split, each subtask can be traveled through, obtains data block corresponding to each subtask
Rowid.With each rowid first 15 (data object numberings, data file numbering and data block number in row address)
As data block identifier, the subtask split is grouped using data block as dimension, subtask is distributed to multiple
In business set, in order to prevent row lock conflict, data block is different corresponding to the subtask in same set of tasks, specific task-set
The number of conjunction can determine according to being actually needed with equipment disposal ability etc..Finally, set of tasks is distributed into thread pool
In, set of tasks is fished for from thread pool by computing cluster, concurrent processing is carried out to the subtask in task-set.
In practical application, computing cluster can configure a status data table, treat that more height of concurrent processing are appointed for recording
The processing state of business, processing state include untreated, processing completion or processing failure etc..In each subtask of concurrent processing
Afterwards, if subtask is successfully processed or processing failure is, it is necessary to which the state that subtask is updated in status data table is completion or processing
Failure.Assuming that carrying out data renewal in status data table, then renewal process is related to occupying to the ITL transaction slots of data block.
And the present embodiment can be the status data for first deleting the subtask in status data table, increase one newly again afterwards and appoint for the son
The status data of business, occupying to the ITL transaction slots of data block it is not related to due to deleting and increasing newly, so as to reduce capable lock
Conflict.
As seen from the above-described embodiment, in the present embodiment, the subtask of concurrent processing is grouped according to the dimension of data block,
Same data block will not be directed to by ensureing the business of same time concurrent processing, be taken so as to reduce ITL transaction slots.On the other hand,
After subtask processing is completed, not by the way of direct more new state data table, but use and delete Added Business list again
According to mode, increasing data newly will write to new data block, and the ITL grooves of legacy data block will be available and make for other business
With so as to effectively reduce row lock conflict.
Corresponding with the embodiment of foregoing task concurrent processing method, this specification additionally provides task concurrent processing device
And its embodiment for the computing device applied.
The embodiment of this specification task concurrent processing device can be applied on the computing device, such as server or terminal
Equipment.Device embodiment can be realized by software, can also be realized by way of hardware or software and hardware combining.With software
It is by non-volatile memories by the processor of file process where it as the device on a logical meaning exemplified by realization
Corresponding computer program instructions read what operation in internal memory was formed in device.For hardware view, as shown in figure 4, being this
A kind of hardware structure diagram of computing device where specification task concurrent processing device, except the processor 410 shown in Fig. 4, interior
Deposit outside 430, network interface 420 and nonvolatile memory 440, the computing device in embodiment where device 431, lead to
Often according to the actual functional capability of the computing device, other hardware can also be included, this is repeated no more.
As shown in figure 5, Fig. 5 is a kind of task concurrent processing device of this specification according to an exemplary embodiment
Block diagram, described device include：
Task determining module 51, is used for：The multiple tasks of pending concurrent processing are determined, the task is corresponding with the task
Data block to be dealt with when being performed；
Task allocating module 52, is used for：The multiple task is distributed into one or more set of tasks, wherein, together
Data block corresponding to task in one set of tasks is different；
Concurrent processing module 53, is used for：Task in same set of tasks is subjected to concurrent processing.
Optionally, the task allocating module, is additionally operable to：
The data block identifier of data block corresponding to the task is determined, using the data block identifier by the multiple task
Distribute into one or more set of tasks, wherein, being corresponding with identical data block identification of the task is distributed to different task-sets
In conjunction.
Optionally, the task allocating module, is additionally operable to：
The row address that column catalogue area stores in the data block is read, with the data object numbering in row address, data text
Part is numbered and data block number determines the data block identifier.
Optionally, the processing state recording of the multiple tasks for treating concurrent processing is in a status data table；
Described device also includes status data table update module, is used for：
After the task in same set of tasks is carried out into concurrent processing, the processing state of the task is obtained；
Delete processing state of the task before concurrent processing is carried out recorded in the status data table；
Increase processing state of the task after concurrent processing is carried out newly in the status data table.
Optionally, the data block is the data block of oracle database management system.
As shown in fig. 6, Fig. 6 is another task concurrent processing device of this specification according to an exemplary embodiment
Block diagram, described device includes：
Task fishes for module 61, is used for：Pending one or more processing tasks are fished for, the processing task is corresponding with
The tables of data of data processing need to be carried out；
Task splits module 62, is used for：The processing task is split as multiple subtasks, determines pending concurrent processing
Multiple subtasks, the subtask be corresponding with the subtask perform when tables of data to be dealt with data block；
Task allocating module 63, is used for：The multiple subtask is distributed into one or more set of tasks, wherein,
Data block corresponding to subtask in same set of tasks is different；
Concurrent processing module 64, is used for：One or more of set of tasks are committed in thread pool, by same
Subtask in business set carries out concurrent processing.
Accordingly, this specification also provides a kind of computing device, including：
Processor；
For storing the memory of processor-executable instruction；
Wherein, the processor is configured as：
The multiple tasks of pending concurrent processing are determined, the task is corresponding with number to be dealt with when the task is performed
According to block；
The multiple task is distributed into one or more set of tasks, wherein, the task pair in same set of tasks
The data block answered is different；
Task in same set of tasks is subjected to concurrent processing.
Accordingly, this specification also provides a kind of computing device, including：
Processor；
For storing the memory of processor-executable instruction；
Wherein, the processor is configured as：
Pending one or more processing tasks are fished for, the processing task is corresponding with the data that need to carry out data processing
Table；
The processing task is split as multiple subtasks, determines multiple subtasks of pending concurrent processing, the son
Task is corresponding with the data block in tables of data to be dealt with when the subtask performs；
The multiple subtask is distributed into one or more set of tasks, wherein, the son in same set of tasks is appointed
Data block corresponding to business is different；
One or more of set of tasks are committed in thread pool, the subtask in same set of tasks is carried out simultaneously
Hair processing.
The function of modules and the implementation process of effect specifically refer to and step are corresponded in the above method in said apparatus
Implementation process, it will not be repeated here.
For device embodiment, because it corresponds essentially to embodiment of the method, so related part is real referring to method
Apply the part explanation of example.Device embodiment described above is only schematical, wherein described be used as separating component
The module of explanation can be or may not be physically separate, can be as the part that module is shown or can also
It is not physical module, you can with positioned at a place, or can also be distributed on multiple mixed-media network modules mixed-medias.Can be according to reality
Need to select some or all of module therein to realize the purpose of this specification scheme.Those of ordinary skill in the art are not
In the case of paying creative work, you can to understand and implement.
It is above-mentioned that this specification specific embodiment is described.Other embodiments are in the scope of the appended claims
It is interior.In some cases, the action recorded in detail in the claims or step can be come according to different from the order in embodiment
Perform and still can realize desired result.In addition, the process described in the accompanying drawings not necessarily require show it is specific suitable
Sequence or consecutive order could realize desired result.In some embodiments, multitasking and parallel processing be also can
With or be probably favourable.
Those skilled in the art will readily occur to this specification after considering specification and putting into practice the invention applied here
Other embodiments.This specification is intended to any modification, purposes or adaptations of this specification, these modifications,
Purposes or adaptations follow the general principle of this specification and not applied in the art including this specification
Common knowledge or conventional techniques.Description and embodiments be considered only as it is exemplary, the true scope of this specification and
Spirit is pointed out by following claim.
It should be appreciated that the precision architecture that this specification is not limited to be described above and is shown in the drawings,
And various modifications and changes can be being carried out without departing from the scope.The scope of this specification is only limited by appended claim
System.
The preferred embodiment of this specification is the foregoing is only, it is all in this explanation not to limit this specification
Within the spirit and principle of book, any modification, equivalent substitution and improvements done etc., the model of this specification protection should be included in
Within enclosing.
Claims (14)
1. a kind of task concurrent processing method, methods described include：
The multiple tasks of pending concurrent processing are determined, the task is corresponding with data to be dealt with when the task is performed
Block；
The multiple task is distributed into one or more set of tasks, wherein, corresponding to the task in same set of tasks
Data block is different；
Task in same set of tasks is subjected to concurrent processing.
2. it is according to the method for claim 1, described to distribute the multiple task into one or more set of tasks, its
In, data block corresponding to the task in same set of tasks is different, including：
The data block identifier of data block corresponding to the task is determined, is distributed the multiple task using the data block identifier
Into one or more set of tasks, wherein, being corresponding with identical data block identification of the task is distributed into different set of tasks.
3. according to the method for claim 2, the data block identifier for determining data block corresponding to the task, including：
The row address that column catalogue area stores in the data block is read, is compiled with data object numbering, the data file in row address
Number and data block number determine the data block identifier.
4. according to the method for claim 1, the processing state recording of the multiple tasks for treating concurrent processing is in a state
In tables of data；
Methods described also includes：
After the task in same set of tasks is carried out into concurrent processing, the processing state of the task is obtained；
Delete processing state of the task before concurrent processing is carried out recorded in the status data table；
Increase processing state of the task after concurrent processing is carried out newly in the status data table.
5. method according to any one of claims 1 to 4, the data block includes the number in oracle database management system
According to block.
6. a kind of task concurrent processing method, methods described include：
Pending one or more processing tasks are fished for, the processing task is corresponding with the tables of data that need to carry out data processing；
The processing task is split as multiple subtasks, determines multiple subtasks of pending concurrent processing, the subtask
It is corresponding with the data block in tables of data to be dealt with when the subtask performs；
The multiple subtask is distributed into one or more set of tasks, wherein, the subtask pair in same set of tasks
The data block answered is different；
One or more of set of tasks are committed in thread pool, the subtask in same set of tasks is concurrently located
Reason.
7. a kind of task concurrent processing device, methods described include：
Task determining module, is used for：The multiple tasks of pending concurrent processing are determined, the task is corresponding with the task and is performed
When data block to be dealt with；
Task allocating module, it is used for：The multiple task is distributed into one or more set of tasks, wherein, same task
Data block corresponding to task in set is different；
Concurrent processing module, is used for：Task in same set of tasks is subjected to concurrent processing.
8. device according to claim 7, the task allocating module, are additionally operable to：
The data block identifier of data block corresponding to the task is determined, is distributed the multiple task using the data block identifier
Into one or more set of tasks, wherein, being corresponding with identical data block identification of the task is distributed into different set of tasks.
9. device according to claim 8, the task allocating module, are additionally operable to：
The row address that column catalogue area stores in the data block is read, is compiled with data object numbering, the data file in row address
Number and data block number determine the data block identifier.
10. device according to claim 7, the processing state recording of the multiple tasks for treating concurrent processing is in a state
In tables of data；
Described device also includes status data table update module, is used for：
After the task in same set of tasks is carried out into concurrent processing, the processing state of the task is obtained；
Delete processing state of the task before concurrent processing is carried out recorded in the status data table；
Increase processing state of the task after concurrent processing is carried out newly in the status data table.
11. according to any described device of claim 7 to 10, the data block is included in oracle database management system
Data block.
12. a kind of task concurrent processing device, described device include：
Task fishes for module, is used for：Pending one or more processing tasks are fished for, the processing task, which is corresponding with, to be needed to carry out
The tables of data of data processing；
Task splits module, is used for：The processing task is split as multiple subtasks, determines the multiple of pending concurrent processing
Subtask, the subtask are corresponding with the data block in tables of data to be dealt with when the subtask performs；
Task allocating module, it is used for：The multiple subtask is distributed into one or more set of tasks, wherein, same
Data block corresponding to subtask in business set is different；
Concurrent processing module, is used for：One or more of set of tasks are committed in thread pool, by same set of tasks
In subtask carry out concurrent processing.
13. a kind of computing device, including：
Processor；
For storing the memory of processor-executable instruction；
Wherein, the processor is configured as：
The multiple tasks of pending concurrent processing are determined, the task is corresponding with data to be dealt with when the task is performed
Block；
The multiple task is distributed into one or more set of tasks, wherein, corresponding to the task in same set of tasks
Data block is different；
Task in same set of tasks is subjected to concurrent processing.
14. a kind of computing device, including：
Processor；
For storing the memory of processor-executable instruction；
Wherein, the processor is configured as：
Pending one or more processing tasks are fished for, the processing task is corresponding with the tables of data that need to carry out data processing；
The processing task is split as multiple subtasks, determines multiple subtasks of pending concurrent processing, the subtask
It is corresponding with the data block in tables of data to be dealt with when the subtask performs；
The multiple subtask is distributed into one or more set of tasks, wherein, the subtask pair in same set of tasks
The data block answered is different；
One or more of set of tasks are committed in thread pool, the subtask in same set of tasks is concurrently located
Reason.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710734931.8A CN107608773B (en) | 2017-08-24 | 2017-08-24 | Task concurrent processing method and device and computing equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710734931.8A CN107608773B (en) | 2017-08-24 | 2017-08-24 | Task concurrent processing method and device and computing equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107608773A true CN107608773A (en) | 2018-01-19 |
| CN107608773B CN107608773B (en) | 2020-08-04 |
Family
ID=61065784
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710734931.8A Active CN107608773B (en) | 2017-08-24 | 2017-08-24 | Task concurrent processing method and device and computing equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107608773B (en) |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108595251A (en) * | 2018-05-10 | 2018-09-28 | 腾讯科技（深圳）有限公司 | Dynamic Graph update method, device, storage engines interface and program medium |
| CN109492000A (en) * | 2018-10-09 | 2019-03-19 | 郑州云海信息技术有限公司 | A kind of clone method and system of database table |
| CN109800063A (en) * | 2019-01-25 | 2019-05-24 | 深圳乐信软件技术有限公司 | Business method for parallel processing, device, server, storage medium |
| CN109933606A (en) * | 2019-03-19 | 2019-06-25 | 上海达梦数据库有限公司 | A kind of database update method, apparatus, equipment and storage medium |
| CN110011832A (en) * | 2019-03-04 | 2019-07-12 | 杭州迪普科技股份有限公司 | A kind of configuration distributing method and device of plan target |
| CN110162531A (en) * | 2018-02-28 | 2019-08-23 | 山东衡昊信息技术有限公司 | A kind of distribution concurrent data processing assignment decisions method |
| CN110222018A (en) * | 2019-05-14 | 2019-09-10 | 联动优势科技有限公司 | Data summarization executes method and device |
| CN110865977A (en) * | 2019-11-05 | 2020-03-06 | 中盈优创资讯科技有限公司 | Method and device for operating HDFS directory by multiple programs |
| CN111124697A (en) * | 2020-01-02 | 2020-05-08 | 中国航空工业集团公司西安航空计算技术研究所 | Task synchronization and execution method facing distributed platform |
| WO2019228568A3 (en) * | 2019-09-12 | 2020-07-09 | Alibaba Group Holding Limited | Log-structured storage systems |
| CN111414362A (en) * | 2020-02-28 | 2020-07-14 | 平安科技（深圳）有限公司 | Data reading method, device, equipment and storage medium |
| CN112181981A (en) * | 2020-09-21 | 2021-01-05 | 苏州浪潮智能科技有限公司 | Database table data splitting method and device and storage medium |
| CN113485809A (en) * | 2021-07-14 | 2021-10-08 | 上海通联金融服务有限公司 | Method for solving problem of high time consumption of business process |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6691166B1 (en) * | 1999-01-07 | 2004-02-10 | Sun Microsystems, Inc. | System and method for transferring partitioned data sets over multiple threads |
| CN1937574A (en) * | 2005-09-19 | 2007-03-28 | 北京大学 | Network flow classifying, state tracking and message processing device and method |
| CN101556534A (en) * | 2009-04-21 | 2009-10-14 | 浪潮电子信息产业股份有限公司 | Large-scale data parallel computation method with many-core structure |
| CN102073546A (en) * | 2010-12-13 | 2011-05-25 | 北京航空航天大学 | Task-dynamic dispatching method under distributed computation mode in cloud computing environment |
| US20120310985A1 (en) * | 2011-06-01 | 2012-12-06 | Jim Gale | Systems and methods for replication replay in a relational database |
| CN104021145A (en) * | 2014-05-16 | 2014-09-03 | 华为技术有限公司 | Mixed service concurrent access method and device |
| CN105022662A (en) * | 2014-04-22 | 2015-11-04 | 中国银联股份有限公司 | Distributive task allocation method |
| CN105677903A (en) * | 2016-02-05 | 2016-06-15 | 华为技术有限公司 | Data acquisition method and device as well as computer device |
| CN105893126A (en) * | 2016-03-29 | 2016-08-24 | 华为技术有限公司 | Task scheduling method and device |
| US20170083372A1 (en) * | 2015-09-17 | 2017-03-23 | Silicon Motion, Inc. | Memory device and data read method thereof |
| CN106663028A (en) * | 2014-07-09 | 2017-05-10 | 谷歌公司 | Dynamic shard allocation adjustment |
| CN106844018A (en) * | 2015-12-07 | 2017-06-13 | 阿里巴巴集团控股有限公司 | A kind of task processing method, apparatus and system |
-
2017
- 2017-08-24 CN CN201710734931.8A patent/CN107608773B/en active Active
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6691166B1 (en) * | 1999-01-07 | 2004-02-10 | Sun Microsystems, Inc. | System and method for transferring partitioned data sets over multiple threads |
| CN1937574A (en) * | 2005-09-19 | 2007-03-28 | 北京大学 | Network flow classifying, state tracking and message processing device and method |
| CN101556534A (en) * | 2009-04-21 | 2009-10-14 | 浪潮电子信息产业股份有限公司 | Large-scale data parallel computation method with many-core structure |
| CN102073546A (en) * | 2010-12-13 | 2011-05-25 | 北京航空航天大学 | Task-dynamic dispatching method under distributed computation mode in cloud computing environment |
| US20120310985A1 (en) * | 2011-06-01 | 2012-12-06 | Jim Gale | Systems and methods for replication replay in a relational database |
| CN105022662A (en) * | 2014-04-22 | 2015-11-04 | 中国银联股份有限公司 | Distributive task allocation method |
| CN104021145A (en) * | 2014-05-16 | 2014-09-03 | 华为技术有限公司 | Mixed service concurrent access method and device |
| CN106663028A (en) * | 2014-07-09 | 2017-05-10 | 谷歌公司 | Dynamic shard allocation adjustment |
| US20170083372A1 (en) * | 2015-09-17 | 2017-03-23 | Silicon Motion, Inc. | Memory device and data read method thereof |
| CN106844018A (en) * | 2015-12-07 | 2017-06-13 | 阿里巴巴集团控股有限公司 | A kind of task processing method, apparatus and system |
| CN105677903A (en) * | 2016-02-05 | 2016-06-15 | 华为技术有限公司 | Data acquisition method and device as well as computer device |
| CN105893126A (en) * | 2016-03-29 | 2016-08-24 | 华为技术有限公司 | Task scheduling method and device |
Cited By (21)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110162531B (en) * | 2018-02-28 | 2021-04-16 | 山东衡昊信息技术有限公司 | Distributed concurrent data processing task decision method |
| CN110162531A (en) * | 2018-02-28 | 2019-08-23 | 山东衡昊信息技术有限公司 | A kind of distribution concurrent data processing assignment decisions method |
| CN108595251A (en) * | 2018-05-10 | 2018-09-28 | 腾讯科技（深圳）有限公司 | Dynamic Graph update method, device, storage engines interface and program medium |
| CN108595251B (en) * | 2018-05-10 | 2022-11-22 | 腾讯科技（深圳）有限公司 | Dynamic graph updating method, device, storage engine interface and program medium |
| CN109492000A (en) * | 2018-10-09 | 2019-03-19 | 郑州云海信息技术有限公司 | A kind of clone method and system of database table |
| CN109800063A (en) * | 2019-01-25 | 2019-05-24 | 深圳乐信软件技术有限公司 | Business method for parallel processing, device, server, storage medium |
| CN110011832A (en) * | 2019-03-04 | 2019-07-12 | 杭州迪普科技股份有限公司 | A kind of configuration distributing method and device of plan target |
| CN110011832B (en) * | 2019-03-04 | 2022-01-25 | 杭州迪普科技股份有限公司 | Configuration issuing method and device for planned tasks |
| CN109933606B (en) * | 2019-03-19 | 2021-04-27 | 上海达梦数据库有限公司 | Database modification method, device, equipment and storage medium |
| CN109933606A (en) * | 2019-03-19 | 2019-06-25 | 上海达梦数据库有限公司 | A kind of database update method, apparatus, equipment and storage medium |
| CN110222018A (en) * | 2019-05-14 | 2019-09-10 | 联动优势科技有限公司 | Data summarization executes method and device |
| WO2019228568A3 (en) * | 2019-09-12 | 2020-07-09 | Alibaba Group Holding Limited | Log-structured storage systems |
| US11423015B2 (en) | 2019-09-12 | 2022-08-23 | Advanced New Technologies Co., Ltd. | Log-structured storage systems |
| CN110865977A (en) * | 2019-11-05 | 2020-03-06 | 中盈优创资讯科技有限公司 | Method and device for operating HDFS directory by multiple programs |
| CN111124697A (en) * | 2020-01-02 | 2020-05-08 | 中国航空工业集团公司西安航空计算技术研究所 | Task synchronization and execution method facing distributed platform |
| CN111124697B (en) * | 2020-01-02 | 2023-09-08 | 中国航空工业集团公司西安航空计算技术研究所 | Task synchronization and execution method for distributed platform |
| CN111414362A (en) * | 2020-02-28 | 2020-07-14 | 平安科技（深圳）有限公司 | Data reading method, device, equipment and storage medium |
| WO2021169496A1 (en) * | 2020-02-28 | 2021-09-02 | 平安科技（深圳）有限公司 | Data reading method, apparatus, and device, and storage medium |
| CN111414362B (en) * | 2020-02-28 | 2023-11-10 | 平安科技（深圳）有限公司 | Data reading method, device, equipment and storage medium |
| CN112181981A (en) * | 2020-09-21 | 2021-01-05 | 苏州浪潮智能科技有限公司 | Database table data splitting method and device and storage medium |
| CN113485809A (en) * | 2021-07-14 | 2021-10-08 | 上海通联金融服务有限公司 | Method for solving problem of high time consumption of business process |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107608773B (en) | 2020-08-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107608773A (en) | task concurrent processing method, device and computing device | |
| JP6542785B2 (en) | Implementation of semi-structured data as first class database element | |
| EP2924594B1 (en) | Data encoding and corresponding data structure in a column-store database | |
| US9942321B2 (en) | Identity-to-account correlation and synchronization | |
| JP7133647B2 (en) | DATA PROCESSING METHOD, APPARATUS AND COMPUTER-READABLE STORAGE MEDIUM | |
| CN101329676B (en) | Data paralleling abstracting method and apparatus and database system | |
| Lin et al. | 1xn pattern for pruning convolutional neural networks | |
| US10817495B2 (en) | Systems and methods for performing data processing operations using variable level parallelism | |
| CN106528717A (en) | Data processing method and system | |
| WO2015184762A1 (en) | Database query method and device | |
| US20140101167A1 (en) | Creation of Inverted Index System, and Data Processing Method and Apparatus | |
| CA2352306A1 (en) | Method and apparatus for concurrent dbms table operations | |
| WO2014067449A1 (en) | System and method for flexible distributed massively parallel processing (mpp) database | |
| CN110688361A (en) | Data migration method, electronic device and computer equipment | |
| JP6098294B2 (en) | Information concealment device and information concealment method | |
| Lwin et al. | Non-redundant dynamic fragment allocation with horizontal partition in Distributed Database System | |
| DE112021003031T5 (en) | ARCHIVE ACCELERATOR-ONLY DATABASE TABLES | |
| KR20120082176A (en) | Data processing method of database management system and system thereof | |
| CN112037874B (en) | Distributed data processing method based on mapping reduction | |
| CN110851437A (en) | Storage method, device and equipment | |
| US11487731B2 (en) | Read iterator for pre-fetching nodes of a B-tree into memory | |
| CN107515867A (en) | The generation method and device that data storage, querying method and the device and a kind of rowKey of a kind of NoSQL databases combine entirely | |
| CN109388638B (en) | Method and system for distributed massively parallel processing of databases | |
| Vo et al. | Scaling up through parallel and distributed computing | |
| Agullo et al. | A preliminary out-of-core extension of a parallel multifrontal solver |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20200923 Address after: Cayman Enterprise Centre, 27 Hospital Road, George Town, Grand Cayman Islands Patentee after: Innovative advanced technology Co.,Ltd. Address before: Cayman Enterprise Centre, 27 Hospital Road, George Town, Grand Cayman Islands Patentee before: Advanced innovation technology Co.,Ltd. Effective date of registration: 20200923 Address after: Cayman Enterprise Centre, 27 Hospital Road, George Town, Grand Cayman Islands Patentee after: Advanced innovation technology Co.,Ltd. Address before: A four-storey 847 mailbox in Grand Cayman Capital Building, British Cayman Islands Patentee before: Alibaba Group Holding Ltd. |
|
| TR01 | Transfer of patent right |