CN108172268A - A kind of batch data processing method, device, terminal and storage medium - Google Patents
A kind of batch data processing method, device, terminal and storage medium Download PDFInfo
- Publication number
- CN108172268A CN108172268A CN201711443926.8A CN201711443926A CN108172268A CN 108172268 A CN108172268 A CN 108172268A CN 201711443926 A CN201711443926 A CN 201711443926A CN 108172268 A CN108172268 A CN 108172268A
- Authority
- CN
- China
- Prior art keywords
- task
- data
- data processing
- processing request
- handling result
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Abstract
The embodiment of the invention discloses a kind of batch data processing method, device, terminal and storage mediums.This method includes：Data processing request is received, wherein, data processing request includes multiple data item；At least one data group of processing data processing request is determined according to data item and default rule of classification, wherein, a data group includes at least one data item；The task of synchronous execution is determined according to configuration parameter, data component is assigned to task, wherein, a task distributes a data group；Task is performed, and determine the handling result of data processing request according to the processing type of synchronization of data processing request.So as to be effectively shortened processing time, treatment effeciency is improved.
Description
Technical field
The present embodiments relate to data processing technique more particularly to a kind of batch data processing method, device, terminal and
Storage medium.
Background technology
With the development of artificial intelligence technology and data processing technique, Clinical Decision Support Systems obtains in the field of medicine
Extensive use.Clinical Decision Support Systems is a kind of system of special adjuvant clinical diagnosis.Clinician is by by patient
Sign data is input in Clinical Decision Support Systems and is retrieved, and is goed deep into according to the retrieval result of Clinical Decision Support Systems
Analysing patient's condition makes dicision of diagnosis and treatment the most appropriate.So as to effectively solve the confinement problems of clinician's knowledge, reduce
Malpractice improves quality of medical care.
In the prior art, when being retrieved according to patient sign data, single table of generally use single database or multilist
Retrieval mode improves recall precision by database technologys means such as indexes.However, when patient sign data amount is larger
When, when being retrieved using existing retrieval mode or means, retrieval rate is slower, and recall precision is low.
Invention content
The embodiment of the present invention provides a kind of batch data processing method, device, terminal and storage medium, during shortening processing
Between, improve treatment effeciency.
In a first aspect, an embodiment of the present invention provides a kind of batch data processing method, including：
Data processing request is received, wherein, the data processing request includes multiple data item；
Determine to handle at least one data group of the data processing request according to the data item and default rule of classification,
Wherein, a data group includes at least one data item；
The task of synchronous execution is determined according to configuration parameter, the data component is assigned to the task, wherein, one is appointed
Business one data group of distribution；
The task is performed, and determine the data processing request according to the processing type of synchronization of the data processing request
Handling result.
Second aspect, the embodiment of the present invention additionally provide a kind of batch data processing unit, including：
Data processing request receiving module, for receiving data processing request, wherein, the data processing request includes more
A data item；
Data group determining module, for determining that handling the data processing asks according to the data item and default rule of classification
At least one data group asked, wherein, a data group includes at least one data item；
The data component for determining the task of synchronous execution according to configuration parameter, is assigned to institute by task determining module
Task is stated, wherein, a task distributes a data group；
Handling result determining module, for performing the task according to the processing type of synchronization of the data processing request,
And determine the handling result of the data processing request.
The third aspect, the embodiment of the present invention additionally provide a kind of terminal, including：
One or more processors；
Memory, for storing one or more programs；
When one or more of programs are performed by one or more of processors so that one or more of processing
Device realizes the batch data processing method as described in any embodiment of the present invention.
Fourth aspect, the embodiment of the present invention additionally provide a kind of computer readable storage medium, are stored thereon with computer
Program realizes the batch data processing method as described in any embodiment of the present invention when the program is executed by processor.
The embodiment of the present invention is by the way that according to default rule of classification, multiple data item of the data processing request of reception are carried out
Grouping determines at least one data group of processing data processing request, and the task of synchronous execution is determined according to configuration parameter, often
A task distributes a data group, and task is performed, and determine handling result according to the processing type of synchronization of data processing request, from
And processing time is effectively shortened using synchronous perform of task, improve treatment effeciency.
Description of the drawings
Fig. 1 is a kind of flow chart for batch data processing method that the embodiment of the present invention one provides；
Fig. 2 is a kind of flow chart of batch data processing method provided by Embodiment 2 of the present invention；
Fig. 3 is a kind of structure diagram for batch data processing unit that the embodiment of the present invention three provides；
Fig. 4 is a kind of structure diagram for terminal that the embodiment of the present invention four provides.
Specific embodiment
The present invention is described in further detail with reference to the accompanying drawings and examples.It is understood that this place is retouched
The specific embodiment stated is used only for explaining the present invention rather than limitation of the invention.It also should be noted that in order to just
Part related to the present invention rather than entire infrastructure are illustrated only in description, attached drawing.
Embodiment one
Fig. 1 is the flow chart of a kind of batch data processing method that the embodiment of the present invention one provides, and the present embodiment is applicable
In to having a case that the batch data of structuring is handled, particularly for the data in Clinical Decision Support Systems
Processing, while can be used in the application scenarios that other need to handle batch data.This method can be by lot number
It is performed according to processing unit, which can be realized by the mode of software and/or hardware, be integrated in data processing terminal.
This method specifically includes following steps：
S110, data processing request is received, wherein, data processing request includes multiple data item.
Wherein, user can input pending batch data in request page, according to the pending batch data of input
Generate data processing request.User or third-party application can be handled by the request page trigger data asks.Optionally,
Processing unit receives the pending batch data in data processing request by calling network service interface.In the present embodiment
Batch data has the characteristics that structuring, can be divided into multiple data item.Such as batch data can be a large amount of clinical sign numbers
According to being made of multiple data item.Illustratively, data item can be but not limited to body temperature, blood pressure or heart rate of patient etc..
S120, at least one data group that processing data processing request is determined according to data item and default rule of classification,
In, a data group includes at least one data item.
Wherein, the group technology that data item is defined in rule of classification is preset.According to default rule of classification, to data processing
Multiple data item in request are grouped, so that it is determined that at least one data group.It illustratively, will be in data processing request
Heart rate and blood pressure the two data item can be divided into one group, be determined as a data group or by the body in data processing request
Temperature is individually determined as a data group.The grouping situation of data item is determined by default rule of classification in the present embodiment.Optionally, it draws
Divided data group can be by being previously stored with data Groups List, which is to all numbers according to default rule of classification
It is grouped what is formed according to item, the data item in data processing request with data Groups List is matched, will be arranged with data group
At least one data item of any data group successful match is determined as a data group in table.Optionally, data group is divided may be used also
To be by identifying pre-set mark in data item, the data item with like-identified is determined as a data group.
S130, task of synchronous execution is determined according to configuration parameter, task is assigned to by data component, wherein, a task
Distribute a data group.
Wherein, configuration parameter can include but is not limited to the maximum number of tasks of synchronous execution, wherein the synchronous maximum performed
Number of tasks can be configured according to the actual conditions of processor.Illustratively, the synchronous maximum number of tasks performed can be set
The maximum number of tasks that can be handled simultaneously to be less than processor, so that the processor can also carry out while data processing is carried out
Other operations.The task quantity of synchronous execution can be determined according to configuration parameter.If specifically, synchronous execution in configuration parameter
Maximum number of tasks is more than the quantity of data group, then the quantity of data group is determined as to the task quantity of synchronous execution, is only needed at this time
The handling result of data processing request can be obtained by carrying out a subsynchronous execution.If the synchronous maximum performed in parameter is configured to appoint
Business number is less than the quantity of data group, then maximum number of tasks is determined as to the task quantity of synchronous execution, needs duty cycle at this time
It performs repeatedly, divides equally until all data groups and be assigned in task.It is performed by the cycle of task, system card can be avoided the occurrence of
The phenomenon that pausing, while improve the performance of processor.
Optionally, when without the concern for the process performance of processor, directly the quantity of data group can also be determined as
The synchronous task quantity performed so that each data group for handling data processing request is respectively assigned in corresponding task, so as to
Only by a subsynchronous execution, the handling result of data processing request can be obtained, is performed without recycling, it is simple and convenient.
Optionally, S130 includes：It determines that current sync performs according to the first quantity of configuration parameter and current data group to appoint
Second quantity of business, wherein the second quantity is less than or equal to the first quantity；The task of the second quantity is created, selects the second quantity
The data component of selection is assigned to task by data group；Correspondingly, after the completion of tasks carrying, task is destroyed.
Wherein, current data group refers to currently also unappropriated data group, and the first quantity of current data group is with execution
The change of number and change.Optionally, the first of the maximum number of tasks performed and current data group is synchronized according in configuration parameter
Quantity determines that current sync performs the second quantity of task.If the synchronous maximum number of tasks performed is less than the first of current data group
Maximum number of tasks is then determined as the second quantity of current sync execution task by quantity；If the synchronous maximum number of tasks performed is big
In the first quantity of current data group, then the first quantity of current data group is determined as to the second number of current sync execution task
Amount does not exceed maximum number of tasks to ensure that current sync performs the second quantity of task, the phenomenon that so as to avoid the occurrence of interim card,
And improve process performance.The task of the second quantity the second quantity of establishment of task is performed according to determining current sync.This reality
It, can be in current data group if applying first quantity of second quantity less than current data group that current sync in example performs task
In randomly select the data group of the second quantity.The data group of selection can be randomly assigned into the task of establishment, and is ensured every
A task distributes a data group.
It should be noted that after current task synchronization has performed, detect whether there is also unappropriated data group, if
Be, then using the current data group of also unappropriated data group subsynchronous execution as under, and will also not according to above-mentioned same procedure
The data component of distribution, which is assigned in task, to be continued with.If it is not, then destroy the task of establishment, to discharge memory space,
It avoids causing the wasting of resources.
Illustratively, it is assumed that determining data group is 15, and the synchronous maximum number of tasks performed is 8, then for first
For subsynchronous execution, determine that the second quantity that current sync performs task is 8 according to the first quantity 15 of current data group, and
8 tasks are created, randomly selected 8 data components are assigned in 8 tasks from 15 current data groups, each task distribution
One data group.After the completion of the first subsynchronous execution, also 7 data groups are also unallocated, so needing to carry out second
It is synchronous to perform.For the second subsynchronous execution, determine that current sync performs task according to the first quantity 7 of current data group
The second quantity for 7, this 7 data groups are only distributed into 7 tasks.After the completion of second subsynchronous execution, it is not present
Also unappropriated data group need to only be destroyed the task of establishment at this time.
S140, task is performed according to the processing type of synchronization of data processing request, and determines the processing of data processing request
As a result.
Wherein, the processing type of data processing request can be but not limited to retrieve and store.According to processing type of synchronization
Each task is performed, the task handling result returned according to each task determines the handling result of data processing request.Passing through will
Batch data is distributed to multiple tasks, and synchronous execution multiple tasks, so as to be effectively shortened processing time, in same hardware
In the case of resource, the ratio that processing time shortens is determined by the task quantity performed.
By the shortening of data processing time quickly number can be provided for decision support in Clinical Decision Support Systems
According to support, so as to quickly medical staff be helped to carry out medical decision making.The shortening of data processing time is in the severe field of medicine
In seem even more important.
Optionally, S140 includes：The corresponding data storage knot of data item in each task is determined according to preset mapping relation
Point, and task processing is carried out in data storage node according to the processing type of data processing request, obtain appointing for each task
Business handling result；
According to the task handling result of each task, the handling result of data processing request is determined.
Wherein, data storage node is the data cell for storing data item, can be database list row formula, can also
For self contained data base.Preset mapping relation refers to the mapping relations between data item and data storage node, wherein, a number
A data storage node is only corresponded to according to item, a data storage node can correspond to a data item, can also correspond to multiple
Data item.Preset mapping relation can be stored in a manner of configuration file or database table in the present embodiment.It is predetermined to reflect
The relationship of penetrating can be indicated in a manner of asterisk wildcard.Illustratively, blood * corresponding datas storage node 1, then it represents that using blood to open
The equal corresponding data storage node 1 of data item of head.Each data item in each task can be determined according to preset mapping relation
Corresponding data storage node, and data processing is carried out to data item in corresponding data storage node according to processing type.
Each task can obtain corresponding task handling result after the completion of the processing of data storage node.When in data processing request
All data item handled in corresponding data storage node after the completion of, data are determined according to all task handling results
The handling result of request is handled, so as to complete the processing of batch data.The ratio that processing time shortens is according to the number of tasks of execution
Amount and the quantity of data storage node determine.Optionally, after the handling result of data processing request is determined, by the handling result
User terminal is sent to, with to user's display processing result.
Optionally, task processing is carried out in data storage node according to the processing type of data processing request, obtained every
The task handling result of a task, including：
If handling type as storage, the data item in each task is stored into corresponding data storage node；And
When storing successfully, the storage successful information of generation is determined as task handling result；
If handling type as retrieval, the data item in each task is examined in corresponding data storage node
Rope, and the retrieval data of return are determined as task handling result.
Wherein, when handling type as storage, each data item in the data group of task distribution is deposited in corresponding data
When storage node is stored, if each data item stores successfully, the storage successful information of generation is determined as at task
Manage result.If any data item storage failure caused by the reasons such as not connected network generates storage failure information, and will generation
Storage failure information and corresponding data item be determined as the task handling result of the task.It optionally, can be according to storage
The data item of failure regenerates data processing request, is stored again.When handling type as retrieval, the number of task distribution
It is retrieved in corresponding data storage node, and the data item that will be stored in data storage node according to the data item in group
Data as retrieval data.If there are at least two data item in the task, and data item each in the task is corresponding
Retrieval data merge, and the retrieval data after merging are back in the task, the task handling result as the task.
Optionally, according to the task handling result of each task, the handling result of data processing request is determined, including：
If handling type as storage, when the task handling result of each task is storage successful information, will store
Successful information is determined as the handling result of data processing request；
If handling type as retrieval, the retrieval data of each task are merged, retrieval data set is determined, will retrieve
Data set is determined as the handling result of data processing request.
Wherein, it when handling type as storage, if being to store the task of failure information there are task handling result, will deposit
Handling result of the data item of failure information and all storage failures as data processing request is stored up, so that user is according to storage
The data item of failure re-starts storage.When handling type as retrieval, since the data item of each task is different, so each
The retrieval data of task are also different.The retrieval data of each task are subjected to simple merging and obtain retrieval data set, this is retrieved
Handling result of the data set as data processing request.
The embodiment of the present invention is divided multiple data item of the data processing request of reception according to default rule of classification
Group determines at least one data group of processing data processing request, and the task of synchronous execution is determined according to configuration parameter, each
Task distributes a data group, and task is performed, and determine handling result according to the processing type of synchronization of data processing request, so as to
Processing time is effectively shortened using synchronous perform of multiple tasks, improves treatment effeciency.
Embodiment two
Fig. 2 is a kind of flow chart of batch data processing method provided by Embodiment 2 of the present invention, and the present embodiment is above-mentioned
It is optimized on the basis of embodiment：The task of synchronous execution is determined according to configuration parameter, data component is assigned to task, is wrapped
It includes：Obtain task pool in can application task third quantity, wherein, can application task be idle state task；According to
First quantity of three quantity and current data group determine current sync perform task the 4th quantity, wherein the 4th quantity be less than or
Equal to third quantity, and the 4th quantity is less than or equal to the first quantity；Choose the 4th quantity can application task, and select the 4th
The data group of quantity, by the data component of selection be assigned to selection can application task.
After optimization, a kind of batch data processing method specifically includes：
S210, data processing request is received, wherein, data processing request includes multiple data item.
S220, at least one data group that processing data processing request is determined according to data item and default rule of classification,
In, a data group includes at least one data item.
S230, obtain task pool in can application task third quantity, wherein, can application task for idle state appoint
Business.
Wherein, a task pool is pre-created according to the maximum number of tasks performed synchronous in configuration parameter.Wherein, task pool
In task quantity can be equal to or less than maximum number of tasks, to ensure that the task quantity performed not exceed maximum task
Number, avoids the occurrence of Caton phenomenon.There are one status indicators to represent current state for each task.The state of task can be sky
Not busy state, operating status or closed state.When the state of task is in off state, represent that the task is recycling destruction,
To reduce the task quantity in task pool, the memory space of task pool is reduced, so as to improve the speed of service.Can application task refer to
It is can the distributing data item of the task.The task of idle state can be used as can application task, by the task quantity of idle state
Be determined as can application task third quantity.
S240, determine that current sync performs the 4th number of task according to the first quantity of third quantity and current data group
Amount, wherein the 4th quantity is less than or equal to third quantity, and the 4th quantity is less than or equal to the first quantity.
Wherein, if can the third quantity of application task be less than first quantity of current data group, by can application task
Third quantity is determined as the 4th quantity that current sync performs task.If can application task third quantity be more than current data group
The first quantity, then by the first quantity of current data group be determined as current sync perform task the 4th quantity.
S250, choose the 4th quantity can application task, and select the 4th quantity data group, by the data component of selection
Be assigned to selection can application task.
Wherein, if current sync perform task the 4th quantity for can application task third quantity, choose task pool
In it is all can application task as synchronous performing for task, and randomly choose in current data group the data of the 4th quantity
Group, with ensure one can application task distribute a data group.If the 4th quantity that current sync performs task is current data
First quantity of group then randomly selects the task of idle state of the 4th quantity as synchronous performing for task in task pool.
Optionally, the data component of selection is being assigned to can further including after application task for selection：
By selection can the state of application task be set as operating status；
Correspondingly, after the completion of the operation task of operating status performs, the state of operation task is set as idle state.
Wherein, selection can after application task distributes the data group of selection one by one, can application task state from idle shape
State is set as operating status.After the completion of the tasks carrying of operating status, the operating status of operation task is set as idle shape
State, and put back in task pool, it is chosen when receiving data processing request so as to next time.Building by task pool in the present embodiment
It is vertical, the problem of being required to re-create and destroy task when receiving data processing request every time is avoided, further shortens place
The time is managed, improves treatment effeciency.
S260, task is performed according to the processing type of synchronization of data processing request, and determines the processing of data processing request
As a result.
It should be noted that in currently can application task perform after the completion of, if also being repeated there are unappropriated data group
Step S230 to S260 is performed to be assigned to unappropriated data component and can continue to execute in application task, it is unallocated until being not present
Data group when stop.When there is no during unappropriated data group, represent that handling data processing request completes, and data processing is asked
The corresponding handling result of each data item in asking merges, and is determined as the handling result of data processing request.
The embodiment of the present invention determines by pre-establishing task pool, according to the state of task in task pool with data group synchronous
Perform can application task, the place of task and determining data processing request is performed according to the processing type of synchronization of data processing request
Manage result.The present embodiment places back in task pool after the completion of tasks carrying, by task, so that the data processing for handling next time please
It asks, avoids the problem of being required to re-create and destroy task when receiving data processing request every time, further shorten place
The time is managed, improves treatment effeciency.
Embodiment three
Fig. 3 is the structure diagram of a kind of batch data processing unit that the embodiment of the present invention three provides, and the present embodiment can
Suitable for having a case that the batch data of structuring is handled, the structure of the device specifically includes：Data processing request
Receiving module 310, data group determining module 320, task determining module 330 and handling result determining module 340.
Wherein, data processing request receiving module 310, for receiving data processing request, wherein, data processing request packet
Include multiple data item；Data group determining module 320, for determining that processing data processing please according to data item and default rule of classification
At least one data group asked, wherein, a data group includes at least one data item；Task determining module 330, for basis
Configuration parameter determines the task of synchronous execution, and data component is assigned to task, wherein, a task distributes a data group；Place
Result determining module 340 is managed, for performing task according to the processing type of synchronization of data processing request, and determines that data processing please
The handling result asked.
Optionally, task determining module 330, including：
Second quantity determination unit, for determining that current sync is held according to the first quantity of configuration parameter and current data group
Second quantity of row task, wherein the second quantity is less than or equal to the first quantity；
Task creation unit for creating the task of the second quantity, selects the data group of the second quantity, by the data of selection
Component is assigned to task；
Correspondingly, task determining module 330 further includes：
Task destroys unit, for after the completion of tasks carrying, destroying task.
Optionally, task determining module 330, including：
Third number obtainment unit, for obtain in task pool can application task third quantity, wherein, can apply appoint
Being engaged in as idle state for task；
4th quantity determination unit, for determining that current sync is held according to the first quantity of third quantity and current data group
4th quantity of row task, wherein the 4th quantity is less than or equal to third quantity, and the 4th quantity is less than or equal to the first quantity；
Can application task selection unit, for choose the 4th quantity can application task, and select the 4th quantity data
Group, by the data component of selection be assigned to selection can application task.
Optionally, which further includes：
First task state setting module, for the data component of selection is assigned to selection can after application task,
By selection can the state of application task be set as operating status；
Correspondingly, the device further includes：
Second task status setup module, after the completion of being performed in the operation task of operating status, by operation task
State is set as idle state.
Optionally, handling result determining module 340, including：
Data storage node determination unit, for determining the corresponding number of data item in each task according to preset mapping relation
According to storage node；
Task handling result acquiring unit, for according to the processing type of data processing request in data storage node into
The processing of row task obtains the task handling result of each task；
Handling result determination unit for the task handling result according to each task, determines the place of data processing request
Manage result.
Optionally, task handling result acquiring unit, is specifically used for：
If handling type as storage, the data item in each task is stored into corresponding data storage node；And
When storing successfully, the storage successful information of generation is determined as task handling result；
If handling type as retrieval, the data item in each task is examined in corresponding data storage node
Rope, and the retrieval data of return are determined as task handling result.
Optionally, handling result determination unit is specifically used for：
If handling type as storage, when the task handling result of each task is storage successful information, will store
Successful information is determined as the handling result of data processing request；
If handling type as retrieval, the retrieval data of each task are merged, retrieval data set is determined, will retrieve
Data set is determined as the handling result of data processing request.
The said goods can perform the batch data processing method that any embodiment of the present invention is provided, and have execution lot number
According to the corresponding function module of processing method and advantageous effect.
Example IV
Fig. 4 is a kind of structure diagram for terminal that the embodiment of the present invention four provides.Referring to Fig. 4, which includes：
One or more processors 410；
Memory 420, for storing one or more programs；
When one or more programs are performed by one or more processors 410 so that one or more processors 410 are realized
Such as the batch data processing method that any embodiment proposes in above-described embodiment.
In Fig. 4 by taking a processor 410 as an example；Processor 410 and memory 420 in terminal can by bus or its
He connects mode, in Fig. 4 for being connected by bus.
Memory 420 is used as a kind of computer readable storage medium, and journey is can perform available for storage software program, computer
Sequence and module, if the corresponding program instruction/module of batch data processing method in the embodiment of the present invention is (for example, lot number
According to data processing request receiving module 310, data group determining module 320, task determining module 330 and the processing in processing unit
As a result determining module 340).Processor 410 is stored in software program, instruction and module in memory 420 by operation, from
And perform various function application and the data processing of terminal, that is, realize above-mentioned batch data processing method.
Memory 420 mainly includes storing program area and storage data field, wherein, storing program area can store operation system
Application program needed for system, at least one function；Storage data field can be stored uses created data etc. according to terminal.This
Outside, memory 420 can include high-speed random access memory, can also include nonvolatile memory, for example, at least one
Disk memory, flush memory device or other non-volatile solid state memory parts.In some instances, memory 420 can be into one
Step is included relative to the remotely located memory of processor 410, these remote memories can pass through network connection to terminal.On
The example for stating network includes but not limited to internet, intranet, LAN, mobile radio communication and combinations thereof.
The batch data processing method that the terminal that the present embodiment proposes is proposed with above-described embodiment belongs to same inventive concept,
The technical detail of detailed description not can be found in above-described embodiment in the present embodiment, and the present embodiment has execution batch data
The identical advantageous effect of processing method.
Embodiment five
The present embodiment provides a kind of computer readable storage mediums, are stored thereon with computer program, which is handled
The batch data processing method as described in any embodiment of the present invention is realized when device performs.
By the description above with respect to embodiment, it is apparent to those skilled in the art that, the present invention
It can be realized by software and required common hardware, naturally it is also possible to which by hardware realization, but the former is more in many cases
Good embodiment.Based on such understanding, what technical scheme of the present invention substantially in other words contributed to the prior art
Part can be embodied in the form of software product, which can be stored in computer readable storage medium
In, floppy disk, read-only memory (Read-Only Memory, ROM), random access memory (Random such as computer
Access Memory, RAM), flash memory (FLASH), hard disk or CD etc., including some instructions with so that a computer is set
Standby (can be personal computer, server or the network equipment etc.) performs the method described in each embodiment of the present invention.
Note that it above are only presently preferred embodiments of the present invention and institute's application technology principle.It will be appreciated by those skilled in the art that
The present invention is not limited to specific embodiment here, various apparent variations, again can be carried out for a person skilled in the art
Adjustment and replacement are without departing from protection scope of the present invention.Therefore, although the present invention has been carried out by above example compared with
For detailed description, but the present invention is not limited only to above example, without departing from the inventive concept, can be with
Including other more equivalent embodiments, and the scope of the present invention is determined by scope of the appended claims.
Claims (10)
1. a kind of batch data processing method, which is characterized in that including：
Data processing request is received, wherein, the data processing request includes multiple data item；
Determine to handle at least one data group of the data processing request according to the data item and default rule of classification,
In, a data group includes at least one data item；
The task of synchronous execution is determined according to configuration parameter, the data component is assigned to the task, wherein, a task point
With a data group；
The task is performed, and determine the place of the data processing request according to the processing type of synchronization of the data processing request
Manage result.
2. according to the method described in claim 1, it is characterized in that, the task of synchronous execution is determined according to configuration parameter, by institute
It states data component and is assigned to the task, including：
Determine that current sync performs the second quantity of task according to the first quantity of configuration parameter and current data group, wherein described
Second quantity is less than or equal to first quantity；
The task of second quantity is created, the data group of the second quantity is selected, the data component of selection is assigned to the task；
Correspondingly, after the completion of the tasks carrying, the task is destroyed.
3. according to the method described in claim 1, it is characterized in that, the task of synchronous execution is determined according to configuration parameter, by institute
It states data component and is assigned to the task, including：
Obtain task pool in can application task third quantity, wherein, it is described can application task be idle state task；
Determine that current sync performs the 4th quantity of task according to the first quantity of the third quantity and current data group, wherein
4th quantity is less than or equal to the third quantity, and the 4th quantity is less than or equal to first quantity；
Choose the 4th quantity can application task, and the data group of the 4th quantity is selected, by the data component of selection
Be assigned to selection can application task.
4. according to the method described in claim 3, it is characterized in that, appoint the data component of selection is assigned to applying for selection
After business, further include：
By selection can the state of application task be set as operating status；
Correspondingly, after the completion of the operation task of the operating status performs, the state of the operation task is set as idle
State.
5. according to the method described in claim 1, it is characterized in that, the processing type of synchronization according to the data processing request is held
The row task, and determine the handling result of the data processing request, including：
The corresponding data storage node of data item in each task is determined, and according to the data processing according to preset mapping relation
The processing type of request carries out task processing in the data storage node, obtains the task handling result of each task；
According to the task handling result of each task, the handling result of the data processing request is determined.
6. according to the method described in claim 5, it is characterized in that, according to the processing type of the data processing request described
Task processing is carried out in data storage node, obtains the task handling result of each task, including：
If handling type as storage, the data item in each task is stored into corresponding data storage node；And in depositing
When storing up successfully, the storage successful information of generation is determined as task handling result；
If handling type as retrieval, the data item in each task is retrieved in corresponding data storage node,
And the retrieval data of return are determined as task handling result.
7. according to the method described in claim 6, it is characterized in that, according to the task handling result of each task, determine described
The handling result of data processing request, including：
If handling type as storage, when the task handling result of each task is storage successful information, by the storage
Successful information is determined as the handling result of the data processing request；
If handling type as retrieval, the retrieval data of each task are merged, retrieval data set are determined, by the retrieval
Data set is determined as the handling result of the data processing request.
8. a kind of batch data processing unit, which is characterized in that including：
Data processing request receiving module, for receiving data processing request, wherein, the data processing request includes multiple numbers
According to item；
Data group determining module, for determining to handle the data processing request according to the data item and default rule of classification
At least one data group, wherein, a data group includes at least one data item；
The data component for determining the task of synchronous execution according to configuration parameter, is assigned to described appoint by task determining module
Business, wherein, a task distributes a data group；
Handling result determining module, for performing the task according to the processing type of synchronization of the data processing request, and really
The handling result of the fixed data processing request.
9. a kind of terminal, which is characterized in that the terminal includes：
One or more processors；
Memory, for storing one or more programs；
When one or more of programs are performed by one or more of processors so that one or more of processors are real
The now batch data processing method as described in any in claim 1-7.
10. a kind of computer readable storage medium, is stored thereon with computer program, which is characterized in that the program is by processor
The batch data processing method as described in any in claim 1-7 is realized during execution.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711443926.8A CN108172268A (en) | 2017-12-27 | 2017-12-27 | A kind of batch data processing method, device, terminal and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711443926.8A CN108172268A (en) | 2017-12-27 | 2017-12-27 | A kind of batch data processing method, device, terminal and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108172268A true CN108172268A (en) | 2018-06-15 |
Family
ID=62518223
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711443926.8A Pending CN108172268A (en) | 2017-12-27 | 2017-12-27 | A kind of batch data processing method, device, terminal and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108172268A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110162532A (en) * | 2019-05-09 | 2019-08-23 | 中国银行股份有限公司 | Transaction data processing method and equipment |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103678303A (en) * | 2012-08-31 | 2014-03-26 | 国际商业机器公司 | Method and system used for group set inquiry of database |
| CN103810223A (en) * | 2012-11-15 | 2014-05-21 | 中国科学院软件研究所 | Data grouping-based memory data organization query method |
| CN104657212A (en) * | 2015-02-13 | 2015-05-27 | 北京亚信数据有限公司 | Task scheduling method and system |
| CN105224289A (en) * | 2014-07-03 | 2016-01-06 | 阿里巴巴集团控股有限公司 | A kind of action message matching process and equipment |
| CN105308579A (en) * | 2013-07-01 | 2016-02-03 | 株式会社日立制作所 | Series data parallel analysis infrastructure and parallel distributed processing method therefor |
-
2017
- 2017-12-27 CN CN201711443926.8A patent/CN108172268A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103678303A (en) * | 2012-08-31 | 2014-03-26 | 国际商业机器公司 | Method and system used for group set inquiry of database |
| CN103810223A (en) * | 2012-11-15 | 2014-05-21 | 中国科学院软件研究所 | Data grouping-based memory data organization query method |
| CN105308579A (en) * | 2013-07-01 | 2016-02-03 | 株式会社日立制作所 | Series data parallel analysis infrastructure and parallel distributed processing method therefor |
| CN105224289A (en) * | 2014-07-03 | 2016-01-06 | 阿里巴巴集团控股有限公司 | A kind of action message matching process and equipment |
| CN104657212A (en) * | 2015-02-13 | 2015-05-27 | 北京亚信数据有限公司 | Task scheduling method and system |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110162532A (en) * | 2019-05-09 | 2019-08-23 | 中国银行股份有限公司 | Transaction data processing method and equipment |
| CN110162532B (en) * | 2019-05-09 | 2021-06-04 | 中国银行股份有限公司 | Transaction data processing method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11416307B2 (en) | System and method for processing task resources | |
| US20080030764A1 (en) | Server parallel aggregation | |
| CN108683516A (en) | A kind of upgrade method of application example, device and system | |
| CN105049268A (en) | Distributed computing resource allocation system and task processing method | |
| CN111880936B (en) | Resource scheduling method, device, container cluster, computer equipment and storage medium | |
| CN111061432B (en) | Service migration method, device, equipment and readable storage medium | |
| CN108491163B (en) | Big data processing method and device and storage medium | |
| CN112241316A (en) | Method and device for distributed scheduling application | |
| CN111695876A (en) | Approval process processing method, device, equipment and storage medium | |
| CN109241037A (en) | A kind of method, apparatus that realizing database processing, equipment and storage medium | |
| WO2018079162A1 (en) | Information processing system | |
| CN111163140A (en) | Method, apparatus and computer readable storage medium for resource acquisition and allocation | |
| CN108172268A (en) | A kind of batch data processing method, device, terminal and storage medium | |
| CN112948077A (en) | Batch processing method, device, equipment and storage medium | |
| CN111506388B (en) | Container performance detection method, container management platform and computer storage medium | |
| CN116881012A (en) | Container application vertical capacity expansion method, device, equipment and readable storage medium | |
| CN108647090B (en) | Resource allocation method and device and server cluster | |
| CN106598726A (en) | Multi-task management system and distributed deployment method thereof | |
| CN110188258A (en) | The method and device of external data is obtained using crawler | |
| CN111767345B (en) | Modeling data synchronization method, modeling data synchronization device, computer equipment and readable storage medium | |
| CN114924888A (en) | Resource allocation method, data processing method, device, equipment and storage medium | |
| CN109617954B (en) | Method and device for creating cloud host | |
| CN111209118A (en) | Method and device for determining resource allowance, storage medium and electronic equipment | |
| CN111858234A (en) | Task execution method, device, equipment and medium | |
| CN109522124A (en) | Storage management system loading method, device, equipment and readable storage medium storing program for executing |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180615 |
|
| RJ01 | Rejection of invention patent application after publication |