CN108038215A - Data processing method and system - Google Patents
Data processing method and system Download PDFInfo
- Publication number
- CN108038215A CN108038215A CN201711401731.7A CN201711401731A CN108038215A CN 108038215 A CN108038215 A CN 108038215A CN 201711401731 A CN201711401731 A CN 201711401731A CN 108038215 A CN108038215 A CN 108038215A
- Authority
- CN
- China
- Prior art keywords
- operator
- server
- master server
- data processing
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
Abstract
The embodiment of the invention discloses a kind of data processing method and system.Wherein, data processing method includes：When master server receives data processing request input by user, the executive plan tree being made of at least one operator is generated based on the data processing request；Master server according to executive plan tree when performing operation corresponding with each operator, if it is judged that the operation corresponding to active operator meets preset condition, then operator and the corresponding data sending of AND operator are subjected to data processing at least one secondary server；Master server receives handling result of the secondary server to data, and carries out integration processing to handling result according to executive plan tree.Above-mentioned technical proposal solves the problems, such as to lift data-handling capacity and dilatation is complicated time-consuming by dilatation in available data processing method, optimizes the data processing method of existing database, realize to the elasticity processing of database data and efficient process.
Description
Technical field
The present embodiments relate to technical field of data processing, more particularly to a kind of data processing method and system.
Background technology
The operation of many relational databases, such as sorts, classification polymerize, connection needs consume substantial amounts of memory and CPU money
Source, when data volume is very big, the disposal ability of unit cannot be met the requirements.
Traditional solution is to use MPP cluster, i.e., so-called database MPP
(massively parallel processing, MPP) cluster, its core concept are：Data by certain mode, such as
Data are distributed on each node of cluster according to the hashed value of some fields in table, cooperate with work by multiple nodes during execution
Make, realize labyrinth query language (Structured Query Language, SQL) function under mass data.But
It is that the shortcomings that MPP clusters is that framework is dumb, increases or to reduce node all extremely difficult.Because it is related to the weight of data
New distribution, when data volume reaches TB grade, redistribution data are a processes taken very much, and on-line rapid estimation also seriously affects pair
It is outer that lasting service ability is provided.
For shared storage data-base cluster (typical such as Oracle RAC), increase new node needs complicated configuration
Journey, the database administrator (Database Administrator, DBA) that cluster upgrade usually requires specialty are supported to avoid latent
Risk, nor the disposal ability of single complexity SQL can be lifted.
With the rise of internet, big data etc., in order to solve the problems, such as the dilatation of database, many distributed data base sides
Case is suggested, and such as the NOSQL Database Systems based on KEY/VALUE models and point storehouse based on message-oriented middleware divide table model
Deng.These models require the fundamental characteristics some original relational databases, such as complexity SQL capability improvings to application layer,
And its dilatation is also more complicated primarily directed to the capacity of database, its dilation process.
The content of the invention
An embodiment of the present invention provides a kind of data processing method and system, solves existing database data processing method
In by dilatation to lift data-handling capacity and complicated time-consuming dilatation the problem of, with realize to the data in database
Efficient process.
In a first aspect, an embodiment of the present invention provides a kind of data processing method, this method includes：
When master server receives data processing request input by user, generated based on the data processing request by extremely
The executive plan tree of few operator composition；
The master server according to the executive plan tree when performing operation corresponding with each operator, if the master
Server judges that the operation corresponding to active operator meets preset condition, then the master server is by the active operator
And data sending corresponding with the active operator carries out data processing at least one secondary server；
The master server receives handling result of the secondary server to the data, and according to the executive plan
Tree carries out integration processing to the handling result.
Further, in master server by the operator and data sending corresponding with the operator at least one
Before a secondary server carries out data processing, further include：
Server calls request is sent to registration module by master server, and receive registration module feedback with it is described
Server calls ask the configuration information of corresponding at least one secondary server；Wherein, the registration module includes main service
Device or registrar.
Further, the method further includes：
When secondary server starts, the secondary server sends registration request to registration module；
Registration module receives the registration request, and stores the server in the secondary server inventory of current active
Configuration information.
Further, the operation judged corresponding to active operator meets preset condition, including：
The complexity of operation of the master server according to corresponding to each operator determines each object run symbol, if current behaviour
When work symbol accords with for the object run, then judge that the operation corresponding to active operator meets preset condition.
Further, the complexity of operation of the master server according to corresponding to each operator determines each object run
Symbol, including：
Master server obtains the active operator in the executive plan tree, and estimates the active operator institute in advance
The corresponding required target resource of operation；Wherein, the operation that the target resource is included corresponding to each operator is required
Committed memory and/or execution time；
If the target resource exceedes default resource occupation threshold value, master server will with the target resource corresponding to
Operator as object run accord with.
Second aspect, the embodiment of the present invention additionally provide a kind of data handling system, which includes：Master server and auxiliary
Help server；Wherein, the master server includes：
Operator acquisition module, for when receiving data processing request input by user, based on the data processing
Request to generate the executive plan tree being made of at least one operator；
Operation judges module, for according to the executive plan tree perform operation corresponding with each operator when, if
Judge that the operation corresponding to active operator meets preset condition, then by the active operator and with the current operation
Accord with corresponding data sending and carry out data processing at least one secondary server；
As a result module is integrated, is held for receiving handling result of the secondary server to the data, and according to described
Row plan tree carries out integration processing to the handling result.
Further, the system further includes：Registration module and calling module；Wherein, the calling module, is configured at
In the master server, in master server by the operator and data sending corresponding with the operator at least
Before one secondary server carries out data processing, server calls request is sent to the registration mould by the master server
Block, and receive the configuration of at least one secondary server corresponding with server calls request of the registration module feedback
Information；The registration module includes master server or registrar.
Further, the system further includes：
Registration request transmitting element, is configured in the secondary server, for when secondary server starts, to registration
Module sends registration request；
Configuration information storage unit, is configured in the registration module, lives for receiving the registration request, and currently
The configuration information of the server is stored in dynamic secondary server inventory.
Further, the operation judges module is specifically used for：
The complexity of operation according to corresponding to each operator determines each object run symbol, if active operator is institute
When stating object run symbol, then judge that the operation corresponding to active operator meets preset condition.
Further, the operation judgment device is specifically additionally operable to：
The active operator in the executive plan tree is obtained, and estimates the behaviour corresponding to the active operator in advance
Make required target resource；Wherein, the target resource includes the required committed memory of operation corresponding to each operator
And/or perform the time；
If the target resource exceedes default resource occupation threshold value, master server will with the target resource corresponding to
Operator as object run accord with.
The technical solution of the embodiment of the present invention, by being generated when master server according to the data processing request received by one
The executive plan tree of a or multiple operator compositions, the data manipulation that can be met with a response corresponding to the data processing request, into
And master server is according to the executive plan tree when performing operation corresponding with each operator, when judging active operator institute
When corresponding operation meets preset condition, then operator and the corresponding data sending of AND operator are carried out to secondary server
Data processing, needs to carry out data transmission between each server, master server only exists different from existing distributed system
Interacted when needing with secondary server, and secondary server only needs to receive data and operator to be treated, it is main
Server can select one to be performed in unison with a data processing to several secondary servers as needed, receive the auxiliary afterwards
Server carries out integration processing according to executive plan tree to the handling results of data to the handling result, to complete whole
Data processing, solve in existing database data processing method and operated by dilatation to lift data-handling capacity and dilatation
The problem of complicated and time consumption, realize to the elasticity processing of database data and efficient process.
Brief description of the drawings
In order to clearly illustrate the technical solution of exemplary embodiment of the present, below to required in description embodiment
The attached drawing to be used does a simple introduction.Obviously, the attached drawing introduced is the part of the embodiment of the invention to be described
Attached drawing, rather than whole attached drawings, for those of ordinary skill in the art, without creative efforts, may be used also
To obtain other attached drawings according to these attached drawings.
A kind of flow diagram for data processing method that Fig. 1 embodiment of the present invention one is provided；
Fig. 2 is a kind of flow diagram for data processing method that the embodiment of the present invention two is provided；
Fig. 3 is a kind of flow diagram for data processing method that the embodiment of the present invention three is provided；
Fig. 4 is a kind of structure diagram for data handling system that the embodiment of the present invention four is provided；
Fig. 5 is a kind of structure diagram for terminal that the embodiment of the present invention four is provided.
Embodiment
The present invention is described in further detail with reference to the accompanying drawings and examples.It is understood that this place is retouched
The specific embodiment stated is used only for explaining the present invention, rather than limitation of the invention.It also should be noted that in order to just
It illustrate only part related to the present invention rather than entire infrastructure in description, attached drawing.
Embodiment one
Fig. 1 is a kind of flow diagram for data processing method that the embodiment of the present invention one is provided, and the present embodiment can fit
For the situation for handling the data in database, it is particularly suitable for carrying out the data in database complicated data fortune
The situation of calculation, this method can be performed by data handling system.
As shown in Figure 1, the method for the present embodiment specifically includes：
S110, when master server receives data processing request input by user, based on the data processing request give birth to
Into the executive plan tree being made of at least one operator.
Wherein, master server can be a node in unit database or data-base cluster.It is exemplary
Ground, operator specifically may include a group operator, attended operation symbol, scan operation symbol, filter operation symbol, sorting operation symbol and throw
Shadow operator etc..
Typically, data processing request input by user can be based on structured query language (Structured Query
Language, SQL) etc. database language realize.SQL is the operation commands set for aiming at database and establishing.Work as master server
When have received the sentence of data processing request input by user, master server can generate executive plan, which includes
The step of being performed to realize the data processing request and operation.I.e. master server receives user's SQL data processing requests
Afterwards, the processing such as syntax and semantics analysis can be carried out to data processing request, obtains processing corresponding with the data processing request
Process, and the association between each step in processing procedure and each step, generate and are held by what multiple operators formed
Row plan tree.Specifically, the read-only tree data structure that executive plan tree can be made of several SQL operators.Wherein,
Each node of executive plan tree is each operator, and association of the set membership between each node between each operator is closed
System.It should be noted that system can create corresponding performing environment before performing according to the executive plan generated.
Alternatively, it can be specifically later root time to perform operation corresponding with each operator according to the executive plan tree
Go through order and perform the corresponding operation of each operator in the executive plan tree.
S120, the master server according to the executive plan tree perform operation corresponding with each operator when, if
The master server judges that the operation corresponding to active operator meets preset condition, then the master server is by the operation
Symbol and data sending corresponding with the operator carry out data processing at least one secondary server.
Some node in unit Database Systems or data-base cluster, in the process of implementation, if it find that current behaviour
Make to need substantial amounts of memory and cpu resource, and there are available idle computing resources in current network, just current operation is split
Go to hold to these utilizable computers into multiple tasks, and these tasks and data distribution corresponding with these tasks
OK.
Alternatively, judge that the operation corresponding to active operator meets that preset condition includes：Master server is according to each behaviour
The complexity for according with corresponding operation determines each object run symbol, if active operator accords with for the object run,
Then judge that the operation corresponding to active operator meets preset condition.Alternatively, master server can be according to each operator
During generation plan tree, the complexity of the operation corresponding to a operator is estimated, and then by the higher operator of complexity
Accorded with as object run.
In order to improve the efficiency of data processing, system can just pre-set each type of operation when system starts
The resource predetermined threshold value of symbol, or resource preset value only is equipped with to some species of operator, for example, sorting operation symbol at most accounts for
With how many million memories etc..When traveling through executive plan tree and going to some operator, according to the current number to be processed of the operator
According to statistical informations such as amounts, estimation, which performs the corresponding operation of the operator, needs resource to be used, with the default money of this operator
Source takes threshold value and is compared, to judge whether the operation corresponding to active operator meets preset condition.
In view of that during data processing, may may require that for relative complex data processing and take more money
Source, such as may have higher requirement to memory and CPU, or need to spend the more time.Specifically, according to each operator
The complexity of corresponding operation determines each object run symbol, it may include：Master server is obtained in the executive plan tree
Active operator, and the required target resource of operation corresponding to the active operator is estimated in advance；If the target
Resource exceedes default resource occupation threshold value, then master server is grasped with the operator corresponding to the target resource as target
Accord with.Wherein, the target resource includes the required committed memory of operation corresponding to each operator and/or performs the time.
Correspondingly, predetermined threshold value can be committed memory threshold value or perform time threshold.The concrete numerical value of predetermined threshold value can basis
The memory configurations of actual conditions such as master server and the requirement to processing time etc. are configured, and are not limited herein.Wherein, institute
It can be that the required committed memory of operation corresponding to each operator surpasses to state target resource to exceed default resource occupation threshold value
Cross committed memory threshold value, or the operation corresponding to each operator is required performs the time and exceed and perform time threshold；Also may be used
To be, the required committed memory of operation corresponding to each operator exceedes committed memory threshold value, and corresponding to each operator
Operation it is required perform the time exceed perform time threshold.
Alternatively, judge that the operation corresponding to active operator meets that preset condition can also be by user rule of thumb
Preset each object run symbol；If active operator accords with for the object run, judge that active operator institute is right
The operation answered meets preset condition.
It is understood that master server can select one or more secondary servers as needed.If main service
Device judges that the operation corresponding to active operator meets preset condition, then master server by the operator and with the behaviour
Make to accord with corresponding data sending to one or more secondary servers progress data processings.The particular number of secondary server can be with
Configured according to the actual requirement of data processing, do not limited herein.
It should be noted that optional in secondary server is all or part of data for being not required to prestore database, only
Need to receive data and operator that master server is sent, perform corresponding operation, and report the data processing after performing operation
As a result.The advantages of this arrangement are as follows secondary server needs only to interact with master server and according to master server institute
The operator of transmission completes the processing to received data, and secondary server requires no knowledge about more information, it is not required that with
Other secondary servers interact, and whole data handling procedure more has elasticity, flexibility, more preferable management.
S130, master server receive handling result of the secondary server to the data, and perform meter according to described
Draw tree and integration processing is carried out to the handling result.
It is probably the corresponding operator of last root node in view of the active operator transmitted by master server, it is corresponding
In final data processed result, it is also possible to the intermediate node of executive plan number, corresponding to intermediate treatment parameter.Therefore, it is main
Server is when receiving secondary server to the handling results of data, it is necessary to by currently grasping in handling result and executive plan tree
It is combined as symbol, docks received handling result and carry out integration processing.
The technical solution of the present embodiment, meter is performed by the way that master server generation is corresponding with the data processing request received
Tree is drawn, and then when according to executive plan tree execution operation corresponding with each operator, when judging active operator institute
When corresponding operation meets preset condition, then active operator and data sending corresponding with active operator are taken to auxiliary
Business device carries out data processing, needs to carry out data transmission between each server different from existing distributed system, main clothes
Business device is only interacted with secondary server when needed, and secondary server only needs to receive data to be treated and behaviour
Accord with, master server can select one to be performed in unison with a data processing to several secondary servers as needed, receive afterwards
The secondary server carries out integration processing according to executive plan tree to the handling results of data to the handling result, with
Complete whole data processings, solve in existing database data processing method by dilatation come lifted data-handling capacity and
The problem of dilatation is complicated time-consuming, realizes to the elasticity processing of database data and efficient process.
Embodiment two
Fig. 2 is a kind of flow diagram for data processing method that the embodiment of the present invention two is provided.The skill of the present embodiment
For art scheme in the technology of above-described embodiment, optional is by the operator and corresponding with the operator in master server
Before data sending carries out data processing at least one secondary server, further include：Master server asks server calls
Registration module is sent to, and receives at least one auxiliary corresponding with server calls request of the registration module feedback
The configuration information of server；Wherein, the registration module includes master server or registrar.
As shown in Fig. 2, the method for the present embodiment specifically includes：
S210, when master server receives data processing request input by user, based on the data processing request give birth to
Into the executive plan tree being made of at least one operator.
S220, master server according to the executive plan tree when performing operation corresponding with each operator, if described
Master server judges that the operation corresponding to active operator meets preset condition, then server calls are asked to send out by master server
Registration module is given, and receives at least one auxiliary clothes corresponding with server calls request of the registration module feedback
The configuration information of business device.
Wherein, the registration module includes master server or registrar.That is, registration module can be arranged in main clothes
It is engaged in device, or registration module is used as using single registrar.
The configuration information of at least one secondary server corresponding with server calls request is fed back in registration module
Before, further include：The secondary server is registered, for obtaining the configuration information of secondary server.Can be specifically,
When secondary server starts, the secondary server sends registration request to registration module；Registration module receives the registration
Ask, and the configuration information of the server is stored in the secondary server inventory of current active.Wherein, configuration information can be with
Attribute information and link information including secondary server, such as the identity of secondary server, IP address and port information
Deng.
For the ease of being counted and being managed to secondary server, alternatively, registration module can preset time point or
Every preset time or the configuration information of each secondary server of real time scan.Further, can also be according to secondary server
Registration request establish the secondary server inventory of current active；When finding no longer movable secondary server, then from
The configuration information of the secondary server is deleted in the secondary server inventory of current active.It should be noted that current active
The secondary server that secondary server inventory can be understood as in secondary server inventory can be called at any time.
In order to realize the quick calling of secondary server, improve the efficiency of data processing, can master server startup after,
Backstage starts a thread, and interval preset time obtains the secondary server inventory of current active to registration module.
In the present embodiment, after server calls request is sent to registration module by master server, can also include：Note
The configuration information of each secondary server that volume module is asked and prestored according to server calls are received, is determined and institute
State server calls and ask corresponding at least one secondary server, and the configuration information of at least one secondary server is fed back
To master server.For example, the requirement to server configuration included in being asked according to server calls, determines auxiliary clothes
The scope being engaged in residing for the configuration information of device, and then determine to be adapted to the secondary server for responding server calls request.
S230, the master server arrive the active operator and data sending corresponding with the active operator
At least one secondary server carries out data processing.
S240, master server receive handling result of the secondary server to the data, and perform meter according to described
Draw tree and integration processing is carried out to the handling result.
The technical solution of the present embodiment, the call request of secondary server is sent by master server to registration module, and
Receive registration module feedback secondary server configuration information, can by independent registration module to each secondary server into
Row effectively manages so that master server can it considerably easier and simpler call secondary server to carry out data processing, further
Improve the efficiency of data processing.
Embodiment three
A kind of flow diagram of the preferred embodiment for data processing method that Fig. 3 is provided by the embodiment of the present invention.With number
Exemplified by being calculated according to the data in storehouse, as shown in figure 3, the method for the present embodiment specifically includes：
After S301, master server MDB receive the data processing request write based on SQL of user, according to the number
Analyzed according to the syntax and semantics of processing request, generate the executive plan tree being made of multiple operators.
Each operator after S302, MDB in the order executive plan tree of root traversal, obtains in executive plan tree and works as prosthomere
The corresponding active operator of point.
S303, judge whether the load of active operator exceedes predetermined threshold value, and has the secondary server ADB of activity can
With if so, then performing step S304；Otherwise, S305 is performed.
Wherein, load includes memory shared by active operator or CPU etc..Predetermined threshold value is with loading included parameter phase
Corresponding, it can also be multiple that can be one, two, and concrete numerical value can be configured according to actual conditions, not limited herein
It is fixed.
S304, master server MDB create an elastometer operator plan based on active operator, and auxiliary from activity
Help in server A DB inventories and select part or all of node, send elastometer operator plan, perform S306.
Specifically, master server MDB can adjust the performing environment of active operator, and EDIS is added before active operator
Operator, adds EGAT operators after active operator.By taking active operator is sequence SORT as an example, elastometer operator meter
Draw as follows：EDIS->SORT->EGAT.Wherein, EDIS is used for MDB to ADB transmission data, and EGAT is used for MDB and is grasped from ADB collections
Deal with result.
Alternatively, MDB can select part or all of node from movable ADB inventories, send elastometer operator plan.
MDB is divided active operator data to be treated, and each ADB chosen is distributed to by EDIS operators.
S305, perform the corresponding operation of active operator, after operation is finished, performs S309.
It is understood that when master server can quickly perform the corresponding operation of active operator, can directly perform,
Secondary server need not be sent to, to save the time of operator data sending, ensures the high efficiency of data processing.
After S306, ADB receive elastometer operator plan, performing environment is created, performs S307.
S307, ADB collect data to be treated, perform operation corresponding with the operator, and implementing result is sent out
To MDB, S308 is performed.
As it was previously stated, ADB can collect this ADB data to be treated by EDIS operators, and then perform and the behaviour
Make to accord with corresponding operation, data are handled, and implementing result is issued into MDB by EGAT operators.
S308, MDB collect the implementing result of each ADB and carry out integration processing, perform S309.
Similarly, MDB can collect the implementing result of each ADB by EGAT operators.
S309, judge whether the corresponding node of active operator is last node, and S302 is performed if it is not, then returning；
If so, perform S310.
If the corresponding node of active operator is not last node, need to continue the order time of later root traversal
The next node in executive plan tree is gone through, that is, obtains the associated next node of present node in executive plan tree and is used as and work as prosthomere
Point, repeats S302-S309, until judging the corresponding node of active operator for last node.
It should be noted that if the corresponding node of active operator is last node, root later is illustrated
All nodes in the complete executive plan tree of order traversal of traversal, that is, performed the corresponding operation of all operators, corresponded at this time
Node should be root node in executive plan tree, corresponding should be last data processed result.
S310, output data handling result.
The technical solution of the present embodiment, in SQL implementation procedures, if it find that current operation need substantial amounts of memory and
Cpu resource, and have available idle computing resources in current network, just current SQL operations split into multiple tasks and
Corresponding data distribution recycles result of calculation and is integrated to these utilizable secondary servers, so as to fulfill complexity
The elastic calculation of SQL.
Example IV
Fig. 4 is a kind of structure diagram for data handling system that the embodiment of the present invention is provided.As shown in figure 4, this reality
Applying the system of example includes：Master server 410 and at least one secondary server 420；The master server includes：Operator obtains
Module, operation judges module and result integrate module.
Wherein, operator acquisition module, for when receiving data processing request input by user, based on the data
Processing requests to generate the executive plan tree being made of at least one operator；Operation judges module, for according to the execution
When plan tree performs operation corresponding with each operator, if the master server judges the operation corresponding to active operator
Meet preset condition, then the master server is by the active operator and data sending corresponding with the active operator
Data processing is carried out at least one secondary server；As a result module is integrated, for receiving the secondary server to the number
According to handling result, and integration processing is carried out to the handling result according to the executive plan tree.
The technical solution of the present embodiment, operation corresponding with the data processing request received is obtained by master server
Accord with, the data manipulation carried out required for user can be obtained, and then master server is according to executive plan tree execution and respectively
During the corresponding operation of operator, when judging that the operation corresponding to active operator meets preset condition, then by operator with
And the corresponding data sending of AND operator carries out data processing to secondary server, needed different from existing distributed system
Carry out data transmission between each server, master server is only interacted with secondary server when needed, and aids in clothes
Business device only needs to receive data and operator to be treated, and master server can select one to arrive several assistant services as needed
Device is performed in unison with a data processing, receives handling result of the secondary server to data afterwards, and according to executive plan
Tree carries out integration processing to the handling result, to complete whole data processings, solves existing database data processing side
In method by dilatation to lift data-handling capacity and complicated time-consuming dilatation the problem of, realize the bullet to database data
Property processing and efficient process.
Based on the above technical solutions, the system may also include：Registration module and calling module；Wherein, institute
State calling module, be configured in the master server, for master server by the operator and with the operator pair
Before the data sending answered carries out data processing at least one secondary server, the master server asks server calls
The registration module is sent to, and receives the corresponding at least one with server calls request of the registration module feedback
The configuration information of secondary server；The registration module includes master server or registrar.
On the basis of above-mentioned each technical solution, the system may also include：Registration request transmitting element and with confidence
Cease storage unit.Wherein, registration request transmitting element, is configured in the secondary server, for starting when secondary server
When, send registration request to registration module；Configuration information storage unit, is configured in the registration module, described for receiving
Registration request, and store in the secondary server inventory of current active the configuration information of the server.
On the basis of above-mentioned each technical solution, the operation judges module is particularly used in：
The complexity of operation according to corresponding to each operator determines each object run symbol, if active operator is institute
When stating object run symbol, then judge that the operation corresponding to active operator meets preset condition.
On the basis of above-mentioned each technical solution, the operation judgment device specifically can be additionally used in：
The active operator in the executive plan tree is obtained, and estimates the behaviour corresponding to the active operator in advance
Make required target resource；Wherein, the target resource includes the required committed memory of operation corresponding to each operator
And/or perform the time；
If the target resource exceedes default resource occupation threshold value, master server will with the target resource corresponding to
Operator as object run accord with.
Above device can perform the data processing method that any embodiment of the present invention is provided, and possesses and performs above method phase
The function module and beneficial effect answered.Not ins and outs of detailed description in the present embodiment, reference can be made to institute of the embodiment of the present invention
The data processing method of offer.
Embodiment five
A kind of structure diagram for terminal that Fig. 5 is provided by the embodiment of the present invention five.Fig. 5 is shown suitable for being used for realizing
The block diagram of the exemplary terminal 512 of embodiment of the present invention.The terminal 512 that Fig. 5 is shown is only an example, should not be to this hair
The function and use scope of bright embodiment bring any restrictions.
As shown in figure 5, terminal 512 is showed in the form of universal computing device.The component of terminal 512 can include but unlimited
In：One or more processor or processor 516, storage device 528, for storing one or more programs, connection is not
The bus 518 of homologous ray component (including storage device 428 and processor 516).
Bus 518 represents the one or more in a few class bus structures, including memory bus or Memory Controller,
Peripheral bus, graphics acceleration port, processor or the local bus using any bus structures in a variety of bus structures.Lift
For example, these architectures include but not limited to industry standard architecture (ISA) bus, microchannel architecture (MAC)
Bus, enhanced isa bus, Video Electronics Standards Association (VESA) local bus and periphery component interconnection (PCI) bus.
Terminal 512 typically comprises various computing systems computer-readable recording medium.These media can be it is any can be by terminal
512 usable mediums accessed, including volatile and non-volatile medium, moveable and immovable medium.
Storage device 528 can include the computer system readable media of form of volatile memory, such as arbitrary access
Memory (RAM) 530 and/or cache memory 532.Terminal 512 may further include other removable/nonremovable
, volatile/non-volatile computer system storage medium.Only as an example, it is not removable to can be used for read-write for storage system 534
Dynamic, non-volatile magnetic media (Fig. 5 do not show, commonly referred to as " hard disk drive ").Although not shown in Fig. 5, it can provide
For the disc driver to moving non-volatile magnetic disk (such as " floppy disk ") read-write, and to moving anonvolatile optical disk
The CD drive of (such as CD-ROM, DVD-ROM or other optical mediums) read-write.In these cases, each driver can
To be connected by one or more data media interfaces with bus 518.Memory 528 can include at least one program and produce
Product, the program product have one group of (for example, at least one) program module, these program modules are configured to perform of the invention each
The function of embodiment.
Program/utility 540 with one group of (at least one) program module 542, can be stored in such as memory
In 528, such program module 542 includes but not limited to operating system, one or more application program, other program modules
And routine data, the realization of network environment may be included in each or certain combination in these examples.Program module 542
Usually perform the function and/or method in embodiment described in the invention.
Terminal 512 can also be logical with one or more external equipments 514 (such as keyboard, sensing equipment, display 524 etc.)
Letter, can also enable a user to the equipment communication interacted with the terminal 512 with one or more, and/or with causing the terminal 512
Any equipment (such as network interface card, the modem etc.) communication that can be communicated with one or more of the other computing device.This
Kind communication can be carried out by input/output (I/O) interface 522.Also, terminal 512 can also by network adapter 520 with
One or more network (such as LAN (LAN), wide area network (WAN) and/or public network, such as internet) communication.Such as
Shown in figure, network adapter 520 is communicated by bus 518 with other modules of terminal 512.It should be understood that although do not show in figure
Go out, terminal 512 can be combined and use other hardware and/or software module, included but not limited to：It is microcode, device driver, superfluous
Remaining processor, external disk drive array, RAID system, tape drive and data backup storage system etc..
Processor 516 is stored in the program in storage device 528 by operation, so as to perform various functions application and number
According to processing, such as realize the inspection method that the embodiment of the present invention is provided.
In addition, the embodiment of the present invention, which additionally provides one kind, includes computer-readable recording medium, computer is stored thereon with
Program, is used to perform a kind of data processing method, this method includes when which is executed by processor：When master server receives
During data processing request input by user, the execution meter being made of at least one operator is generated based on the data processing request
Draw tree；Master server according to the executive plan tree when performing operation corresponding with each operator, if the master server
Judge that the operation corresponding to active operator meets preset condition, then the master server by the active operator and with
The corresponding data sending of the active operator carries out data processing at least one secondary server；Described in master server receives
Secondary server carries out at integration the handling result according to the executive plan tree handling results of the data
Reason.
Optionally, which can be also used for performing the present invention times when being performed by computer processor
The technical solution for the data processing method that meaning embodiment is provided.
Expression or logic and/or step described otherwise above herein in flow charts, for example, being considered use
In the order list for the executable instruction for realizing logic function, may be embodied in any computer-readable recording medium,
For instruction execution system, device or equipment (such as computer based system including the system of processor or other can be from finger
The system for making execution system, device or equipment instruction fetch and execute instruction) use, or combine these instruction execution systems, device
Or equipment and use.For the purpose of this specification, " computer-readable recording medium " can be it is any can include, store, communicating,
Propagate or transmission program for instruction execution system, device or equipment or with reference to these instruction execution systems, device or equipment and
The device used.
The more specifically example (non-exhaustive list) of computer-readable recording medium includes following：With one or more
The electrical connection section (electronic device) of wiring, portable computer diskette box (magnetic device), random access memory (RAM) are read-only to deposit
Reservoir (ROM), erasable edit read-only storage (EPROM or flash memory), fiber device, and portable optic disk are only
Read memory (CDROM).In addition, computer-readable recording medium can even is that the paper or its that can print described program on it
His suitable medium, because can be for example by carrying out optical scanner to paper or other media, then into edlin, interpretation or must
Handled when wanting with other suitable methods electronically to obtain described program, be then stored in computer storage
In.
It should be appreciated that each several part of the present invention can be realized with hardware, software, firmware or combinations thereof.Above-mentioned
In embodiment, software that multiple steps or method can be performed in memory and by suitable instruction execution system with storage
Or firmware is realized.If, and in another embodiment, can be with well known in the art for example, realized with hardware
Any one of row technology or their combination are realized：With the logic gates for realizing logic function to data-signal
Discrete logic, have suitable combinational logic gate circuit application-specific integrated circuit, programmable gate array (PGA), scene
Programmable gate array (FPGA) etc..
In the description of this specification, reference term " one embodiment ", " some embodiments ", " example ", " specifically show
The description of example " or " some examples " etc. means specific features, structure, material or the spy for combining the embodiment or example description
Point is contained at least one embodiment of the present invention or example.In the present specification, schematic expression of the above terms is not
Necessarily refer to identical embodiment or example.Moreover, particular features, structures, materials, or characteristics described can be any
One or more embodiments or example in combine in an appropriate manner.
Note that it above are only presently preferred embodiments of the present invention and institute's application technology principle.It will be appreciated by those skilled in the art that
The invention is not restricted to specific embodiment described here, can carry out for a person skilled in the art various obvious changes,
Readjust and substitute without departing from protection scope of the present invention.Therefore, although being carried out by above example to the present invention
It is described in further detail, but the present invention is not limited only to above example, without departing from the inventive concept, also
It can include other more equivalent embodiments, and the scope of the present invention is determined by scope of the appended claims.
Claims (10)
- A kind of 1. data processing method, it is characterised in that including：When master server receives data processing request input by user, generated based on the data processing request by least one The executive plan tree of a operator composition；The master server according to the executive plan tree when performing operation corresponding with each operator, if the main service Device judges that the operation corresponding to active operator meets preset condition, then the master server by the active operator and Data sending corresponding with the active operator carries out data processing at least one secondary server；The master server receives handling result of the secondary server to the data, and according to the executive plan tree pair The handling result carries out integration processing.
- 2. according to the method described in claim 1, it is characterized in that, master server by the operator and with the operation Accord with corresponding data sending at least one secondary server carry out data processing before, further include：Server calls request is sent to registration module by master server, and receiving registration module feedback with the service The configuration information of the corresponding at least one secondary server of device call request；Wherein, the registration module include master server or Registrar.
- 3. according to the method described in claim 2, it is characterized in that, further include：When secondary server starts, the secondary server sends registration request to registration module；Registration module receives the registration request, and stores matching somebody with somebody for the server in the secondary server inventory of current active Confidence ceases.
- 4. according to any methods of claim 1-3, it is characterised in that the behaviour judged corresponding to active operator Work meets preset condition, including：The complexity of operation of the master server according to corresponding to each operator determines each object run symbol, if active operator When being accorded with for the object run, then judge that the operation corresponding to active operator meets preset condition.
- 5. according to the method described in claim 4, it is characterized in that, operation of the master server according to corresponding to each operator Complexity determine each object run symbol, including：Master server obtains the active operator in the executive plan tree, and estimates in advance corresponding to the active operator The required target resource of operation；Wherein, the target resource includes the required occupancy of operation corresponding to each operator Memory and/or execution time；If the target resource exceedes default resource occupation threshold value, master server is by the behaviour corresponding to the target resource Make symbol to accord with as object run.
- A kind of 6. data handling system, it is characterised in that including：Master server and secondary server；Wherein, the master server Including：Operator acquisition module, for when receiving data processing request input by user, based on the data processing request Generate the executive plan tree being made of at least one operator；Operation judges module, for according to the executive plan tree perform operation corresponding with each operator when, if described Master server judges that the operation corresponding to active operator meets preset condition, then the master server is by the current operation Symbol and data sending corresponding with the active operator carry out data processing at least one secondary server；As a result module is integrated, meter is performed for receiving handling result of the secondary server to the data, and according to described Draw tree and integration processing is carried out to the handling result.
- 7. system according to claim 6, it is characterised in that further include：Registration module and calling module；Wherein, it is described Calling module, is configured in the master server, in master server by the operator and corresponding with the operator Data sending at least one secondary server carry out data processing before, the master server by server calls ask send out The registration module is given, and receives the corresponding at least one auxiliary with server calls request of the registration module feedback Help the configuration information of server；The registration module includes master server or registrar.
- 8. system according to claim 7, it is characterised in that further include：Registration request transmitting element, is configured in the secondary server, for when secondary server starts, to registration module Send registration request；Configuration information storage unit, is configured in the registration module, for receiving the registration request, and in current active The configuration information of the server is stored in secondary server inventory.
- 9. according to any systems of claim 6-8, it is characterised in that the operation judges module is specifically used for：The complexity of operation according to corresponding to each operator determines each object run symbol, if active operator is the mesh When marking operator, then judge that the operation corresponding to active operator meets preset condition.
- 10. system according to claim 9, it is characterised in that the operation judgment device is specifically additionally operable to：The active operator in the executive plan tree is obtained, and estimates the operation institute corresponding to the active operator in advance The target resource needed；Wherein, the target resource include each operator corresponding to the required committed memory of operation and/or Perform the time；If the target resource exceedes default resource occupation threshold value, master server is by the behaviour corresponding to the target resource Make symbol to accord with as object run.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711401731.7A CN108038215A (en) | 2017-12-22 | 2017-12-22 | Data processing method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711401731.7A CN108038215A (en) | 2017-12-22 | 2017-12-22 | Data processing method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108038215A true CN108038215A (en) | 2018-05-15 |
Family
ID=62100322
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711401731.7A Pending CN108038215A (en) | 2017-12-22 | 2017-12-22 | Data processing method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108038215A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109754868A (en) * | 2018-12-18 | 2019-05-14 | 杭州深睿博联科技有限公司 | Data processing method and device for medical image |
| CN110569257A (en) * | 2019-09-16 | 2019-12-13 | 上海达梦数据库有限公司 | data processing method, corresponding device, equipment and storage medium |
| CN110851534A (en) * | 2019-11-15 | 2020-02-28 | 上海达梦数据库有限公司 | Data processing method, system and storage medium |
| CN111091473A (en) * | 2019-11-25 | 2020-05-01 | 泰康保险集团股份有限公司 | Insurance problem analysis and processing method and device |
| CN113448967A (en) * | 2021-07-20 | 2021-09-28 | 威讯柏睿数据科技(北京)有限公司 | Method and device for accelerating database operation |
| CN114925092A (en) * | 2022-05-09 | 2022-08-19 | 北京达佳互联信息技术有限公司 | Data processing method and device, electronic equipment and storage medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050240570A1 (en) * | 2004-04-22 | 2005-10-27 | Oracle International Corporation | Partial query caching |
| CN102984012A (en) * | 2012-12-10 | 2013-03-20 | 青岛海信传媒网络技术有限公司 | Management method and system for service resources |
| CN103377292A (en) * | 2013-07-02 | 2013-10-30 | 华为技术有限公司 | Database result set caching method and device |
| US20130325841A1 (en) * | 2012-06-05 | 2013-12-05 | Tanvir Ahmed | Sql transformation-based optimization techniques for enforcement of data access control |
| CN106156301A (en) * | 2016-06-30 | 2016-11-23 | 上海达梦数据库有限公司 | A kind of processing method and processing device of big field data |
| CN107301205A (en) * | 2017-06-01 | 2017-10-27 | 华南理工大学 | A kind of distributed Query method in real time of big data and system |
| CN107329837A (en) * | 2016-09-21 | 2017-11-07 | 广州特道信息科技有限公司 | Method and unit, the distribution NewSQL Database Systems of a kind of load balancing |
-
2017
- 2017-12-22 CN CN201711401731.7A patent/CN108038215A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050240570A1 (en) * | 2004-04-22 | 2005-10-27 | Oracle International Corporation | Partial query caching |
| US20130325841A1 (en) * | 2012-06-05 | 2013-12-05 | Tanvir Ahmed | Sql transformation-based optimization techniques for enforcement of data access control |
| CN102984012A (en) * | 2012-12-10 | 2013-03-20 | 青岛海信传媒网络技术有限公司 | Management method and system for service resources |
| CN103377292A (en) * | 2013-07-02 | 2013-10-30 | 华为技术有限公司 | Database result set caching method and device |
| CN106156301A (en) * | 2016-06-30 | 2016-11-23 | 上海达梦数据库有限公司 | A kind of processing method and processing device of big field data |
| CN107329837A (en) * | 2016-09-21 | 2017-11-07 | 广州特道信息科技有限公司 | Method and unit, the distribution NewSQL Database Systems of a kind of load balancing |
| CN107301205A (en) * | 2017-06-01 | 2017-10-27 | 华南理工大学 | A kind of distributed Query method in real time of big data and system |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109754868A (en) * | 2018-12-18 | 2019-05-14 | 杭州深睿博联科技有限公司 | Data processing method and device for medical image |
| CN110569257A (en) * | 2019-09-16 | 2019-12-13 | 上海达梦数据库有限公司 | data processing method, corresponding device, equipment and storage medium |
| CN110569257B (en) * | 2019-09-16 | 2022-04-01 | 上海达梦数据库有限公司 | Data processing method, corresponding device, equipment and storage medium |
| CN110851534A (en) * | 2019-11-15 | 2020-02-28 | 上海达梦数据库有限公司 | Data processing method, system and storage medium |
| CN110851534B (en) * | 2019-11-15 | 2022-09-06 | 上海达梦数据库有限公司 | Data processing method, system and storage medium |
| CN111091473A (en) * | 2019-11-25 | 2020-05-01 | 泰康保险集团股份有限公司 | Insurance problem analysis and processing method and device |
| CN113448967A (en) * | 2021-07-20 | 2021-09-28 | 威讯柏睿数据科技(北京)有限公司 | Method and device for accelerating database operation |
| CN113448967B (en) * | 2021-07-20 | 2022-02-08 | 威讯柏睿数据科技(北京)有限公司 | Method and device for accelerating database operation |
| CN114925092A (en) * | 2022-05-09 | 2022-08-19 | 北京达佳互联信息技术有限公司 | Data processing method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108038215A (en) | Data processing method and system | |
| US8037185B2 (en) | Dynamic application placement with allocation restrictions, vertical stacking and even load distribution | |
| US8271523B2 (en) | Coordination server, data allocating method, and computer program product | |
| US9813490B2 (en) | Scheduled network communication for efficient re-partitioning of data | |
| US20100198855A1 (en) | Providing parallel result streams for database queries | |
| CN110308980A (en) | Batch processing method, device, equipment and the storage medium of data | |
| EP3816877A1 (en) | Model-based prediction method and device | |
| CN103984713B (en) | A kind of financial data querying method based on cloud computing | |
| CN107704597A (en) | Relevant database to Hive ETL script creation methods | |
| CN106528683A (en) | Index segmenting equalization based big data cloud search platform and method thereof | |
| US7707581B2 (en) | Method and system for managing server load to execute transactions of an application program on multiple servers | |
| US20120166492A1 (en) | Database transfers using constraint free data | |
| Hu et al. | Output-optimal massively parallel algorithms for similarity joins | |
| US20150269234A1 (en) | User Defined Functions Including Requests for Analytics by External Analytic Engines | |
| CN113821332B (en) | Method, device, equipment and medium for optimizing efficiency of automatic machine learning system | |
| CN107070645A (en) | Compare the method and system of the data of tables of data | |
| CN105872082B (en) | Fine granularity resource response system based on container cluster load-balancing algorithm | |
| CN110442454A (en) | A kind of resource regulating method, device and computer equipment | |
| US7827132B2 (en) | Peer based event conversion | |
| CN116450355A (en) | Multi-cluster model training method, device, equipment and medium | |
| CN110069319A (en) | A kind of multiple target dispatching method of virtual machine and system towards cloudlet resource management | |
| CN115563160A (en) | Data processing method, data processing device, computer equipment and computer readable storage medium | |
| CN115147183A (en) | Chip resource management method, device, equipment and storage medium based on cloud platform | |
| CN111680069B (en) | Database access method and device | |
| CN113382075A (en) | Enterprise information management platform, management method, electronic device and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180515 |
|
| RJ01 | Rejection of invention patent application after publication |