CN106550045A - A kind of distributed data processing method and component - Google Patents
A kind of distributed data processing method and component Download PDFInfo
- Publication number
- CN106550045A CN106550045A CN201611051440.5A CN201611051440A CN106550045A CN 106550045 A CN106550045 A CN 106550045A CN 201611051440 A CN201611051440 A CN 201611051440A CN 106550045 A CN106550045 A CN 106550045A
- Authority
- CN
- China
- Prior art keywords
- data processing
- tenant
- server
- pel
- processing task
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
- H04L67/1004—Server selection for load balancing
- H04L67/1012—Server selection for load balancing based on compliance of requirements or conditions with available server resources
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Distributed data processing method disclosed by the invention and component, when data processing task model is built, controlling stream pel is added in visualization tasks editing machine, controlling stream pel is combined with sequential processing pel, so as to realize the logic control function in data processing task implementation procedure.Receive and parse through the resource bid information of tenant, obtain the quantity of tenant's required service device, the running status of current each registered server is obtained again, according to the running status of current each registered server, the corresponding multiple servers of server resource for needed for the tenant distributes with the tenant, cluster is set up based on the multiple servers for distributing to tenant, each server data processing task in data processing task model being sent in cluster, each server distribution formula is made to perform data processing task, avoid the in short supply or idle of server resource, meet the flexible allocation on demand of server resource.
Description
Technical field
The present invention relates to data processing field, more particularly, to a kind of distributed data processing method and component.
Background technology
Informationization is, based on modern communicationses, network, database technology, each key element of institute's object of study to be collected to data
Storehouse, for a kind of skill that the closely bound up various actions of specific crowd life, work, study, aid decision etc. and the mankind combine
Art.Informationalized essence is the processing procedure of data, therefore, data handling component becomes indispensable important of Information-based Item
Ingredient.
Existing data handling component is realized based on B/S patterns, builds data processing by visual mode
The links of data processing have been carried out abstract by journey, describe the links of data processing using unified model, using many
Thread and cluster, support the distributed treatment of data, but, this data handling component takes unified distribution resource component cluster
Mode perform data processing task, resource distribution mode is single, causes resource allocation uneven, may cause data processing amount
The server resource that big user is assigned to is inadequate, and the server resource that the little user of data processing amount is assigned to has the spare time
Put, resource utilization is relatively low, it is impossible to meet the multifarious demand of resource allocation.
The content of the invention
In view of this, the invention provides a kind of distributed data processing method and component, using multi-tenant technology, according to
Tenant data process demand distributes corresponding server resource, distributed execution data processing task.
Concrete scheme is as follows：
A kind of distributed data processing method, methods described are applied to distributed data processing component, and methods described includes：
According to the edit instruction of tenant, data processing task model is generated in visualization tasks editing machine；
The resource bid information of the tenant is received and parsed through, the server resource needed for the tenant is obtained；
Obtain the running status of current each registered server；
According to the running status of current each registered server, for needed for the tenant distributes with the tenant
The corresponding multiple servers of server resource；
Cluster is set up based on the multiple described server for distributing to the tenant, by the data processing task model
Each described server that data processing task is sent in the cluster, makes each described server respectively while performing described
Data processing task.
Preferably, in the edit instruction according to tenant, data processing task is generated in visualization tasks editing machine
Model, also includes before：
Multiple tenants are created, the corresponding tenant's information of each described tenant is generated respectively, and tenant's information is carried out
Safeguard；Tenant's information includes：Tenant identification, administrator password, source database and target database.
Preferably, multiple sequential processing pels and multiple controlling stream pels are provided with the visualization tasks editing machine；Institute
The edit instruction according to tenant is stated, data processing task model is generated in visualization tasks editing machine, including：
According to edit instruction of the tenant to sequential processing pel each described, data processing sequence is generated, at the data
Reason processing sequence corresponds to data importing, data detection, data processing and the data of Data processing and derives four-stage；
According to edit instruction of the tenant to the controlling stream pel, will be relative with the edit instruction to the controlling stream pel
The multiple described controlling stream pel answered is combined with the multiple described sequential processing pel arranged by the data processing sequence,
Generate data processing task model；The controlling stream pel includes：Terminate operation pel, computational chart line number pel, variable is set
Pel, inspection field value pel, execution SQL pels, conversion pel and beginning pel.
Preferably, the running status of the registered server includes whether the server is allocated data processing and appoints
Business；The running status of current each the registered server of the basis, for needed for the tenant distributes with the tenant
The corresponding multiple servers of server resource, including：
According to the running status of current each registered server, multiple clothes of unassigned data processing task are obtained
Business device；
The number of servers needed for the tenant is determined according to the server resource needed for the tenant；
From multiple servers of the unassigned data processing task, it is the service of tenant's distribution requirement
Device, and the running status for distributing to the server of the tenant is revised as being allocated data processing task.
Preferably, it is described that cluster is set up based on the multiple described server for distributing to the tenant, by the data processing
Data processing task in task model is assigned to each server in the cluster, make each described server respectively while
The data processing task is performed, including：
Cluster is set up based on the multiple described server for distributing to the tenant, determine master server in the cluster and
Each is from server；
The data processing task model is sent to into the master server；
Control the master server data processing task model is assigned to described in each from server；
Control to parse the data processing task model from server described in each, obtain data processing sequence and
Control logic, makes to carry out data processing according to the data processing sequence and the control logic from server described in each.
A kind of distributed data processing component, the component include：
Signal generating unit, for the edit instruction according to tenant, generates data processing task in visualization tasks editing machine
Model；
Resolution unit, for receiving and parsing through the resource bid information of the tenant, obtains the service needed for the tenant
Device resource；
Acquiring unit, for obtaining the running status of current each registered server；
Allocation unit, for the running status according to current each registered server, is tenant's distribution
The multiple servers corresponding with the server resource needed for the tenant；
Transmitting element, for setting up cluster based on the multiple described server for distributing to the tenant, at the data
Each described server that data processing task in reason task model is sent in the cluster, makes each described server point
Perform the data processing task not simultaneously.
Preferably, the component also includes：
Tenant's administrative unit, for creating multiple tenants, generates the corresponding tenant's information of each described tenant respectively, and right
Tenant's information is safeguarded；Tenant's information includes：Tenant identification, administrator password, source database and target database.
Preferably, the signal generating unit includes：
First generates subelement, for the edit instruction according to tenant to sequential processing pel each described, generates data
Processing sequence, the data processing processing sequence is corresponding to the data importing of Data processing, data detection, data processing sum
According to derivation four-stage；
Second generate subelement, for the edit instruction according to tenant to the controlling stream pel, will with to the control
The multiple described controlling stream pel that the edit instruction of flow graph unit is corresponding is multiple described with what is arranged by the data processing sequence
Sequential processing pel is combined, and generates data processing task model；The controlling stream pel includes：Terminate operation pel, meter
Table row number pel is calculated, variable pel is set, is checked field value pel, is performed SQL pels, conversion pel and start pel.
Preferably, the allocation unit includes：
Subelement is obtained, for the running status according to current each registered server, unassigned data is obtained
Multiple servers of process task；
Determination subelement, for the server count needed for the tenant is determined according to the server resource needed for the tenant
Amount；
Distribution subelement, for the multiple servers from the unassigned data processing task in, be the tenant point
Server with requirement, and the running status for distributing to the server of the tenant is revised as being allocated data processing
Task.
Preferably, the transmitting element includes：
Subelement is set up, and for cluster being set up based on the multiple described server for distributing to the tenant, determines the collection
Master server in group is with each from server；
Subelement is sent, for the data processing task model is sent to the master server；
The data processing task model is assigned to each institute for controlling the master server by the first control subelement
State from server；
Second control subelement, for controlling to solve the data processing task model from server described in each
Analysis, obtains data processing sequence and control logic, makes described in each from server according to the data processing sequence and the control
Logic processed carries out data processing.
Relative to prior art, beneficial effects of the present invention are as follows：
Distributed data processing method disclosed by the invention and component, when data processing task model is built, visual
Controlling stream pel is added in changing task editor, controlling stream pel is combined with sequential processing pel, so as to realize counting
According to the logic control function in process task implementation procedure.The resource bid information of tenant is received and parsed through, is obtained needed for tenant
Server resource, then obtain the running status of current each registered server, each is described registered according to current
The running status of server, the corresponding multiple servers of server resource for needed for the tenant distributes with the tenant,
Cluster is set up based on the multiple servers for distributing to tenant, the data processing task in data processing task model is sent to into collection
Each server in group, make each server distribution formula perform data processing task, it is to avoid server resource in short supply or
It is idle, meet the flexible allocation on demand of server resource.
Description of the drawings
In order to be illustrated more clearly that the embodiment of the present invention or technical scheme of the prior art, below will be to embodiment or existing
Accompanying drawing to be used needed for having technology description is briefly described, it should be apparent that, drawings in the following description are only this
Inventive embodiment, for those of ordinary skill in the art, on the premise of not paying creative work, can be with basis
The accompanying drawing of offer obtains other accompanying drawings.
Fig. 1 is a kind of distributed data processing method flow diagram disclosed in the embodiment of the present invention；
Fig. 2 is a kind of another method flow diagram of distributed data processing method disclosed in the embodiment of the present invention；
Fig. 3 is a kind of another method flow diagram of distributed data processing method disclosed in the embodiment of the present invention；
Fig. 4 is a kind of distributed data processing modular construction schematic diagram disclosed in the embodiment of the present invention；
Fig. 5 is another kind of distributed data processing modular construction schematic diagram disclosed in the embodiment of the present invention.
Specific embodiment
Below in conjunction with the accompanying drawing in the embodiment of the present invention, the technical scheme in the embodiment of the present invention is carried out clear, complete
Site preparation is described, it is clear that described embodiment is only a part of embodiment of the invention, rather than the embodiment of whole.It is based on
Embodiment in the present invention, it is every other that those of ordinary skill in the art are obtained under the premise of creative work is not made
Embodiment, belongs to the scope of protection of the invention.
Fig. 1 is referred to, present embodiment discloses a kind of distributed data processing method, methods described is applied to distributed number
According to process assembly, methods described specifically includes following steps：
Step S101：According to the edit instruction of tenant, data processing task model is generated in visualization tasks editing machine；
It should be noted that the present embodiment adopts multi-tenant technology, multi-tenant is the proprietary term of field of cloud calculation, at this
In embodiment, tenant's one operation system of correspondence, data isolation between tenant, multiple tenant's shared server resources.
Also, it should be noted that the present embodiment provides visualization tasks editing machine for tenant, tenant can be on visualization circle
Input editing instruction on face, generates data processing task model.
Specifically, the visualization tasks editing machine based on flex technologies realize, for aid in tenant pass through graphical method
Edlin is entered to data processing task model, the functional requirements of data processing task and performance demand can be carried out entirely
Graphical configuration management, functional requirements include link composition, data flow and layout format of data processing task etc.；Performance
Demand includes that the thread usage quantity to link, server usage quantity and data number of channels are configured.The visualization
The data processing task model of generation is stored in the way of xml by task editor, is provided with the visualization tasks editing machine
Multiple sequential processing pels and multiple controlling stream pels, refer to Fig. 2, and the concrete implementation procedure of step S101 is as follows：
Step S201：According to edit instruction of the tenant to sequential processing pel each described, data processing sequence, institute are generated
State data importing, data detection, data processing and data derivation four that data processing processing sequence corresponds to Data processing
Stage；
Specifically, the sequential processing pel includes being input into, exports, counts, script and calculating 5 class pels, corresponds to respectively
Four-stage is derived in data importing, data detection, data processing and data in Data processing, and to Data processing ratio
The demand such as more typical incremental crawler and error handling processing provides targetedly solution.
Step S202：According to edit instruction of the tenant to the controlling stream pel, by with the volume to the controlling stream pel
Collect the corresponding multiple described controlling stream pel of instruction and the multiple described sequential processing figure by data processing sequence arrangement
Unit is combined, and generates data processing task model；The controlling stream pel includes：Terminate operation pel, computational chart line number figure
First, setting variable pel, inspection field value pel, execution SQL pels, conversion pel and beginning pel.
Specifically, the service application in order that data handling procedure is more fitted, realizes in data processing task implementation procedure
Middle service logic judges that increase controlling stream pel in visualization tasks editing machine, the concrete function of various controlling stream pels is such as
Under：
Terminate operation pel：Judge whether to meet data processing task termination condition, terminate at the data if meeting
Reason task.
Computational chart line number pel：The SQL of tenant's configuration is performed, computational chart line number, for the line number of statistical disposition data, is carried
Whether supply subsequent primitives or comparison expression, the line number and final line number for calculating processing data match, and verification data is complete
Property.
Variable pel is set：The setting of some temporary variables in calculating process is completed, for storing some process datas,
These data can be with final output to object table, it is also possible to be only used for calculating process.Configuration end mainly completes variable name and initial
The setting of value.
Check field value pel：Calculating factor check before calculating for expression formula, checks and calculates whether the factor meets meter
Calculation condition, can filter to the ineligible calculating factor, and output is not inconsistent the calculating factor of condition, is easy to O＆M people
Member checks.Configuration end completes the definition for checking field, and the configuration of test condition.
Perform SQL pels：One section of grammatical SQL statement of operation.Configuration end mainly completes the SQL for needing operation
The relevant configuration of sentence.
Conversion pel：To reduce the complexity of individual data process task and strengthening the reusability of data processing task, hold
Other data processing tasks that row conversion pel has been configured before being mainly used to call.Configuration end need complete with other data
The configuration of reason task interface, and the configuration of related context parameter.
Start pel：Task model is parsed, beginning pel is found, first business pel is performed.
Step S102：The resource bid information of the tenant is received and parsed through, the server money needed for the tenant is obtained
Source；
Step S103：Obtain the running status of current each registered server；
Specifically, after resource bid information is received, obtain whether current each registered server is assigned to rent
Family, so that the running status of current each server of basis is tenant's distribution server resource.
Step S104：According to the running status of current each registered server, it is tenant's distribution and institute
State the corresponding multiple servers of server resource needed for tenant；
It should be noted that the concrete implementation procedure of step S104 is as follows：
According to the running status of current each registered server, multiple clothes of unassigned data processing task are obtained
Business device；
The number of servers needed for the tenant is determined according to the server resource needed for the tenant；
From multiple servers of the unassigned data processing task, it is the service of tenant's distribution requirement
Device, and the running status for distributing to the server of the tenant is revised as being allocated data processing task.
Step S105：Cluster is set up based on the multiple described server for distributing to the tenant, the data processing is appointed
Data processing task in business model is sent to each server in the cluster, makes each described server respectively while holding
The row data processing task.
Specifically, the concrete implementation procedure of step S105 is as follows：
Cluster is set up based on the multiple described server for distributing to the tenant, determine master server in the cluster and
Each is from server；
It should be noted that the master server can be determined and from service according to the IP address of each server in cluster
Device, wherein, the minimum server of IP address in cluster is defined as into master server, and other servers in cluster are defined as
From server.
The data processing task model is sent to into the master server；
Control the master server data processing task model is assigned to described in each from server；
Control to parse the data processing task model from server described in each, obtain data processing sequence and
Control logic, makes to carry out data processing according to the data processing sequence and the control logic from server described in each.
Distributed data processing method disclosed in the present embodiment, when data processing task model is built, appoints in visualization
Controlling stream pel is added in business editing machine, controlling stream pel is combined with sequential processing pel, so as to realize at data
Logic control function during reason tasks carrying.The resource bid information of tenant is received and parsed through, the clothes needed for tenant are obtained
Business device resource, then the running status of current each registered server is obtained, according to current each described registered service
The running status of device, the corresponding multiple servers of server resource for needed for the tenant distributes with the tenant, is based on
The multiple servers for distributing to tenant set up cluster, and the data processing task in data processing task model is sent in cluster
Each server, make each server distribution formula perform data processing task, it is to avoid server resource it is in short supply or idle,
Meet the flexible allocation on demand of server resource.
Fig. 3 is referred to, was also included before step S101：
Step S106：Multiple tenants are created, the corresponding tenant's information of each described tenant is generated respectively, and to the tenant
Information is safeguarded；Tenant's information includes：Tenant identification, administrator password, source database and target database.
It should be noted that tenant's information can also include：Tenant describes and creation time.Tenant identification is sign
The unique mark of tenant, the tenant identification can be the combination of tenant's title and tenant ID.Source database is at tenant data
Data after treatment are saved in target database after data processing task terminates by the data source during reason.
Maintenance to tenant's information includes：Inquiry tenant, modification tenant, deletion tenant, tenant's authorization service and multi-tenant
Isolation.Wherein, tenant's authorization service provides security guarantee for the operation of tenant, by " tenant ID+ tenant administrator password+tenant
Creation time+encrypted characters string " is massaged this password and generates token, and tenant downloads token to be needed to provide correct administrator password, rents
Family needs to carry out safety check according to information such as tenant identification, token and passwords, after verification passes through when server resource is applied for
Could be that tenant distributes corresponding resource.Multi-tenant isolation increases the isolation of tenant's id field by the data between different tenants.
For multiple server registration server accounts, each server info is generated respectively, and to the server
Information is safeguarded；The server info includes：Server identification, server ip address, Service-Port and running status.
It should be noted that by multiple server registration server accounts, so as to when tenant applies for server resource, root
According to each server registered in advance, whether the current of current each server of inquiry is allocated data processing task, according to rent
The resource bid information at family, is tenant's distribution server resource.
Maintenance is carried out to server info includes：Inquiry server and deletion server.Wherein, delete service implement body
For：Server is deleted from tenant, while deleting the data processing task for being assigned to the server.
Fig. 4 is referred to, based on a kind of distributed data processing method disclosed in above-described embodiment, the present embodiment correspondence is open
A kind of distributed data processing is set up, and the component includes：
Signal generating unit 101, for the edit instruction according to tenant, in visualization tasks editing machine generates data processing and appoints
Business model；
Resolution unit 102, for receiving and parsing through the resource bid information of the tenant, obtains the clothes needed for the tenant
Business device resource；
Acquiring unit 103, for obtaining the running status of current each registered server；
Allocation unit 104, for the running status according to current each registered server, is the tenant point
With the multiple servers corresponding with the server resource needed for the tenant；
Transmitting element 105, for setting up cluster based on the multiple described server for distributing to the tenant, by the data
Data processing task in process task model is sent to each server in the cluster, makes each server difference
Perform the data processing task simultaneously.
It should be noted that distributed data processing component disclosed in the present embodiment also includes that data processing process shows list
Unit, for showing each link is consumed in whole data handling procedure process time, to make distributed data processing group
The operation maintenance personnel of part quickly positions the performance bottleneck of whole data handling procedure, takes corresponding means to complete data processing and appoints
The Performance tuning work of business operation.
Distributed data processing component disclosed in the present embodiment, signal generating unit 101 build data processing task model when,
Controlling stream pel is added in visualization tasks editing machine, controlling stream pel is combined with sequential processing pel, so as to reality
Logic control function in present data processing task implementation procedure.Resolution unit 102 receives and parses through the resource bid of tenant
Information, obtains the server resource needed for tenant, and acquiring unit 103 obtains the operation shape of current each registered server
State, running status of the allocation unit 104 according to current each registered server is tenant's distribution and the rent
The corresponding multiple servers of server resource needed for family, transmitting element 105 is based on the multiple server groups for distributing to tenant
Cluster is built, each server data processing task in data processing task model being sent in cluster services each
Device distributed execution data processing task, it is to avoid server resource it is in short supply or idle, meet server resource on demand
Flexible allocation.
Fig. 5 is referred to, the distributed data processing component also includes：
Tenant's administrative unit 106, for creating multiple tenants, generates the corresponding tenant's information of each described tenant respectively,
And tenant's information is safeguarded；Tenant's information includes：Tenant identification, administrator password, source database and number of targets
According to storehouse.
The signal generating unit 101 includes：
First generates subelement 107, for the edit instruction according to tenant to sequential processing pel each described, generates number
According to processing sequence, the data processing processing sequence corresponding to the data importing of Data processing, data detection, data processing and
Data derive four-stage；
Second generate subelement 108, for the edit instruction according to tenant to the controlling stream pel, will with to the control
The corresponding multiple described controlling stream pel of the first edit instruction of flow graph processed and the multiple institutes by data processing sequence arrangement
State sequential processing pel to be combined, generate data processing task model；The controlling stream pel includes：Termination operation pel,
Computational chart line number pel, setting variable pel, inspection field value pel, execution SQL pels, conversion pel and beginning pel.
The allocation unit 104 includes：
Subelement 109 is obtained, for the running status according to current each registered server, unassigned number is obtained
According to multiple servers of process task；
Determination subelement 110, for the service needed for the tenant is determined according to the server resource needed for the tenant
Device quantity；
Distribution subelement 111, for the multiple servers from the unassigned data processing task in, be the tenant
The server of distribution requirement, and the running status for distributing to the server of the tenant is revised as being allocated at data
Reason task.
The transmitting element 105 includes：
Subelement 112 is set up, for cluster being set up based on the multiple described server for distributing to the tenant, it is determined that described
Master server in cluster is with each from server；
Subelement 113 is sent, for the data processing task model is sent to the master server；
The data processing task model is assigned to respectively by the first control subelement 114 for controlling the master server
It is individual described from server；
Second control subelement 115, for controlling to carry out the data processing task model from server described in each
Parsing, obtains data processing sequence and control logic, makes each described from server according to the data processing sequence and described
Control logic carries out data processing.
The foregoing description of the disclosed embodiments, enables professional and technical personnel in the field to realize or using the present invention.
Various modifications to these embodiments will be apparent for those skilled in the art, as defined herein
General Principle can be realized without departing from the spirit or scope of the present invention in other embodiments.Therefore, the present invention
The embodiments shown herein is not intended to be limited to, and is to fit to and principles disclosed herein and features of novelty phase one
The most wide scope for causing.
Claims (10)
1. a kind of distributed data processing method, it is characterised in that methods described is applied to distributed data processing component, described
Method includes：
According to the edit instruction of tenant, data processing task model is generated in visualization tasks editing machine；
The resource bid information of the tenant is received and parsed through, the server resource needed for the tenant is obtained；
Obtain the running status of current each registered server；
According to the running status of current each registered server, the clothes for needed for the tenant distributes with the tenant
The corresponding multiple servers of business device resource；
Cluster is set up based on the multiple described server for distributing to the tenant, by the data in the data processing task model
Each described server that process task is sent in the cluster, makes each described server respectively while performing the data
Process task.
2. method according to claim 1, it is characterised in that in the edit instruction according to tenant, appoint in visualization
Data processing task model is generated in business editing machine, is also included before：
Multiple tenants are created, the corresponding tenant's information of each described tenant is generated respectively, and tenant's information is safeguarded；
Tenant's information includes：Tenant identification, administrator password, source database and target database.
3. method according to claim 1, it is characterised in that be provided with multiple orders in the visualization tasks editing machine
Reason pel and multiple controlling stream pels；The edit instruction according to tenant, generates at data in visualization tasks editing machine
Reason task model, including：
According to edit instruction of the tenant to sequential processing pel each described, data processing sequence is generated, at the data processing
Make data importing, data detection, data processing and data derivation four-stage that sequence corresponds to Data processing in order；
According to edit instruction of the tenant to the controlling stream pel, will be corresponding with the edit instruction to the controlling stream pel
Multiple controlling stream pels are combined with the multiple described sequential processing pel arranged by the data processing sequence, are generated
Data processing task model；The controlling stream pel includes：Terminate operation pel, computational chart line number pel, arrange variable pel,
Field value pel is checked, SQL pels, conversion pel is performed and is started pel.
4. method according to claim 1, it is characterised in that the running status of the registered server includes described
Whether server is allocated data processing task；The running status of current each the registered server of the basis, be
The tenant distributes the multiple servers corresponding with the server resource needed for the tenant, including：
According to the running status of current each registered server, multiple services of unassigned data processing task are obtained
Device；
The number of servers needed for the tenant is determined according to the server resource needed for the tenant；
From multiple servers of the unassigned data processing task, it is the server of tenant's distribution requirement,
And be revised as being allocated data processing task by the running status for distributing to the server of the tenant.
5. method according to claim 1, it is characterised in that described based on the multiple described service for distributing to the tenant
Device sets up cluster, each service data processing task in the data processing task model being assigned in the cluster
Device, makes each described server respectively while performing the data processing task, including：
Cluster is set up based on the multiple described server for distributing to the tenant, determine master server in the cluster and each
From server；
The data processing task model is sent to into the master server；
Control the master server data processing task model is assigned to described in each from server；
Control to parse the data processing task model from server described in each, obtain data processing sequence and control
Logic, makes to carry out data processing according to the data processing sequence and the control logic from server described in each.
6. a kind of distributed data processing component, it is characterised in that the component includes：
Signal generating unit, for the edit instruction according to tenant, generates data processing task model in visualization tasks editing machine；
Resolution unit, for receiving and parsing through the resource bid information of the tenant, obtains the server money needed for the tenant
Source；
Acquiring unit, for obtaining the running status of current each registered server；
Allocation unit, for the running status according to current each registered server, is tenant's distribution and institute
State the corresponding multiple servers of server resource needed for tenant；
Transmitting element, for setting up cluster based on the multiple described server for distributing to the tenant, the data processing is appointed
Each described server that data processing task in business model is sent in the cluster, makes each described server same respectively
Data processing task described in Shi Zhihang.
7. component according to claim 6, it is characterised in that the component also includes：
Tenant's administrative unit, for creating multiple tenants, generates the corresponding tenant's information of each described tenant respectively, and to described
Tenant's information is safeguarded；Tenant's information includes：Tenant identification, administrator password, source database and target database.
8. component according to claim 6, it is characterised in that the signal generating unit includes：
First generates subelement, for the edit instruction according to tenant to sequential processing pel each described, generates data processing
Sequentially, the data processing processing sequence is led corresponding to the data importing of Data processing, data detection, data processing and data
Go out four-stage；
Second generate subelement, for the edit instruction according to tenant to the controlling stream pel, will with to the controlling stream graph
The corresponding multiple described controlling stream pel of first edit instruction and the multiple described order by data processing sequence arrangement
Process pel to be combined, generate data processing task model；The controlling stream pel includes：Terminate operation pel, computational chart
Line number pel, setting variable pel, inspection field value pel, execution SQL pels, conversion pel and beginning pel.
9. component according to claim 6, it is characterised in that the allocation unit includes：
Subelement is obtained, for the running status according to current each registered server, unassigned data processing is obtained
Multiple servers of task；
Determination subelement, for the number of servers needed for the tenant is determined according to the server resource needed for the tenant；
Distribution subelement, for the multiple servers from the unassigned data processing task in, be tenant distribution institute
The server of quantity is needed, and the running status for distributing to the server of the tenant is revised as being allocated data processing times
Business.
10. component according to claim 6, it is characterised in that the transmitting element includes：
Subelement is set up, for cluster being set up based on the multiple described server for distributing to the tenant, in determining the cluster
Master server and each from server；
Subelement is sent, for the data processing task model is sent to the master server；
First control subelement, for control the master server by the data processing task model be assigned to described in each from
Server；
Second control subelement, for controlling to parse the data processing task model from server described in each, obtains
To data processing sequence and control logic, make described in each from server according to the data processing sequence and the control logic
Carry out data processing.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611051440.5A CN106550045A (en) | 2016-11-24 | 2016-11-24 | A kind of distributed data processing method and component |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611051440.5A CN106550045A (en) | 2016-11-24 | 2016-11-24 | A kind of distributed data processing method and component |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN106550045A true CN106550045A (en) | 2017-03-29 |
Family
ID=58395103
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611051440.5A Pending CN106550045A (en) | 2016-11-24 | 2016-11-24 | A kind of distributed data processing method and component |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106550045A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110020021A (en) * | 2018-09-03 | 2019-07-16 | 福建星瑞格软件有限公司 | Based on visual data flow processing method |
| CN111611236A (en) * | 2020-05-28 | 2020-09-01 | 宁波和利时智能科技有限公司 | Data analysis method and system |
| CN114661406A (en) * | 2022-04-08 | 2022-06-24 | 杭州半云科技有限公司 | Dynamic configuration visualization platform and method based on multi-tenant architecture |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102135883A (en) * | 2011-03-14 | 2011-07-27 | 山东大学 | Software-as-a-service (SaaS) application generation and deployment supporting method and device |
| CN102438049A (en) * | 2011-12-31 | 2012-05-02 | 东软集团股份有限公司 | System resource allocation method and device for software-as-a-service (SaaS) application system |
| US20140136712A1 (en) * | 2012-04-12 | 2014-05-15 | Lg Cns Co., Ltd. | Cloud resources as a service multi-tenant data model |
| CN104050201A (en) * | 2013-03-15 | 2014-09-17 | 伊姆西公司 | Method and equipment for managing data in multi-tenant distributive environment |
| CN104360859A (en) * | 2014-11-14 | 2015-02-18 | 中国建设银行股份有限公司 | Visual service development method and system |
| CN204695386U (en) * | 2015-06-01 | 2015-10-07 | 北京圆通慧达管理软件开发有限公司 | Towards the management information system of many tenants |
| CN105072182A (en) * | 2015-08-10 | 2015-11-18 | 北京佳讯飞鸿电气股份有限公司 | Load balancing method, load balancer and user terminal |
-
2016
- 2016-11-24 CN CN201611051440.5A patent/CN106550045A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102135883A (en) * | 2011-03-14 | 2011-07-27 | 山东大学 | Software-as-a-service (SaaS) application generation and deployment supporting method and device |
| CN102438049A (en) * | 2011-12-31 | 2012-05-02 | 东软集团股份有限公司 | System resource allocation method and device for software-as-a-service (SaaS) application system |
| US20140136712A1 (en) * | 2012-04-12 | 2014-05-15 | Lg Cns Co., Ltd. | Cloud resources as a service multi-tenant data model |
| CN104050201A (en) * | 2013-03-15 | 2014-09-17 | 伊姆西公司 | Method and equipment for managing data in multi-tenant distributive environment |
| CN104360859A (en) * | 2014-11-14 | 2015-02-18 | 中国建设银行股份有限公司 | Visual service development method and system |
| CN204695386U (en) * | 2015-06-01 | 2015-10-07 | 北京圆通慧达管理软件开发有限公司 | Towards the management information system of many tenants |
| CN105072182A (en) * | 2015-08-10 | 2015-11-18 | 北京佳讯飞鸿电气股份有限公司 | Load balancing method, load balancer and user terminal |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110020021A (en) * | 2018-09-03 | 2019-07-16 | 福建星瑞格软件有限公司 | Based on visual data flow processing method |
| CN110020021B (en) * | 2018-09-03 | 2023-04-07 | 福建星瑞格软件有限公司 | Visualization-based data stream processing method |
| CN111611236A (en) * | 2020-05-28 | 2020-09-01 | 宁波和利时智能科技有限公司 | Data analysis method and system |
| CN114661406A (en) * | 2022-04-08 | 2022-06-24 | 杭州半云科技有限公司 | Dynamic configuration visualization platform and method based on multi-tenant architecture |
| CN114661406B (en) * | 2022-04-08 | 2024-03-22 | 杭州半云科技有限公司 | Dynamic configuration visualization platform and method based on multi-tenant architecture |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103425778B (en) | A kind of intelligent development platform of database application system | |
| CN109308582A (en) | A kind of manufacturing execution system framework based on modularization Yu serviceization | |
| WO2019081085A1 (en) | Method and control system for controlling and/or monitoring devices | |
| CN104580284B (en) | Traffic assignments device and method for distributing business | |
| CN109740851A (en) | Design for Automatic Control Engineering work management method, platform and the system quickly delivered | |
| CN106276457B (en) | The method of elevator data management | |
| CN105260203B (en) | A kind of Hadoop deployment and collocation method based on model | |
| CN107423952A (en) | A kind of expansible DevOps continues delivery system and its method | |
| CN104731521B (en) | Method and apparatus for configuration data center | |
| WO2019081071A1 (en) | Method and control system for controlling and/or monitoring devices | |
| CN106550045A (en) | A kind of distributed data processing method and component | |
| CN109299083A (en) | A kind of data governing system | |
| CN106302483A (en) | Decentralized management method and system | |
| CN105786611A (en) | Method and device for task scheduling of distributed cluster | |
| CN106789186A (en) | A kind of regulation management method and device, UMP monitoring systems and monitoring method | |
| CN107623599A (en) | A kind of method and system of data configuration | |
| CN111880921A (en) | Job processing method and device based on rule engine and computer equipment | |
| CN109614190A (en) | Resources configuration management method and platform | |
| CN106354507A (en) | Enterprise-level application management system and method for operating same | |
| CN103677841B (en) | IETM code generation method and device based on element-level template | |
| CN102999810B (en) | A kind of bank application system management platform and authority control method | |
| CN108008977B (en) | Software development service platform oriented to function outsourcing | |
| WO2024140916A1 (en) | Internet of things permission matrix model and interaction method thereof | |
| CN112686580B (en) | Workflow definition method and system capable of customizing flow | |
| CN113986924A (en) | Intelligent data filling method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20170329 |
|
| RJ01 | Rejection of invention patent application after publication |