CN108153642A - A kind of method that selection calculate node is loaded according to operation - Google Patents
A kind of method that selection calculate node is loaded according to operation Download PDFInfo
- Publication number
- CN108153642A CN108153642A CN201611111263.5A CN201611111263A CN108153642A CN 108153642 A CN108153642 A CN 108153642A CN 201611111263 A CN201611111263 A CN 201611111263A CN 108153642 A CN108153642 A CN 108153642A
- Authority
- CN
- China
- Prior art keywords
- data
- node
- service
- calculate node
- load
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/3006—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Quality & Reliability (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The present invention relates to a kind of methods that selection calculate node is loaded according to operation, selection one is loaded according to the operation of each calculate node and serves as data serving node, data service preparation from control node is separated, mitigates the operation load of control node, accelerates system processing speed.
Description
【Technical field】
Selection calculate node method is loaded the invention belongs to data processing field more particularly to according to operation.
【Background technology】
Recently as the development of network technology, computing system can usually provide a variety of external services, in the prior art
For multi-node system, data service is typically to be provided by control node, when more since data service will expend
Between, control node, which carries out data service work, will necessarily occupy its comparable computing resource, since control node will also be appointed
Scheduling, distribution and resources control of business etc., therefore the processing load of control node has been aggravated, it will to the hardware configuration of control node
It asks very high, if operation load requirement is not achieved in control node configuration, is easy to cause control node crash, systemic breakdown.
Based on the above problem, there is an urgent need for a kind of new data service providing methods now, mitigate the operation load of control node,
Quickening system processing speed.
【Invention content】
In order to solve the above problem of the prior art, the present invention proposes data service side in a kind of multi-node system
Method.
The technical solution adopted by the present invention is as follows：
A kind of method that selection calculate node is loaded according to operation, which is characterized in that this method comprises the following steps：
(1) the load monitoring thread in control node monitors the operation load of each calculate node in real time；
(2) when control node receives data service request, above-mentioned load monitoring thread is current to each calculate node
The Real-time Monitoring Data of load operation conditions is analyzed and is compared, and selects the calculate node of wherein minimum operation load as number
According to service node；
(3) the data serving node operation service thread realizes the offer of service.
Beneficial effects of the present invention include：Selection one is loaded according to the operation of each calculate node and serves as data service section
Point separates data service work from control node, mitigates the operation load of control node, accelerates system processing speed
Degree.
【Description of the drawings】
Attached drawing described herein is to be used to provide further understanding of the present invention, and forms the part of the application, but
It does not constitute improper limitations of the present invention, in the accompanying drawings：
Fig. 1 is the structure chart of present system.
Fig. 2 is the flow chart for the method that the present invention loads selection calculate node according to operation.
【Specific embodiment】
Come that the present invention will be described in detail below in conjunction with attached drawing and specific embodiment, illustrative examples therein and say
It is bright to be only used for explaining the present invention, but not as a limitation of the invention.
It is the multi-node system that the present invention is applied referring to attached drawing 1, which includes a control node and multiple calculating
Node.
Referring to attached drawing 2, a kind of method that selection calculate node is loaded according to operation, this method comprises the following steps：
(1) the load monitoring thread in control node monitors the operation load of each calculate node in real time；
In one embodiment, following step can also be included before step (1)：By the node in system according to property
Multiple calculate nodes and a control node can be divided into, the calculate node group in the distributed system as computer resource
Into a group of planes, computer resource is provided to control node, and it is clear to be stored with scheduling thread, load monitoring thread, data in control node
Thread is washed, can also store data mining thread, wherein control node is responsible for task scheduling in system, each calculate node
Load monitoring and service is externally provided, in one embodiment, the service includes data mining service, in other implementations
In mode, other can also be included and externally serviced；
(2) when control node receives data service request, in embodiments of the present invention, the service is data cleansing
Service, in other embodiments or the services such as data mining, data query, above-mentioned load monitoring thread is to each
The Real-time Monitoring Data of calculate node present load operating status is analyzed and is compared, the meter of selection wherein minimum operation load
Operator node is as data cleansing node, in one embodiment, if there are 1 control node and 5 calculate nodes, Ge Geji
The system resource that the current operation task of operator node occupies respectively is 60%, 65%, 70%, 75%, 80%, then selects minimum fortune
The calculate node of row load 60% is as data cleansing node；The data cleansing thread that control node is stored sends to above-mentioned
Data cleansing node is data cleansing node in embodiments of the present invention, in other embodiments, or other data
Service node；
(3) the data cleansing node receives above-mentioned data cleansing thread and is stored in local and runs, and realizes data cleansing
Task, the data cleansing thread of the application is realized by data cleansing abstract component in one embodiment, is implemented at one
In mode, cleaning serviced component includes data normalization module, wrong data searching module, data de-duplication module, data
Association, data merging, data analysis, data enhancing module in other embodiments, can also increase and set other clear
Wash serviced component.
In one embodiment, for solving, multi-source data standard is skimble-scamble to ask the data normalization module
Topic, according to the unified and standard describing mode of the data warehouse pre-established, realizes full storage data standardized format；The error number
It is used to searching and deleting unreasonable data, illogical data and inconsistency data according to searching module；The data de-duplication mould
Block is used to identifying and deleting approximately duplicated data.
In one embodiment, the data analysis module is used to carry out initial data according to user-defined pattern
Correlation analysis is targetedly analyzed according to user-defined personality analysis demand；The data enhancing module is used for
Using external dictionaries and rule, incomplete data in initial data, the field omitted are supplemented or in a manner of increasing field
Add additional information.
In one embodiment, the data association module is used to find and identify related data and be associated, than
Such as be relevant to the age of same name field, professional field is associated, establish incidence relation；The data combiners block is used for
It was found that it and identifies homogeneous data and merges, for example multiple purchaser records under same date merge, and carry out purchase number
Amount merges addition or the quantity purchase of a middle of the month same article merges addition.
By the above method, the present invention loads selection one according to the operation of each calculate node and serves as data service section
Point separates data service from control node, mitigates the operation load of control node, accelerates system processing speed.
The above is only the better embodiment of the present invention, therefore all constructions according to described in present patent application range,
The equivalent change or modification that feature and principle are done, is included in the range of present patent application.
Claims (4)
- A kind of 1. method that selection calculate node is loaded according to operation, which is characterized in that this method comprises the following steps：(1) the load monitoring thread in control node monitors the operation load of each calculate node in real time；(2) when control node receives data service request, above-mentioned load monitoring thread is to each calculate node present load The Real-time Monitoring Data of operating status is analyzed and is compared, and the calculate node of wherein minimum operation load is selected to be taken as data Business node；(3) the data serving node operation service thread realizes the offer of service.
- 2. the method according to claim 1 that selection calculate node is loaded according to operation, which is characterized in that the service is Data mining service.
- 3. according to claim 1 load selection calculate node according to operation, which is characterized in that the service is looked into for data Ask service.
- 4. according to claim 1 load selection calculate node according to operation, which is characterized in that the service is clear for data The service of washing.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611111263.5A CN108153642A (en) | 2016-12-02 | 2016-12-02 | A kind of method that selection calculate node is loaded according to operation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611111263.5A CN108153642A (en) | 2016-12-02 | 2016-12-02 | A kind of method that selection calculate node is loaded according to operation |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108153642A true CN108153642A (en) | 2018-06-12 |
Family
ID=62468554
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611111263.5A Pending CN108153642A (en) | 2016-12-02 | 2016-12-02 | A kind of method that selection calculate node is loaded according to operation |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108153642A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109542360A (en) * | 2018-12-03 | 2019-03-29 | 郑州云海信息技术有限公司 | Data delete method, apparatus, equipment, system and computer readable storage medium again |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104331421A (en) * | 2014-10-14 | 2015-02-04 | 安徽四创电子股份有限公司 | High-efficiency processing method and system for big data |
| CN105094982A (en) * | 2014-09-23 | 2015-11-25 | 航天恒星科技有限公司 | Multi-satellite remote sensing data processing system |
-
2016
- 2016-12-02 CN CN201611111263.5A patent/CN108153642A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105094982A (en) * | 2014-09-23 | 2015-11-25 | 航天恒星科技有限公司 | Multi-satellite remote sensing data processing system |
| CN104331421A (en) * | 2014-10-14 | 2015-02-04 | 安徽四创电子股份有限公司 | High-efficiency processing method and system for big data |
Non-Patent Citations (1)
| Title |
|---|
| 赵蕴华等: "《服务于科技创新的专利分析实践与案例》", 31 August 2015 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109542360A (en) * | 2018-12-03 | 2019-03-29 | 郑州云海信息技术有限公司 | Data delete method, apparatus, equipment, system and computer readable storage medium again |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106776984B (en) | A kind of cleaning method of distributed system mining data | |
| Vera-Baquero et al. | Real-time business activity monitoring and analysis of process performance on big-data domains | |
| CN110008018B (en) | Batch task processing method, device and equipment | |
| CN106095940A (en) | A kind of data migration method of task based access control load | |
| CN107464151B (en) | Order data processing method and device for high-concurrency service | |
| TW200931285A (en) | Method, system and apparatus for combining distributed computational data | |
| CN111881221A (en) | Method, device and equipment for customer portrait in logistics service | |
| CN108804712A (en) | Data export method and device | |
| CN105989163A (en) | Data real-time processing method and system | |
| US20120089734A1 (en) | Allocation of resources between web services in a composite service | |
| CN109739919A (en) | A kind of front end processor and acquisition system for electric system | |
| CN112148779B (en) | Method, device and storage medium for determining business index | |
| CN110389989B (en) | Data processing method, device and equipment | |
| CN112559301A (en) | Service processing method, storage medium, processor and electronic device | |
| US8392577B2 (en) | Reduction of message flow between bus-connected consumers and producers | |
| Lin et al. | Research on building an innovative electric power marketing business application system based on cloud computing and microservices architecture technologies | |
| Fan et al. | Adaptive task scheduling in storm | |
| CN110266555A (en) | Method for analyzing web site service request | |
| CN110196868A (en) | Based on distributed work order flow monitoring method | |
| CN117633116A (en) | Data synchronization method, device, electronic equipment and storage medium | |
| Li et al. | Performance analysis of service clouds serving composite service application jobs | |
| CN108153642A (en) | A kind of method that selection calculate node is loaded according to operation | |
| US10067778B2 (en) | Management system, recording medium and method for managing virtual machines | |
| Wu et al. | Design and implementation of business-driven BI platform based on cloud computing | |
| CN108156207A (en) | A kind of data processing method of multi-node system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: 101399 No. 2 East Airport Road, Shunyi Airport Economic Core Area, Beijing (1st, 5th and 7th floors of Industrial Park 1A-4) Applicant after: Zhongke Star Map Co., Ltd. Address before: 101399 Building 1A-4, National Geographic Information Technology Industrial Park, Guomen Business District, Shunyi District, Beijing Applicant before: Space Star Technology (Beijing) Co., Ltd. |
|
| CB02 | Change of applicant information | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180612 |
|
| RJ01 | Rejection of invention patent application after publication |