CN112698949A - Multi-source heterogeneous flexible balanced acquisition method, electronic equipment and storage medium - Google Patents
Multi-source heterogeneous flexible balanced acquisition method, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN112698949A CN112698949A CN202011635306.6A CN202011635306A CN112698949A CN 112698949 A CN112698949 A CN 112698949A CN 202011635306 A CN202011635306 A CN 202011635306A CN 112698949 A CN112698949 A CN 112698949A
- Authority
- CN
- China
- Prior art keywords
- data acquisition
- acquisition
- service
- data
- acquisition service
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 28
- 238000004590 computer program Methods 0.000 claims description 4
- 238000013508 migration Methods 0.000 abstract description 7
- 230000005012 migration Effects 0.000 abstract description 7
- 238000013480 data collection Methods 0.000 description 6
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003032 molecular docking Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
- G06F9/5088—Techniques for rebalancing the load in a distributed system involving task migration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/505—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the load
Abstract
The invention provides a multi-source heterogeneous flexible balanced acquisition method, which comprises the following steps: registering IP, determining position, acquiring information of an acquisition object, and acquiring data. The balanced distribution of the acquisition threads is realized through a Hash algorithm, when the data acquisition service expands the capacity, only the acquisition thread of the previous node of the capacity expansion node needs to be migrated, when the data acquisition service contracts the capacity, only the acquisition thread of the capacity reduction node needs to be migrated, and the balanced distribution of the acquisition threads has no influence on other normal acquisition nodes. The invention relates to an electronic device and a storage medium, which are used for executing a multi-source heterogeneous flexible balanced acquisition method. The invention ensures the minimum migration of the acquisition threads and the data, is suitable for the distributed clusters, avoids the migration of a large amount of data, reduces the pressure of the server and increases the stability of the acquisition service.
Description
Technical Field
The invention relates to the technical field of data acquisition of the Internet of things, in particular to a multi-source heterogeneous flexible balanced acquisition method, electronic equipment and a storage medium.
Background
At present, more and more protocols and devices for data acquisition service docking of the internet of things of the system are used. The traditional main and standby data acquisition service has a single machine performance bottleneck. Some data acquisition has a single-machine performance bottleneck, and when dynamic service expansion is carried out, long-time interruption of acquisition service is caused, and acquisition threads and data are migrated in large quantities, so that the pressure of a server is increased, and important data of an acquisition object is lost for a long time.
Disclosure of Invention
In order to overcome the defects of the prior art, the invention aims to provide a multi-source heterogeneous flexible balanced acquisition method, which avoids mass data migration, reduces the pressure of a server and increases the stability of acquisition service.
The invention provides a multi-source heterogeneous flexible balanced acquisition method, which comprises the following steps:
registering the IP, and registering the IP of the node to a registration center when the data acquisition service is started;
determining the position, and after the data acquisition service receives the IP list of the current service notified by the registration center, identifying each data acquisition service IP to obtain the position of each data acquisition service;
acquiring acquisition object information, and pulling all object information to be acquired by the data acquisition service;
searching data acquisition service, wherein the data acquisition service traverses the acquisition object information, determines the position of the data acquisition service through an acquisition object IP, and searches the distributed data acquisition service through the determined position;
and data acquisition, namely starting a thread to acquire data when the searched data acquisition service is the current data acquisition service IP.
Further, in the step of determining the position, the data acquisition service IPs are calculated by using a Hash algorithm to obtain the position of each data acquisition service in a Hash ring.
Further, in the step of searching for data collection service, the collection object IP calculates a Hash value by using a Hash algorithm, and determines the position of the data on the Hash ring, and from this position, the data is searched clockwise along the ring, and the encountered node is the data collection service allocated to the node.
Further, in the data acquisition step, if the found data acquisition service is not the current data acquisition service IP, skipping is performed.
And further, the method also comprises data acquisition service capacity expansion, when the data acquisition service capacity expansion is carried out, a newly added service registration node IP is registered to a registration center, and the step of determining the position, the step of acquiring the acquired object information, the step of searching the data acquisition service and the step of acquiring the data are repeated.
Further, in the data collection service capacity expansion step, if the collection thread which is not started by the node is calculated and the node starts collection, the connection is closed, and the thread is destroyed.
And further, the method also comprises data acquisition service capacity reduction, and when the data acquisition service capacity reduction is carried out, the registration center is informed to remove the node, the acquisition thread of the node is closed, and the position determining step, the acquisition object information obtaining step, the data acquisition service searching step and the data acquisition step are repeated.
Further, the acquisition object information includes an acquisition protocol, an IP, and a port.
An electronic device, comprising: a processor;
a memory; and a program, wherein the program is stored in the memory and configured to be executed by a processor, the program comprising instructions for performing a multi-source heterogeneous flexible equilibrium acquisition method.
A computer-readable storage medium having stored thereon a computer program for execution by a processor of a multi-source heterogeneous flexible balanced acquisition method.
Compared with the prior art, the invention has the beneficial effects that:
the invention realizes the balanced distribution of the acquisition threads through the Hash algorithm, only needs to migrate the acquisition thread of the previous node of the capacity expansion node when the capacity expansion of the data acquisition service is carried out, and only needs to migrate the acquisition thread of the capacity reduction node when the capacity reduction of the data acquisition service is carried out, thereby having no influence on other normal acquisition nodes, ensuring the minimum migration of the acquisition thread and data, being suitable for distributed clusters, avoiding the migration of a large amount of data, reducing the pressure of a server and increasing the stability of the acquisition service.
The foregoing description is only an overview of the technical solutions of the present invention, and in order to make the technical solutions of the present invention more clearly understood and to implement them in accordance with the contents of the description, the following detailed description is given with reference to the preferred embodiments of the present invention and the accompanying drawings. The detailed description of the present invention is given in detail by the following examples and the accompanying drawings.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the invention without limiting the invention. In the drawings:
fig. 1 is a flow chart of a multi-source heterogeneous flexible equilibrium acquisition method of the present invention.
Detailed Description
The present invention will be further described with reference to the accompanying drawings and the detailed description, and it should be noted that any combination of the embodiments or technical features described below can be used to form a new embodiment without conflict.
A multi-source heterogeneous flexible equilibrium acquisition method is shown in figure 1 and comprises the following steps:
and registering the IP, and registering the IP of the node to a registration center when the data acquisition service is started. The node is the data acquisition service.
And after receiving the node registration information, the registration center informs all nodes of the IP list of the current service.
And determining the position, and identifying each data acquisition service IP after the data acquisition service receives the IP list of the current service notified by the registration center to obtain the position of each data acquisition service. Specifically, each data collection service IP is calculated using a Hash algorithm, so that the position of each data collection service on the Hash ring is determined.
Acquiring acquisition object information, and pulling all object information to be acquired from the cache by the data acquisition service, wherein the acquisition object information comprises configuration information such as acquisition protocols, IP (Internet protocol), ports and the like.
And searching data acquisition services, wherein the data acquisition services traverse the information of the acquisition object, determine the position of the data acquisition services through the IP of the acquisition object, and search the distributed data acquisition services through the determined position. Specifically, the Hash value of the collection object IP is calculated by using a Hash algorithm, the position of the data on the Hash ring is determined, the data is searched clockwise along the ring from the position, and the encountered node is the data collection service allocated to the node, which is assumed to be node a.
Data acquisition, namely starting a thread to acquire data when the calculated node A is the current data acquisition service IP; otherwise, skipping.
When the data acquisition service expands the capacity, registering a newly added service registration node IP to a registration center, and repeating the steps of determining the position, acquiring the information of the acquired object, searching the data acquisition service and acquiring the data. And if the collection thread which is not started by the node is calculated and the collection is started by the node, closing the connection and destroying the thread.
When the data acquisition service is reduced in volume, the registration center is informed to remove the node, the acquisition thread of the node is closed, and the steps of determining the position, acquiring the acquisition object information, searching the data acquisition service and acquiring the data are repeated.
An electronic device, comprising: a processor;
a memory; and a program, wherein the program is stored in the memory and configured to be executed by the processor, the program comprising instructions for performing a multi-source heterogeneous flexible balanced acquisition method.
A computer-readable storage medium having stored thereon a computer program for execution by a processor of a multi-source heterogeneous flexible balanced acquisition method.
The invention realizes the balanced distribution of the acquisition threads through the Hash algorithm, only needs to migrate the acquisition thread of the previous node of the capacity expansion node when the capacity expansion of the data acquisition service is carried out, and only needs to migrate the acquisition thread of the capacity reduction node when the capacity reduction of the data acquisition service is carried out, thereby having no influence on other normal acquisition nodes, ensuring the minimum migration of the acquisition thread and data, being suitable for distributed clusters, avoiding the migration of a large amount of data, reducing the pressure of a server and increasing the stability of the acquisition service.
The foregoing is merely a preferred embodiment of the invention and is not intended to limit the invention in any manner; those skilled in the art can readily practice the invention as shown and described in the drawings and detailed description herein; however, those skilled in the art should appreciate that they can readily use the disclosed conception and specific embodiments as a basis for designing or modifying other structures for carrying out the same purposes of the present invention without departing from the scope of the invention as defined by the appended claims; meanwhile, any changes, modifications, and evolutions of the equivalent changes of the above embodiments according to the actual techniques of the present invention are still within the protection scope of the technical solution of the present invention.
Claims (10)
1. A multi-source heterogeneous flexible equilibrium acquisition method is characterized by comprising the following steps:
registering the IP, and registering the IP of the node to a registration center when the data acquisition service is started;
determining the position, and after the data acquisition service receives the IP list of the current service notified by the registration center, identifying each data acquisition service IP to obtain the position of each data acquisition service;
acquiring acquisition object information, and pulling all object information to be acquired by the data acquisition service;
searching data acquisition service, wherein the data acquisition service traverses the acquisition object information, determines the position of the data acquisition service through an acquisition object IP, and searches the distributed data acquisition service through the determined position;
and data acquisition, namely starting a thread to acquire data when the searched data acquisition service is the current data acquisition service IP.
2. The multi-source heterogeneous flexible balanced acquisition method according to claim 1, characterized in that: in the step of determining the position, each data acquisition service IP is calculated by using a Hash algorithm to obtain the position of each data acquisition service in a Hash ring.
3. The multi-source heterogeneous flexible balanced acquisition method according to claim 2, characterized in that: in the step of searching data acquisition service, the IP of the acquisition object calculates a Hash value by using a Hash algorithm, and determines the position of the data on the Hash ring, and the data is searched clockwise along the ring from the position, and the encountered node is the data acquisition service distributed to the node.
4. The multi-source heterogeneous flexible balanced acquisition method according to claim 1, characterized in that: in the data acquisition step, if the searched data acquisition service is not the current data acquisition service IP, skipping.
5. The multi-source heterogeneous flexible balanced acquisition method according to claim 3, characterized in that: and the method also comprises data acquisition service capacity expansion, wherein when the data acquisition service capacity expansion is carried out, a newly added service registration node IP is registered to a registration center, and the step of determining the position, the step of acquiring the acquired object information, the step of searching the data acquisition service and the step of acquiring the data are repeated.
6. The multi-source heterogeneous flexible balanced acquisition method of claim 5, wherein: in the data acquisition service capacity expansion step, if the acquisition thread which is not started by the node is calculated and the node starts acquisition, the connection is closed, and the thread is destroyed.
7. The multi-source heterogeneous flexible balanced acquisition method according to claim 3, characterized in that: and the method also comprises data acquisition service capacity reduction, wherein when the data acquisition service capacity reduction is carried out, the registration center is informed to remove the node, the acquisition thread of the node is closed, and the step of determining the position, the step of acquiring the acquired object information, the step of searching the data acquisition service and the step of acquiring the data are repeated.
8. The multi-source heterogeneous flexible balanced acquisition method according to claim 1, characterized in that: the collection object information includes a collection protocol, an IP and a port.
9. An electronic device, characterized by comprising: a processor;
a memory; and a program, wherein the program is stored in the memory and configured to be executed by the processor, the program comprising instructions for carrying out the method according to any one of claims 1-8.
10. A computer-readable storage medium having stored thereon a computer program, characterized in that: the computer program is executed by a processor for performing the method according to any of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011635306.6A CN112698949A (en) | 2020-12-31 | 2020-12-31 | Multi-source heterogeneous flexible balanced acquisition method, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011635306.6A CN112698949A (en) | 2020-12-31 | 2020-12-31 | Multi-source heterogeneous flexible balanced acquisition method, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112698949A true CN112698949A (en) | 2021-04-23 |
Family
ID=75513784
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011635306.6A Pending CN112698949A (en) | 2020-12-31 | 2020-12-31 | Multi-source heterogeneous flexible balanced acquisition method, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112698949A (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020002614A1 (en) * | 1998-03-20 | 2002-01-03 | Sun Microsystems Inc. | Dynamic lookup service in distributed system |
| CN104301425A (en) * | 2014-10-24 | 2015-01-21 | 南方电网科学研究院有限责任公司 | Object registration center service cluster implementation method based on object type and Hash ring |
| WO2017028697A1 (en) * | 2015-08-17 | 2017-02-23 | 阿里巴巴集团控股有限公司 | Method and device for growing or shrinking computer cluster |
| CN111858033A (en) * | 2020-06-28 | 2020-10-30 | 上海缔安科技股份有限公司 | Load balancing method based on cluster and multiple processes |
-
2020
- 2020-12-31 CN CN202011635306.6A patent/CN112698949A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020002614A1 (en) * | 1998-03-20 | 2002-01-03 | Sun Microsystems Inc. | Dynamic lookup service in distributed system |
| CN104301425A (en) * | 2014-10-24 | 2015-01-21 | 南方电网科学研究院有限责任公司 | Object registration center service cluster implementation method based on object type and Hash ring |
| WO2017028697A1 (en) * | 2015-08-17 | 2017-02-23 | 阿里巴巴集团控股有限公司 | Method and device for growing or shrinking computer cluster |
| CN111858033A (en) * | 2020-06-28 | 2020-10-30 | 上海缔安科技股份有限公司 | Load balancing method based on cluster and multiple processes |
Non-Patent Citations (1)
| Title |
|---|
| 高聪等著: "《工业大数据融合体系结构与关键技术》", vol. 1, 机械工业出版社, pages: 65 - 69 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110865867B (en) | Method, device and system for discovering application topological relation | |
| US8595262B1 (en) | Resource resolution in computing environments using directed graphs | |
| US11341202B2 (en) | Efficient method of location-based content management and delivery | |
| US20150248443A1 (en) | Hierarchical host-based storage | |
| US20200272610A1 (en) | Method, apparatus, device and medium for storing and querying data | |
| CN110673928B (en) | Thread binding method, thread binding device, storage medium and server | |
| CN106951179B (en) | Data migration method and device | |
| US11500879B2 (en) | Method, device, and program product for managing index of streaming data storage system | |
| US11405861B2 (en) | Network repository function controller | |
| US9798674B2 (en) | N-ary tree for mapping a virtual memory space | |
| US20160006633A1 (en) | Monitoring item selection method and device, and storage medium | |
| CN111857539A (en) | Method, apparatus and computer program product for managing a storage system | |
| CN110945496A (en) | System and method for state object data store | |
| US10082978B2 (en) | Distributed shared log storage system having an adapter for heterogenous big data workloads | |
| CN111459650B (en) | Method, apparatus and medium for managing memory of dedicated processing resource | |
| US11288237B2 (en) | Distributed file system with thin arbiter node | |
| US10031777B2 (en) | Method and system for scheduling virtual machines in integrated virtual machine clusters | |
| US8341368B2 (en) | Automatic reallocation of structured external storage structures | |
| US20190215281A1 (en) | Fenced Clone Applications | |
| CN112698949A (en) | Multi-source heterogeneous flexible balanced acquisition method, electronic equipment and storage medium | |
| US10970285B2 (en) | Grid topology change in a distributed data grid when iterating on the contents of the data grid | |
| US20160098428A1 (en) | Creating and handling identification for a resource in a configuration database | |
| CN108228272B (en) | WEB container generation processing method, equipment and server | |
| CN113596098B (en) | Session retrieval method, apparatus, device and computer readable storage medium | |
| KR20160126148A (en) | Database read method and apparatus using t-tree index for improving read performance |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210423 |