CN109302438B - Radio monitoring big data concurrent processing system - Google Patents
Radio monitoring big data concurrent processing system Download PDFInfo
- Publication number
- CN109302438B CN109302438B CN201711491399.8A CN201711491399A CN109302438B CN 109302438 B CN109302438 B CN 109302438B CN 201711491399 A CN201711491399 A CN 201711491399A CN 109302438 B CN109302438 B CN 109302438B
- Authority
- CN
- China
- Prior art keywords
- sensor control
- server
- load
- task
- sensor
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000012544 monitoring process Methods 0.000 title claims abstract description 33
- 230000004044 response Effects 0.000 claims abstract description 6
- 238000003672 processing method Methods 0.000 claims description 5
- 230000004888 barrier function Effects 0.000 claims description 3
- 238000000034 method Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
- H04L67/025—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP] for remote control or remote monitoring of applications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/12—Protocols specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
Abstract
The invention relates to a radio monitoring big data concurrent processing system, which comprises: load balancing server: the execution tasks of the sensors are distributed to the sensor control servers in an equalizing mode, and the execution efficiency of the sensor control servers for concurrent processing of big data and the response capability of the system are improved; and meanwhile, disaster guarantee processing is provided, and once a certain sensor control server fails, monitoring tasks which are being executed by the sensor control server can be dynamically distributed to other sensor control servers to be executed. For a more complex monitoring system, a standby load balancing server can be provided, and when a main load balancing server fails, the standby load balancing server can be started to monitor and distribute tasks of a sensor control server. The invention introduces a plurality of sensor control services to relieve the pressure of the sensor control services. And the execution efficiency of each sensor control server and the response capability of the system are improved.
Description
Technical Field
The invention relates to the technical field of network big data processing, in particular to a radio monitoring big data concurrent processing system.
Background
The traditional radio monitoring system mainly controls a plurality of isomorphic nodes to execute radio monitoring tasks through a single sensor control server, and a Web server is responsible for providing presentation of user data and storing data acquired by the sensors in a database server in real time.
The traditional deployment mode can normally process the monitoring request of a user when the number of sensors in the system is small. However, as the deployment of the radio monitoring network goes deep, the number of sensors connected in the network is continuously increased and the types of the sensors are more diversified, so that the sensor control service can be caused to run in full load or even overload when a large number of sensors simultaneously perform tasks, and in addition, once the sensor control server fails, all the sensors cannot normally perform task execution and data acquisition. Therefore, it is necessary to introduce multiple sensor control servers to share the pressure of multiple sensor task execution and data acquisition.
Disclosure of Invention
The invention overcomes the defects existing in the prior art, and solves the technical problems that: a load balancing method for concurrent processing of big data is provided.
In order to solve the technical problems, the invention adopts the following technical scheme: a radio monitoring big data concurrent processing method comprises the following steps:
s1) requesting, wherein a Web server receives a data acquisition request sent by a client;
s2) analyzing the request, and analyzing the client request into tasks of different sensors;
s3) load balancing, wherein a load balancing server calculates the load of a task according to the type of the sensor and the task type; sequentially distributing tasks to the proper sensor control servers;
searching whether a sensor control server is already present to manage the sensor related to the task, and if the sensor related to the task is not present, directly distributing the task to the sensor control server with the minimum load;
if the task load exists, judging whether the sensor control server can accept the new task load, if so, distributing the task to the sensor control server, and if not, directly distributing the task to the sensor control server with the minimum load in other servers;
the load balancing server maintains a load index table to record the load of each sensor control server, and when the task of the sensor control server is executed, the load balancing server is also required to update the load index table, so that the sensor control server with the minimum load can be obtained according to the load index table;
when a new task is allocated to a certain sensor control server, the load index of the sensor control server also changes, and the sensor control server may not keep the load to the minimum any more;
s4) feeding back results, wherein the load balancing server is responsible for receiving task execution results of the sensor control servers, and feeding back the results to the client through the Web server after merging processing.
A radio monitoring big data concurrency processing system for implementing a radio monitoring big data concurrency processing method, comprising:
the user: a dispatcher of the sensor task and an observer of the collected data; accessing a radio monitoring system through an lnternet by adopting terminal equipment;
a sensor: heterogeneous nodes are formed by sensors of different types of the same equipment manufacturer or sensors of different equipment manufacturers;
a firewall: as a security barrier between the external network and the internal network;
the server switch: connecting different servers;
sensor control server: the data receiving and transmitting of the sensors are respectively controlled by a plurality of sensor control servers, and the specific sensors are controlled by which sensor control server and are dynamically distributed by a load balancing server by adopting a load balancing algorithm when the sensor tasks are distributed;
load balancing server: the execution tasks of the sensors are distributed to the sensor control servers in an equalizing mode, and the execution efficiency of the sensor control servers for concurrent processing of big data and the response capability of the system are improved; and meanwhile, disaster guarantee processing is provided, and once a certain sensor control server fails, monitoring tasks which are being executed by the sensor control server can be dynamically distributed to other sensor control servers to be executed. For a more complex monitoring system, a standby load balancing server can be provided, and when a main load balancing server fails, the standby load balancing server can be started to monitor and distribute tasks of a sensor control server;
the Web server: providing browsing service of information on the Web client; because the number of the existing monitoring users is limited, a Web server is adopted to receive a task list issued by a user, data acquired by a sensor are presented to the user, and the number of the users is increased along with the increase of the future users, and the task list is expanded into a plurality of Web servers and a load balancing server;
database server: the data collected by the sensor is stored in real time, and the data is realized in a database cluster mode due to the large data volume.
The invention also provides a processing method of the novel radio monitoring big data concurrency processing system, which comprises the following steps:
step 1, requesting, wherein a Web server receives a data acquisition request sent by a client;
step 2, analyzing the request, and analyzing the client request into tasks of different sensors;
step 3, load balancing, namely calculating the load of the task according to the type of the sensor and the task type by a load balancing server; sequentially distributing tasks to the proper sensor control servers;
searching whether a sensor control server is already present to manage the sensor related to the task, and if the sensor related to the task is not present, directly distributing the task to the sensor control server with the minimum load;
if the task load exists, judging whether the sensor control server can accept the new task load, if so, distributing the task to the sensor control server, and if not, directly distributing the task to the sensor control server with the minimum load in other servers;
the load balancing server maintains a load index table to record the load of each sensor control server, and when the task of the sensor control server is executed, the load balancing server is also required to update the load index table, so that the sensor control server with the minimum load can be obtained according to the load index table;
when a new task is allocated to a certain sensor control server, the load index of the sensor control server also changes, and the sensor control server may not keep the load to the minimum any more;
and 4, feeding back a result, wherein the load balancing server is responsible for receiving the task execution result of each sensor control server, and feeding back the result to the client through the Web server after the combination processing.
Compared with the prior art, the invention has the following beneficial effects: the invention introduces a plurality of sensor control services to relieve the pressure of the sensor control services. Meanwhile, a load balancing server is added and used for distributing requests of the sensors for executing tasks to the sensor control servers in a balanced mode, and therefore execution efficiency of the sensor control servers and response capacity of the system are improved.
Drawings
The invention is described in further detail below with reference to the accompanying drawings.
Fig. 1 is a topology diagram of a radio monitoring system of the present invention.
FIG. 2 is a flow chart of system task allocation in accordance with the present invention.
Fig. 3 is a load balancing sub-flow diagram of the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention; all other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
As shown in fig. 1, a radio monitoring big data concurrency processing system of the present invention includes:
the user: a dispatcher of the sensor task and an observer of the collected data. Desktop computers, portable computers, tablet computers, portable terminals and other devices can be adopted to access the radio monitoring system through lnternet
A sensor: heterogeneous nodes. Is composed of sensors of different types of the same equipment manufacturer or sensors of different equipment manufacturers
A firewall: as a security barrier between external and internal networks, enforcing network security policies, and preventing internal information from escaping
The server switch: different servers are communicated, and efficiency of network data transmission is improved
Sensor control server: the data receiving and transmitting of the sensors are respectively controlled by a plurality of sensor control servers, and the specific sensor is controlled by which sensor control server and is dynamically distributed by a load balancing server by adopting a load balancing algorithm when the sensor task is distributed
Load balancing server: and the execution tasks of the sensors are distributed to the sensor control servers in an equalizing manner, so that the execution efficiency of the sensor control servers for concurrent processing of big data and the response capability of the system are improved. And meanwhile, disaster guarantee processing is provided, and once a certain sensor control server fails, monitoring tasks which are being executed by the sensor control server can be dynamically distributed to other sensor control servers to be executed. For a more complex monitoring system, a standby load balancing server can be provided, and when a main load balancing server fails, the standby load balancing server can be started to realize task monitoring and allocation of a sensor control server
The Web server: and providing a browsing service of information on the Web client, adopting a Web server to receive a task list issued by a user due to the limited number of the existing monitoring users, and presenting data acquired by the sensor to the user. With the increase of the number of future users, the system is expanded into a plurality of Web servers and load balancing servers. Web load balancing is not within the description of this document
Database server: the data collected by the sensor is stored in real time, and the data is realized in a database cluster mode due to the large data volume.
As shown in fig. 2, the method for processing radio monitoring big data concurrency of the invention comprises the following steps:
(1) Request for
And the Web server receives the data acquisition request sent by the client.
(2) Resolving requests
The client requests are parsed into tasks of different sensors.
(3) Load balancing
And calculating the load of the task according to the type of the sensor and the task type by the load balancing server.
Tasks are assigned in turn to the appropriate sensor control server.
First, it is searched whether or not there is already a sensor related to the task managed by the sensor control server. If not, the task is directly distributed to the sensor control server with the least load.
Second, if already present, it is determined whether the sensor control server is able to accept the new task load. If so, a task is assigned to the sensor control server. If not, the task is directly distributed to the sensor control server with the least load in other servers.
Finally, the load balancing server maintains a load index table to record the load of each sensor control server. When the task of the sensor control server is completed, the load balancing server is also requested to update the load index table. Therefore, according to this load index table, the sensor control server with the smallest load can be obtained.
When a new task is assigned to a sensor control server, its load index will also change. The sensor control server may no longer keep the load to a minimum.
(4) Feedback of results
The load balancing server is responsible for receiving the task execution results of the sensor control servers, and feeds back the results to the client through the Web server after the combination processing.
Task allocation principle:
(1) The existing relation between the sensor and the control server is not changed as much as possible; the control server switching the management sensor may affect the execution efficiency of the task.
(2) The control server with the smallest load capacity is distributed to the control server with the smallest load capacity as far as possible; the control server may already have a sensor to manage before managing the new sensor.
As shown in fig. 3, in the load balancing sub-process of the present invention, the sensor control server is responsible for managing the execution of tasks performed by the sensor. From the sensor point of view, the greater the number of sensors managed by the sensor control server, the greater the load. From a task perspective, the more tasks the sensor control server manages, the more load.
In addition, the type of sensor and the content of the task are also two very important factors affecting the sensor control server load.
(1) The same sensor, different tasks. Different types of tasks are issued to the same type of sensor, and the amount of data reported by the sensor is different. For example, there is a very significant difference in the amount of data between the status information of the sensor and the traffic data collected by the sensor.
(2) Different sensors, the same task. Generally, different sensors originate from different manufacturers. The same task, different manufacturers have different implementations.
(3) Different sensors, different tasks
Because of the heterogeneous nodes in the system, the limit of the sensor control server load is determined by the load generated by the sensor task, rather than by the type of sensor. Therefore, the load index of the sensor control server can more accurately reflect the load accepted by the server.
The load calculation mode is as follows:
(1) Load index
And the load index is used for indicating the load of the sensor control server. One sensor controls the load of the server to be equal to the sum of the loads of the tasks it manages. The formula for calculating the load index is as follows. Wherein y represents a load index, i represents the number of tasks, n represents the number of tasks carried by the sensor control server, and x represents a task load index.
(2) Task load index
The task load index is used for indicating the load generated by a task. That is, the sensor control server adds a load added to the task. The larger the index, the greater the load added. The functional formula for calculating the task load index is as follows. Wherein x represents a task load index, x 1 Indicating sensor type, x 2 Indicating the type of task being performed by the sensor.
x＝F(x 1 ，x 2 )
Typically, the mission load index is measured by a specific combination of sensors and mission.
Task allocation principle
(1) The existing relation between the sensor and the control server is not changed as much as possible; the control server switching the management sensor may affect the execution efficiency of the task.
(2) The control server with the smallest load capacity is distributed to the control server with the smallest load capacity as far as possible; the control server may already have sensors to be managed before managing the new sensors.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and not for limiting the same; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some or all of the technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit of the invention.
Claims (2)
1. A radio monitoring big data concurrency processing method 1, characterized by comprising the following steps:
s1) requesting, wherein a Web server receives a data acquisition request sent by a client;
s2) analyzing the request, and analyzing the client request into tasks of different sensors;
s3) load balancing, namely calculating the load of the task by a load balancing server according to the type of the sensor and the task type;
searching whether a sensor control server is already present to manage the sensor related to the task, and if the sensor related to the task is not present, directly distributing the task to the sensor control server with the minimum load;
if the task load exists, judging whether the sensor control server can accept the new task load, if so, distributing the task to the sensor control server, and if not, directly distributing the task to the sensor control server with the minimum load in other servers;
the load balancing server maintains a load index table to record the load of each sensor control server, and when the task of the sensor control server is executed, the load balancing server is also required to update the load index table, so that the sensor control server with the minimum load can be obtained according to the load index table;
when a new task is allocated to a certain sensor control server, the load index of the sensor control server also changes, and the sensor control server may not keep the load to the minimum any more;
s4) feeding back results, wherein the load balancing server is responsible for receiving task execution results of the sensor control servers, and feeding back the results to the client through the Web server after merging processing.
2. A radio monitoring big data concurrency processing system for implementing a radio monitoring big data concurrency processing method as defined in claim 1, comprising:
the user: a dispatcher of the sensor task and an observer of the collected data; a terminal device is adopted to access a radio monitoring system through the Internet;
a sensor: heterogeneous nodes are formed by sensors of different types of the same equipment manufacturer or sensors of different equipment manufacturers;
a firewall: as a security barrier between the external network and the internal network;
the server switch: connecting different servers;
sensor control server: the data receiving and transmitting of the sensors are respectively controlled by a plurality of sensor control servers, and the specific sensors are controlled by which sensor control server and are dynamically distributed by a load balancing server by adopting a load balancing algorithm when the sensor tasks are distributed;
load balancing server: the execution tasks of the sensors are distributed to the sensor control servers in an equalizing mode, and the execution efficiency of the sensor control servers for concurrent processing of big data and the response capability of the system are improved; meanwhile, disaster guarantee processing is provided, and once a certain sensor control server fails, monitoring tasks which are being executed by the sensor control server can be dynamically distributed to other sensor control servers to be executed; for a more complex monitoring system, a standby load balancing server can be provided, and when a main load balancing server fails, the standby load balancing server can be started to monitor and distribute tasks of a sensor control server;
the Web server: providing browsing service of information on the Web client; because the number of the existing monitoring users is limited, a Web server is adopted to receive a task list issued by a user, data acquired by a sensor are presented to the user, and the number of the users is increased along with the increase of the future users, and the task list is expanded into a plurality of Web servers and a load balancing server;
database server: the data collected by the sensor is stored in real time, and the data is realized in a database cluster mode due to the large data volume.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711491399.8A CN109302438B (en) | 2017-12-30 | 2017-12-30 | Radio monitoring big data concurrent processing system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711491399.8A CN109302438B (en) | 2017-12-30 | 2017-12-30 | Radio monitoring big data concurrent processing system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109302438A CN109302438A (en) | 2019-02-01 |
| CN109302438B true CN109302438B (en) | 2023-10-31 |
Family
ID=65168193
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711491399.8A Active CN109302438B (en) | 2017-12-30 | 2017-12-30 | Radio monitoring big data concurrent processing system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109302438B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2005148911A (en) * | 2003-11-12 | 2005-06-09 | Nec Corp | Load distribution method and device, system and its program |
| CN105141541A (en) * | 2015-09-23 | 2015-12-09 | 浪潮(北京)电子信息产业有限公司 | Task-based dynamic load balancing scheduling method and device |
| CN105139118A (en) * | 2015-08-19 | 2015-12-09 | 国网山东省电力公司东营供电公司 | Distribution network fault first-aid repair power failure information reporting system and method |
| CN107104858A (en) * | 2017-06-09 | 2017-08-29 | 携程旅游信息技术（上海）有限公司 | The monitoring system of Web SiteServer LBSs |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2015187946A1 (en) * | 2014-06-05 | 2015-12-10 | KEMP Technologies Inc. | Adaptive load balancer and methods for intelligent data traffic steering |
| JP6225960B2 (en) * | 2015-08-18 | 2017-11-08 | コニカミノルタ株式会社 | Network system, load suppression control program, and load suppression control method |
| CN207853938U (en) * | 2017-12-30 | 2018-09-11 | 上海创远仪器技术股份有限公司 | A kind of novel radio pyroelectric monitor big data concurrent processing system |
-
2017
- 2017-12-30 CN CN201711491399.8A patent/CN109302438B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2005148911A (en) * | 2003-11-12 | 2005-06-09 | Nec Corp | Load distribution method and device, system and its program |
| CN105139118A (en) * | 2015-08-19 | 2015-12-09 | 国网山东省电力公司东营供电公司 | Distribution network fault first-aid repair power failure information reporting system and method |
| CN105141541A (en) * | 2015-09-23 | 2015-12-09 | 浪潮(北京)电子信息产业有限公司 | Task-based dynamic load balancing scheduling method and device |
| CN107104858A (en) * | 2017-06-09 | 2017-08-29 | 携程旅游信息技术（上海）有限公司 | The monitoring system of Web SiteServer LBSs |
Non-Patent Citations (1)
| Title |
|---|
| SDN-based scheduling strategy on load balancing of virtual sensor resources in sensor-cloud;F. Banaie;《2016 8th International Symposium on Telecommunications (IST)》;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109302438A (en) | 2019-02-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10855545B2 (en) | Centralized resource usage visualization service for large-scale network topologies | |
| US9971823B2 (en) | Dynamic replica failure detection and healing | |
| US9647904B2 (en) | Customer-directed networking limits in distributed systems | |
| US9369389B1 (en) | Dynamic isolation of shared resources | |
| US9813491B2 (en) | Highly available network filer with automatic load balancing and performance adjustment | |
| EP2563062B1 (en) | Long connection management apparatus and link resource management method for long connection communication | |
| CA2891339C (en) | Distributed caching cluster management | |
| US8447757B1 (en) | Latency reduction techniques for partitioned processing | |
| CN111124670A (en) | Tactical cloud-oriented micro-service platform | |
| US9075660B2 (en) | Apparatus and method for providing service availability to a user via selection of data centers for the user | |
| GB2407887A (en) | Automatically modifying fail-over configuration of back-up devices | |
| US20060224731A1 (en) | Method for measuring server performance, system for measuring server performance and computer programs therefor | |
| US9875061B2 (en) | Distributed backup system | |
| US20150215394A1 (en) | Load distribution method taking into account each node in multi-level hierarchy | |
| CN103607424A (en) | Server connection method and server system | |
| CN111556135A (en) | Request scheduling method, system and device and electronic equipment | |
| US10771372B2 (en) | Transmitting test traffic on a communication link | |
| WO2020094064A1 (en) | Performance optimization method, device, apparatus, and computer readable storage medium | |
| US20160043894A1 (en) | Automatic reconfiguration of network parameters during file system failover | |
| CN109302438B (en) | Radio monitoring big data concurrent processing system | |
| CN109510730A (en) | Distributed system and its monitoring method, device, electronic equipment and storage medium | |
| CN110611688B (en) | Method, electronic device and computer program product for searching nodes | |
| KR101275995B1 (en) | Data storing system and data processing method of informational household appliance | |
| US10812390B2 (en) | Intelligent load shedding of traffic based on current load state of target capacity | |
| EP3306471B1 (en) | Automatic server cluster discovery |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |