CN112988505A - Cloud computing-based general real-time big data monitoring and early warning system - Google Patents
Cloud computing-based general real-time big data monitoring and early warning system Download PDFInfo
- Publication number
- CN112988505A CN112988505A CN202110181011.4A CN202110181011A CN112988505A CN 112988505 A CN112988505 A CN 112988505A CN 202110181011 A CN202110181011 A CN 202110181011A CN 112988505 A CN112988505 A CN 112988505A
- Authority
- CN
- China
- Prior art keywords
- data
- module
- real
- time
- early warning
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/3006—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems
Landscapes
- Engineering & Computer Science (AREA)
- Computing Systems (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Mathematical Physics (AREA)
- Quality & Reliability (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a cloud computing-based big data universal real-time monitoring and early warning system, which comprises a data acquisition and caching module, a data reporting module, a data aggregation gateway analysis module, a data stream time sequence persistence module, a real-time data stream computing module, an early warning notification module and a monitoring visual interface module; the data collection and caching module is connected with the data reporting module, the data reporting module is connected with the data convergence gateway analysis module, and the data convergence gateway analysis module is connected with the task queue cluster. The cloud service system improves the operation stability and the service reliability of the whole cloud service system, can realize early warning protection in advance of an accident, quick response of faults in the accident and root tracing of faults after the accident, promotes the cloud user experience of the whole user for cloud computing, big data processing, real-time computing and machine learning, promotes the updating and development of industrial technology, and responds to national calls; the service range is expanded infinitely, the industry integration is promoted, and the access to a three-party platform and the application are facilitated.
Description
Technical Field
The invention relates to the field of cloud computing data communication real-time computing, in particular to a cloud computing-based big data universal real-time monitoring and early warning system.
Background
With the continuous development of information technology, cloud computing becomes a new computing mode following distributed computing, parallel computing, grid computing and the like, services such as resource renting, service outsourcing and application hosting can be provided for users, and the cloud computing is a hotspot in the development of information technology due to the advantages of simplicity, convenience, economy, easy expandability and the like, but brings convenience to users and simultaneously brings great challenges to the safety and maintenance of user information safety assets. At present, solutions are urgently needed to be provided for automatic operation and maintenance and real-time alarm early warning, and the key and the urgency of the solutions are not ignored for a long time.
In the cloud computing virtual environment, a plurality of virtual machines exist in a physical server, and each virtual machine bears a different service system. Meanwhile, the flow among different virtual machines in the same physical server. And different service scenarios and dynamic complex service scenarios, very high requirements are put forward on the monitoring and warning system.
Disclosure of Invention
The invention aims to improve the operation stability and service reliability of the cloud service overall system, realize early warning protection in advance of accidents, quick response to faults in the accidents and root tracing of faults after the accidents, promote cloud user experience on the overall user for cloud computing, big data processing, real-time computing and machine learning,
promote the updating and development of the industrial technology;
the cloud computing-based big data universal real-time monitoring and early warning system is realized by the following technical scheme: the system comprises a data acquisition and caching module, a data reporting module, a data aggregation gateway analysis module, a data stream time sequence persistence module, a real-time data stream calculation module, an early warning notification module and a monitoring visual interface module;
the data collection and caching module is connected with the data reporting module, the data reporting module is connected with the data convergence gateway analysis module, and the data convergence gateway analysis module is connected with the task queue cluster; the task queue cluster is connected with the data stream time sequence persistence module, the task queue cluster is connected with the data storage cluster, and the data storage cluster is connected with the monitoring visual interface module; the real-time data flow calculation module and the early warning notification module are connected with the task queue cluster.
As a preferred technical scheme, the data reporting module includes a data header portion and a data body portion, and the data header portion is generalized by a separate data protocol header field.
As a preferred technical scheme, the data flow calculation module mainly stores data and historical data in a large-scale memory cluster pool, and mainly comprises real-time online tasks and offline tasks; the real-time online task mainly processes corresponding real-time data and needs to be triggered immediately and inform a user of the message type in time; the main processing of the off-line task is off-line report and prediction type task, and the data does not need to be informed to the user in real time.
The invention has the beneficial effects that: the cloud service system improves the operation stability and the service reliability of the whole cloud service system, can realize early warning protection in advance of an accident, quick response of faults in the accident and root tracing of faults after the accident, promotes the cloud user experience of the whole user for cloud computing, big data processing, real-time computing and machine learning, promotes the updating and development of industrial technology, and responds to national calls; the service range is expanded infinitely, the industry integration is promoted, and the access to a three-party platform and the application are facilitated.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a schematic diagram of the system of the present invention;
FIG. 2 is a schematic diagram of a protocol framework of the present invention;
fig. 3 is a schematic diagram of an alarm process according to the present invention.
Detailed Description
All of the features disclosed in this specification, or all of the steps in any method or process so disclosed, may be combined in any combination, except combinations of features and/or steps that are mutually exclusive.
Any feature disclosed in this specification (including any accompanying claims, abstract and drawings), may be replaced by alternative features serving equivalent or similar purposes, unless expressly stated otherwise. That is, unless expressly stated otherwise, each feature is only an example of a generic series of equivalent or similar features.
In the description of the present invention, it is to be understood that the terms "one end", "the other end", "outside", "upper", "inside", "horizontal", "coaxial", "central", "end", "length", "outer end", and the like, indicate orientations or positional relationships based on those shown in the drawings, and are used only for convenience in describing the present invention and for simplicity in description, and do not indicate or imply that the device or element being referred to must have a particular orientation, be constructed in a particular orientation, and be operated, and thus, should not be construed as limiting the present invention.
Further, in the description of the present invention, "a plurality" means at least two, e.g., two, three, etc., unless specifically limited otherwise.
The use of terms such as "upper," "above," "lower," "below," and the like in describing relative spatial positions herein is for the purpose of facilitating description to describe one element or feature's relationship to another element or feature as illustrated in the figures. The spatially relative positional terms may be intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the figures. For example, if the device in the figures is turned over, elements described as "below" or "beneath" other elements or features would then be oriented "above" the other elements or features. Thus, the exemplary term "below" can encompass both an orientation of above and below. The device may be otherwise oriented and the spatially relative descriptors used herein interpreted accordingly.
In the present invention, unless otherwise explicitly specified or limited, the terms "disposed," "sleeved," "connected," "penetrating," "plugged," and the like are to be construed broadly, e.g., as a fixed connection, a detachable connection, or an integral part; can be mechanically or electrically connected; they may be directly connected or indirectly connected through intervening media, or they may be connected internally or in any other suitable relationship, unless expressly stated otherwise. The specific meanings of the above terms in the present invention can be understood by those skilled in the art according to specific situations.
As shown in fig. 1-3, the cloud computing-based general real-time big data monitoring and early warning system of the present invention includes a data acquisition and caching module, a data reporting module, a data aggregation gateway parsing module, a data stream timing persistence module, a real-time data stream computing module, an early warning notification module, and a monitoring visualization interface module;
the data acquisition and caching module is used for acquiring various concerned type index data serialization and encryption compressed data by the client, and the data entering the data reporting module is reported to the cloud computing system by the reporting module in a form of a two-in-flow encryption compressed data flow;
the data convergence gateway analysis module is used for receiving the encrypted compressed binary data stream from the data reporting module, decrypting the data, disassembling the head of the data packet, reading the flag bit of the data packet, determining the type of the data packet, and putting the data body into a corresponding queue;
the data flow time sequence persistence module is used for receiving data contents in the designated queue, each module designs ideas according to a single principle, each module does a small amount of work as much as possible, and the data contents are correspondingly processed; thereby achieving high-concurrency and high-performance data processing; and a super-large data processing cluster is formed by combining distributed micro-service independent service application deployment;
the real-time data flow calculation module is used for receiving data, storing the real-time data and historical data in a memory pool, and dividing real-time source data and the historical data into a real-time online warning task and an offline warning task according to a warning model:
real-time on-line alarm task, comparing real-time current data with historical data through a real-time computing system, obtaining alarm push according to an alarm rule algorithm, writing the alarm push message back to a message queue, and then pushing the message to various third party access parties through an early warning notification module;
an offline early warning task: predicting alarm information to be generated by the machine according to the real-time data and the historical data according to the early warning model and the early warning model generated by autonomous machine learning, thereby achieving the aim of early warning;
monitoring the visual interface module: and the monitoring system is used for providing the generated historical alarm report and the concerned monitoring index data to a three-party platform in an access visualization display mode in a RestFul API (application program interface) mode.
In this embodiment, the data reporting module includes a data header portion and a data body portion; the data header is deployed and generalized by a single data protocol header field (the types of the concerned indexes are marked on the data header by various products and data, and the product flag bit of the data header identifies the products, so that the data types can be rapidly determined by a data convergence gateway analysis module and are delivered to a corresponding data queue, and the purpose of rapidly processing the data is achieved); the project mainly realizes that business module direction expansion can be realized by plugging in various functions; to support the expansion of more service types to achieve more generalization.
In this embodiment, the real-time data stream calculation module mainly stores data and historical data in a large memory cluster pool, so as to realize real-time calculation; the method mainly comprises the following steps of real-time online tasks and offline tasks:
the real-time online task mainly processes corresponding real-time data, and needs to be triggered immediately and inform a user of the message type in time;
in the first online task scenario, a user system monitoring threshold alarm is split by the data aggregation gateway analysis module 3 and then put into a message queue. Receiving the data by a real-time data stream calculation module and then setting an alarm threshold rule according to a user;
triggering an alarm when the CPU utilization rate reaches 90% within 5 minutes continuously according to a set rule such as CPU utilization rate;
when the real-time data stream calculation module receives the current reported value collected by the data collection caching module, the current utilization rate of the CPU in the data packet is solved;
firstly, judging whether 90% of the data reaches a set threshold value, if so, marking the trigger rule of the data with a bit of 1, and judging whether the trigger rule of the data meets the requirement of reaching the set threshold value for 5 minutes continuously from historical data;
if the continuous 5 minutes reach the trigger value, the marking bit of whether the data needs to push the message is 1;
if the trigger value is not reached within 5 minutes, storing the database in the memory to become historical data;
if the condition message is judged to be a message needing to be pushed, the resource waste condition caused by malicious attack or other reasons is still protected according to an anti-flooding mechanism (if the index alarm value is pushed within one hour, the message is not pushed; the message is pushed after the next hour);
when a push message is triggered and no message has been pushed within an hour. Writing the push message back to a push alarm message queue;
and when the early warning notification module 6 receives the push message data in the warning message queue. Pushing to a unified platform according to a pushing mode set by a user;
when the visual plug-in is required to be displayed to the user for visualization, the visual plug-in is customized according to the requirements of the user by the monitoring visual interface module 7;
the main processing of the offline task is offline report and prediction type task, and the data can be informed to the user without real time.
In the embodiment, the plug-in customization and access permission of the third-party platform are realized; and returning the concerned monitoring index data or report data to the data in an interface form so as to meet the requirements of data visualization data large screen and the like.
The above description is only an embodiment of the present invention, but the scope of the present invention is not limited thereto, and any changes or substitutions that are not thought of through the inventive work should be included in the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope defined by the claims.
Claims (3)
1. The utility model provides a based on general type real time monitoring early warning system of cloud computing big data which characterized in that: the system comprises a data acquisition and caching module, a data reporting module, a data aggregation gateway analysis module, a data stream time sequence persistence module, a real-time data stream calculation module, an early warning notification module and a monitoring visual interface module;
the data collection and caching module is connected with a data reporting module, the data reporting module is connected with a data convergence gateway analysis module, and the data convergence gateway analysis module is connected with a task queue cluster; the task queue cluster is connected with a data stream time sequence persistence module, the task queue cluster is connected with a data storage cluster, and the data storage cluster is connected with a monitoring visual interface module; the real-time data flow calculation module and the early warning notification module are connected with the task queue cluster.
2. The cloud computing big data general real-time monitoring and early warning system based on claim 1, characterized in that: the data reporting module comprises a data head part and a data body part, and the data head part is generalized by a single data protocol head field.
3. The cloud computing big data general real-time monitoring and early warning system based on claim 1, characterized in that: the data flow calculation module is mainly used for storing data and historical data in a large-scale memory cluster pool and mainly comprises real-time online tasks and offline tasks; the real-time online task mainly processes corresponding real-time data and needs to be triggered immediately and inform a user of the message type in time; the main processing of the off-line task is off-line report forms and prediction type tasks, and the data does not need to be informed to a user in real time.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110181011.4A CN112988505A (en) | 2021-02-08 | 2021-02-08 | Cloud computing-based general real-time big data monitoring and early warning system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110181011.4A CN112988505A (en) | 2021-02-08 | 2021-02-08 | Cloud computing-based general real-time big data monitoring and early warning system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112988505A true CN112988505A (en) | 2021-06-18 |
Family
ID=76393093
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110181011.4A Pending CN112988505A (en) | 2021-02-08 | 2021-02-08 | Cloud computing-based general real-time big data monitoring and early warning system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112988505A (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130238791A1 (en) * | 2011-11-10 | 2013-09-12 | CopperEgg Corporation | System for Monitoring Elastic Cloud-Based Computing Systems as a Service |
| CN105610605A (en) * | 2015-12-18 | 2016-05-25 | 成都广达新网科技股份有限公司 | Message reverse push method, network management system alarm method and state update method |
| CN107395669A (en) * | 2017-06-01 | 2017-11-24 | 华南理工大学 | A kind of collecting method and system based on the real-time distributed big data of streaming |
| CN107491375A (en) * | 2017-08-18 | 2017-12-19 | 国网山东省电力公司信息通信公司 | Equipment detection and fault early warning system and method under a kind of cloud computing environment |
| CN108335075A (en) * | 2018-03-02 | 2018-07-27 | 华南理工大学 | A kind of processing system and method for Logistics Oriented big data |
| CN110912773A (en) * | 2019-11-25 | 2020-03-24 | 深圳晶泰科技有限公司 | Cluster monitoring system and monitoring method for multiple public cloud computing platforms |
| CN111984498A (en) * | 2020-07-24 | 2020-11-24 | 华东计算技术研究所（中国电子科技集团公司第三十二研究所） | Server cluster monitoring and management system |
-
2021
- 2021-02-08 CN CN202110181011.4A patent/CN112988505A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130238791A1 (en) * | 2011-11-10 | 2013-09-12 | CopperEgg Corporation | System for Monitoring Elastic Cloud-Based Computing Systems as a Service |
| CN105610605A (en) * | 2015-12-18 | 2016-05-25 | 成都广达新网科技股份有限公司 | Message reverse push method, network management system alarm method and state update method |
| CN107395669A (en) * | 2017-06-01 | 2017-11-24 | 华南理工大学 | A kind of collecting method and system based on the real-time distributed big data of streaming |
| CN107491375A (en) * | 2017-08-18 | 2017-12-19 | 国网山东省电力公司信息通信公司 | Equipment detection and fault early warning system and method under a kind of cloud computing environment |
| CN108335075A (en) * | 2018-03-02 | 2018-07-27 | 华南理工大学 | A kind of processing system and method for Logistics Oriented big data |
| CN110912773A (en) * | 2019-11-25 | 2020-03-24 | 深圳晶泰科技有限公司 | Cluster monitoring system and monitoring method for multiple public cloud computing platforms |
| CN111984498A (en) * | 2020-07-24 | 2020-11-24 | 华东计算技术研究所（中国电子科技集团公司第三十二研究所） | Server cluster monitoring and management system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108200123B (en) | Internet of things industrial cloud monitoring system based on safety inspection equipment | |
| CA3080027C (en) | Monitoring and controlling of distributed machines | |
| US9934462B1 (en) | Visualizing deep neural networks | |
| US11182870B2 (en) | System and method for collective and collaborative navigation by a group of individuals | |
| CN107241211A (en) | Improve the method and system of relevance between data center's overlay network and bottom-layer network | |
| CN104301147A (en) | Method for monitoring service and process activities in service application system | |
| CN110347694B (en) | Equipment monitoring method, device and system based on Internet of things | |
| WO2023246347A1 (en) | Digital twin processing method and digital twin system | |
| CN105099763B (en) | Equipment goes offline based reminding method and device | |
| CN104573904A (en) | Data visualizing system for monitoring user and software behaviors during network transaction | |
| CN110995859A (en) | Intelligent transformer substation supporting platform system based on ubiquitous Internet of things | |
| CN111143167B (en) | Alarm merging method, device, equipment and storage medium for multiple platforms | |
| CN103973484B (en) | A kind of operation management system based on network topology structure | |
| CN104572405A (en) | Pc server operation system and database operation environment monitoring alarm system | |
| CN106487597A (en) | A kind of service monitoring system and method based on Zookeeper | |
| CN113507691B (en) | Information pushing system and method based on power distribution network cross-region service | |
| CN104539449B (en) | A kind of failure information processing method and relevant apparatus | |
| CN106445789A (en) | Monitoring visualizing method and system | |
| CN104793570A (en) | Portable motor train unit fault processing support equipment and portable motor train unit fault processing support system | |
| CN202841168U (en) | Network resource monitoring system | |
| CN109587130B (en) | Integrated operation support system based on RTI space-time consistency | |
| CN112988505A (en) | Cloud computing-based general real-time big data monitoring and early warning system | |
| WO2020037634A1 (en) | Information monitoring system and method for industrial control device network, computer readable storage medium, and computer device | |
| WO2023273461A1 (en) | Robot operating state monitoring system, and method | |
| CN116260702A (en) | Method, device, computer equipment and storage medium for data monitoring |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210618 |