CN111090646B - Electromagnetic data processing platform - Google Patents
Electromagnetic data processing platform Download PDFInfo
- Publication number
- CN111090646B CN111090646B CN201911002384.XA CN201911002384A CN111090646B CN 111090646 B CN111090646 B CN 111090646B CN 201911002384 A CN201911002384 A CN 201911002384A CN 111090646 B CN111090646 B CN 111090646B
- Authority
- CN
- China
- Prior art keywords
- data
- module
- electromagnetic
- sent
- analysis
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2291—User-Defined Types; Storage management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2465—Query processing support for facilitating data mining operations in structured databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/248—Presentation of query results
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y04—INFORMATION OR COMMUNICATION TECHNOLOGIES HAVING AN IMPACT ON OTHER TECHNOLOGY AREAS
- Y04S—SYSTEMS INTEGRATING TECHNOLOGIES RELATED TO POWER NETWORK OPERATION, COMMUNICATION OR INFORMATION TECHNOLOGIES FOR IMPROVING THE ELECTRICAL POWER GENERATION, TRANSMISSION, DISTRIBUTION, MANAGEMENT OR USAGE, i.e. SMART GRIDS
- Y04S10/00—Systems supporting electrical power generation, transmission or distribution
- Y04S10/50—Systems or methods supporting the power network operation or management, involving a certain degree of interaction with the load-side end user applications
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Probability & Statistics with Applications (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The embodiment of the invention provides an electromagnetic data processing platform, which comprises: the system comprises a data collection module, a data storage module, a data cache module, a real-time calculation module, a statistics cache module, a visualization module and an analysis mining module; the method is used for collecting data, storing data, caching data, counting data, storing counting results, visually displaying data and analyzing data respectively. The embodiment of the invention realizes the collection and storage of a large amount of electromagnetic data by using a big data related processing technology; the problem of mismatching of data reading-in and writing-out speeds is solved; the real-time performance of analysis is improved; visually displaying electromagnetic data; providing a machine learning algorithm and a data mining technology, so that electromagnetic data processing is more intelligent; the deployment flow of the electromagnetic data processing platform is simpler.
Description
Technical Field
The invention relates to the technical field of big data processing, in particular to an electromagnetic data processing platform.
Background
With the rapid development of mobile communication industry, electromagnetic spectrum resources are widely developed, electronic equipment relying on electromagnetic waves is widely applied, and electromagnetic space data security is also paid unprecedented attention.
In recent years, the electromagnetic environment becomes more complex, and the acquired electromagnetic space data has the remarkable characteristics of big data, for example, the electromagnetic space data has explosive growth, so that the big data has a larger scale; the data sources are wide, so that the big data presents diversity; the data is analyzed in real time, and the data is input, processed and discarded to take effect immediately, so that the big data processing presents high speed; by mining out data valuable for predictive analysis of future trends and patterns from a large amount of uncorrelated data of various types, the value of large data processing is embodied.
However, conventional electromagnetic signal processing techniques have become increasingly inadequate in terms of processing scale and processing speed for everyday practical needs. The processing of electromagnetic data presents problems such as an increase in the amount of electromagnetic data resulting in inefficient storage of data, inability to process electromagnetic data in multiple formats, poor real-time computing performance, inability to provide real-time alert functions, severe reliance on manual analysis, and insufficient mining of historical data.
Disclosure of Invention
The embodiment of the invention provides an electromagnetic data processing platform which is used for solving the problems that the efficiency of data storage is low, electromagnetic data in various formats cannot be processed due to the increase of the electromagnetic data quantity, the real-time calculation performance is poor, a real-time alarm function cannot be provided, manual analysis is seriously relied on, and historical data is not fully mined.
The embodiment of the invention provides an electromagnetic data processing platform, which comprises: the system comprises a data collection module, a data storage module, a data cache module, a real-time calculation module, a statistics cache module, a visualization module and an analysis mining module;
the data collection module is used for monitoring messages sent by the front-end acquisition equipment; collecting electromagnetic trace data and electromagnetic alarm data by using a logstack, and sending the collected electromagnetic trace data and electromagnetic alarm data to the data caching module and the data storage module;
the data storage module is used for receiving the electromagnetic trace data and the electromagnetic alarm data sent by the data collection module, storing the electromagnetic trace data and the electromagnetic alarm data by utilizing an elastic search and retrieving the electromagnetic trace data and the electromagnetic alarm data; the electromagnetic trace data and the electromagnetic alarm data are sent to the analysis mining module and the statistic caching module;
the data caching module is used for receiving the electromagnetic trace data and the electromagnetic alarm data sent by the data collecting module and caching the data by using Kafka; transmitting the received electromagnetic trace data and electromagnetic alarm data to the real-time computing module, the visualization module and the analysis mining module;
the real-time computing module is used for receiving the electromagnetic trace data and the electromagnetic alarm data sent by the data caching module; receiving the front-end acquisition equipment related information sent by the statistics cache module; counting the electromagnetic trace data and the electromagnetic alarm data by using a storm, and sending a counting result to the counting cache module;
the statistical buffer module is used for receiving the statistical result sent by the real-time calculation module and storing the statistical result in the statistical buffer module in a key-value form by utilizing Redis; storing the front-end acquisition equipment related information extracted from the data storage module and sending the front-end acquisition equipment related information to the real-time calculation module; sending the statistical result to the visualization module;
the visualization module is used for receiving the electromagnetic trace data and the electromagnetic alarm data sent by the data caching module, the statistical result sent by the statistical caching module and the analysis result sent by the analysis mining module, and visually displaying the electromagnetic trace data and the electromagnetic alarm data, the statistical result and the analysis result;
the analysis mining module is used for analyzing the historical data in the data storage module and the cache data in the data cache module by utilizing an integrated machine learning algorithm and a data mining technology, and sending an analysis result to the visualization module.
Optionally, the electromagnetic data processing platform further comprises: and the configuration management module is used for sending configuration information to the data collection module.
Optionally, the analysis mining module is further configured to generate configuration information according to an analysis result of analyzing the historical data in the data storage module and the cached data in the data caching module; and sending the configuration information to the configuration management module.
Optionally, the configuration management module is further configured to receive configuration information sent by the analysis mining module.
Optionally, the data collection module is further configured to receive configuration information sent by the configuration management module.
Optionally, the configuration information includes: electromagnetic trace data, electromagnetic alert data, voice data, and/or bandwidth.
The embodiment of the invention respectively uses the Logstar and the elastic search to collect and store a large amount of electromagnetic data in real time; the cache technologies such as kafka and Redis are respectively used, so that the problem of mismatching of data reading-in and writing-out speeds is solved, and the robustness of the system is improved; the storm is used for real-time calculation and analysis, so that the real-time performance of the analysis is improved, and the labor cost is reduced; the visual display platform is provided, electromagnetic trace data is dynamically displayed on a large screen in the form of a line graph, electromagnetic alarm data in the trace can be detected by applying an abnormal signal detection algorithm, statistics is carried out, and data support is provided for decisions of management staff. Providing a machine learning algorithm and a data mining technology, so that electromagnetic data processing is more intelligent; the electromagnetic data processing platform has simple deployment flow, comprises equipment throwing, IP configuration, local area network construction, basic information of the equipment addition in the platform, online test and the like, and can complete deployment and subsequent supervision and maintenance work only by two to three maintenance and supervision personnel under the general condition, thereby greatly improving the supervision efficiency and saving the labor cost.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the prior art, the following description will briefly explain the drawings used in the embodiments or the description of the prior art, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained according to these drawings without inventive effort for a person skilled in the art.
The invention combines the related technology of big data processing, collects, stores and caches a large amount of electromagnetic data in real time, analyzes the electromagnetic data in real time, counts the data and visually displays the electromagnetic data in multiple dimensions, and alarms abnormal electromagnetic data. And data support is provided for decisions of management personnel.
FIG. 1 is a schematic flow chart of interaction between modules of an electromagnetic data processing platform according to an embodiment of the present invention;
FIG. 2 is a schematic flow chart of a data buffering module according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of another process for interaction between modules of an electromagnetic data processing platform according to an embodiment of the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
FIG. 1 is a schematic flow chart of interaction between modules of an electromagnetic data processing platform according to an embodiment of the present invention, including: the system comprises a data collection module 11, a data storage module 12, a data cache module 13, a real-time calculation module 14, a statistics cache module 15, a visualization module 16 and an analysis mining module 17;
the data collection module 11 is configured to monitor a message sent by the front-end acquisition device; collecting electromagnetic trace data and electromagnetic alarm data by using logstack, and transmitting the collected electromagnetic trace data and electromagnetic alarm data to the data caching module 13 and the data storage module 12;
in the embodiment of the invention, the front-end acquisition equipment is deployed at each place needing to acquire electromagnetic data, is connected with the data collection module 11, and can interact with the data collection module 11. Logstack is an open source data collection engine that has real-time data transfer capabilities and can process electromagnetic data using logstack technology. The data collection module 11 connected with the front-end acquisition device can monitor the message sent by the front-end acquisition device, can collect electromagnetic trace data and electromagnetic alarm data by using the logmesh technology, and sends the electromagnetic trace data and the electromagnetic alarm data to the data storage module 12 and the data cache module 13.
The data storage module 12 is configured to receive the electromagnetic trace data and the electromagnetic alarm data sent by the data collection module 11, store the electromagnetic trace data and the electromagnetic alarm data by using an elastic search, and retrieve the electromagnetic trace data and the electromagnetic alarm data; transmitting the electromagnetic trace data and the electromagnetic alarm data to the analysis mining module 17 and the statistical buffer module 15;
in the embodiment of the present invention, the data storage module 12 is a data persistent storage module provided by an electromagnetic data processing platform, and supports the persistent storage operation of the platform by adopting an elastic search component. The elastiscearch is an open-source high-expansion distributed full-text search engine, has good expansibility, and can be expanded to hundreds of servers to process PB-level data. And can store data and retrieve data in real time. Because the deployment of the whole electromagnetic environment involves a plurality of environmental information distributed on devices and the like in various places, the information is stored in an elastic search for analysis and management staff to inquire, so that the states and the running states of all the devices are mastered. Meanwhile, the data storage module 12 can receive the electromagnetic trace data and the electromagnetic alarm data sent by the data collection module 11, store the data collected by the data collection module 11 by using an elastic search, and retrieve the electromagnetic trace data and the electromagnetic alarm data. And may send the electromagnetic trace data and the electromagnetic alert data to the analysis mining module 17 and the statistical caching module 15 for subsequent data analysis.
The data caching module 13 is configured to receive the electromagnetic trace data and the electromagnetic alarm data sent by the data collecting module 11, and cache the data by using Kafka; transmitting the received electromagnetic trace data and electromagnetic alarm data to the real-time computing module 14, the visualization module 16 and the analysis mining module 17;
in the embodiment of the present invention, the collection device cluster at the front end continuously submits real-time data to the data collection module 11, and after data aggregation, the data is subjected to real-time calculation and sent to each module at the background, such as the visualization module 16 and the analysis mining module 17. The server cannot process a large amount of data in time, which can cause congestion, jamming, data loss and even system breakdown. kafka is a distributed message queue with high performance, persistence, multi-copy backup, lateral expansion capability, and generally plays roles of decoupling, peak clipping, and asynchronous processing in architecture design. The invention adds the kafka message queue as the data caching module 13 to relieve the problem that the production rate and the consumption rate of electromagnetic data cannot be synchronous, and performs asynchronous processing of the message to ensure that a large amount of real-time data is not lost. The data caching module 13 may receive the electromagnetic trace data and the electromagnetic alarm data sent by the data collecting module 11, and cache the data by using Kafka; and may send the received electromagnetic trace data and electromagnetic alert data to the real-time computing module 14, the visualization module 16, and the analysis mining module 17; the data buffer module 13 may forward the real-time message of the data collecting module 11, and the visualization module 16 and the analysis mining module 17 may subscribe electromagnetic alarm data and electromagnetic trace data from the kafka message queue, so as to dynamically display and analyze the message of each module in real time. The flow chart of the data buffer module 13 is shown in fig. 2.
The real-time computing module 14 is configured to receive the electromagnetic trace data and the electromagnetic alarm data sent by the data caching module 13; receiving the front-end acquisition equipment related information sent by the statistics cache module 15; counting the electromagnetic trace data and the electromagnetic alarm data by using a storm, and sending a counting result to the counting cache module 15;
in the embodiment of the present invention, the real-time computing module 14 is mainly responsible for real-time statistics of electromagnetic trace data and electromagnetic alarm data, and sends the statistics to the statistics cache module 15. And the system can also be used for receiving the electromagnetic trace data and the electromagnetic alarm data sent by the data caching module 13 and the front-end acquisition equipment related information sent by the statistics caching module 15. If the conventional database mode is adopted, a large number of frequent database I/O operations are generated, and meanwhile, the situation of querying the same content in a short time can occur, so that the platform use efficiency is greatly reduced, and the electromagnetic trace data and the electromagnetic alarm data cannot be displayed in real time by the visualization module 16. The Storm real-time stream processing tool is a real-time data processing framework, has the characteristics of low delay, high availability, easy expansion, no data loss and the like, and simultaneously, the Storm also provides a simple programming model similar to MapReduce, thereby being convenient for development. Therefore, the real-time calculation module 14 adopts storm to perform corresponding field extraction and information statistics on the real-time message subscribed from kafka, organizes the statistics result into different key-value information, and sends the key-value information to the statistics cache module 15.
The statistics buffer module 15 is configured to receive the statistics result sent by the real-time computing module 14, and store the statistics result in the statistics buffer module 15 in a key-value form by using Redis; saving the front-end acquisition device related information extracted from the data storage module 12 and transmitting the front-end acquisition device related information to the real-time computing module 14; sending the statistics to the visualization module 16;
in the embodiment of the present invention, the statistical buffer module 15 adopts the Redis buffer technology, which is mainly responsible for receiving the statistical result sent by the real-time computing module 14, and Redis is essentially a Key-Value type memory database. Because of the pure memory operation, redis has excellent performance, can process more than 10 ten thousand read-write operations per second, and is a Key-Value database with the fastest known performance. The statistics caching module 15 stores the statistics of the real-time computing module 14 in the statistics caching module 15 in a key-value form, and simultaneously sends the statistics to the visualization module 16 to provide data support for the visualization module 16. Meanwhile, the real-time calculation module 14 needs to frequently read the relevant information of the front-end acquisition device stored in the data storage module 12, and the relevant information of the front-end acquisition device needing to be repeatedly read can be extracted from the database and stored in the statistics cache module 15, so that the calculation efficiency of the platform is greatly improved. In addition, front-end acquisition device related information extracted from the data storage module 12 is saved and sent to the real-time computing module 14.
The visualization module 16 is configured to receive the electromagnetic trace data and the electromagnetic alarm data sent by the data caching module 13, the statistical result sent by the statistical caching module 15, and the analysis result sent by the analysis mining module 17, and visually display the electromagnetic trace data and the electromagnetic alarm data, the statistical result, and the analysis result;
in the embodiment of the present invention, first, the visualization module 16 may receive the electromagnetic trace data and the electromagnetic alarm data sent by the data caching module 13, the statistical result sent by the statistical caching module 15, and the analysis result sent by the analysis mining module 17. Second, the visualization module 16 is also responsible for visual presentation of electromagnetic trace data, electromagnetic alert data, statistical results, and analysis results. The visualization module 16 clearly and effectively displays the meaning of the electromagnetic environment represented by the electromagnetic trace data and the electromagnetic alarm data by means of a graphical means, so that analysts and management staff can quickly grasp the key point information and the abnormal information, and the electromagnetic environment is controlled. The visualization module 16 displays the electromagnetic environment in real time from different visualization levels, different times and different statistical dimensions of fields.
The analysis mining module 17 is configured to analyze the historical data in the data storage module 12 and the cached data in the data caching module 13 by using an integrated machine learning algorithm and a data mining technology, and send the analysis result to the visualization module 16.
In the embodiment of the present invention, the analysis mining module 17 provides an intelligent service for the platform. The analysis mining module 17 integrates a common machine learning algorithm and a data mining technology, can analyze the historical data in the data storage module 12 and the cached data in the data caching module 13, mine potential rules, and provide support for analysis and decision-making by decision-making staff. The analysis result of the analysis mining module 17 can be sent to the visualization module 16 for visual display. The embodiment of the invention respectively uses the Logstar and the elastic search to collect and store a large amount of electromagnetic data in real time; the cache technologies such as kafka and Redis are respectively used, so that the problem of mismatching of data reading-in and writing-out speeds is solved, and the robustness of the system is improved; the storm is used for real-time calculation and analysis, so that the real-time performance of the analysis is improved, and the labor cost is reduced; a visual display platform is provided, and electromagnetic trace data is dynamically displayed on a large screen in the form of a line graph. Providing a machine learning algorithm and a data mining technology, so that electromagnetic data processing is more intelligent; the electromagnetic data processing platform has simple deployment flow, comprises equipment throwing, IP configuration, local area network construction, basic information of the equipment addition in the platform, online test and the like, and can complete deployment and subsequent supervision and maintenance work only by two to three maintenance and supervision personnel under the general condition, thereby greatly improving the supervision efficiency and saving the labor cost.
Further, on the basis of the platform embodiment, the electromagnetic data processing platform further includes: a configuration management module 31, configured to send configuration information to the data collection module 11.
In the embodiment of the present invention, the configuration management module 31 is shown in fig. 3. The data collection module 11 is connected to a front-end collection device. Some parts of the data collection module 11 are present in the front-end acquisition device for collecting data. The configuration management module 31 may send configuration information to the data collection module 11 to control some parts of the data collection module 11 in the front-end acquisition device to collect specific data.
It should be noted that, the configuration management module 31 provides a set of management tools for platform deployment management, and a background manager may perform device management, system monitoring, etc. on the background management module, so as to facilitate grasping the current running status of the whole platform cluster.
The embodiment of the invention controls the collection of specific data through configuration information, and is convenient for background management personnel to manage equipment and monitor a system through a set of management tools.
Further, on the basis of the platform embodiment, the analysis mining module 17 is further configured to generate configuration information according to an analysis result of analyzing the historical data in the data storage module 12 and the cached data in the data caching module 13; the configuration information is sent to the configuration management module 31.
In an embodiment of the present invention, the analysis mining module 17 may analyze the history data in the data storage module 12 and the cache data in the data cache module 13. And generates configuration information according to the analysis result and transmits the configuration information to the configuration management module 31.
The configuration information generated by the analysis mining module according to the analysis result can be used for controlling the data collection module to collect specific data.
Further, on the basis of the platform embodiment, the configuration management module is further configured to receive the configuration information sent by the analysis mining module.
In the embodiment of the present invention, the analysis mining module 17 may generate more appropriate configuration information according to the analysis result of analyzing the history data in the data storage module 12 and the cached data in the data caching module 13 and send the configuration information to the configuration management module 31. Accordingly, the configuration management module 31 may receive the configuration information sent by the analysis mining module 17.
The configuration management module of the embodiment of the invention can receive the configuration information from the analysis mining module, and the configuration information can be used for controlling the data collection module to collect specific data.
Further, on the basis of the platform embodiment, the data collection module is further configured to receive configuration information sent by the configuration management module.
In the embodiment of the present invention, the data collection module 11 may receive the configuration information sent from the configuration management module 31, and collect data according to the configuration information.
The embodiment of the invention controls the data collection module to collect data through the configuration information sent by the configuration management module.
Further, on the basis of the platform embodiment, the configuration information includes: electromagnetic trace data, electromagnetic alert data, voice data, and/or bandwidth.
In an embodiment of the present invention, the configuration information sent by the configuration management module 31 includes electromagnetic trace data, electromagnetic alarm data, voice data and/or bandwidth. It should be noted that the configuration information includes, but is not limited to, the electromagnetic trace data, the electromagnetic alarm data, the voice data, and/or the bandwidth described above.
The configuration information in the embodiment of the invention can be used for controlling the data collection module to collect specific data.
The apparatus embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
From the above description of the embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by means of software plus necessary general hardware platforms, or of course may be implemented by means of hardware. Based on this understanding, the foregoing technical solution may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a computer readable storage medium, such as ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method described in the respective embodiments or some parts of the embodiments.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and are not limiting; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims (5)
1. An electromagnetic data processing platform, comprising:
the system comprises a data collection module, a data storage module, a data cache module, a real-time calculation module, a statistics cache module, a visualization module and an analysis mining module;
the data collection module is used for monitoring messages sent by the front-end acquisition equipment; collecting electromagnetic trace data and electromagnetic alarm data by using a logstack, and sending the collected electromagnetic trace data and electromagnetic alarm data to the data caching module and the data storage module;
the data storage module is used for receiving the electromagnetic trace data and the electromagnetic alarm data sent by the data collection module, storing the electromagnetic trace data and the electromagnetic alarm data by utilizing an elastic search and retrieving the electromagnetic trace data and the electromagnetic alarm data; the electromagnetic trace data and the electromagnetic alarm data are sent to the analysis mining module and the statistic caching module;
the data caching module is used for receiving the electromagnetic trace data and the electromagnetic alarm data sent by the data collecting module and caching the data by using Kafka; transmitting the received electromagnetic trace data and electromagnetic alarm data to the real-time computing module, the visualization module and the analysis mining module;
the real-time computing module is used for receiving the electromagnetic trace data and the electromagnetic alarm data sent by the data caching module; receiving the front-end acquisition equipment related information sent by the statistics cache module; counting the electromagnetic trace data and the electromagnetic alarm data by using a storm, and sending a counting result to the counting cache module;
the statistical buffer module is used for receiving the statistical result sent by the real-time calculation module and storing the statistical result in the statistical buffer module in a key-value form by utilizing Redis; storing the front-end acquisition equipment related information extracted from the data storage module and sending the front-end acquisition equipment related information to the real-time calculation module; sending the statistical result to the visualization module;
the visualization module is used for receiving the electromagnetic trace data and the electromagnetic alarm data sent by the data caching module, the statistical result sent by the statistical caching module and the analysis result sent by the analysis mining module, and visually displaying the electromagnetic trace data and the electromagnetic alarm data, the statistical result and the analysis result;
the analysis mining module is used for analyzing the historical data in the data storage module and the cache data in the data cache module by utilizing an integrated machine learning algorithm and a data mining technology, and sending an analysis result to the visualization module;
wherein, electromagnetic data processing platform still includes: the configuration management module is used for sending configuration information to the data collection module, and controlling the collection of specific data through the configuration information to form a set of management tool for background management personnel to manage equipment and monitor a system.
2. The electromagnetic data processing platform of claim 1, wherein the analysis mining module is further configured to generate configuration information according to an analysis result of analyzing the historical data in the data storage module and the cached data in the data caching module; and sending the configuration information to the configuration management module.
3. The electromagnetic data processing platform of claim 2, wherein the configuration management module is further configured to receive configuration information sent by the analysis mining module.
4. The electromagnetic data processing platform of claim 1, wherein the data collection module is further configured to receive configuration information sent by the configuration management module.
5. The electromagnetic data processing platform of any of claims 2-4, wherein the configuration information comprises: electromagnetic trace data, electromagnetic alert data, voice data, and/or bandwidth.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911002384.XA CN111090646B (en) | 2019-10-21 | 2019-10-21 | Electromagnetic data processing platform |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911002384.XA CN111090646B (en) | 2019-10-21 | 2019-10-21 | Electromagnetic data processing platform |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111090646A CN111090646A (en) | 2020-05-01 |
| CN111090646B true CN111090646B (en) | 2023-07-28 |
Family
ID=70393399
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911002384.XA Active CN111090646B (en) | 2019-10-21 | 2019-10-21 | Electromagnetic data processing platform |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111090646B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111914014A (en) * | 2020-08-17 | 2020-11-10 | 深圳市联恒星科技有限公司 | Big data platform and application thereof |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN205902093U (en) * | 2016-06-24 | 2017-01-18 | 南京欣网通信科技股份有限公司 | Mobile network optimizing system based on big data platform |
| CN106709003A (en) * | 2016-12-23 | 2017-05-24 | 长沙理工大学 | Hadoop-based mass log data processing method |
| CN108334556A (en) * | 2017-12-31 | 2018-07-27 | 江苏易润信息技术有限公司 | A kind of method and system of analysis internet finance massive logs |
| CN109460393A (en) * | 2018-11-08 | 2019-03-12 | 浙江工业大学 | Visualization system is repaired in a kind of preliminary examination based on big data in advance |
| CN109977125A (en) * | 2019-04-09 | 2019-07-05 | 福建奇点时空数字科技有限公司 | A kind of big data safety analysis plateform system based on network security |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9846690B2 (en) * | 2009-03-02 | 2017-12-19 | International Business Machines Corporation | Automating interrogative population of electronic forms using a real-time communication platform |
| US10607484B2 (en) * | 2017-12-28 | 2020-03-31 | Intel Corporation | Privacy-preserving distributed visual data processing |
-
2019
- 2019-10-21 CN CN201911002384.XA patent/CN111090646B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN205902093U (en) * | 2016-06-24 | 2017-01-18 | 南京欣网通信科技股份有限公司 | Mobile network optimizing system based on big data platform |
| CN106709003A (en) * | 2016-12-23 | 2017-05-24 | 长沙理工大学 | Hadoop-based mass log data processing method |
| CN108334556A (en) * | 2017-12-31 | 2018-07-27 | 江苏易润信息技术有限公司 | A kind of method and system of analysis internet finance massive logs |
| CN109460393A (en) * | 2018-11-08 | 2019-03-12 | 浙江工业大学 | Visualization system is repaired in a kind of preliminary examination based on big data in advance |
| CN109977125A (en) * | 2019-04-09 | 2019-07-05 | 福建奇点时空数字科技有限公司 | A kind of big data safety analysis plateform system based on network security |
Non-Patent Citations (2)
| Title |
|---|
| Benchmarking streaming computation engines: Storm, flink and spark streaming;Sanket Chintapalli等;《2016 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW)》;第1789-1792页 * |
| 物联网大数据存储与管理技术研究;郝行军;《中国博士学位论文全文数据库 信息科技辑》(第9期);第I138-50页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111090646A (en) | 2020-05-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11150974B2 (en) | Anomaly detection using circumstance-specific detectors | |
| CN110347116B (en) | Machine tool state monitoring system and monitoring method based on operation data flow | |
| CN107070692A (en) | A kind of cloud platform monitoring service system analyzed based on big data and method | |
| CN110691070B (en) | Network abnormity early warning method based on log analysis | |
| CN111858526B (en) | Failure time space prediction method and system based on information system log | |
| CN111290913A (en) | Fault location visualization system and method based on operation and maintenance data prediction | |
| US11899444B2 (en) | Real-time alerts and transmission of selected signal samples under a dynamic capacity limitation | |
| CN105204971A (en) | Dynamic monitoring interval adjustment method based on Naive Bayes classification technology | |
| KR102087959B1 (en) | Artificial intelligence operations system of telecommunication network, and operating method thereof | |
| CN111090646B (en) | Electromagnetic data processing platform | |
| CN107332680A (en) | A kind of system monitoring method and device | |
| CN109660396A (en) | A kind of method for monitoring network and device | |
| CN111666978B (en) | Intelligent fault early warning system for IT system operation and maintenance big data | |
| CN112749153A (en) | Industrial network data management system | |
| CN117194919A (en) | Production data analysis system | |
| CN110753049A (en) | Safety situation sensing system based on industrial control network flow | |
| CN108713310A (en) | Method and system for information security data in online and transmission to be compressed and optimized | |
| CN104243192A (en) | Fault treatment method and system | |
| CN113282920B (en) | Log abnormality detection method, device, computer equipment and storage medium | |
| CN113312321A (en) | Abnormal monitoring method for traffic and related equipment | |
| CN111209314A (en) | System for processing massive log data of power information system in real time | |
| CN111190790A (en) | Cloud computing cluster monitoring method and system based on peak prediction | |
| CN115080363B (en) | System capacity evaluation method and device based on service log | |
| CN110769069A (en) | Method and device for pushing alarm information | |
| CN114428711A (en) | Data detection method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |