CN111090646A - Electromagnetic data processing platform - Google Patents
Electromagnetic data processing platform Download PDFInfo
- Publication number
- CN111090646A CN111090646A CN201911002384.XA CN201911002384A CN111090646A CN 111090646 A CN111090646 A CN 111090646A CN 201911002384 A CN201911002384 A CN 201911002384A CN 111090646 A CN111090646 A CN 111090646A
- Authority
- CN
- China
- Prior art keywords
- data
- module
- electromagnetic
- sent
- caching
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2291—User-Defined Types; Storage management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2465—Query processing support for facilitating data mining operations in structured databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/248—Presentation of query results
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y04—INFORMATION OR COMMUNICATION TECHNOLOGIES HAVING AN IMPACT ON OTHER TECHNOLOGY AREAS
- Y04S—SYSTEMS INTEGRATING TECHNOLOGIES RELATED TO POWER NETWORK OPERATION, COMMUNICATION OR INFORMATION TECHNOLOGIES FOR IMPROVING THE ELECTRICAL POWER GENERATION, TRANSMISSION, DISTRIBUTION, MANAGEMENT OR USAGE, i.e. SMART GRIDS
- Y04S10/00—Systems supporting electrical power generation, transmission or distribution
- Y04S10/50—Systems or methods supporting the power network operation or management, involving a certain degree of interaction with the load-side end user applications
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Probability & Statistics with Applications (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
An embodiment of the present invention provides an electromagnetic data processing platform, including: the system comprises a data collection module, a data storage module, a data caching module, a real-time calculation module, a statistic caching module, a visualization module and an analysis mining module; the data acquisition and storage device is respectively used for collecting data, storing data, caching data, counting data, storing counting results, visually displaying data and analyzing data. The embodiment of the invention realizes the collection and storage of a large amount of electromagnetic data by using a big data correlation processing technology; the problem that the data reading speed and the data writing speed are not matched is solved; the real-time performance of analysis is improved; visually displaying the electromagnetic data; a machine learning algorithm and a data mining technology are provided, so that electromagnetic data processing is more intelligent; the deployment process of the electromagnetic data processing platform is simple.
Description
Technical Field
The invention relates to the technical field of big data processing, in particular to an electromagnetic data processing platform.
Background
With the rapid development of mobile communication, electromagnetic spectrum resources are widely developed, electronic devices relying on electromagnetic waves are widely applied, and the security guarantee of electromagnetic space data is paid unprecedented attention.
In recent years, the electromagnetic environment becomes more complex, and the acquired electromagnetic space data has the remarkable characteristic of big data, for example, the electromagnetic space data is explosively increased, so that the big data is large in scale; the data source is wide, so that the big data presents diversity; the data is analyzed in real time, and the data input, processing and discarding take effect immediately, so that the big data processing presents high speed; the data valuable for predicting and analyzing future trends and patterns is mined from a large amount of irrelevant data of various types, so that the value of big data processing is embodied.
However, conventional electromagnetic signal processing techniques have become increasingly unable to meet daily practical requirements in terms of processing scale and processing speed. The processing of electromagnetic data presents a number of problems, such as, increased electromagnetic data volume resulting in inefficient data storage, inability to process multiple formats of electromagnetic data, poor real-time computing performance, inability to provide real-time alarm functionality, severe reliance on manual analysis, and inadequate mining of historical data.
Disclosure of Invention
The embodiment of the invention provides an electromagnetic data processing platform, which is used for solving the problems of low data storage efficiency, incapability of processing electromagnetic data in various formats, poor real-time computing performance, incapability of providing a real-time alarm function, severe dependence on manual analysis and insufficient mining of historical data caused by the increase of the amount of electromagnetic data.
An embodiment of the present invention provides an electromagnetic data processing platform, including: the system comprises a data collection module, a data storage module, a data caching module, a real-time calculation module, a statistic caching module, a visualization module and an analysis mining module;
the data collection module is used for monitoring messages sent by the front-end acquisition equipment; collecting electromagnetic trace data and electromagnetic alarm data by using the Logstash, and sending the collected electromagnetic trace data and the collected electromagnetic alarm data to the data caching module and the data storage module;
the data storage module is used for receiving the electromagnetic trace data and the electromagnetic alarm data sent by the data collection module, storing the electromagnetic trace data and the electromagnetic alarm data by using an elastic search and retrieving the electromagnetic trace data and the electromagnetic alarm data; sending the electromagnetic trace data and the electromagnetic alarm data to the analysis mining module and the statistics caching module;
the data caching module is used for receiving the electromagnetic trace data and the electromagnetic alarm data sent by the data collecting module and caching the data by using Kafka; sending the received electromagnetic trace data and electromagnetic alarm data to the real-time computation module, the visualization module, and the analysis mining module;
the real-time computing module is used for receiving the electromagnetic trace data and the electromagnetic alarm data sent by the data caching module; receiving the relevant information of the front-end acquisition equipment sent by the statistics cache module; counting the electromagnetic trace data and the electromagnetic alarm data by using storm, and sending a counting result to the counting cache module;
the statistics cache module is used for receiving the statistics result sent by the real-time calculation module and storing the statistics result in the statistics cache module in a key-value form by utilizing Redis; storing the relevant information of the front-end acquisition equipment extracted from the data storage module and sending the relevant information of the front-end acquisition equipment to the real-time computing module; sending the statistical result to the visualization module;
the visualization module is used for receiving the electromagnetic trace data and the electromagnetic alarm data sent by the data caching module, the statistical result sent by the statistical caching module and the analysis result sent by the analysis mining module, and visually displaying the electromagnetic trace data and the electromagnetic alarm data, the statistical result and the analysis result;
the analysis mining module is used for analyzing the historical data in the data storage module and the cache data in the data cache module by utilizing an integrated machine learning algorithm and a data mining technology, and sending an analysis result to the visualization module.
Optionally, the electromagnetic data processing platform further includes: and the configuration management module is used for sending configuration information to the data collection module.
Optionally, the analysis mining module is further configured to generate configuration information according to an analysis result of analyzing the historical data in the data storage module and the cache data in the data cache module; and sending the configuration information to the configuration management module.
Optionally, the configuration management module is further configured to receive configuration information sent by the analysis mining module.
Optionally, the data collection module is further configured to receive configuration information sent by the configuration management module.
Optionally, the configuration information includes: electromagnetic trace data, electromagnetic alarm data, voice data, and/or bandwidth.
The embodiment of the invention respectively uses the Logstash and the Elasticissearch to collect and store a large amount of electromagnetic data in real time; the cache technologies such as kafka and Redis are respectively used, so that the problem that the data reading speed and the data writing speed are not matched is solved, and the robustness of the system is improved; the storm is used for real-time calculation and analysis, so that the analysis instantaneity is improved, and the labor cost is reduced; and a visual display platform is provided, electromagnetic trace data is dynamically displayed on a large screen in a line graph form, electromagnetic alarm data in traces can be detected by applying an abnormal signal detection algorithm, statistics is carried out, and data support is provided for decisions of managers. A machine learning algorithm and a data mining technology are provided, so that electromagnetic data processing is more intelligent; the deployment process of the electromagnetic data processing platform is simple, and comprises equipment release, IP configuration, local area network establishment, basic information of the equipment addition in the platform, online test and the like, and generally only two to three maintenance and supervision personnel are needed to complete deployment and subsequent supervision and maintenance, so that supervision efficiency is greatly improved, and labor cost is saved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and those skilled in the art can also obtain other drawings according to the drawings without creative efforts.
The invention combines the related technology of big data processing to collect, store and cache a large amount of electromagnetic data in real time, analyze in real time, count data and display in a multi-dimensional visualization way, and alarm abnormal electromagnetic data. And data support is provided for the decision of management personnel.
FIG. 1 is a flow chart illustrating interaction between modules of an electromagnetic data processing platform according to an embodiment of the present invention;
fig. 2 is a schematic flow chart of a data caching module according to an embodiment of the present invention;
fig. 3 is another schematic flow chart illustrating interaction between modules of an electromagnetic data processing platform according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Fig. 1 is a schematic flowchart illustrating interaction between modules of an electromagnetic data processing platform according to an embodiment of the present invention, where the flowchart includes: the system comprises a data collection module 11, a data storage module 12, a data caching module 13, a real-time calculation module 14, a statistic caching module 15, a visualization module 16 and an analysis mining module 17;
the data collection module 11 is configured to monitor a message sent by a front-end acquisition device; collecting electromagnetic trace data and electromagnetic alarm data by using the Logstash, and sending the collected electromagnetic trace data and the collected electromagnetic alarm data to the data caching module 13 and the data storage module 12;
in the embodiment of the present invention, the front-end acquisition device is deployed in each place where electromagnetic data needs to be acquired, is connected to the data collection module 11, and can interact with the data collection module 11. The Logstash is an open-source data collection engine, has real-time data transmission capability, and can process electromagnetic data by adopting a Logstash technology. The data collection module 11 connected to the front-end collection device may monitor a message sent by the front-end collection device, may also collect electromagnetic trace data and electromagnetic alarm data by using a Logstash technique, and sends the electromagnetic trace data and the electromagnetic alarm data to the data storage module 12 and the data cache module 13.
The data storage module 12 is configured to receive the electromagnetic trace data and the electromagnetic alarm data sent by the data collection module 11, store the electromagnetic trace data and the electromagnetic alarm data by using an Elasticsearch, and retrieve the electromagnetic trace data and the electromagnetic alarm data; sending the electromagnetic trace data and the electromagnetic alarm data to the analysis mining module 17 and the statistics cache module 15;
in the embodiment of the present invention, the data storage module 12 is a data persistent storage module provided by an electromagnetic data processing platform, and provides support for persistent storage work of the platform by using an Elasticsearch component. The elastic search is an open-source high-expansion distributed full-text search engine, has good expansibility, and can be expanded to hundreds of servers to process PB-level data. And data can be stored and retrieved in real time. Since the deployment of the whole electromagnetic environment involves a lot of environmental information such as devices distributed in various regions, the information is stored in an elastic search for being inquired by analysis and management personnel, so that the states and operating conditions of all the devices can be grasped. Meanwhile, the data storage module 12 can receive the electromagnetic trace data and the electromagnetic alarm data sent by the data collection module 11, store the data collected by the data collection module 11 by using an Elasticsearch, and retrieve the electromagnetic trace data and the electromagnetic alarm data. And may send the electromagnetic trace data and the electromagnetic alarm data to the analysis mining module 17 and the statistics cache module 15 for subsequent data analysis.
The data caching module 13 is configured to receive the electromagnetic trace data and the electromagnetic alarm data sent by the data collection module 11, and cache the data by using Kafka; sending the received electromagnetic trace data and electromagnetic alarm data to the real-time calculation module 14, the visualization module 16 and the analysis mining module 17;
in the embodiment of the present invention, the front-end collection device cluster continuously submits real-time data to the data collection module 11, and after the data are aggregated, the real-time data are calculated and sent to the modules in the background, such as the visualization module 16 and the analysis mining module 17. The server cannot process a large amount of inrush data in time, which can cause congestion, blockage, data loss and even system crash. The kafka is a distributed message queue, has high-performance, persistence, multi-copy backup and horizontal expansion capability, and generally plays roles of decoupling, peak clipping and asynchronous processing in architecture design. The invention adds the kafka message queue as the data cache module 13 to relieve the problem that the production rate and the consumption rate of the electromagnetic data cannot be synchronous, and the asynchronous processing of the messages is carried out to ensure that a large amount of real-time data is not lost. The data caching module 13 may receive the electromagnetic trace data and the electromagnetic alarm data sent by the data collecting module 11, and cache the data by using Kafka; and may send the received electromagnetic trace data and electromagnetic alarm data to the real-time calculation module 14, the visualization module 16, and the analysis mining module 17; the data caching module 13 may forward the real-time message of the data collecting module 11, and the visualization module 16 and the analysis mining module 17 may subscribe the electromagnetic alarm data and the electromagnetic trace data from the kafka message queue, and perform real-time dynamic display and analysis of the message of each module. The flow chart of the data caching module 13 is shown in fig. 2.
The real-time computing module 14 is configured to receive the electromagnetic trace data and the electromagnetic alarm data sent by the data caching module 13; receiving the relevant information of the front-end acquisition equipment sent by the statistics cache module 15; the storm is utilized to count the electromagnetic trace data and the electromagnetic alarm data, and the statistical result is sent to the statistical cache module 15;
in the embodiment of the present invention, the real-time computing module 14 is mainly responsible for real-time statistics of electromagnetic trace data and electromagnetic alarm data, and sends the statistics result to the statistics caching module 15. The electromagnetic trace data and the electromagnetic alarm data sent by the data caching module 13 and the front-end acquisition device related information sent by the statistics caching module 15 may also be received. Part of the functions of the visualization module 16 require statistical information of electromagnetic trace data and electromagnetic alarm data, and if a conventional database mode is adopted, a large amount of frequent database I/O operations are generated, and meanwhile, the condition of querying the same content in a short time occurs, so that the use efficiency of the platform is greatly reduced, and the visualization module 16 cannot display the electromagnetic trace data and the electromagnetic alarm data in real time. The Storm real-time stream processing tool is a real-time data processing framework and has the characteristics of low delay, high availability, easiness in expansion, no data loss and the like, and meanwhile, the Storm also provides a simple programming model similar to MapReduce and is convenient to develop. Therefore, the real-time calculation module 14 performs corresponding field extraction and information statistics on the real-time messages subscribed from kafka by using storm, organizes the statistical results into different key-value information, and sends the key-value information to the statistics cache module 15.
The statistics cache module 15 is configured to receive the statistics result sent by the real-time computing module 14, and store the statistics result in the statistics cache module 15 in a key-value form by using Redis; storing the front-end acquisition device related information extracted from the data storage module 12 and sending the front-end acquisition device related information to the real-time computation module 14; sending the statistical result to the visualization module 16;
in the embodiment of the present invention, the statistics cache module 15 adopts a Redis cache technology, and is mainly responsible for receiving the statistics result sent by the real-time computing module 14, and Redis essentially a Key-Value type memory database. Because of the pure memory operation, the Redis has excellent performance, can process more than 10 ten thousand read-write operations per second, and is a Key-Value database with the fastest known performance. The statistics cache module 15 stores the statistics result of the real-time calculation module 14 in the statistics cache module 15 in a key-value form, and simultaneously sends the statistics result to the visualization module 16, so as to provide data support for the visualization module 16. Meanwhile, the calculation of the real-time calculation module 14 needs to frequently read the relevant information of the front-end acquisition device stored in the data storage module 12, and the relevant information of the front-end acquisition device which needs to be repeatedly read can be extracted from the database and stored in the statistics cache module 15, so that the calculation efficiency of the platform is greatly improved. In addition, the front-end collecting device-related information extracted from the data storage module 12 is saved and sent to the real-time computing module 14.
The visualization module 16 is configured to receive the electromagnetic trace data and the electromagnetic warning data sent by the data caching module 13, the statistical result sent by the statistical caching module 15, and the analysis result sent by the analysis mining module 17, and visually display the electromagnetic trace data and the electromagnetic warning data, the statistical result, and the analysis result;
in an embodiment of the present invention, first, the visualization module 16 may receive the electromagnetic trace data and the electromagnetic alarm data sent by the data caching module 13, the statistical result sent by the statistical caching module 15, and the analysis result sent by the analysis mining module 17. Secondly, the visualization module 16 is also responsible for the visualization of electromagnetic trace data, electromagnetic alarm data, statistical results, and analysis results. The visualization module 16 clearly and effectively shows the electromagnetic environment meanings represented by the electromagnetic trace data and the electromagnetic alarm data by means of a graphical means, so that the analysts and the managers can quickly grasp the key point information and the abnormal information, and the electromagnetic environment is controlled. The visualization module 16 displays the electromagnetic environment in real time from different visualization levels and different time and field statistical dimensions.
The analysis mining module 17 is configured to analyze the historical data in the data storage module 12 and the cache data in the data cache module 13 by using an integrated machine learning algorithm and a data mining technology, and send an analysis result to the visualization module 16.
In the embodiment of the present invention, the analysis mining module 17 provides an intelligent service for the platform. The analysis mining module 17 integrates a common machine learning algorithm and a common data mining technology, and can analyze the historical data in the data storage module 12 and the cache data in the data cache module 13, mine a potential rule, and provide support for the decision of an analysis and decision maker. The analysis result of the analysis mining module 17 may be sent to the visualization module 16 for visualization display. The embodiment of the invention respectively uses the Logstash and the Elasticissearch to collect and store a large amount of electromagnetic data in real time; the cache technologies such as kafka and Redis are respectively used, so that the problem that the data reading speed and the data writing speed are not matched is solved, and the robustness of the system is improved; the storm is used for real-time calculation and analysis, so that the analysis instantaneity is improved, and the labor cost is reduced; and providing a visual display platform, and dynamically displaying the electromagnetic trace data on a large screen in a line graph mode. A machine learning algorithm and a data mining technology are provided, so that electromagnetic data processing is more intelligent; the deployment process of the electromagnetic data processing platform is simple, and comprises equipment release, IP configuration, local area network establishment, basic information of the equipment addition in the platform, online test and the like, and generally only two to three maintenance and supervision personnel are needed to complete deployment and subsequent supervision and maintenance, so that supervision efficiency is greatly improved, and labor cost is saved.
Further, on the basis of the above platform embodiment, the electromagnetic data processing platform further includes: a configuration management module 31, configured to send configuration information to the data collection module 11.
In the embodiment of the present invention, the configuration management module 31 is shown in fig. 3. It should be noted that the data collection module 11 is connected to a front-end acquisition device. Some parts of the data collection module 11 are present in the front-end acquisition device for acquiring data. The configuration management module 31 may send configuration information to the data collection module 11 to control some parts of the data collection module 11 in the front-end collection device to collect specific data.
It should be further noted that the configuration management module 31 provides a set of management tools to manage platform deployment, and a background manager can manage equipment and monitor a system in the background management module, so as to conveniently master the current operating status of the whole platform cluster.
The embodiment of the invention controls the acquisition of specific data through the configuration information, and facilitates the management of equipment and the monitoring of a system by background management personnel through a set of management tools.
Further, on the basis of the above platform embodiment, the analysis mining module 17 is further configured to generate configuration information according to an analysis result of analyzing the historical data in the data storage module 12 and the cache data in the data cache module 13; the configuration information is sent to the configuration management module 31.
In the embodiment of the present invention, the analysis mining module 17 may analyze the historical data in the data storage module 12 and the cache data in the data cache module 13. And generates configuration information according to the analysis result and transmits the configuration information to the configuration management module 31.
The configuration information generated by the analysis mining module according to the analysis result can be used for controlling the data collection module to collect the specific data.
Further, on the basis of the above platform embodiment, the configuration management module is further configured to receive the configuration information sent by the analysis mining module.
In the embodiment of the present invention, the analysis mining module 17 may generate more appropriate configuration information according to an analysis result of analyzing the historical data in the data storage module 12 and the cache data in the data cache module 13, and send the configuration information to the configuration management module 31. Thus, the configuration management module 31 may receive the configuration information sent by the analysis mining module 17.
The configuration management module of the embodiment of the invention can receive the configuration information from the analysis mining module, and the configuration information can be used for controlling the data collection module to collect specific data.
Further, on the basis of the above platform embodiment, the data collection module is further configured to receive configuration information sent by the configuration management module.
In this embodiment of the present invention, the data collection module 11 may receive the configuration information sent by the configuration management module 31, and collect data according to the configuration information.
According to the embodiment of the invention, the data collection of the data collection module is controlled through the configuration information sent by the configuration management module.
Further, on the basis of the above platform embodiment, the configuration information includes: electromagnetic trace data, electromagnetic alarm data, voice data, and/or bandwidth.
In an embodiment of the present invention, the configuration information sent by the configuration management module 31 includes electromagnetic trace data, electromagnetic alarm data, voice data and/or bandwidth. It is noted that the configuration information includes, but is not limited to, the electromagnetic trace data, electromagnetic alarm data, voice data, and/or bandwidth described above.
The configuration information in the embodiments of the present invention may be used to control the collection of specific data by the data collection module.
The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (6)
1. An electromagnetic data processing platform, comprising:
the system comprises a data collection module, a data storage module, a data caching module, a real-time calculation module, a statistic caching module, a visualization module and an analysis mining module;
the data collection module is used for monitoring messages sent by the front-end acquisition equipment; collecting electromagnetic trace data and electromagnetic alarm data by using the Logstash, and sending the collected electromagnetic trace data and the collected electromagnetic alarm data to the data caching module and the data storage module;
the data storage module is used for receiving the electromagnetic trace data and the electromagnetic alarm data sent by the data collection module, storing the electromagnetic trace data and the electromagnetic alarm data by using an elastic search and retrieving the electromagnetic trace data and the electromagnetic alarm data; sending the electromagnetic trace data and the electromagnetic alarm data to the analysis mining module and the statistics caching module;
the data caching module is used for receiving the electromagnetic trace data and the electromagnetic alarm data sent by the data collecting module and caching the data by using Kafka; sending the received electromagnetic trace data and electromagnetic alarm data to the real-time computation module, the visualization module, and the analysis mining module;
the real-time computing module is used for receiving the electromagnetic trace data and the electromagnetic alarm data sent by the data caching module; receiving the relevant information of the front-end acquisition equipment sent by the statistics cache module; counting the electromagnetic trace data and the electromagnetic alarm data by using storm, and sending a counting result to the counting cache module;
the statistics cache module is used for receiving the statistics result sent by the real-time calculation module and storing the statistics result in the statistics cache module in a key-value form by utilizing Redis; storing the relevant information of the front-end acquisition equipment extracted from the data storage module and sending the relevant information of the front-end acquisition equipment to the real-time computing module; sending the statistical result to the visualization module;
the visualization module is used for receiving the electromagnetic trace data and the electromagnetic alarm data sent by the data caching module, the statistical result sent by the statistical caching module and the analysis result sent by the analysis mining module, and visually displaying the electromagnetic trace data and the electromagnetic alarm data, the statistical result and the analysis result;
the analysis mining module is used for analyzing the historical data in the data storage module and the cache data in the data cache module by utilizing an integrated machine learning algorithm and a data mining technology, and sending an analysis result to the visualization module.
2. The electromagnetic data processing platform of claim 1, further comprising: and the configuration management module is used for sending configuration information to the data collection module.
3. The electromagnetic data processing platform of claim 2, wherein the analytics mining module is further configured to generate configuration information based on analysis results of analyzing historical data in the data storage module and cached data in the data caching module; and sending the configuration information to the configuration management module.
4. The electromagnetic data processing platform of claim 3, wherein the configuration management module is further configured to receive configuration information sent by the analytics mining module.
5. The electromagnetic data processing platform of claim 2, wherein the data collection module is further configured to receive configuration information sent by the configuration management module.
6. An electromagnetic data processing platform according to any of claims 2-5, wherein said configuration information comprises: electromagnetic trace data, electromagnetic alarm data, voice data, and/or bandwidth.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911002384.XA CN111090646B (en) | 2019-10-21 | 2019-10-21 | Electromagnetic data processing platform |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911002384.XA CN111090646B (en) | 2019-10-21 | 2019-10-21 | Electromagnetic data processing platform |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111090646A true CN111090646A (en) | 2020-05-01 |
| CN111090646B CN111090646B (en) | 2023-07-28 |
Family
ID=70393399
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911002384.XA Active CN111090646B (en) | 2019-10-21 | 2019-10-21 | Electromagnetic data processing platform |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111090646B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111914014A (en) * | 2020-08-17 | 2020-11-10 | 深圳市联恒星科技有限公司 | Big data platform and application thereof |
| CN118277625A (en) * | 2024-06-04 | 2024-07-02 | 北京珊瑚礁科技有限公司 | Distributed electromagnetic big data processing system and method |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN205902093U (en) * | 2016-06-24 | 2017-01-18 | 南京欣网通信科技股份有限公司 | Mobile network optimizing system based on big data platform |
| CN106709003A (en) * | 2016-12-23 | 2017-05-24 | 长沙理工大学 | Hadoop-based mass log data processing method |
| US20180113847A1 (en) * | 2009-03-02 | 2018-04-26 | International Business Machines Corporation | Automating interrogative population of electronic forms using a real-time communication system |
| CN108334556A (en) * | 2017-12-31 | 2018-07-27 | 江苏易润信息技术有限公司 | A kind of method and system of analysis internet finance massive logs |
| US20190034716A1 (en) * | 2017-12-28 | 2019-01-31 | Intel Corporation | Privacy-preserving demographics identification |
| CN109460393A (en) * | 2018-11-08 | 2019-03-12 | 浙江工业大学 | Visualization system is repaired in a kind of preliminary examination based on big data in advance |
| CN109977125A (en) * | 2019-04-09 | 2019-07-05 | 福建奇点时空数字科技有限公司 | A kind of big data safety analysis plateform system based on network security |
-
2019
- 2019-10-21 CN CN201911002384.XA patent/CN111090646B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180113847A1 (en) * | 2009-03-02 | 2018-04-26 | International Business Machines Corporation | Automating interrogative population of electronic forms using a real-time communication system |
| CN205902093U (en) * | 2016-06-24 | 2017-01-18 | 南京欣网通信科技股份有限公司 | Mobile network optimizing system based on big data platform |
| CN106709003A (en) * | 2016-12-23 | 2017-05-24 | 长沙理工大学 | Hadoop-based mass log data processing method |
| US20190034716A1 (en) * | 2017-12-28 | 2019-01-31 | Intel Corporation | Privacy-preserving demographics identification |
| CN108334556A (en) * | 2017-12-31 | 2018-07-27 | 江苏易润信息技术有限公司 | A kind of method and system of analysis internet finance massive logs |
| CN109460393A (en) * | 2018-11-08 | 2019-03-12 | 浙江工业大学 | Visualization system is repaired in a kind of preliminary examination based on big data in advance |
| CN109977125A (en) * | 2019-04-09 | 2019-07-05 | 福建奇点时空数字科技有限公司 | A kind of big data safety analysis plateform system based on network security |
Non-Patent Citations (4)
| Title |
|---|
| SANKET CHINTAPALLI等: "Benchmarking streaming computation engines: Storm, flink and spark streaming", 《2016 IEEE INTERNATIONAL PARALLEL AND DISTRIBUTED PROCESSING SYMPOSIUM WORKSHOPS (IPDPSW)》 * |
| 吕彬等: "基于大数据的电磁频谱数据分析技术" * |
| 吕彬等: "大数据在信息安全领域的应用分析" * |
| 郝行军: "物联网大数据存储与管理技术研究", 《中国博士学位论文全文数据库 信息科技辑》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111914014A (en) * | 2020-08-17 | 2020-11-10 | 深圳市联恒星科技有限公司 | Big data platform and application thereof |
| CN118277625A (en) * | 2024-06-04 | 2024-07-02 | 北京珊瑚礁科技有限公司 | Distributed electromagnetic big data processing system and method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111090646B (en) | 2023-07-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11150974B2 (en) | Anomaly detection using circumstance-specific detectors | |
| CN104407964B (en) | A kind of centralized monitoring system and method based on data center | |
| CN107704387B (en) | Method, device, electronic equipment and computer readable medium for system early warning | |
| KR102087959B1 (en) | Artificial intelligence operations system of telecommunication network, and operating method thereof | |
| US20210311468A1 (en) | Real-time alerts and transmission of selected signal samples under a dynamic capacity limitation | |
| CN112749153B (en) | Industrial network data management system | |
| CN115776449B (en) | Train Ethernet communication state monitoring method and system | |
| CN114091704B (en) | Alarm suppression method and device | |
| CN116668380A (en) | Message processing method and device of convergence diverter equipment | |
| CN114461792A (en) | Alarm event correlation method, device, electronic equipment, medium and program product | |
| CN110677271B (en) | Big data alarm method, device, equipment and storage medium based on ELK | |
| CN114428711A (en) | Data detection method, device, equipment and storage medium | |
| CN115809119A (en) | Monitoring method, system and device for container arrangement engine | |
| CN111090646A (en) | Electromagnetic data processing platform | |
| CN111209314A (en) | System for processing massive log data of power information system in real time | |
| CN114881167A (en) | Abnormality detection method, abnormality detection device, electronic apparatus, and medium | |
| CN111339466A (en) | Interface management method and device, electronic equipment and readable storage medium | |
| CN113312321A (en) | Abnormal monitoring method for traffic and related equipment | |
| CN111708846A (en) | Multi-terminal data management method and device | |
| CN111190790A (en) | Cloud computing cluster monitoring method and system based on peak prediction | |
| CN115080363B (en) | System capacity evaluation method and device based on service log | |
| CN110958137A (en) | Traffic management method and device and electronic equipment | |
| CN109947623A (en) | A kind of monitoring system and method shared for government data | |
| CN115408236A (en) | Log data auditing system, method, equipment and medium | |
| CN112579394A (en) | Log processing system and method applied to internet finance and computer equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |