CN116701525A - Early warning method and system based on real-time data analysis and electronic equipment - Google Patents
Early warning method and system based on real-time data analysis and electronic equipment Download PDFInfo
- Publication number
- CN116701525A CN116701525A CN202210192774.3A CN202210192774A CN116701525A CN 116701525 A CN116701525 A CN 116701525A CN 202210192774 A CN202210192774 A CN 202210192774A CN 116701525 A CN116701525 A CN 116701525A
- Authority
- CN
- China
- Prior art keywords
- data
- early warning
- production data
- target
- index
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 48
- 238000007405 data analysis Methods 0.000 title claims abstract description 24
- 238000004519 manufacturing process Methods 0.000 claims abstract description 172
- 238000004364 calculation method Methods 0.000 claims abstract description 35
- 230000002776 aggregation Effects 0.000 claims description 25
- 238000004220 aggregation Methods 0.000 claims description 25
- 230000006399 behavior Effects 0.000 claims description 18
- 238000003860 storage Methods 0.000 claims description 18
- 230000004931 aggregating effect Effects 0.000 claims description 11
- 230000000007 visual effect Effects 0.000 claims description 7
- 230000004044 response Effects 0.000 claims description 5
- 238000012545 processing Methods 0.000 abstract description 12
- 238000004458 analytical method Methods 0.000 abstract description 7
- 238000011161 development Methods 0.000 abstract description 5
- 230000002159 abnormal effect Effects 0.000 abstract description 4
- 230000008569 process Effects 0.000 description 10
- 238000004590 computer program Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 6
- 238000012544 monitoring process Methods 0.000 description 6
- 230000009471 action Effects 0.000 description 4
- 238000013480 data collection Methods 0.000 description 4
- 230000008901 benefit Effects 0.000 description 3
- 230000005856 abnormality Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000006116 polymerization reaction Methods 0.000 description 2
- 238000007792 addition Methods 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 230000002354 daily effect Effects 0.000 description 1
- 238000013144 data compression Methods 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 230000003203 everyday effect Effects 0.000 description 1
- 238000003780 insertion Methods 0.000 description 1
- 230000037431 insertion Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000002085 persistent effect Effects 0.000 description 1
- 238000007619 statistical method Methods 0.000 description 1
- 238000000547 structure data Methods 0.000 description 1
- 230000001131 transforming effect Effects 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Landscapes
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The application provides an early warning method, system and electronic equipment based on real-time data analysis, and belongs to the technical field of data processing. According to the embodiment of the application, the newly added and/or changed data is stored and processed in real time through the high-throughput distributed publishing and subscribing message system, so that real-time and comprehensive collection of enterprise big data is realized; meanwhile, the initial production data are sequentially split and aggregated to respectively obtain intermediate production data and target production data, the intermediate production data and the target production data are stored in a distributed release subscription message system, and the data are stored in a layered manner, so that the data structure can be simplified, the repeated development and the repeated calculation can be reduced, and the generation speed of early warning information can be greatly improved; according to the target production data, early warning information corresponding to the preset early warning indexes is generated, immediate data query, statistics and analysis services can be provided, and the corresponding early warning information can be quickly responded and timely sent under the condition that the data is abnormal.
Description
Technical Field
The application relates to the technical field of data processing, in particular to an early warning method, system and electronic equipment based on real-time data analysis.
Background
The early warning system is a system which is used for an enterprise to analyze data by collecting production data, monitor the variation trend of risk factors, evaluate the degree of deviation of various risk states from an early warning line, send early warning signals to a decision layer and take pre-control countermeasures in advance. The early warning system is an effective means of minimizing the risk-induced losses. Has become one of the important measures to ensure the business operations of enterprises and create the greatest benefit. The main function of the early warning system is to obtain related information and operation data through various monitoring means, process and analyze the data, and make preliminary judgment on future trends through a proper evaluation method. When the judging result meets the requirement of the early warning rule, the alarm system is triggered, and the alarm system gives out accident alarm according to the preset alarm level.
With the rapid development of enterprises, the data volume is larger and larger, and especially the production data such as transaction data, user behavior data and the like generated by electronic commerce, finance and train enterprises every day are huge. For the early warning system, the real-time performance of the data is very important, and the huge data volume is faced, which means that stronger computing power and response speed are required.
However, in view of the limited technical development of the enterprises, the early warning systems of most of the enterprises at present collect partial data of the enterprises, the data volume is relatively small, the monitoring indexes are relatively less, the data collection is incomplete, partial risks are possibly omitted, and the enterprises are lost; the method is limited by the calculation force of the system, is difficult to calculate large data volume, is slow in calculation, and even is blocked; by adopting an off-line calculation and analysis mode, the problems of untimely early warning and untimely response caused by low instantaneity exist.
Disclosure of Invention
The application provides an early warning method, system and electronic equipment based on real-time data analysis, which are used for solving the problems of low real-time performance and untimely early warning caused by incomplete data acquisition in enterprise production and difficulty in real-time calculation of a large amount of data.
In order to solve the problems, the application adopts the following technical scheme:
in a first aspect, an embodiment of the present application provides an early warning method based on real-time data analysis, where the method includes:
acquiring newly added and/or changed initial production data in real time;
storing the initial production data through a distributed publish-subscribe messaging system;
the initial production data is divided and aggregated in sequence to respectively obtain intermediate production data and target production data, and the intermediate production data and the target production data are stored in the distributed publishing and subscribing message system;
and generating early warning information corresponding to a preset early warning index according to the target production data.
In an embodiment of the present application, obtaining initial production data of new additions and/or changes in real time includes:
acquiring newly added and/or changed service data in a database in real time, or acquiring newly added and/or changed user behavior data of a front-end page in real time;
analyzing the business data and/or the user behavior data, and taking the analyzed business data and/or the analyzed user behavior data as the initial production data.
In one embodiment of the present application, the distributed publish-subscribe messaging system includes an original data layer, a data detail layer, and a data aggregation layer; the data aggregation layer is used for storing the target production data obtained by aggregating the intermediate production data.
In an embodiment of the present application, the splitting and aggregating the initial production data sequentially to obtain intermediate production data and target production data, and storing the intermediate production data and the target production data in the distributed publish-subscribe message system, including:
reading the initial production data stored in the original data layer by adopting a Flink calculation engine, shunting the initial production data according to the data type of the initial production data to obtain the intermediate production data, and storing the intermediate production data to the data detail layer;
and reading the intermediate production data by adopting the Flink calculation engine, aggregating the data of the same subject in the intermediate production data to obtain the target production data, and storing the target production data into the data aggregation layer.
In an embodiment of the present application, generating early warning information corresponding to a preset early warning indicator according to the target production data includes:
storing the target production data into a pre-configured column database so that the column database generates index data corresponding to the target production data;
selecting corresponding target index data from the index data according to a preset early warning index;
and displaying early warning information corresponding to the preset early warning index according to the target index data.
In an embodiment of the present application, the early warning information includes an early warning threshold corresponding to each preset early warning indicator, and the method further includes:
in response to a configuration instruction for changing the preset early warning index, or,
and responding to a configuration instruction for changing the early warning threshold value, and changing the early warning threshold value.
In an embodiment of the present application, the early warning information includes an alarm level corresponding to each preset early warning indicator, and the method further includes:
outputting alarm information corresponding to the alarm level under the condition that any target index data in the alarm information is larger than a corresponding alarm threshold value; wherein, different alarm grades correspond to different alarm information.
In a second aspect, based on the same inventive concept, an embodiment of the present application provides an early warning system based on real-time data analysis, the system including: the system comprises a real-time data acquisition module, a data layering storage module, a real-time calculation module and a data application module; the data hierarchical storage module comprises a pre-configured distributed publish-subscribe message system, wherein the distributed publish-subscribe message system comprises an original data layer, a data detail layer and a data aggregation layer;
the real-time data acquisition module is used for acquiring newly added and/or changed initial production data in real time and transmitting the initial production data to the data layering storage module;
the data layering storage module is used for receiving the initial production data and storing the initial production data to the original data layer;
the real-time computing module is used for reading the initial production data from the original data layer through a Flink computing engine, sequentially splitting and aggregating the initial production data to respectively obtain intermediate production data and target production data, storing the intermediate production data into the data detail layer, and storing the target production data into the data aggregation layer; the data aggregation layer is also used for transmitting the target production data in the data aggregation layer to the data application module;
the data application module is used for receiving the target production data and storing the target production data into a pre-configured column database so that the column database generates index data corresponding to the target production data.
In one embodiment of the present application, the system further comprises: the visual early warning module and the early warning notification module; wherein,,
the visual early warning module is used for selecting corresponding target index data from the index data according to a preset early warning index; displaying early warning information corresponding to the preset early warning index according to the target index data; the early warning information comprises an early warning threshold value and an alarm level corresponding to each preset early warning index;
the early warning notification module is used for outputting warning information corresponding to the warning level under the condition that any target index data in the early warning information is larger than a corresponding early warning threshold value; wherein, different alarm grades correspond to different alarm information.
In a third aspect, based on the same inventive concept, an embodiment of the present application provides an electronic device, including a processor and a memory, where the memory stores machine executable instructions executable by the processor, and the processor is configured to execute the machine executable instructions to implement the early warning method based on real-time data analysis according to the first aspect of the present application.
Compared with the prior art, the application has the following advantages:
according to the early warning method based on real-time data analysis, a large amount of continuously added and/or changed data is stored and processed in real time through a high-throughput distributed publishing and subscribing message system, so that real-time and comprehensive collection of enterprise big data is realized; meanwhile, the initial production data are sequentially split and aggregated to respectively obtain intermediate production data and target production data, the intermediate production data and the target production data are stored in a distributed release subscription message system, and the data are stored in a layered manner, so that the data structure can be simplified, the repeated development and the repeated calculation can be reduced, and the generation speed of early warning information can be greatly improved; and generating early warning information corresponding to the preset early warning indexes according to the target production data, providing immediate data query, statistics and analysis service, and rapidly responding and timely sending out the corresponding early warning information under the condition that the data is abnormal. The embodiment of the application can realize real-time acquisition, real-time calculation and real-time update of early warning information of enterprise big data, and early warning in time, and has the characteristics of high throughput, low delay and high performance.
Drawings
In order to more clearly illustrate the embodiments of the application or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a flowchart of steps of an early warning method based on real-time data analysis in an embodiment of the present application.
Fig. 2 is a schematic diagram of a functional module of an early warning system based on real-time data analysis in an embodiment of the application.
Reference numerals: 201-a real-time data acquisition module; 202-a data hierarchical storage module; 203-a real-time computing module; 204-a data application module; 205-a visual early warning module; 206-an early warning notification module.
Detailed Description
The following description of the embodiments of the present application will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present application, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.
Referring to fig. 1, there is shown an early warning method based on real-time data analysis according to the present application, which may include the steps of:
s101: and acquiring newly added and/or changed initial production data in real time.
In this embodiment, the initial production data refers to a series of business data and user behavior data generated by an enterprise during a production operation process. The business data can be divided into various business data according to different business ranges of enterprises, and the business data corresponding to production and manufacturing enterprises can comprise data such as capacity data, progress data, quality data and cost data, and the business data corresponding to Internet enterprises can comprise data such as order data, transaction data, flow data and complaint data; the user behavior data refers to recorded user interaction behavior information of the user on the webpage or the application, and specifically may include information such as clicking operation, browsing operation, stay time and the like of the user on the webpage or the application.
It should be noted that, the service data is generally stored in a database configured in advance by the enterprise; the user behavior data can be obtained through embedding points in a web/app front-end page, wherein the embedding points are used for collecting some information in a specific process in the application and tracking the use condition of the application, and are used for further optimizing products or providing data support for operation, and the embedding points are roughly divided into two parts, wherein one part is used for counting the access condition of the application page, namely page statistics, and reporting along with the occurrence of the page access action; the other part is to count the operation behavior in the application and report the operation in the page.
In this embodiment, to ensure that the newly added and/or changed initial production data can be obtained comprehensively in real time, S101 may specifically include the following sub-steps:
s101-1: and acquiring newly added and/or changed service data in the database in real time, or acquiring newly added and/or changed user behavior data of the front-end page in real time.
S101-2: and analyzing the business data and/or the user behavior data, and taking the analyzed business data and/or user behavior data as initial production data.
In this embodiment, the user behavior data collected from the front-end page generally adopts data with a specific structure, such as JSON (JavaScript Object Notation, JS object numbered musical notation, a lightweight data exchange format) data, so that the user behavior data needs to be parsed to obtain parsed user behavior data for subsequent data analysis and processing.
In this embodiment, for the service data stored in the database, a flank-CDC technique may be used to monitor the newly added and/or changed service data in the database in real time. The flank-CDC may read the full and incremental changes directly from the database, such as MySQL, postgreSQL, to monitor and capture database changes, including data or data table insertions, updates, and deletions, and record these changes in their order of occurrence. When the Flink-CDC is used, connection is established with the MySQL database according to the IP address information, port number, user name, password, database name and other information of the designated MySQL database, after the service data in the MySQL database are changed, the changed service data can be read in real time, then the acquired service data are analyzed, and the analyzed service data are sent to a distributed publishing and subscribing message system for storage so as to carry out subsequent data analysis and processing.
S102: the initial production data is stored via a distributed publish-subscribe messaging system.
In this embodiment, the distributed publish-subscribe messaging system may be constructed based on Kafka, which is a high throughput distributed publish-subscribe messaging component, and its biggest feature is that a large amount of data flowing in the network can be processed in real time to meet various requirements. Kafka can process hundreds of thousands of messages per second, its delay is at least a few milliseconds, supporting data backup to prevent data loss, and supporting thousands of clients to read and write simultaneously.
In this embodiment, considering that the real-time calculation of a huge amount of initial production data is realized, huge calculation pressure is brought, because, to effectively reduce the repeated calculation of data and improve the generation speed of the target production data, the distributed publish-subscribe message system is structurally optimized, and is divided into an original data layer, a data detail layer and a data aggregation layer, where the original data layer is used for storing the initial production data, the data detail layer is used for storing intermediate production data obtained after splitting the initial production data, and the data aggregation layer is used for storing the target production data obtained after aggregating the intermediate production data.
In this embodiment, by hierarchically managing data, each layer of data may represent different data granularity, and each time a statistical requirement is newly increased, calculation from initial production data may not be needed, but intermediate data in a data detail layer or target production data in a data aggregation layer may be directly called as required, so as to effectively reduce repeated development and repeated calculation, and improve data generation and query speed.
S103: and sequentially splitting and aggregating the initial production data to respectively obtain intermediate production data and target production data, and storing the intermediate production data and the target production data into a distributed publishing and subscribing message system.
In this embodiment, considering that the initial production data may contain more useless data, before the initial production data is shunted, an ETL (Extract-Transform-Load) operation may be performed on the initial production data, that is, a process of extracting (Extract), transforming (Transform), and loading (Load) the initial production data from the data collection end to the original data layer. Through ETL operation, scattered and standard non-uniform initial production data in enterprises can be integrated together, empty data and illegal data are filtered out, and analysis basis is provided for subsequent processing.
In this embodiment, the initial production data subjected to the ETL process will be split according to the data type of the initial production data, where the data type may include, but is not limited to, order data type, transaction data type, flow data type, complaint data type, page access data type, and the like, and the intermediate production data obtained after the split will be stored in the data detail layer of Kafka. After obtaining the intermediate production data, the intermediate production data may be further subjected to light polymerization according to the subject matter, and the target production data obtained after the light polymerization may be stored in the data aggregation layer of Kafka.
It should be noted that in Kafka, messages may be categorized in units of topics (Topic), a sender is called Producer, a receiver is called Consumer, a Producer is responsible for sending messages to a specific Topic (each message sent to the Kafka cluster is assigned a Topic), and a Consumer is responsible for subscribing to a Topic and consuming.
In this embodiment, different topics may be formulated according to the actual business of the enterprise, for example, for an e-commerce enterprise, a member topic, a coupon topic, an activity topic, etc. may be set. The intermediate data is further aggregated according to the subject, so that the data aggregation layer of Kafka can meet more real-time query requirements while relieving the query pressure of the data detail layer.
S104: and generating early warning information corresponding to the preset early warning index according to the target production data.
In this embodiment, based on the target production data, an immediate data query, statistics and analysis service may be provided, and when an abnormality occurs in the target production data, the abnormality information may be obtained at the first time, and corresponding early warning information may be generated.
For enterprises with large daily output data, the obtained target production data is large in volume and data types. Accordingly, various types of index data can be correspondingly acquired based on the target production data. And by setting the preset early warning indexes, target index data corresponding to the preset early warning indexes can be screened out from various early warning indexes, and early warning information corresponding to the target index data is generated based on the relation between the target index data and the early warning threshold value.
In the embodiment, the early warning information can be visualized, so that related personnel can clearly and comprehensively know the early warning information at the first time, and the early warning response speed under abnormal conditions is improved. For example, a corresponding report and/or a statistical chart can be generated according to the target index data, and the report and/or the statistical chart is displayed in the monitoring large screen in real time.
In a possible embodiment, to meet the calculation requirement of a large amount of initial production data and to increase the calculation speed, S103 may specifically include the following sub-steps:
s103-1: and reading the initial production data stored in the original data layer by adopting a Flink calculation engine, shunting the initial production data according to the data type of the initial production data to obtain intermediate production data, and storing the intermediate production data into a data detail layer.
S103-2: and reading the intermediate production data by adopting a Flink calculation engine, aggregating the data of the same subject in the intermediate production data to obtain target production data, and storing the target production data into a data aggregation layer.
In this embodiment, the calculation of each layer of data in the distributed publish-subscribe message system may be performed by the Flink calculation engine. It should be noted that the link calculation engine is a high throughput, low latency, high performance distributed big data processing engine, and can perform stateful calculation on unbounded data streams and bounded data streams, and perform fast calculation on data sizes of various sizes.
In this embodiment, a flank calculation engine is used to perform stateful calculation on continuously generated unbounded data streams, i.e., initial production data.
Specifically, the data can be calculated in real time by adopting a distributed computing framework based on the Flink computing engine, and a large number of computing tasks are parallel to thousands of tasks distributed in a cluster and executed simultaneously; meanwhile, the task executes all the calculations by accessing the local memory state, and the task is always kept in the memory, or when the state size exceeds the available memory, the task is kept in the structure data on the access efficient disk. The flank adopts an asynchronous and incremental check point algorithm, so that the minimum influence on processing delay can be ensured, the consistency of the primary state when faults occur is ensured by periodically and asynchronously checking the local state to persistent storage, namely, even if faults or downtime occur, the task can be continuously executed on the basis of the failed task node by reading the task state when the task is restarted, the task is prevented from being lost or recalculated, and the task calculation efficiency is effectively improved.
In the embodiment, the Flink computing engine is used for storing the initial production data and carrying out the splitting operation and the aggregation operation on the intermediate data, so that the computing requirements of high throughput, low delay and high performance can be fully met.
In one possible embodiment, S104 may specifically include the following substeps:
s104-1: and storing the target production data into a pre-configured column database so that the column database generates index data corresponding to the target production data.
S104-2: and selecting corresponding target index data from the index data according to a preset early warning index.
S104-3: and displaying the early warning information corresponding to the preset early warning index according to the target index data.
In this embodiment, the generation of the early warning information may be implemented by using Clickhouse. It should be noted that the click house is a column database management system for online analysis (OLAP), which enables an analyst to quickly, consistently, and interactively observe information from various aspects for the purpose of understanding data in depth. ClickHouse is used as a column-oriented database with the fastest data processing in the field of in-memory databases, the performance of the ClickHouse exceeds that of most column-oriented databases at present, and various scenes of data statistical analysis are supported: support SQL-like queries, support numerous library functions, support arrays (arrays) and nested data structures, support database copy-in-place deployment. In the clickHouse, for efficient use of the CPU, data is not only stored by column, but also processed by vector; the data compression space is large, and IO can be effectively reduced; the method has the advantages of high throughput, high writing speed and the like, and is suitable for updating large data.
In the present embodiment, the lightly aggregated target production data is stored in the clickHouse, and an immediate data query, statistics, and analysis service can be provided. As one preferable scheme, the Flink calculation engine can be called to calculate index data, then the index data is analyzed and processed, target index data corresponding to a preset early warning index is selected from the index data, and then early warning information corresponding to the preset early warning index is obtained. The early warning information can be displayed in a report form, and displayed report data can be updated in real time.
In one example, an electronic business performs splitting and aggregation based on initial production data collection to obtain target production data, where the index data corresponding to the target production data includes: commodity indexes, complaint indexes, wind control indexes, marketing activities indexes, trade indexes, flow indexes, shopping cart indexes, order placing indexes and payment indexes. Among the above indexes, the complaint index, the wind control index, the order setting index and the payment index need to be monitored in an important way, and the four important indexes can be pre-configured as preset early warning indexes, so that each time the data corresponding to the four important indexes changes, the corresponding early warning information is updated, an effective early warning mechanism is formed, and the purposes of real-time acquisition, real-time calculation and real-time updating are achieved.
In a possible implementation manner, the early warning information is provided with an early warning threshold value corresponding to each preset early warning index, and the method may further include the following steps:
s105: and responding to the configuration instruction for changing the preset early warning index, and changing the preset early warning index, or responding to the configuration instruction for changing the early warning threshold, and changing the early warning threshold.
In this embodiment, the preset early warning index and the early warning threshold corresponding to the preset early warning index may be set according to the actual monitoring requirement.
For example, when a business is newly added to a certain enterprise, a corresponding preset early warning index can be added for the business, and a corresponding early warning threshold can be configured for the business according to actual monitoring needs. Or when the enterprise gives up a certain service or the certain service is not important any more, the preset early warning index corresponding to the service can be deleted, so that the data of the service are not displayed in the early warning information any more. It should be noted that, although the data of the service will not be displayed in the early warning information, the data of the service will be stored in the distributed publish-subscribe message system in a layered manner, and used in other data analysis requirements, so as to ensure the comprehensiveness of enterprise data collection.
In a possible implementation manner, the early warning information may further include an alarm level corresponding to each preset early warning indicator, and the method may further include the following steps:
s106: outputting alarm information corresponding to the alarm level under the condition that any target index data in the alarm information is larger than a corresponding alarm threshold value; wherein, different alarm grades correspond to different alarm information.
In this embodiment, different alarm modes may be set based on the severity of the abnormal indicator, where the alarm information may include corresponding information including a preset early warning indicator, a current indicator value, an alarm level, an alarm time, details of a problem, a source system, and the like.
In the embodiment, the alarm information can be notified to related personnel through mail, short message, APP, telephone, micro message or a third party platform and the like, and meanwhile, the alarm information can be displayed in a large monitoring screen in real time to remind the related personnel of timely paying attention to taking corresponding measures.
In one example, a user performs illegal or fraudulent operations in an e-commerce platform and a financial platform, and then the data can be quickly screened out through real-time calculation of big data and then processed by an air control department. In another example, if it is detected that the order data, transaction data or click data generated by the user for a certain commodity is higher or increased, it is indicated that the user has a stronger purchase intention for a certain commodity, then these "business opportunities" may be pushed to the customer service department, so that the customer service can actively follow up.
In the embodiment, the large data volume of the enterprise is collected in real time, calculated in real time, and simultaneously high throughput, low delay and high performance are supported, so that wind control or marketing can be quickly and timely obtained early warning information, and various countermeasures can be taken; the early warning information can also be formed into a statistical report, and then the large screen is monitored for real-time display and update, so that related personnel can know the latest progress of the event in time; the related responsible person can be informed by mail, short message, APP, telephone, micro message or third party platform, so as to take countermeasures quickly.
In a second aspect, referring to fig. 2, based on the same inventive concept, an embodiment of the present application provides an early warning system based on real-time data analysis, the system including: a real-time data acquisition module 201, a data hierarchical storage module 202, a real-time calculation module 203 and a data application module 204; wherein the data hierarchical storage module 202 comprises a pre-configured distributed publish-subscribe message system, the distributed publish-subscribe message system comprising an original data layer, a data detail layer and a data aggregation layer;
the real-time data acquisition module 201 is configured to acquire newly added and/or changed initial production data in real time, and send the initial production data to the data hierarchical storage module 202;
a data layering storage module 202 for receiving the initial production data and storing the initial production data to an original data layer;
the real-time computing module 203 is configured to read initial production data from the original data layer through the link computing engine, sequentially split and aggregate the initial production data to obtain intermediate production data and target production data, store the intermediate production data into the data detail layer, and store the target production data into the data aggregation layer; and is further configured to send the target production data in the data aggregation layer to the data application module 204;
the data application module 204 is configured to receive the target production data, and store the target production data in a pre-configured column database, so that the column database generates index data corresponding to the target production data.
In one possible embodiment, the early warning system based on real-time data analysis may further include: a visual early warning module 205 and an early warning notification module 206; wherein,,
the visual early warning module 205 is configured to select corresponding target index data from the index data according to a preset early warning index; displaying early warning information corresponding to a preset early warning index according to the target index data; the early warning information comprises an early warning threshold value and an alarm level corresponding to each preset early warning index;
the early warning notification module 206 is configured to output warning information corresponding to the warning level when any one of the target index data in the early warning information is greater than the corresponding early warning threshold; wherein, different alarm grades correspond to different alarm information.
It should be noted that, the specific implementation manner of the early warning system based on real-time data analysis according to the embodiment of the present application refers to the specific implementation manner of the early warning system method based on real-time data analysis set forth in the first aspect of the embodiment of the present application, and is not described herein again.
In a third aspect, based on the same inventive concept, an embodiment of the present application provides an electronic device, including a processor and a memory, where the memory stores machine executable instructions capable of being executed by the processor, and the processor is configured to execute the machine executable instructions to implement the early warning system method based on real-time data analysis according to the first aspect of the present application.
It should be noted that, the specific implementation manner of the electronic device according to the embodiment of the present application refers to the specific implementation manner of the early warning system method based on real-time data analysis according to the first aspect of the embodiment of the present application, which is not described herein.
It will be apparent to those skilled in the art that embodiments of the present application may be provided as a method, apparatus, or computer program product. Accordingly, embodiments of the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, embodiments of the application may take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.
Embodiments of the present application are described with reference to flowchart illustrations and/or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
While preferred embodiments of the present application have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. It is therefore intended that the following claims be interpreted as including the preferred embodiment and all such alterations and modifications as fall within the scope of the embodiments of the application.
Finally, it is further noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or terminal. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or terminal device comprising the element.
The foregoing describes in detail a method, a system and an electronic device for early warning based on real-time data analysis, and specific examples are applied to illustrate the principles and embodiments of the present application, and the description of the foregoing examples is only used to help understand the method and core idea of the present application; meanwhile, as those skilled in the art will have variations in the specific embodiments and application scope in accordance with the ideas of the present application, the present description should not be construed as limiting the present application in view of the above.
Claims (10)
1. An early warning method based on real-time data analysis, which is characterized by comprising the following steps:
acquiring newly added and/or changed initial production data in real time;
storing the initial production data through a distributed publish-subscribe messaging system;
the initial production data is divided and aggregated in sequence to respectively obtain intermediate production data and target production data, and the intermediate production data and the target production data are stored in the distributed publishing and subscribing message system;
and generating early warning information corresponding to a preset early warning index according to the target production data.
2. The method of claim 1, wherein obtaining the newly added and/or changed initial production data in real time comprises:
acquiring newly added and/or changed service data in a database in real time, or acquiring newly added and/or changed user behavior data of a front-end page in real time;
analyzing the business data and/or the user behavior data, and taking the analyzed business data and/or the analyzed user behavior data as the initial production data.
3. The method of claim 1, wherein the distributed publish-subscribe messaging system comprises a raw data layer, a data detail layer, and a data aggregation layer; the data aggregation layer is used for storing the target production data obtained by aggregating the intermediate production data.
4. A method according to claim 3, wherein the splitting and aggregating the initial production data sequentially to obtain intermediate production data and target production data, respectively, and storing the intermediate production data and the target production data to the distributed publish-subscribe messaging system comprises:
reading the initial production data stored in the original data layer by adopting a Flink calculation engine, shunting the initial production data according to the data type of the initial production data to obtain the intermediate production data, and storing the intermediate production data to the data detail layer;
and reading the intermediate production data by adopting the Flink calculation engine, aggregating the data of the same subject in the intermediate production data to obtain the target production data, and storing the target production data into the data aggregation layer.
5. The method of claim 1, wherein generating pre-warning information corresponding to a pre-set pre-warning indicator based on the target production data comprises:
storing the target production data into a pre-configured column database so that the column database generates index data corresponding to the target production data;
selecting corresponding target index data from the index data according to a preset early warning index;
and displaying early warning information corresponding to the preset early warning index according to the target index data.
6. The method of claim 5, wherein the pre-warning information includes a pre-warning threshold corresponding to each preset pre-warning indicator, the method further comprising:
in response to a configuration instruction for changing the preset early warning index, or,
and responding to a configuration instruction for changing the early warning threshold value, and changing the early warning threshold value.
7. The method of claim 5, wherein the pre-warning information includes a warning level corresponding to each preset pre-warning indicator, the method further comprising:
outputting alarm information corresponding to the alarm level under the condition that any target index data in the alarm information is larger than a corresponding alarm threshold value; wherein, different alarm grades correspond to different alarm information.
8. An early warning system based on real-time data analysis, the system comprising: the system comprises a real-time data acquisition module, a data layering storage module, a real-time calculation module and a data application module; the data hierarchical storage module comprises a pre-configured distributed publish-subscribe message system, wherein the distributed publish-subscribe message system comprises an original data layer, a data detail layer and a data aggregation layer;
the real-time data acquisition module is used for acquiring newly added and/or changed initial production data in real time and transmitting the initial production data to the data layering storage module;
the data layering storage module is used for receiving the initial production data and storing the initial production data to the original data layer;
the real-time computing module is used for reading the initial production data from the original data layer through a Flink computing engine, sequentially splitting and aggregating the initial production data to respectively obtain intermediate production data and target production data, storing the intermediate production data into the data detail layer, and storing the target production data into the data aggregation layer; the data aggregation layer is also used for transmitting the target production data in the data aggregation layer to the data application module;
the data application module is used for receiving the target production data and storing the target production data into a pre-configured column database so that the column database generates index data corresponding to the target production data.
9. The system of claim 8, wherein the system further comprises: the visual early warning module and the early warning notification module; wherein,,
the visual early warning module is used for selecting corresponding target index data from the index data according to a preset early warning index; displaying early warning information corresponding to the preset early warning index according to the target index data; the early warning information comprises an early warning threshold value and an alarm level corresponding to each preset early warning index;
the early warning notification module is used for outputting warning information corresponding to the warning level under the condition that any target index data in the early warning information is larger than a corresponding early warning threshold value; wherein, different alarm grades correspond to different alarm information.
10. An electronic device comprising a processor and a memory, the memory storing machine executable instructions executable by the processor for executing the machine executable instructions to implement the method of any of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210192774.3A CN116701525A (en) | 2022-02-28 | 2022-02-28 | Early warning method and system based on real-time data analysis and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210192774.3A CN116701525A (en) | 2022-02-28 | 2022-02-28 | Early warning method and system based on real-time data analysis and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116701525A true CN116701525A (en) | 2023-09-05 |
Family
ID=87826364
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210192774.3A Pending CN116701525A (en) | 2022-02-28 | 2022-02-28 | Early warning method and system based on real-time data analysis and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116701525A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117349322A (en) * | 2023-12-05 | 2024-01-05 | 摩尔元数(福建)科技有限公司 | SPC real-time analysis method and system based on analysis control chart |
| CN117851501A (en) * | 2023-12-29 | 2024-04-09 | 奥格科技股份有限公司 | Data analysis method, device, equipment and storage medium |
-
2022
- 2022-02-28 CN CN202210192774.3A patent/CN116701525A/en active Pending
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117349322A (en) * | 2023-12-05 | 2024-01-05 | 摩尔元数(福建)科技有限公司 | SPC real-time analysis method and system based on analysis control chart |
| CN117349322B (en) * | 2023-12-05 | 2024-03-08 | 摩尔元数(福建)科技有限公司 | SPC real-time analysis method and system based on analysis control chart |
| CN117851501A (en) * | 2023-12-29 | 2024-04-09 | 奥格科技股份有限公司 | Data analysis method, device, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111984499B (en) | Fault detection method and device for big data cluster | |
| US20240086421A1 (en) | Method and Apparatus for Monitoring an In-memory Computer System | |
| US10666525B2 (en) | Distributed multi-data source performance management | |
| US20210176136A1 (en) | Continuous data sensing of functional states of networked computing devices to determine efficiency metrics for servicing electronic messages asynchronously | |
| US8600955B2 (en) | Method and apparatus for monitoring an in-memory computer system | |
| CN111339175B (en) | Data processing method, device, electronic equipment and readable storage medium | |
| US20100070981A1 (en) | System and Method for Performing Complex Event Processing | |
| CN116701525A (en) | Early warning method and system based on real-time data analysis and electronic equipment | |
| CN112948397A (en) | Data processing system, method, device and storage medium | |
| CN104991939A (en) | Transaction data monitoring method and system | |
| CN112052134A (en) | Service data monitoring method and device | |
| CN113032252A (en) | Method and device for collecting buried point data, client device and storage medium | |
| CN114253806A (en) | Access stratum log collection, analysis and early warning system | |
| CN114090529A (en) | Log management method, device, system and storage medium | |
| CN111414355A (en) | Offshore wind farm data monitoring and storing system, method and device | |
| CN116795631A (en) | Service system monitoring alarm method, device, equipment and medium | |
| CN112825165A (en) | Project quality management method and device | |
| CN110677271A (en) | Big data alarm method, device, equipment and storage medium based on ELK | |
| CN113986656B (en) | Power grid data safety monitoring system based on data center platform | |
| CN112448840B (en) | Communication data quality monitoring method, device, server and storage medium | |
| CN114625763A (en) | Information analysis method and device for database, electronic equipment and readable medium | |
| CN114049036A (en) | Data computing platform, method, device and storage medium | |
| CN112527887A (en) | Visual operation and maintenance method and device applied to Gbase database | |
| Fu et al. | A Study of Brand Power Simulation by Time Series Based on Flink and Kafka Framework | |
| CN117056184A (en) | Detection method and device of database system, storage medium and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |