CN113420043A - Data real-time monitoring method, device, equipment and storage medium - Google Patents
Data real-time monitoring method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN113420043A CN113420043A CN202110691329.7A CN202110691329A CN113420043A CN 113420043 A CN113420043 A CN 113420043A CN 202110691329 A CN202110691329 A CN 202110691329A CN 113420043 A CN113420043 A CN 113420043A
- Authority
- CN
- China
- Prior art keywords
- data
- early warning
- real
- time
- monitoring
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2425—Iterative querying; Query formulation based on the results of a preceding query
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/283—Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
Abstract
The invention relates to the field of data analysis and discloses a method, a device, equipment and a storage medium for real-time data monitoring. The method comprises the following steps: creating a message transmission channel between a production environment and a real-time computing platform, and receiving real-time data generated by the production environment through the message transmission channel; preprocessing the real-time data to obtain aggregate data of early warning monitoring dimensions; performing multilayer logic operation on the aggregated data to obtain early warning monitoring index data; and monitoring whether the early warning monitoring index data meets preset early warning conditions in real time, and if so, sending early warning information to an early warning object corresponding to the early warning conditions. The invention can monitor the data in the service scene in real time based on the real-time computing technology, thereby improving the safety and stability of the platform operation.
Description
Technical Field
The present invention relates to the field of data analysis, and in particular, to a method, an apparatus, a device, and a storage medium for real-time monitoring of data.
Background
With the development of internet technology, real-time computing technology has been gradually applied to actual services of internet companies, and services can make timely adjustments through real-time activity data, so that operation efficiency is improved and risk control is performed.
The traditional risk control system adopts a T +1 offline data calculation mode, the timeliness is low, the data of the current day can be known the next day, the data is obviously insufficient for activity effect monitoring and risk control, the timeliness is low, and a large risk exists.
Disclosure of Invention
The invention mainly aims to solve the technical problem that the platform operation risk is high due to low timeliness of data monitoring.
The invention provides a real-time data monitoring method in a first aspect, which comprises the following steps:
creating a message transmission channel between a production environment and a real-time computing platform, and receiving real-time data generated by the production environment through the message transmission channel;
preprocessing the real-time data to obtain aggregate data of early warning monitoring dimensions;
performing multilayer logic operation on the aggregated data to obtain early warning monitoring index data;
and monitoring whether the early warning monitoring index data meets preset early warning conditions in real time, and if so, sending early warning information to an early warning object corresponding to the early warning conditions.
Optionally, in a first implementation manner of the first aspect of the present invention, before the creating a message transmission channel between the production environment and the real-time computing platform, the method further includes:
acquiring a task through a preset dimension table, and asynchronously acquiring target dimension data, wherein the target dimension data is used for indicating service index data related to early warning monitoring dimensions;
importing the target dimension data into a thermal storage data warehouse by adopting a thermal storage import mechanism;
and asynchronously reading the target dimension data in the thermal storage data warehouse, and storing the target dimension data into a cache to obtain a target dimension table.
Optionally, in a second implementation manner of the first aspect of the present invention, the creating a message transmission channel between a production environment and a real-time computing platform, and receiving real-time data generated by the production environment through the message transmission channel includes:
subscribing a data source to be monitored in a production environment, and creating a message transmission channel between the data source and a real-time computing platform;
and monitoring the real-time message transmitted by the message transmission channel, and receiving the real-time data generated by the production environment through the message transmission channel.
Optionally, in a third implementation manner of the first aspect of the present invention, the preprocessing the real-time data to obtain aggregated data of early warning monitoring dimensions includes:
creating a data source table corresponding to the real-time data, and storing the real-time data into the data source table;
performing association query on the data source table and the target dimension table in the cache according to the early warning monitoring dimension to obtain an association query result, and aggregating the association query result and the data source table to obtain first aggregated data;
and performing data cleaning on the first aggregated data to obtain second aggregated data meeting the requirement of the target storage format.
Optionally, in a fourth implementation manner of the first aspect of the present invention, the performing multilayer logic operation on the aggregated data to obtain early warning monitoring index data includes:
adopting a plurality of preset real-time calculation operators to respectively calculate early warning monitoring indexes corresponding to the second aggregation data to obtain a plurality of early warning monitoring index data;
and outputting each early warning monitoring index data to a target database, and storing the early warning monitoring index data by adopting a column type storage method.
Optionally, in a fifth implementation manner of the first aspect of the present invention, the monitoring whether the early warning monitoring index data meets a preset early warning condition in real time, and if so, sending an early warning message to an early warning object corresponding to the early warning condition includes:
reading a plurality of preset early warning conditions, and determining early warning monitoring index data corresponding to the early warning conditions according to the early warning conditions;
and dynamically monitoring whether the early warning monitoring index meets the corresponding early warning condition, and if so, sending an early warning message to an early warning object corresponding to the early warning condition.
Optionally, in a sixth implementation manner of the first aspect of the present invention, after the monitoring whether the early warning monitoring index data meets a preset early warning condition in real time, and if so, sending an early warning message to an early warning object corresponding to the early warning condition, the method further includes:
reading a preset user portrait model, and inputting the early warning monitoring index data into the user portrait model in real time;
respectively judging whether each user accords with each early warning monitoring index in the early warning monitoring index data through the user portrait model to obtain a user list which respectively accords with each early warning monitoring index;
and adding the label corresponding to each early warning monitoring index into the portrait data of each user in the user list, and outputting each label and the user list corresponding to each label.
The second aspect of the present invention provides a real-time data monitoring apparatus, including:
the system comprises a creating module, a data processing module and a data processing module, wherein the creating module is used for creating a message transmission channel between a production environment and a real-time computing platform and receiving real-time data generated by the production environment through the message transmission channel;
the aggregation module is used for preprocessing the real-time data to obtain aggregated data of early warning monitoring dimensions;
the operation module is used for carrying out multilayer logic operation on the aggregated data to obtain early warning monitoring index data;
and the early warning module is used for monitoring whether the early warning monitoring index data meets preset early warning conditions in real time, and if so, sending early warning information to an early warning object corresponding to the early warning conditions.
Optionally, in a first implementation manner of the second aspect of the present invention, the data real-time monitoring apparatus further includes:
the acquisition module is used for acquiring tasks through a preset dimension table and asynchronously acquiring target dimension data, wherein the target dimension data is used for indicating service index data related to early warning monitoring dimensions;
the thermal storage module is used for importing the target dimension data into a thermal storage data warehouse by adopting a thermal storage import mechanism;
and the cache module is used for asynchronously reading the target dimension data in the thermal storage data warehouse and storing the target dimension data into a cache to obtain a target dimension table.
Optionally, in a second implementation manner of the second aspect of the present invention, the creating module is specifically configured to:
subscribing a data source to be monitored in a production environment, and creating a message transmission channel between the data source and a real-time computing platform;
and monitoring the real-time message transmitted by the message transmission channel, and receiving the real-time data generated by the production environment through the message transmission channel.
Optionally, in a third implementation manner of the second aspect of the present invention, the aggregation module is specifically configured to:
creating a data source table corresponding to the real-time data, and storing the real-time data into the data source table;
performing association query on the data source table and the target dimension table in the cache according to the early warning monitoring dimension to obtain an association query result, and aggregating the association query result and the data source table to obtain first aggregated data;
and performing data cleaning on the first aggregated data to obtain second aggregated data meeting the requirement of the target storage format.
Optionally, in a fourth implementation manner of the second aspect of the present invention, the operation module is specifically configured to:
adopting a plurality of preset real-time calculation operators to respectively calculate early warning monitoring indexes corresponding to the second aggregation data to obtain a plurality of early warning monitoring index data;
and outputting each early warning monitoring index data to a target database, and storing the early warning monitoring index data by adopting a column type storage method.
Optionally, in a fifth implementation manner of the second aspect of the present invention, the early warning module is specifically configured to:
reading a plurality of preset early warning conditions, and determining early warning monitoring index data corresponding to the early warning conditions according to the early warning conditions;
and dynamically monitoring whether the early warning monitoring index meets the corresponding early warning condition, and if so, sending an early warning message to an early warning object corresponding to the early warning condition.
Optionally, in a sixth implementation manner of the second aspect of the present invention, the data real-time monitoring apparatus further includes:
the model reading module is used for reading a preset user portrait model and inputting the early warning monitoring index data into the user portrait model in real time;
the module judging module is used for respectively judging whether each user accords with each early warning monitoring index in the early warning monitoring index data through the user portrait model to obtain a user list which respectively accords with each early warning monitoring index;
and the model output module is used for adding the labels corresponding to the early warning monitoring indexes into the portrait data of the users in the user list and outputting the labels and the user list corresponding to the labels.
The third aspect of the present invention provides a real-time data monitoring apparatus, including: a memory and at least one processor, the memory having instructions stored therein; the at least one processor calls the instructions in the memory to enable the data real-time monitoring equipment to execute the data real-time monitoring method.
A fourth aspect of the present invention provides a computer-readable storage medium having stored therein instructions, which, when run on a computer, cause the computer to execute the above-mentioned data real-time monitoring method.
In the technical scheme provided by the invention, in order to improve the timeliness of data monitoring, a message transmission channel, namely a subscription type message queue, between a production environment to be monitored and a real-time computing platform is established, multidimensional data preprocessing is carried out by receiving data transmitted by the message queue in real time to obtain aggregated data required by real-time monitoring index computing, and then monitoring index data is obtained by multilayer logic operation on the aggregated data, thereby realizing monitoring and early warning. The invention can monitor the data in the service scene in real time based on the real-time computing technology, thereby improving the safety and stability of the platform operation.
Drawings
FIG. 1 is a schematic diagram of a first embodiment of a real-time data monitoring method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a second embodiment of a real-time data monitoring method according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a first embodiment of a real-time data monitoring apparatus according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of a second embodiment of a real-time data monitoring apparatus according to an embodiment of the present invention;
fig. 5 is a schematic diagram of an embodiment of a data real-time monitoring device in an embodiment of the present invention.
Detailed Description
The embodiment of the invention provides a method, a device, equipment and a storage medium for real-time monitoring of data. The terms "first," "second," "third," "fourth," and the like in the description and in the claims, as well as in the drawings, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It will be appreciated that the data so used may be interchanged under appropriate circumstances such that the embodiments described herein may be practiced otherwise than as specifically illustrated or described herein. Furthermore, the terms "comprises," "comprising," or "having," and any variations thereof, are intended to cover non-exclusive inclusions, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
For convenience of understanding, a specific flow of the embodiment of the present invention is described below, and referring to fig. 1, a first embodiment of the real-time data monitoring method in the embodiment of the present invention includes:
101. creating a message transmission channel between a production environment and a real-time computing platform, and receiving real-time data generated by the production environment through the message transmission channel;
it is to be understood that the executing subject of the present invention may be a data real-time monitoring device, and may also be a terminal or a server, which is not limited herein. The embodiment of the present invention is described by taking a server as an execution subject.
In this embodiment, the Topic is a channel for transmitting messages between the production environment and the real-time computing platform, and in the platform of the internet of things, the server and the device realize message communication through the Topic. Two messaging models supported by JMS (Java Message Service): queue (Queue) and Topic (Topic), where Topic implements distribution and subscription, and when you distribute a message, all services subscribing to the message can get the message, so that from 0 to many subscribers can get a copy of the message, and only when the message broker receives the message, there is at least one active subscribed person, the copy of the message can be got. And the message queue is used for transmitting messages, so that the real-time performance of data can be ensured.
In this embodiment, the production environment is an online environment, and is a server environment used by a user, and it can generate real data in the use process of the user, and the data is used for real-time calculation, so as to monitor whether various indexes generated by the real data are normal, thereby achieving the purpose of risk control.
Optionally, in an embodiment, before step 101, the method further includes:
acquiring a task through a preset dimension table, and asynchronously acquiring target dimension data, wherein the target dimension data is used for indicating service index data related to early warning monitoring dimensions;
importing the target dimension data into a thermal storage data warehouse by adopting a thermal storage import mechanism;
and asynchronously reading the target dimension data in the thermal storage data warehouse, and storing the target dimension data into a cache to obtain a target dimension table.
In this alternative embodiment, the dimension table (target dimension table) may be viewed as a window of analytical data, with the dimension table containing properties of the fact records in the fact data table, some properties providing descriptive information, some properties specifying how the fact data table data is to be aggregated to provide useful information to the analyst, and the dimension table containing a hierarchy of properties that help aggregate the data. The dimension table contains the relevant detailed information of the specified attributes in the fact table, for example, the target dimension data can be the detailed product attributes, the customer attributes, the storage information and the like. A typical example is to compare logical services to a cube, and the product dimension, the time dimension, and the location dimension are respectively used as different coordinate axes, and the intersection of the coordinate axes is a concrete fact. That is, the fact table is an intersection of a plurality of dimension tables.
In this alternative embodiment, we often have the need to base the original data stream and then associate a large number of external tables to supplement some of the attributes. For example, if the order data is desired to obtain the name of the province where the order receiver is located, and generally the order records an ID of a province, the name attribute of the province needs to be supplemented by querying an external dimension table according to the ID. In order to improve timeliness of the dimension table, a thermal storage association mode is adopted for association query of the dimension table, and the specific implementation mode is that dimension data is imported into thermal storage, the thermal storage is queried through asynchronous IO, and the dimension data is cached in a memory through a Cache mechanism. When the user associates the dimension table, the cache is inquired, if the data does not exist in the cache, the hot storage data warehouse is inquired, and the inquired data is inserted into the cache. Therefore, the query speed of the dimension table can be faster, and the timeliness of real-time calculation is improved.
Optionally, in an embodiment, step 101 specifically includes:
subscribing a data source to be monitored in a production environment, and creating a message transmission channel between the data source and a real-time computing platform;
and monitoring the real-time message transmitted by the message transmission channel, and receiving the real-time data generated by the production environment through the message transmission channel.
In this alternative embodiment, messages are persisted into one Topic in a publish-subscribe messaging system. Unlike a peer-to-peer messaging system, a consumer may subscribe to one or more topics, the consumer may consume all of the data in the Topic, the same piece of data may be consumed by multiple consumers, and the data may not be immediately deleted after being consumed. In a publish-subscribe messaging system, the producer of a message is called a publisher and the consumer of the message is called a subscriber. The publisher of the invention is a production environment and the subscriber is a real-time computing platform.
In this optional embodiment, before receiving real-time data generated in a production environment, a data source to be monitored needs to be subscribed, and message processing can be performed through a publish-subscribe message system.
102. Preprocessing the real-time data to obtain aggregate data of early warning monitoring dimensions;
in this embodiment, after receiving the real-time data returned by the message channel, a series of preprocessing needs to be performed on the data to obtain data meeting the index calculation requirement for the purpose of early warning monitoring. The series of preprocessing includes data extraction, cleaning, conversion and loading, namely an ETL process in a data warehouse technology, and aims to aggregate scattered, messy and standard-inconsistent data together and provide an analysis basis for decision making.
In this embodiment, when obtaining the data source of the dimension table, the data sources of Hive and Hbase may be supported, but these two parts of data are also offline data, and may be written into Mysql as the data source of the dimension table through an offline task and read into the real-time computing platform. The Hive is a data warehouse tool based on Hadoop and used for data extraction, transformation and loading, and the Hive data warehouse tool can map structured data files into a database table, provide SQL query functions and convert SQL statements into MapReduce tasks for execution. The HBase is a distributed and nematic open source database, is a high-reliability, high-performance, nematic and telescopic distributed storage system, and can build a large-scale structured storage cluster on a cheap server by utilizing the HBase technology.
Optionally, in an embodiment, step 102 specifically includes:
creating a data source table corresponding to the real-time data, and storing the real-time data into the data source table;
performing association query on the data source table and the target dimension table in the cache according to the early warning monitoring dimension to obtain an association query result, and aggregating the association query result and the data source table to obtain first aggregated data;
and performing data cleaning on the first aggregated data to obtain second aggregated data meeting the requirement of the target storage format.
In this optional embodiment, the ETL process includes dimension association query of data, and the query method is to query, according to source data (fact table), a dimension table that meets the target early warning monitoring dimension in a dimension table stored in the thermal storage mechanism, and then aggregate the queried dimension table and fact table to obtain complete data. For example, if the early warning monitoring dimension is a time dimension, the time dimension table is inquired according to the fact table, and then the data in the fact table corresponding to the time of the time dimension table is integrated into aggregate data, so that the complete data required by monitoring can be obtained. For further screening, the data is washed to meet the target storage format requirements.
103. Performing multilayer logic operation on the aggregated data to obtain early warning monitoring index data;
in this embodiment, after the complete aggregated data is obtained, the index data concerned by the operation/wind control is calculated, so as to perform data monitoring. The real-time computing platform can meet different data monitoring requirements and provide visual interface configuration risk index parameters, and different index parameters are obtained through different operation methods, so that the logic operation method is not further limited.
In this embodiment, the multi-layer logical operation includes a custom function calculation, for example, if i want to monitor the sales condition of a certain product, then a calculation function of sales/sales volume is formulated according to the existing data (aggregated data), and the early warning monitoring index data, that is, the real-time sales/sales volume of the monitored product, is obtained through the calculation function, so as to prepare for further data early warning.
104. And monitoring whether the early warning monitoring index data meets preset early warning conditions in real time, and if so, sending early warning information to an early warning object corresponding to the early warning conditions.
In this embodiment, after the early warning monitoring index data is obtained, the early warning condition is matched, the multi-platform early warning condition configuration is supported, the real-time computing platform computes the data in real time and provides a real-time computing data calling interface, and the wind-control early warning platform, the data real-time display platform or the operation platform and the like can call the index data in real time through the real-time computing data calling interface to perform customized processing, so that the operation efficiency or the data risk control capability is improved. In addition, the early warning monitoring requirement can also be used for monitoring data by formulating an early warning rule through a real-time computing platform.
In the embodiment of the invention, in order to improve the timeliness of data monitoring, a message transmission channel, namely a subscription type message queue, between a production environment to be monitored and a real-time computing platform is created, data transmitted by the message queue is received in real time, multidimensional data preprocessing is carried out to obtain aggregated data required by real-time monitoring index calculation, and then monitoring index data is obtained through multilayer logic operation on the aggregated data, so that monitoring and early warning are realized. The invention can monitor the data in the service scene in real time based on the real-time computing technology, thereby improving the safety and stability of the platform operation.
Referring to fig. 2, a second embodiment of the real-time data monitoring method according to the embodiment of the present invention includes:
201. creating a message transmission channel between a production environment and a real-time computing platform, and receiving real-time data generated by the production environment through the message transmission channel;
202. preprocessing the real-time data to obtain aggregate data of early warning monitoring dimensions;
203. adopting a plurality of preset real-time calculation operators to respectively calculate early warning monitoring indexes corresponding to the second aggregation data to obtain a plurality of early warning monitoring index data;
204. outputting each early warning monitoring index data to a target database, and storing the early warning monitoring index data by adopting a column type storage method;
205. and monitoring whether the early warning monitoring index data meets preset early warning conditions in real time, and if so, sending early warning information to an early warning object corresponding to the early warning conditions.
In this embodiment, the real-time computation operator refers to an operator in the Flink technology, the Flink is an open source stream processing framework developed by the Apache software foundation, and the core of the framework is a distributed stream data stream engine written by Java and Scala. Flink executes arbitrary stream data programs in a data parallel and pipelined manner, and Flink's pipelined runtime system can execute batch and stream processing programs.
In this embodiment, preferably, a Flink-sql real-time calculation operator may also be used, where the Flink-sql is a simplified calculation model of the Flink real-time calculation, supports operations supported by most traditional databases, such as Union, Join, project, Difference, interaction, and window, provides a large number of operator operations, and mainly completes, for example, query and aggregation operations. Various logical operations can be satisfied and the speed is improved.
In the embodiment, after the index data required by early warning monitoring is obtained, the target data is stored by adopting a column type storage method (such as a ClickHouse and a TIDB database), and the method has higher query efficiency, so that the method has obvious advantages for the task of query efficiency and can be used as a data query interface for the outside.
Optionally, in an embodiment, step 205 specifically includes:
reading a plurality of preset early warning conditions, and determining early warning monitoring index data corresponding to the early warning conditions according to the early warning conditions;
and dynamically monitoring whether the early warning monitoring index meets the corresponding early warning condition, and if so, sending an early warning message to an early warning object corresponding to the early warning condition.
In this optional embodiment, different early warning conditions correspond to different early warning monitoring index data, for example, the early warning condition established by the operation may be whether the number of users who purchase a certain product in a certain time period exceeds a certain threshold, and if the number of users exceeds the certain threshold, a warehouse is arranged for replenishment immediately, so that the condition of commodity outage is avoided. Then, the corresponding monitoring index can be determined through the establishment of the early warning condition, the index data is further dynamically monitored, and the message sent back by the platform is detected in real time, so that the operation efficiency is improved.
Optionally, in an embodiment, after step 205, the method further includes:
reading a preset user portrait model, and inputting the early warning monitoring index data into the user portrait model in real time;
respectively judging whether each user accords with each early warning monitoring index in the early warning monitoring index data through the user portrait model to obtain a user list which respectively accords with each early warning monitoring index;
and adding the label corresponding to each early warning monitoring index into the portrait data of each user in the user list, and outputting each label and the user list corresponding to each label.
In the optional embodiment, the index data can be used for risk monitoring and early warning, and can also be used for perfecting the portrait of the user, so that the actual service is put in more accurately and more timely. In this embodiment, the real-time computing platform may further combine the real-time user image with machine learning, process the on-line production data by the real-time platform, perform index computation of complex dimensionality, write the computation result into the trained machine learning model in real time, and perform real-time personalized recommendation. Meanwhile, the label can be marked on the user in real time, real-time user group data can be configured in operation, and service delivery is facilitated.
In the embodiment of the invention, in order to improve the timeliness of real-time calculation, the processing speed of data is considered from multiple angles, in the aspect of index calculation, multiple operators in the real-time calculation technology are adopted to perform multilayer logic operation, so that the operation speed is improved, and in the aspect of data storage, a column type storage method is adopted, so that the data access speed is improved. The embodiment of the invention can efficiently calculate the real-time data, thereby improving the efficiency of platform wind control or operation.
The above description of the data real-time monitoring method in the embodiment of the present invention, and the following description of the data real-time monitoring apparatus in the embodiment of the present invention refer to fig. 3, where a first embodiment of the data real-time monitoring apparatus in the embodiment of the present invention includes:
a creating module 301, configured to create a message transmission channel between a production environment and a real-time computing platform, and receive real-time data generated by the production environment through the message transmission channel;
the aggregation module 302 is configured to preprocess the real-time data to obtain aggregated data of the early warning monitoring dimension;
the operation module 303 is configured to perform multilayer logic operation on the aggregated data to obtain early warning monitoring index data;
the early warning module 304 is configured to monitor whether the early warning monitoring index data meets a preset early warning condition in real time, and if so, send an early warning message to an early warning object corresponding to the early warning condition.
Optionally, in an embodiment, the data real-time monitoring apparatus further includes:
an obtaining module 305, configured to obtain a task through a preset dimension table, and asynchronously obtain target dimension data, where the target dimension data is used to indicate service index data related to an early warning monitoring dimension;
the thermal storage module 306 is configured to import the target dimension data into a thermal storage data warehouse by using a thermal storage import mechanism;
and the cache module 307 is configured to asynchronously read the target dimension data in the thermal storage data warehouse, and store the target dimension data in a cache to obtain a target dimension table.
Optionally, in an embodiment, the creating module 301 is specifically configured to:
subscribing a data source to be monitored in a production environment, and creating a message transmission channel between the data source and a real-time computing platform;
and monitoring the real-time message transmitted by the message transmission channel, and receiving the real-time data generated by the production environment through the message transmission channel.
Optionally, in an embodiment, the aggregation module 302 is specifically configured to:
creating a data source table corresponding to the real-time data, and storing the real-time data into the data source table;
performing association query on the data source table and the target dimension table in the cache according to the early warning monitoring dimension to obtain an association query result, and aggregating the association query result and the data source table to obtain first aggregated data;
and performing data cleaning on the first aggregated data to obtain second aggregated data meeting the requirement of the target storage format.
In the embodiment of the invention, in order to improve the timeliness of data monitoring, a message transmission channel, namely a subscription type message queue, between a production environment to be monitored and a real-time computing platform is created, data transmitted by the message queue is received in real time, multidimensional data preprocessing is carried out to obtain aggregated data required by real-time monitoring index calculation, and then monitoring index data is obtained through multilayer logic operation on the aggregated data, so that monitoring and early warning are realized. The invention can monitor the data in the service scene in real time based on the real-time computing technology, thereby improving the safety and stability of the platform operation.
Referring to fig. 4, a second embodiment of the real-time data monitoring apparatus according to the embodiment of the present invention includes:
a creating module 301, configured to create a message transmission channel between a production environment and a real-time computing platform, and receive real-time data generated by the production environment through the message transmission channel;
the aggregation module 302 is configured to preprocess the real-time data to obtain aggregated data of the early warning monitoring dimension;
the operation module 303 is configured to perform multilayer logic operation on the aggregated data to obtain early warning monitoring index data;
the early warning module 304 is configured to monitor whether the early warning monitoring index data meets a preset early warning condition in real time, and if so, send an early warning message to an early warning object corresponding to the early warning condition.
Optionally, in an embodiment, the operation module 303 is specifically configured to:
adopting a plurality of preset real-time calculation operators to respectively calculate early warning monitoring indexes corresponding to the second aggregation data to obtain a plurality of early warning monitoring index data;
and outputting each early warning monitoring index data to a target database, and storing the early warning monitoring index data by adopting a column type storage method.
Optionally, in an embodiment, the early warning module 304 is specifically configured to:
reading a plurality of preset early warning conditions, and determining early warning monitoring index data corresponding to the early warning conditions according to the early warning conditions;
and dynamically monitoring whether the early warning monitoring index meets the corresponding early warning condition, and if so, sending an early warning message to an early warning object corresponding to the early warning condition.
Optionally, in an embodiment, the data real-time monitoring apparatus further includes:
the model reading module 308 is used for reading a preset user portrait model and inputting the early warning monitoring index data into the user portrait model in real time;
a module judging module 309, configured to respectively judge whether each user meets each early warning monitoring index in the early warning monitoring index data through the user profile model, so as to obtain a user list meeting each early warning monitoring index;
a model output module 310, configured to add a tag corresponding to each of the early warning monitoring indicators to the portrait data of each user in the user list, and output each of the tags and the user list corresponding to each of the tags.
In the embodiment of the invention, in order to improve the timeliness of real-time calculation, the processing speed of data is considered from multiple angles, in the aspect of index calculation, multiple operators in the real-time calculation technology are adopted to perform multilayer logic operation, so that the operation speed is improved, and in the aspect of data storage, a column type storage method is adopted, so that the data access speed is improved. The embodiment of the invention can efficiently calculate the real-time data, thereby improving the efficiency of platform wind control or operation.
Fig. 3 and fig. 4 describe the data real-time monitoring apparatus in the embodiment of the present invention in detail from the perspective of the modular functional entity, and the data real-time monitoring device in the embodiment of the present invention is described in detail from the perspective of hardware processing.
Fig. 5 is a schematic structural diagram of a data real-time monitoring device according to an embodiment of the present invention, where the data real-time monitoring device 500 may have a relatively large difference due to different configurations or performances, and may include one or more processors (CPUs) 510 (e.g., one or more processors) and a memory 520, and one or more storage media 530 (e.g., one or more mass storage devices) storing applications 533 or data 532. Memory 520 and storage media 530 may be, among other things, transient or persistent storage. The program stored on the storage medium 530 may include one or more modules (not shown), each of which may include a series of instruction operations in the real-time data monitoring device 500. Still further, the processor 510 may be configured to communicate with the storage medium 530 to execute a series of instruction operations in the storage medium 530 on the data real-time monitoring device 500.
The data real-time monitoring apparatus 500 may also include one or more power supplies 540, one or more wired or wireless network interfaces 550, one or more input-output interfaces 560, and/or one or more operating systems 531, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will appreciate that the data real-time monitoring device configuration shown in FIG. 5 does not constitute a limitation of the data real-time monitoring device, and may include more or fewer components than shown, or some components in combination, or a different arrangement of components.
The invention further provides a data real-time monitoring device, which includes a memory and a processor, where the memory stores computer readable instructions, and the computer readable instructions, when executed by the processor, cause the processor to execute the steps of the data real-time monitoring method in the above embodiments.
The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium, and which may also be a volatile computer-readable storage medium, having stored therein instructions, which, when run on a computer, cause the computer to perform the steps of the data real-time monitoring method.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a read-only memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
The above-mentioned embodiments are only used for illustrating the technical solutions of the present invention, and not for limiting the same; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (10)
1. A real-time data monitoring method is characterized by comprising the following steps:
creating a message transmission channel between a production environment and a real-time computing platform, and receiving real-time data generated by the production environment through the message transmission channel;
preprocessing the real-time data to obtain aggregate data of early warning monitoring dimensions;
performing multilayer logic operation on the aggregated data to obtain early warning monitoring index data;
and monitoring whether the early warning monitoring index data meets preset early warning conditions in real time, and if so, sending early warning information to an early warning object corresponding to the early warning conditions.
2. The method of real-time data monitoring according to claim 1, further comprising, prior to said creating a message transmission channel between the production environment and the real-time computing platform:
acquiring a task through a preset dimension table, and asynchronously acquiring target dimension data, wherein the target dimension data is used for indicating service index data related to early warning monitoring dimensions;
importing the target dimension data into a thermal storage data warehouse by adopting a thermal storage import mechanism;
and asynchronously reading the target dimension data in the thermal storage data warehouse, and storing the target dimension data into a cache to obtain a target dimension table.
3. The method for real-time data monitoring according to claim 1, wherein the creating a message transmission channel between a production environment and a real-time computing platform, and the receiving real-time data generated by the production environment through the message transmission channel comprises:
subscribing a data source to be monitored in a production environment, and creating a message transmission channel between the data source and a real-time computing platform;
and monitoring the real-time message transmitted by the message transmission channel, and receiving the real-time data generated by the production environment through the message transmission channel.
4. The real-time data monitoring method according to claim 2, wherein the preprocessing the real-time data to obtain aggregated data of early warning monitoring dimensions comprises:
creating a data source table corresponding to the real-time data, and storing the real-time data into the data source table;
performing association query on the data source table and the target dimension table in the cache according to the early warning monitoring dimension to obtain an association query result, and aggregating the association query result and the data source table to obtain first aggregated data;
and performing data cleaning on the first aggregated data to obtain second aggregated data meeting the requirement of the target storage format.
5. The real-time data monitoring method according to claim 4, wherein the performing multi-layer logic operation on the aggregated data to obtain early warning monitoring index data comprises:
adopting a plurality of preset real-time calculation operators to respectively calculate early warning monitoring indexes corresponding to the second aggregation data to obtain a plurality of early warning monitoring index data;
and outputting each early warning monitoring index data to a target database, and storing the early warning monitoring index data by adopting a column type storage method.
6. The real-time data monitoring method according to claim 1, wherein the real-time monitoring whether the early warning monitoring index data meets a preset early warning condition, and if so, sending an early warning message to an early warning object corresponding to the early warning condition comprises:
reading a plurality of preset early warning conditions, and determining early warning monitoring index data corresponding to the early warning conditions according to the early warning conditions;
and dynamically monitoring whether the early warning monitoring index meets the corresponding early warning condition, and if so, sending an early warning message to an early warning object corresponding to the early warning condition.
7. The real-time data monitoring method according to claim 1, wherein the real-time monitoring whether the early warning monitoring index data meets a preset early warning condition, if yes, after sending an early warning message to an early warning object corresponding to the early warning condition, further comprises:
reading a preset user portrait model, and inputting the early warning monitoring index data into the user portrait model in real time;
respectively judging whether each user accords with each early warning monitoring index in the early warning monitoring index data through the user portrait model to obtain a user list which respectively accords with each early warning monitoring index;
and adding the label corresponding to each early warning monitoring index into the portrait data of each user in the user list, and outputting each label and the user list corresponding to each label.
8. A real-time data monitoring device, comprising:
the system comprises a creating module, a data processing module and a data processing module, wherein the creating module is used for creating a message transmission channel between a production environment and a real-time computing platform and receiving real-time data generated by the production environment through the message transmission channel;
the aggregation module is used for preprocessing the real-time data to obtain aggregated data of early warning monitoring dimensions;
the operation module is used for carrying out multilayer logic operation on the aggregated data to obtain early warning monitoring index data;
and the early warning module is used for monitoring whether the early warning monitoring index data meets preset early warning conditions in real time, and if so, sending early warning information to an early warning object corresponding to the early warning conditions.
9. A real-time data monitoring device, comprising: a memory and at least one processor, the memory having instructions stored therein;
the at least one processor invokes the instructions in the memory to cause the data real-time monitoring device to perform the data real-time monitoring method of any one of claims 1-7.
10. A computer-readable storage medium having instructions stored thereon, wherein the instructions, when executed by a processor, implement a method for real-time monitoring of data according to any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110691329.7A CN113420043A (en) | 2021-06-22 | 2021-06-22 | Data real-time monitoring method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110691329.7A CN113420043A (en) | 2021-06-22 | 2021-06-22 | Data real-time monitoring method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113420043A true CN113420043A (en) | 2021-09-21 |
Family
ID=77789731
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110691329.7A Pending CN113420043A (en) | 2021-06-22 | 2021-06-22 | Data real-time monitoring method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113420043A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113946627A (en) * | 2021-10-27 | 2022-01-18 | 北京科杰科技有限公司 | Data accuracy detection early warning system and method under data real-time synchronization scene |
| CN115134265A (en) * | 2022-05-16 | 2022-09-30 | 北京璇星科技有限公司 | Real-time monitoring and early warning method, device, equipment and storage medium for process |
| CN116166701A (en) * | 2023-03-17 | 2023-05-26 | 湖北坤盈数字科技有限公司 | Service data real-time early warning method, device, equipment and storage medium |
| CN116629805A (en) * | 2023-06-07 | 2023-08-22 | 浪潮智慧科技有限公司 | Water conservancy index service method, equipment and medium for distributed flow batch integration |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110908815A (en) * | 2019-12-03 | 2020-03-24 | 京东数字科技控股有限公司 | Message queue data early warning method, device and system and storage medium |
| CN110990433A (en) * | 2019-11-21 | 2020-04-10 | 深圳马可孛罗科技有限公司 | Real-time service monitoring and early warning method and early warning device |
-
2021
- 2021-06-22 CN CN202110691329.7A patent/CN113420043A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110990433A (en) * | 2019-11-21 | 2020-04-10 | 深圳马可孛罗科技有限公司 | Real-time service monitoring and early warning method and early warning device |
| CN110908815A (en) * | 2019-12-03 | 2020-03-24 | 京东数字科技控股有限公司 | Message queue data early warning method, device and system and storage medium |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113946627A (en) * | 2021-10-27 | 2022-01-18 | 北京科杰科技有限公司 | Data accuracy detection early warning system and method under data real-time synchronization scene |
| CN113946627B (en) * | 2021-10-27 | 2022-04-29 | 北京科杰科技有限公司 | Data accuracy detection early warning system and method under data real-time synchronization scene |
| CN115134265A (en) * | 2022-05-16 | 2022-09-30 | 北京璇星科技有限公司 | Real-time monitoring and early warning method, device, equipment and storage medium for process |
| CN116166701A (en) * | 2023-03-17 | 2023-05-26 | 湖北坤盈数字科技有限公司 | Service data real-time early warning method, device, equipment and storage medium |
| CN116166701B (en) * | 2023-03-17 | 2023-07-25 | 湖北坤盈数字科技有限公司 | Service data real-time early warning method, device, equipment and storage medium |
| CN116629805A (en) * | 2023-06-07 | 2023-08-22 | 浪潮智慧科技有限公司 | Water conservancy index service method, equipment and medium for distributed flow batch integration |
| CN116629805B (en) * | 2023-06-07 | 2023-12-01 | 浪潮智慧科技有限公司 | Water conservancy index service method, equipment and medium for distributed flow batch integration |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11386127B1 (en) | Low-latency streaming analytics | |
| US11645471B1 (en) | Determining a relationship recommendation for a natural language request | |
| CN107577805B (en) | Business service system for log big data analysis | |
| CN113420043A (en) | Data real-time monitoring method, device, equipment and storage medium | |
| CN107908690B (en) | Data processing method based on big data operation analysis | |
| US10565196B2 (en) | Determining a user-specific approach for disambiguation based on an interaction recommendation machine learning model | |
| US11288319B1 (en) | Generating trending natural language request recommendations | |
| CN102902775B (en) | The method and system that internet calculates in real time | |
| CN109074384A (en) | To context metadata ranking to generate related data opinion | |
| US11494395B2 (en) | Creating dashboards for viewing data in a data storage system based on natural language requests | |
| US9965772B2 (en) | System and method for unifying user-level data across different media platforms | |
| US10901811B2 (en) | Creating alerts associated with a data storage system based on natural language requests | |
| CN111125518B (en) | Household appliance information recommendation system and method | |
| CN111881221A (en) | Method, device and equipment for customer portrait in logistics service | |
| US20190034430A1 (en) | Disambiguating a natural language request based on a disambiguation recommendation machine learning model | |
| CN112396462B (en) | Crowd circling method and device based on click house | |
| CN111429241A (en) | Accounting processing method and device | |
| CN110866040A (en) | User portrait generation method, device and system | |
| CN110675238A (en) | Client label configuration method, system, readable storage medium and electronic equipment | |
| CN113505265A (en) | Data query method and device, electronic equipment, storage medium and program product | |
| CN112181678A (en) | Service data processing method, device and system, storage medium and electronic device | |
| Maske et al. | A real time processing and streaming of wireless network data using storm | |
| CN107679097A (en) | A kind of distributed data processing method, system and storage medium | |
| CN116249079A (en) | User return visit method and system based on 5G message | |
| CN111695035B (en) | Recommendation system and multi-algorithm fusion recommendation processing flow |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |