CN114741404A - Early warning information aggregation system and method and electronic equipment - Google Patents
Early warning information aggregation system and method and electronic equipment Download PDFInfo
- Publication number
- CN114741404A CN114741404A CN202111548769.3A CN202111548769A CN114741404A CN 114741404 A CN114741404 A CN 114741404A CN 202111548769 A CN202111548769 A CN 202111548769A CN 114741404 A CN114741404 A CN 114741404A
- Authority
- CN
- China
- Prior art keywords
- early warning
- aggregation
- warning information
- module
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2433—Query languages
- G06F16/244—Grouping and aggregation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2453—Query optimisation
- G06F16/24534—Query rewriting; Transformation
- G06F16/24539—Query rewriting; Transformation using cached or materialised query results
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D30/00—Reducing energy consumption in communication networks
- Y02D30/50—Reducing energy consumption in communication networks in wire-line communication networks, e.g. low power modes or reduced link rate
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides an early warning information aggregation system, a method, electronic equipment and a storage medium, wherein the system comprises a message queue component, a data acquisition module, an early warning calculation module and an early warning aggregation module, wherein the data acquisition module is used for acquiring object data of a monitored object and sending the object data to the message queue component; the early warning calculation module is used for acquiring object data from the message queue component, comparing the object data based on the early warning rule, generating early warning information according to the comparison result and sending the early warning information to the message queue component; the early warning aggregation module is used for acquiring early warning information from the message queue component; aggregating the early warning information to obtain aggregate cache data; judging whether the aggregation cache state meets a preset aggregation ending condition or not; and if so, taking the aggregated cache data obtained by aggregation as final effective early warning information. By applying the embodiment of the invention, the aggregation optimization of the early warning information is realized.
Description
Technical Field
The invention relates to the technical field of big data analysis, in particular to an early warning information aggregation system, an early warning information aggregation method, electronic equipment and a storage medium.
Background
With the popularization of big data technology, more and more data are generated in various industries, and the problems of real-time performance and effectiveness of the data are more and more prominent. For example, various internet data such as face snapshot, vehicle snapshot, civil aviation, railway, passenger transport, accommodation, social security and the like have huge data volume, so that monitoring and early warning for specific personnel from mass data are very important, and the effectiveness problem of real-time early warning is more prominent. Therefore, a method for performing early warning information aggregation is needed to increase data coupling.
Disclosure of Invention
The invention aims to overcome the defects of the prior art, provides an early warning information aggregation system, a method and a device, electronic equipment and a storage medium, and aims to solve the technical problems of analysis of mass early warning information and the like.
The invention is realized by the following steps:
in a first aspect, an embodiment of the present application provides an early warning information aggregation system, where the system includes a message queue component, a data acquisition module, an early warning computation module, and an early warning aggregation module, where the early warning computation module is configured to compute a message queue of the message queue component and the data acquisition module
The data acquisition module is used for acquiring object data of a monitored object; sending the collected object data to the message queue component;
the early warning calculation module is used for acquiring the object data from the message queue component, comparing the object data based on a preset early warning rule and generating early warning information according to a comparison result; sending the generated slave warning information to the message queue component;
the early warning aggregation module is used for acquiring early warning information from the message queue component; aggregating the acquired early warning information to obtain aggregated cache data; determining an aggregation cache state, and judging whether the aggregation cache state meets a preset aggregation ending condition; and if so, taking the aggregated cache data obtained by aggregation as final effective early warning information.
Optionally, the early warning aggregation module is further configured to store the obtained effective early warning information in a database;
and/or the presence of a gas in the atmosphere,
the early warning aggregation module is further configured to judge whether the early warning aggregation module meets a preset cache cleaning condition after obtaining effective early warning information, and if so, cleaning aggregated cache data cached by the early warning aggregation module.
Optionally, the early warning aggregation module aggregates the acquired early warning information to obtain aggregated cache data specifically:
judging whether target early warning information with consistent source and consistent monitored object exists in the cached early warning information aiming at the currently acquired early warning information; and if so, aggregating the currently acquired early warning information and the target early warning information to obtain aggregated cache data.
Optionally, the determining, by the early warning aggregation module, an aggregation cache state specifically includes:
determining the acquisition time of the currently acquired early warning information and the aggregation time period to which the currently acquired early warning information belongs; taking the determined acquisition time and the aggregation time period as an aggregation cache state;
the early warning aggregation module judges whether the aggregation cache state meets a preset aggregation ending condition, and specifically comprises the following steps:
if the acquisition time of the currently acquired early warning information reaches the determined end time of the aggregation time period, determining that the aggregation cache state meets a preset aggregation ending condition;
and if the acquisition time of the currently acquired early warning information does not reach the end time of the determined aggregation time period, determining that the aggregation cache state does not meet the preset aggregation ending condition.
Optionally, the early warning aggregation module aggregates the acquired early warning information to obtain aggregated cache data specifically:
aiming at the currently acquired early warning information, determining a target early warning rule met by the early warning information;
and aggregating all the early warning information meeting the target early warning rule to obtain aggregated cache data.
Optionally, the determining, by the early warning aggregation module, the aggregation cache state specifically includes:
performing integral accumulation calculation on the aggregation process based on a preset distributed calculation strategy to obtain an integral accumulation value; taking the obtained integral accumulated value as an aggregation buffer state;
the early warning aggregation module judges whether the aggregation cache state meets a preset aggregation ending condition, and specifically comprises the following steps:
if the integral accumulated value reaches a preset integral threshold value, determining that the aggregation cache state meets a preset aggregation ending condition;
and if the integral accumulated value does not reach a preset integral threshold value, determining that the aggregation cache state does not meet a preset aggregation ending condition.
In a second aspect, an embodiment of the present application provides an early warning information aggregation method, which is applied to an early warning aggregation module, and the method includes:
acquiring early warning information from a message queue component;
aggregating the acquired early warning information to obtain aggregated cache data;
determining an aggregation cache state, and judging whether the aggregation cache state meets a preset aggregation ending condition; and if so, taking the aggregated cache data obtained by aggregation as final effective early warning information.
Optionally, the method further includes:
storing the obtained effective early warning information to a database;
and/or the presence of a gas in the gas,
after effective early warning information is obtained, whether the early warning aggregation module meets preset cache cleaning conditions or not is judged, and if yes, aggregated cache data cached by the early warning aggregation module is cleaned.
In a third aspect, an embodiment of the present application provides an electronic device, which includes the foregoing data analysis terminal, a memory and a processor;
wherein the memory is connected with the processor and used for storing programs;
the processor is used for realizing the steps of the early warning information aggregation method applied to the early warning aggregation module by operating the program stored in the memory.
In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program, where the computer program causes a computer to execute the foregoing steps of an early warning information aggregation method applied to an early warning aggregation module.
The invention has the following beneficial effects: the embodiment of the application provides an early warning information aggregation system, a method, electronic equipment and a storage medium, wherein the system comprises a message queue component, a data acquisition module, an early warning calculation module and an early warning aggregation module, wherein the data acquisition module is used for acquiring object data of a monitored object; sending the collected object data to a message queue component; the early warning calculation module is used for acquiring object data from the message queue component, comparing the object data based on a preset early warning rule and generating early warning information according to a comparison result; sending the generated slave warning information to a message queue component; the early warning aggregation module is used for acquiring early warning information from the message queue component; aggregating the acquired early warning information to obtain aggregated cache data; determining an aggregation cache state, and judging whether the aggregation cache state meets a preset aggregation ending condition; and if so, taking the aggregated cache data obtained by aggregation as final effective early warning information. The message queue component can be used for being connected with various data sources in a butt joint mode, so that the early warning calculation module, the early warning aggregation module and the data sources can be loosely coupled, the early warning information is obtained from the message queue component, the early warning information is further integrated, and the aggregation optimization of the early warning information is achieved.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure of the embodiments of the application.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a schematic structural diagram of an early warning information aggregation system according to an embodiment of the present invention;
fig. 2 is a schematic flow chart of an early warning information aggregation method according to an embodiment of the present invention;
fig. 3 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The flow diagrams depicted in the figures are merely illustrative and do not necessarily include all of the elements and operations/steps, nor do they necessarily have to be performed in the order depicted. For example, some operations/steps may be decomposed, combined or partially combined, so that the actual execution order may be changed according to the actual situation.
Some embodiments of the present application will be described in detail below with reference to the accompanying drawings. The embodiments described below and the features of the embodiments can be combined with each other without conflict.
Referring to fig. 1, fig. 1 is a schematic flow chart of an early warning information aggregation system according to an embodiment of the present disclosure. The system comprises a message queue component 101, a data acquisition module 102, an early warning calculation module 103, and an early warning aggregation module 104, wherein,
the data acquisition module is used for acquiring object data of a monitored object; sending the collected object data to the message queue component;
the early warning calculation module is used for acquiring the object data from the message queue component, comparing the object data based on a preset early warning rule and generating early warning information according to a comparison result; sending the generated slave warning information to the message queue component;
the early warning aggregation module is used for acquiring early warning information from the message queue component; aggregating the acquired early warning information to obtain aggregated cache data; determining an aggregation cache state, and judging whether the aggregation cache state meets a preset aggregation ending condition; and if so, taking the aggregated cache data obtained by aggregation as final effective early warning information.
The data acquisition module can be equipment with data acquisition function, such as a camera, or other data capturing tools, and equipment with data acquisition function such as face snapshot, vehicle snapshot data, network data, and the like. After the device collects data, the data can be directly pushed to the message queue component in real time. For example, the data acquisition module can acquire data of civil aviation, railway, passenger transport, accommodation, social security, images and the like and push the data to the message queue component in a timed or real-time mode in an incremental mode.
The monitoring object may be an object to be analyzed for early warning, for example, a person, a vehicle, or the like to be analyzed for monitoring. The object data of the monitored object may be data related to the monitored object, for example, if it is determined that the monitored object or the monitored object is a person, the object data of the monitored object may be an identification number of the monitored object, the monitored object or the monitored object is a vehicle, and the object data of the monitored object may be a vehicle number plate; if the monitored object is definitely in the activity range, the object data of the monitored object can be departure destination stations or departure related data of civil aviation, railways, passenger transport and the like; if the monitored object is definitely the snapshot device, for example, the face snapshot device in a certain cell or the face snapshot device lamp in a hospital, the data captured by the snapshot device associated with the school or the hospital may be used as the object data. In addition, the vehicle snapshot range can be used for framing the snapshot equipment on the map according to actual conditions, and data generated by the snapshot equipment can be provided with identity information (identity card numbers or license plate numbers).
In addition, monitoring objects and characteristics can be determined based on data and service requirements, early warning calculation modes of monitoring objects are analyzed and judged, monitoring rules are formed, the formed monitoring rules are cached in a database (such as Redis) and the like, and the monitoring rules can be conveniently modified and cancelled at any time.
The message queue component may be a Kafka service component, which is a high throughput low latency message middleware. Persisting data by writing data to the operating system's page cache, lets the operating system itself decide when to synchronize data to disk. The page buffer of the operating system is distributed in the memory, so the message writing speed is very high, the Kafka does not need to directly interact with the underlying file system, and the Kafka writing operation adopts an additional mode, thereby avoiding the random writing operation of the disk and simultaneously supporting load balancing and failover. The invention can utilize the characteristics of high throughput, low delay, durability, load balance, fault transfer and the like of Kafka, can stably monitor the monitored object with low delay, and has wider application range.
In practical situations, the data sources may be various, the data sources to be docked include storage tools for storing all related object data of the monitored object, and the data sources to be docked can be in a Kafka unified pushing mode, so that loose coupling and timeliness of the early warning calculation module and data pushing are achieved. For example, the message queue component Kafka may interface multiple data sources, such as Mysql, Oracle, PostGreSql, and so on. After the data acquisition module acquires the object data, the object data can be stored in the data sources, specifically, the data sources can be self-contained databases or independent third-party databases, and then the object data can be published to the message queue component through the data sources periodically or in real time.
The early warning calculation module can be a server with a data analysis processing function, and the early warning aggregation module can be a server capable of performing data aggregation analysis. The message queue component can be communicated with the data acquisition module, the early warning calculation module and the early warning aggregation module. It should be noted that the message queue component, the early warning calculation module, and the early warning aggregation module may be all an independent server, or may be a server in a server cluster formed by a plurality of servers, which is not limited in this embodiment of the present application.
Specifically, the early warning calculation module and the early warning aggregation module may be software function modules integrated in a certain server, or may be separate hardware, for example, electronic devices such as a mobile phone, a tablet computer, a notebook computer, a desktop computer, a personal digital assistant, and a server, but are not limited thereto. In addition, the data acquisition module, the early warning calculation module and the early warning aggregation module can be devices in the same local area network with the message queue component, or can be devices in different local area networks.
The early warning computation module may include a computation engine, for example, hosted by a Flink streaming computation engine employing Yarn. And sequentially consuming the data in the Kafka in real time through a calculation engine, performing collision comparison according to a preset early warning rule to generate early warning information, and outputting the early warning information to a Kafka output queue. The early warning rule can comprise key features, and when the key features in the object data are matched with the key features in the early warning rule, the comparison can be regarded as successful, and then an early warning message is generated; if the key features in the object data are not matched with any key feature in the early warning rule, the comparison between the key features and the early warning rule is not successful, and new object data can be continuously obtained for comparison until the comparison ending condition is met. The condition for ending the alignment may be the number of times of reaching the alignment or the time of reaching the alignment, and the like, which is not limited in the present invention.
The early warning rules can be set in advance according to requirements, and can be stored in a database or an early warning calculation module after being set, and the early warning calculation module can acquire the early warning rules by accessing the database or calling the data stored in the early warning calculation module. Or the early warning rule can be generated by the early warning calculation module based on a SQL (structured query language) or regular rule and other rule generation modules according to the actual business scene logic. The invention does not limit the concrete form and content of the early warning rule. For example, the early warning rule generation module mainly generates an early warning rule according to the business requirement: the personnel monitoring and early warning rule mainly comprises: the early warning rule id, the early warning object identification number, the monitoring start time, the monitoring end time and other fields can take the identification number as the key field characteristics; when the object data hits the identity card number in the early warning rule, the comparison is considered to be successful. For another example, the vehicle monitoring and warning rules mainly include: and the fields of the early warning rule id, the early warning object license plate number, the monitoring start time, the monitoring end time and the like can be used for identifying the key field characteristics of the license plate number, and when the object data hits the license plate number in the early warning rule, the comparison can be considered to be successful.
In another implementation manner, the early warning calculation module may compare the data and the early warning rule from Kafka in real time on the flink, compare the time range in the monitoring rule with other early warning generation conditions that need to be judged after hit, generate early warning information and push the early warning information to a Kafka early warning queue after the comparison is successful. By applying the embodiment of the invention, the comprehensiveness of comparison is improved.
The early warning aggregation module can acquire early warning information from the message queue component and aggregate the early warning information to obtain aggregated cache data. It is understood that the aggregated cached data may be data that is aggregated and then cached locally, which may provide data processing efficiency.
The specific aggregation manner is not limited in the present invention, for example, in an implementation manner, the early warning aggregation module aggregates the acquired early warning information to obtain the aggregation cache data specifically:
judging whether target early warning information with consistent source and consistent monitored object exists in the cached early warning information aiming at the currently acquired early warning information; and if so, aggregating the currently acquired early warning information and the target early warning information to obtain aggregated cache data.
It can be understood that the scheme can be executed for multiple rounds, and can continuously acquire the early warning information, the current early warning information is the early warning information acquired in the execution process of the scheme in the round, the next scheme is the processing of the early warning information, and it can be considered that before the early warning information is acquired, if the scheme is executed, the cached early warning information may exist, and then the scheme can be compared with the cached early warning information to determine whether the target early warning information with the consistent source and the consistent monitored object exists in the cached early warning information; if yes, aggregating the currently acquired early warning information and the target early warning information to obtain aggregated cache data; if not, the current early warning information is directly cached, and the step of obtaining the early warning information is executed again. In addition, if the scheme is executed for the first time, it may be considered that there is no cached warning information. If the early warning information is not cached, the early warning information can be directly cached, and the step of obtaining the early warning information is executed again. The above-described method can be considered as a time polymerization method.
The source consistency can be regarded as consistency of data sources of the monitored objects associated with the early warning information, for example, the data sources can be object data captured by the same camera or the same network card. The monitored objects may be considered the same object, such as a person or a vehicle.
If the time aggregation mode is adopted by the early warning aggregation module, the determination of the aggregation cache state by the early warning aggregation module may specifically be: determining the acquisition time of the currently acquired early warning information and the aggregation time period to which the currently acquired early warning information belongs; taking the determined acquisition time and the aggregation time period as an aggregation cache state;
the early warning aggregation module judges whether the aggregation cache state meets a preset aggregation ending condition, and specifically comprises the following steps:
if the acquisition time of the currently acquired early warning information reaches the determined end time of the aggregation time period, determining that the aggregation cache state meets a preset aggregation ending condition;
and if the acquisition time of the currently acquired early warning information does not reach the determined end time of the aggregation time period, determining that the aggregation cache state does not meet the preset aggregation end condition.
The acquisition time of the early warning information may be the acquisition time of the object data of the monitored object associated with the early warning information, or the acquisition time of the early warning information may be the generation time.
The time may be divided into a plurality of time periods in chronological order, for example, the time periods may be divided in units of days or months or hours. And then after the acquisition time of the early warning information is determined, the time period of the acquisition time can be determined to be used as the aggregation time period of the currently acquired early warning information.
When the early warning information in a certain time period is generated for the first time, for example, the early warning aggregation module obtains the first early warning information in the time period, and for the first early warning information, the early warning aggregation module can perform caching so as to generate cached early warning information, and further, when the same monitoring object appears under the same monitoring device, the early warning aggregation module can perform data aggregation with the early warning information cached at present so as to form aggregated cached data. For example, the specific record may be track data or activity data.
After the aggregation cache data are obtained, the early warning aggregation module can determine the aggregation cache state, and further determine whether the cache data obtained by aggregation are used as final effective early warning information according to whether the aggregation cache state meets the preset aggregation ending condition. The preset aggregation ending condition may be set in advance according to a requirement, for example, it may be considered that the preset aggregation ending condition is not satisfied if the acquisition time of the currently acquired early warning information does not reach the end time of the determined aggregation time period; if the preset condition for finishing the aggregation is met, the condition for finishing the aggregation can be determined to be met, and then the cache data obtained by aggregation can be used as final effective early warning information. Otherwise, the early warning information can be obtained again for aggregation.
Further, in order to provide the utilization rate of system resources, the early warning aggregation module is also used for storing the obtained effective early warning information to a database; and/or the presence of a gas in the gas,
and the early warning aggregation module is further used for judging whether the early warning aggregation module meets preset cache cleaning conditions or not after effective early warning information is obtained, and cleaning aggregated cache data cached by the early warning aggregation module if the early warning aggregation module meets the preset cache cleaning conditions.
The obtained effective early warning information is stored in the database, so that subsequent business query can be facilitated, and a quick way is provided for data access and storage.
In order to avoid the waste of unnecessary resources and improve the data processing efficiency, if the time aggregation mode is adopted, the early warning aggregation module can be considered to meet the preset cache cleaning condition after determining that the acquisition time of the early warning information reaches the end time point of the aggregation time period to which the early warning information belongs, and then the aggregation cache data of the aggregation time period can be cleaned, so that the preparation can be made for the aggregation of the data of the next time period. (ii) a Otherwise, the preset cache cleaning condition is not met, and the cache data is not cleaned. Or in other manners, the preset cache cleaning condition may also be that the current time of the preset cache cleaning condition reaches a preset time point set in advance, or that the current memory state value of the early warning aggregation module reaches a preset memory state value, and the like, which is not limited in the present invention.
In another implementation manner, the early warning aggregation module aggregates the acquired early warning information to obtain aggregated cache data, which may be:
aiming at the currently acquired early warning information, determining a target early warning rule met by the early warning information;
and aggregating all the early warning information meeting the target early warning rule to obtain aggregated cache data.
One or more pre-warning rules can be provided, and the pre-warning rule which can be compared successfully with the currently acquired pre-warning information can be determined as the target pre-warning rule. Furthermore, all the early warning information under the target early warning rule can be subjected to data aggregation, and aggregation cache data are obtained. The present invention does not limit the specific data aggregation mode, and may be any tool capable of implementing the data aggregation function.
Specifically, the determining, by the early warning aggregation module, the aggregation cache state may specifically be:
performing integral accumulation calculation on the polymerization process based on a preset distributed calculation strategy to obtain an integral accumulation value; taking the obtained integral accumulated value as an aggregation buffer state;
the early warning aggregation module judges whether the aggregation cache state meets a preset aggregation ending condition, and specifically may be: if the integral accumulated value reaches a preset integral threshold value, determining that the aggregation cache state meets a preset aggregation ending condition; and if the integral accumulated value does not reach a preset integral threshold value, determining that the aggregation cache state does not meet a preset aggregation ending condition.
If the aggregation cache state meets the preset aggregation ending condition, the aggregation cache data obtained by aggregation can be used as final effective early warning information; if not, the early warning information can be obtained again, and the aggregation step is executed.
The distributed computing strategy can be MapReduce, which is a computing model that can decompose a large data processing task into many single tasks that can be executed in parallel in a server cluster, and the computing results of the tasks can be combined together to compute a final result. MapReduce can be divided into Map operation and Reduce operation. Map operations will convert elements in a set from one form to another, in which case the input key-value pairs will be converted to zero to multiple key-value pair outputs. Where the input and output keys must be completely different and the input and output values may be completely different. All key-value pairs for a key are distributed to the same Reduce operation. Rather, the key and all values corresponding to the key are passed to the same Reducer. The purpose of the Reduce process is to convert a set of values into one value (e.g., summing or averaging), or into another set. This Reducer will eventually also produce a key-value pair. Thereby improving data processing efficiency.
The aggregate cache state may include an accumulated value of credits. The integral accumulated value can reflect the effectiveness of the aggregation of the early warning information, the higher the integral accumulated value is, the stronger the aggregation effectiveness of the early warning information can be shown, but when the integral accumulated value reaches a certain threshold value, the integral accumulated value can be considered to be saturated, no further aggregation is needed, and the aggregation cache data obtained by current aggregation is used as the final effective early warning information.
For each monitoring task, a monitoring rule can be set in advance according to requirements, the monitoring rule can include a unique identifier, a time range and the like of a monitored object, and an integral threshold value can be included. According to the time range and the integral threshold of the monitoring rule, after the early warning information is obtained, aggregate cache data is generated according to the target early warning rule, integral accumulation calculation is carried out on the data aggregation process according to the MapReduce idea, and the final effective early warning information can be generated only after the integral threshold is reached. The above method can be considered as an integral polymerization method.
In yet another implementation, the aggregation mode may be divided into a time aggregation mode and an integral aggregation mode according to the aggregation dimension. The early warning aggregation module can receive a user instruction, and selects a dimensionality according to the user instruction to aggregate early warning information and store the early warning information in the database.
In addition, the database may be a part of the early warning information aggregation system, or may be a database independent of the early warning information aggregation system. The database can obtain and store data to be stored sent by other modules, and establishes index information and a data model based on the obtained data to be stored, so that query is facilitated, storage and access speed is improved, and centralized storage and access of the data are realized.
The number of databases may be one or more, each database may include the same or different types of data, and so on, and the invention is not limited to the specific number and types of databases.
By applying the embodiment of the invention, the message queue component can be utilized to be connected with various data sources, so that the early warning calculation module and the data sources can be loosely coupled. And then, according to the actual service requirements, determining a monitoring object and a monitoring rule, and caching the monitoring rule into Redis, so that the monitoring object and the monitoring rule can be conveniently changed and cancelled at any time. And then, the early warning calculation module can acquire object data from the message queue component and perform real-time collision early warning calculation with the early warning rule, the formed early warning information can be output to the message queue component, and the early warning information is further integrated through the early warning aggregation module, so that the aggregation optimization of the early warning information is realized. It can be seen that the whole process data flow has strong real-time performance, is loosely coupled and configurable, and realizes loose coupling, real-time performance and data aggregation.
Corresponding to the embodiment of the personnel management and control system, an embodiment of the present invention provides an early warning information aggregation method, please refer to fig. 2, and fig. 2 is a schematic flow diagram of an early warning information aggregation method according to another embodiment of the present application, where the early warning information aggregation method is applied to the early warning aggregation module.
As shown in fig. 2, the aggregation of the warning information in this embodiment includes steps S210 to S240.
S210, acquiring early warning information from the message queue component;
s220, aggregating the acquired early warning information to obtain aggregated cache data;
s230, determining an aggregation cache state, and judging whether the aggregation cache state meets a preset aggregation ending condition; if yes, executing S240;
and S240, taking the aggregated cache data obtained by aggregation as final effective early warning information.
Therefore, by applying the technical scheme provided by the embodiment of the invention, the early warning information is acquired from the message queue component, and is further integrated, so that the aggregation optimization of the early warning information is realized. And because the message queue component can be connected with different data sources in a butt joint mode, the early warning information is obtained through the message queue component, and loose coupling of the early warning aggregation module and the data sources is achieved.
Optionally, the method further includes:
storing the obtained effective early warning information to a database;
and/or the presence of a gas in the atmosphere,
and after the effective early warning information is obtained, judging whether the early warning aggregation module meets the preset cache cleaning condition, and if so, cleaning the aggregation cache data cached by the early warning aggregation module.
Optionally, aggregating the acquired early warning information to obtain aggregated cache data includes:
judging whether target early warning information with consistent source and consistent monitored object exists in the cached early warning information aiming at the currently acquired early warning information; and if so, aggregating the currently acquired early warning information and the target early warning information to obtain aggregated cache data.
Optionally, determining the aggregation cache state includes:
determining the acquisition time of the currently acquired early warning information and the aggregation time period to which the currently acquired early warning information belongs; taking the determined acquisition time and the aggregation time period as an aggregation cache state;
the early warning aggregation module judges whether the aggregation cache state meets a preset aggregation ending condition, and specifically comprises the following steps:
if the acquisition time of the currently acquired early warning information reaches the determined end time of the aggregation time period, determining that the aggregation cache state meets a preset aggregation ending condition;
and if the acquisition time of the currently acquired early warning information does not reach the end time of the determined aggregation time period, determining that the aggregation cache state does not meet a preset aggregation end condition.
The early warning aggregation module aggregates the acquired early warning information to obtain aggregated cache data specifically as follows:
aiming at the currently acquired early warning information, determining a target early warning rule met by the early warning information;
and aggregating all the early warning information meeting the target early warning rule to obtain aggregated cache data.
Optionally, determining the aggregation cache state includes:
performing integral accumulation calculation on the aggregation process based on a preset distributed calculation strategy to obtain an integral accumulation value; taking the obtained integral accumulated value as an aggregation buffer state;
the early warning aggregation module judges whether the aggregation cache state meets a preset aggregation ending condition, and specifically comprises the following steps:
if the integral accumulated value reaches a preset integral threshold value, determining that the aggregation cache state meets a preset aggregation ending condition;
and if the integral accumulated value does not reach a preset integral threshold value, determining that the aggregation cache state does not meet a preset aggregation ending condition.
All the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on differences from other embodiments. In particular, as for the method embodiment, since it is substantially similar to the system embodiment, the description is simple, and the relevant points can be referred to the partial description of the system embodiment.
An embodiment of the present application further provides an electronic device, please refer to fig. 3, which shows a schematic structural diagram of the electronic device, and the electronic device may include: at least one processor 301, at least one communication interface 302, at least one memory 303 and at least one communication bus 304;
in the embodiment of the present application, the number of the processor 301, the communication interface 302, the memory 303 and the communication bus 304 is at least one, and the processor 301, the communication interface 302 and the memory 303 complete communication with each other through the communication bus 304;
the processor 301 may be a central processing unit CPU, or an application Specific Integrated circuit asic, or one or more Integrated circuits configured to implement embodiments of the present invention, or the like;
the memory 303 may include a high-speed RAM memory, and may further include a non-volatile memory (non-volatile memory) or the like, such as at least one disk memory;
the memory stores programs, the processor can call the programs stored in the memory, and the programs are used for the early warning information aggregation method and comprise the following steps: acquiring early warning information from a message queue component; aggregating the acquired early warning information to obtain aggregated cache data; determining an aggregation cache state, and judging whether the aggregation cache state meets a preset aggregation ending condition; and if so, taking the aggregated cache data obtained by aggregation as final effective early warning information.
By applying the embodiment of the invention and the technical scheme provided by the embodiment of the invention, the early warning information is acquired from the message queue component and is further integrated, so that the aggregation optimization of the early warning information is realized. And because the message queue component can be connected with different data sources in a butt mode, the early warning information is obtained through the message queue component, and loose coupling of the early warning aggregation module and the data sources is achieved.
Alternatively, the detailed function and the extended function of the program may be as described above.
An embodiment of the present application further provides a computer-readable storage medium, where a computer program is stored, and when the computer program is executed by a processor, the processor is enabled to implement the steps of the method for aggregating early warning information applied to the early warning aggregation module provided in the foregoing embodiment.
The computer-readable storage medium may be an internal storage unit of the data analysis terminal according to any of the foregoing embodiments, for example, a hard disk or a memory of the data analysis terminal. The computer readable storage medium may also be an external storage device of the data analysis terminal, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like, provided on the data analysis terminal.
Alternatively, the detailed function and the extended function of the program may be as described above.
It should be understood that the term "and/or" as used in this application and the appended claims refers to any and all possible combinations of one or more of the associated listed items and includes such combinations.
Finally, it should also be noted that, in this document, relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
In the present specification, the embodiments are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (10)
1. The early warning information aggregation system is characterized by comprising a message queue component, a data acquisition module, an early warning calculation module and an early warning aggregation module, wherein the early warning calculation module and the early warning aggregation module are connected with each other through a network
The data acquisition module is used for acquiring object data of a monitored object; sending the collected object data to the message queue component;
the early warning calculation module is used for acquiring the object data from the message queue component, comparing the object data based on a preset early warning rule and generating early warning information according to a comparison result; sending the generated slave warning information to the message queue component;
the early warning aggregation module is used for acquiring early warning information from the message queue component; aggregating the acquired early warning information to obtain aggregated cache data; determining an aggregation cache state, and judging whether the aggregation cache state meets a preset aggregation ending condition; and if so, taking the aggregated cache data obtained by aggregation as final effective early warning information.
2. The early warning information aggregation system of claim 1, wherein the early warning aggregation module is further configured to store the obtained valid early warning information in a database;
and/or the presence of a gas in the atmosphere,
the early warning aggregation module is further configured to judge whether the early warning aggregation module meets a preset cache cleaning condition after obtaining effective early warning information, and if so, cleaning aggregated cache data cached by the early warning aggregation module.
3. The early warning information aggregation system according to claim 1, wherein the early warning aggregation module aggregates the acquired early warning information to obtain aggregated cache data specifically:
judging whether target early warning information with consistent source and consistent monitored object exists in the cached early warning information aiming at the currently acquired early warning information; and if so, aggregating the currently acquired early warning information and the target early warning information to obtain aggregated cache data.
4. The early warning information aggregation system according to claim 3, wherein the early warning aggregation module determines the aggregation cache state specifically as:
determining the acquisition time of the currently acquired early warning information and the aggregation time period of the currently acquired early warning information; taking the determined acquisition time and the aggregation time period as an aggregation cache state;
the early warning aggregation module judges whether the aggregation cache state meets a preset aggregation ending condition, and specifically comprises the following steps:
if the acquisition time of the currently acquired early warning information reaches the determined end time of the aggregation time period, determining that the aggregation cache state meets a preset aggregation ending condition;
and if the acquisition time of the currently acquired early warning information does not reach the end time of the determined aggregation time period, determining that the aggregation cache state does not meet the preset aggregation ending condition.
5. The early warning information aggregation system according to claim 1, wherein the early warning aggregation module aggregates the acquired early warning information to obtain aggregated cache data specifically:
aiming at the currently acquired early warning information, determining a target early warning rule met by the early warning information;
and aggregating all the early warning information meeting the target early warning rule to obtain aggregated cache data.
6. The early warning information aggregation system according to claim 5, wherein the early warning aggregation module determines the aggregation cache state specifically as:
performing integral accumulation calculation on the aggregation process based on a preset distributed calculation strategy to obtain an integral accumulation value; taking the obtained integral accumulated value as an aggregation buffer state;
the early warning aggregation module judges whether the aggregation cache state meets a preset aggregation ending condition, and specifically comprises the following steps:
if the integral accumulated value reaches a preset integral threshold value, determining that the aggregation cache state meets a preset aggregation ending condition;
and if the integral accumulated value does not reach a preset integral threshold value, determining that the aggregation cache state does not meet a preset aggregation ending condition.
7. An early warning information aggregation method is applied to an early warning aggregation module, and comprises the following steps:
acquiring early warning information from a message queue component;
aggregating the acquired early warning information to obtain aggregated cache data;
determining an aggregation cache state, and judging whether the aggregation cache state meets a preset aggregation ending condition; and if so, taking the aggregated cache data obtained by aggregation as final effective early warning information.
8. The method of claim 6, further comprising:
storing the obtained effective early warning information to a database;
and/or the presence of a gas in the gas,
and after the effective early warning information is obtained, judging whether the early warning aggregation module meets the preset cache cleaning condition, and if so, cleaning the aggregation cache data cached by the early warning aggregation module.
9. An electronic device comprising a memory and a processor;
wherein the memory is connected with the processor and used for storing programs;
the processor is configured to implement the steps of the warning information aggregation method according to any one of claims 7 to 8 by executing a program stored in the memory.
10. A computer-readable storage medium storing a computer program, wherein the computer program causes a computer to execute the warning information aggregation method according to any one of claims 7 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111548769.3A CN114741404A (en) | 2021-12-17 | 2021-12-17 | Early warning information aggregation system and method and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111548769.3A CN114741404A (en) | 2021-12-17 | 2021-12-17 | Early warning information aggregation system and method and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114741404A true CN114741404A (en) | 2022-07-12 |
Family
ID=82275655
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111548769.3A Pending CN114741404A (en) | 2021-12-17 | 2021-12-17 | Early warning information aggregation system and method and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114741404A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115373361A (en) * | 2022-10-24 | 2022-11-22 | 江苏智云天工科技有限公司 | Factory production safety early warning method and system based on industrial Internet |
-
2021
- 2021-12-17 CN CN202111548769.3A patent/CN114741404A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115373361A (en) * | 2022-10-24 | 2022-11-22 | 江苏智云天工科技有限公司 | Factory production safety early warning method and system based on industrial Internet |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109558450B (en) | Automobile remote monitoring method and device based on distributed architecture | |
| CN104969213B (en) | Data flow for low latency data access is split | |
| CN111950988B (en) | Distributed workflow scheduling method and device, storage medium and electronic equipment | |
| CN104125163B (en) | Data processing method and device and terminal | |
| CN111309550A (en) | Data acquisition method, system, equipment and storage medium of application program | |
| CN104021194A (en) | Mixed type processing system and method oriented to industry big data diversity application | |
| CN110147470B (en) | Cross-machine-room data comparison system and method | |
| CN112231296B (en) | Distributed log processing method, device, system, equipment and medium | |
| CN110019267A (en) | A kind of metadata updates method, apparatus, system, electronic equipment and storage medium | |
| US11188443B2 (en) | Method, apparatus and system for processing log data | |
| CN111522786A (en) | Log processing system and method | |
| CN113676400B (en) | Cloud gateway system for Internet of things access of environment-friendly data acquisition equipment | |
| US8959229B1 (en) | Intelligently provisioning cloud information services | |
| CN111400288A (en) | Data quality inspection method and system | |
| CN114741404A (en) | Early warning information aggregation system and method and electronic equipment | |
| CN114265904A (en) | Data processing method and cloud computing platform | |
| CN113377777B (en) | Data loading method, device, computer program product and storage medium | |
| CN113420323B (en) | Data sharing method and terminal equipment | |
| CN105162837A (en) | Method and system for improving I/O throughput rate in massive data storage environment | |
| CN113656369A (en) | Log distributed streaming acquisition and calculation method in big data scene | |
| CN109299089A (en) | The calculating and storage method and calculating of a kind of label data of drawing a portrait and storage system | |
| CN111209263A (en) | Data storage method, device, equipment and storage medium | |
| CN115858672A (en) | Power terminal management method and device, electronic equipment and storage medium | |
| CN113010852B (en) | Data metering and counting method and device, electronic equipment and storage medium | |
| CN115664992A (en) | Network operation data processing method and device, electronic equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |