CN113781101A - Event center management method and device based on real-time calculation and readable medium - Google Patents

Event center management method and device based on real-time calculation and readable medium Download PDF

Info

Publication number
CN113781101A
CN113781101A CN202110910281.4A CN202110910281A CN113781101A CN 113781101 A CN113781101 A CN 113781101A CN 202110910281 A CN202110910281 A CN 202110910281A CN 113781101 A CN113781101 A CN 113781101A
Authority
CN
China
Prior art keywords
event
events
real
configuration
pushing
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202110910281.4A
Other languages
Chinese (zh)
Inventor
彭旭
谭必文
何伟
李柯辰
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shenzhen Weike Data Technology Co ltd
Original Assignee
Shenzhen Weike Data Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shenzhen Weike Data Technology Co ltd filed Critical Shenzhen Weike Data Technology Co ltd
Priority to CN202110910281.4A priority Critical patent/CN113781101A/en
Publication of CN113781101A publication Critical patent/CN113781101A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06QINFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00Commerce
    • G06Q30/02Marketing; Price estimation or determination; Fundraising
    • G06Q30/0201Market modelling; Market analysis; Collecting market data

Landscapes

  • Business, Economics & Management (AREA)
  • Strategic Management (AREA)
  • Engineering & Computer Science (AREA)
  • Accounting & Taxation (AREA)
  • Development Economics (AREA)
  • Finance (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Game Theory and Decision Science (AREA)
  • Data Mining & Analysis (AREA)
  • Economics (AREA)
  • Marketing (AREA)
  • Physics & Mathematics (AREA)
  • General Business, Economics & Management (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Debugging And Monitoring (AREA)

Abstract

The invention relates to a technical scheme of an event center management method, a device and a readable medium based on real-time computation, which comprises the following steps: acquiring events from a plurality of data sources through a plurality of different types of data acquisition modes, and sending the events; event processing, namely receiving an event and executing preprocessing, and further executing cleaning, counting and aggregation processing of the event; and subscribing the event, namely publishing the event to a corresponding subscriber according to subscription configuration and event configuration. The invention has the beneficial effects that: the method is simple to use, and when a new service system depends on the basic middle station service or the basic middle station service changes and needs to be updated, the existing service container configuration does not need to be modified; the unified deployment shields some problems of deployment target machine errors that may result during the development process.

Description

Event center management method and device based on real-time calculation and readable medium
Technical Field
The invention relates to the field of computers, in particular to an event center management method and device based on real-time computing and a readable medium.
Background
The current marketing market enters a digital marketing era from the traditional brand marketing, how to reach the content that a user needs to pay key attention to in a more accurate and real-time manner is described in the invention, the event configuration rule is used for acquiring the real-time event, the real-time event is summarized and converted through the real-time streaming calculation, the subscription and the release of the unified event are provided, and the capability center of an event center or an event factory is realized.
Meanwhile, due to the configurable capacity of the event center, the report of the new event can be processed only by interface configuration when the new event is generated, and the new event can be seamlessly pushed to subscribers in various modes after conversion.
The prior art has the following disadvantages: the current event center design is based on a subscription and publishing mode of the message, the subscription of the events and the events which need to be generated cannot be flexibly and dynamically configured, the event center degenerates into a message middleware, meanwhile, the subscription cannot be configured through a certain rule, the subscription is summarized into a series of aggregation events based on the existing streaming events, and the aggregation events can be used for triggering more critical marketing actions, controlling system wind and the like.
Disclosure of Invention
The present invention is directed to solve at least one of the technical problems in the prior art, and provides a method, an apparatus and a readable medium for managing an event center based on real-time computation, which solve the disadvantages of the prior art.
The technical scheme of the invention comprises an event center management method based on real-time calculation, which is characterized by comprising the following steps: acquiring events from a plurality of data sources through a plurality of different types of data acquisition modes, and sending the events; event processing, namely receiving the event and executing preprocessing so as to execute cleaning, counting and aggregation processing of the event; and subscribing the event, namely publishing the event to a corresponding subscriber according to subscription configuration and event configuration.
According to the real-time computing-based event center management method, event collection comprises the following steps: acquiring the events through a plurality of event acquisition standard interfaces, and preprocessing data based on time configuration of event processing, wherein the preprocessing at least comprises event type verification and event content verification; the event collection mode comprises HTTP interface report, asynchronous collection of message middleware, collection of specific files or memory events by an IOT device or a server through a collection client, and configurable full-volume or incremental collection.
According to the event center management method based on real-time computing, the collection standard format of an event comprises an event header and event content, wherein the event header comprises an event identifier, event occurrence time, an event title, an event filtering label, an event classification and an event source; the event content comprises an event service message body, and the event service message body is customized according to the service.
According to the event center management method based on real-time computing, the event collection further comprises: and filtering one or more items in the collected set standard format of the events according to the event filtering tags, discarding the events with missing items in the collected standard format, and generating an event processing log.
According to the event center management method based on real-time computing, event processing comprises the following steps: executing unified standard and cleaning conversion processing on the events collected and collected by the events to obtain standardized events meeting the event configuration, and distributing the standardized events; cleaning the received standardized event, and storing the standardized event in a storage medium supporting streaming calculation; and according to whether the event configuration is matched with the currently stored standardized event or not, or whether an aggregated event exists or not depends on the generation of the currently stored standardized event, and if so, executing the standardized event dynamic configuration.
According to the event center management method based on real-time computing, the aggregation processing comprises the following steps: acquiring a collected event, searching a unique identifier in the event, and persisting the corresponding event to a streaming computing storage engine; reading event rule configuration, and triggering event statistics based on a time window if detecting that an aggregation event is dependent on the unique identifier event; all aggregation-related events in a set time period are counted based on the stream type calculation, and if the number of the aggregation-related events reaches a set value of the aggregation events, the aggregation events are triggered, and no aggregation-related events are generated.
According to the event center management method based on real-time computing, event subscription comprises the following steps: the subscription pushing mode comprises two modes of message middleware pushing and HTTP pushing, multiple times of pushing and delayed pushing are carried out on a subscriber, and all pushing logs are recorded at the same time; the configuration of the visualization subscribing to the push mode comprises the steps of based on an event type, based on a single event and based on a global event; and the event subscription further comprises optional re-pushing of the historical push records.
The method for managing the event center based on real-time computing further comprises the following steps: the pushing of the streaming storage or message middleware is realized through the Kafka middleware.
The technical solution of the present invention also includes an event center management device based on real-time computing, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the method steps when executing the computer program.
The present invention also includes a computer-readable storage medium, in which a computer program is stored, wherein the computer program, when executed by a processor, implements any of the method steps.
The invention has the beneficial effects that: the method is simple to use, and when a new service system depends on the basic middle station service or the basic middle station service changes and needs to be updated, the existing service container configuration does not need to be modified; the unified deployment shields some problems of deployment target machine errors that may result during the development process.
Drawings
The invention is further described below with reference to the accompanying drawings and examples;
FIG. 1 illustrates an overall flow diagram according to an embodiment of the invention;
FIG. 2 illustrates a central management system according to an embodiment of the present invention;
FIG. 3 is a schematic diagram illustrating event subscription publishing according to an embodiment of the present invention;
fig. 4 shows a diagram of an apparatus according to an embodiment of the invention.
Detailed Description
Reference will now be made in detail to the present preferred embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to like elements throughout.
In the description of the present invention, the meaning of a plurality of means is one or more, the meaning of a plurality of means is two or more, and larger, smaller, larger, etc. are understood as excluding the number, and larger, smaller, inner, etc. are understood as including the number.
In the description of the present invention, the consecutive reference numbers of the method steps are for convenience of examination and understanding, and the implementation order between the steps is adjusted without affecting the technical effect achieved by the technical solution of the present invention by combining the whole technical solution of the present invention and the logical relationship between the steps.
In the description of the present invention, unless otherwise explicitly defined, terms such as set, etc. should be broadly construed, and those skilled in the art can reasonably determine the specific meanings of the above terms in the present invention in combination with the detailed contents of the technical solutions.
Fig. 1 shows a general flow diagram according to an embodiment of the invention, the flow comprising: acquiring events from a plurality of data sources through a plurality of different types of data acquisition modes, and sending the events; event processing, namely receiving an event and executing preprocessing, and further executing cleaning, counting and aggregation processing of the event; and subscribing the event, namely publishing the event to a corresponding subscriber according to subscription configuration and event configuration.
As shown in fig. 2, when the basic middleware services application is applied, the corresponding system refers to fig. 2, which mainly includes an event collection module, an event factory, and an event publishing module.
The entire event center consists of 3 large modules:
the event collection module comprises: and the data access module is responsible for providing event acquisition standard interfaces of various modes, and preprocessing data based on the time configuration of the event factory, such as event type verification, content verification and the like.
The event acquisition module provides a plurality of acquisition modes, which specifically comprise the following steps:
(1) and reporting and real-time collecting by the HTTP interface.
(2) The business events are asynchronously collected through the message middleware.
(3) The IOT equipment or the server provides an acquisition SDK by acquiring a specific file or a memory event through an acquisition client agent, and the business is non-invasive by adopting a java probe technology.
(4) The database business data can be collected in configurable full or incremental mode, and the collection of database data changes is monitored by using a self-research collection plug-in.
The standard event format definition of the acquisition includes:
event head: including event identification, event occurrence time, event title, event filtering tag, event classification, event source channel, etc.
The content of the event: and the message body containing the event service can be customized by the service.
The event acquisition module filters out irrelevant events according to the event identification list configured by the events, discards some missing necessary information, such as event titles and data of event sources, and records event processing logs.
Event factory: after the real-time receiving component and the asynchronous receiving component receive the events reported by the event collecting module, the event cleaning and statistics are performed based on the event configuration, and meanwhile, the whole processing engine is integrated with the streaming type computing engine, so that the event aggregation capability can be realized, for example, the total number of certain events generated in a certain time window is dynamically computed, the production of aggregated events is realized through the configured time window and a data threshold, and the specific processing flow is as follows.
(1) And unifying and standardizing the events collected by the event collection module, cleaning and converting the events into standardized events configured by the event configuration module, and then distributing the standardized events to the event processing engine.
(2) The event processing engine is responsible for cleaning received events, storing the events in a storage medium supporting streaming computing, such as Kafka.
(3) Whether the root event configuration matches the current event or not, or whether the aggregated event exists depends on the generation of the current event, and finally, the configured event (including the aggregated event) is dynamically generated.
Reference is made to the following code:
"eventldentify": "order _ money _ sum"/defining the event unique identifier as order _ money _ sum;
"eventType": "aggregate"/defining type as aggregation event;
"childeventuality": "order"/defining sub-events with the aggregate event statistical event unique identifier of order
"aggregate _ filtered": "userld, sum (money)",/defining the statistics field as summing the money field based on the userld field;
"aggregate _ window": "30"/defining a statistical time interval of 30 days;
"aggregate _ threshold": "10000/define the statistical breadth as triggering the aggregation event;
the aggregation event production takes the example that the monthly order target of the user achieves 10000 yuan of accumulated consumption and triggers the user level upgrading event as an example:
(1) the event factory receives the user order event issued by the event collection module, the unique identifier of the event is order (subscription), and the event is persisted to the streaming computing storage engine Kafka.
(2) The event factory reads the event configuration module to issue the existence of the event rule configuration shown in the figure two, finds that the aggregation event depends on the order (subscription) event, and triggers the event statistics based on the time window.
(3) And the event aggregation component counts the sum of the order amount of the user within 30 days based on the streaming calculation, triggers the aggregation event if the sum exceeds 10000 yuan, and does not generate a new event if the sum is less than 10000 yuan.
Event publishing: and receiving the event finally processed by the event factory, and pushing the event to the subscriber in various ways according to the subscription configuration.
FIG. 3 is a schematic diagram illustrating an event subscription publishing according to an embodiment of the present invention, which includes the following steps
(1) The subscription pushing mode comprises two modes of message middleware pushing and HTTP pushing, multiple times of pushing are supported, pushing is delayed to ensure that pushing is successful, and all pushing logs are recorded. As shown in fig. 3, the pushing manner of the system a is continuous pushing for 3 times, each time pushing is fixed by an interval time, the pushing manner is pushing through HTTP, the system B delays pushing through MQ messages for 1 hour, the system C increases the pushing interval by multiple times in an interval pushing manner, total pushing is performed for 3 times, and the pushing manner is HTTP;
(2) the subscription mode supports visual configuration including subscription modes based on event types, single events and global events, as shown in fig. 3, a system a subscribes events classified into order categories, a system B subscribes events identified as orders under all categories, and a system C subscribes events identified as orders under the event categories;
(3) for the history push record, the message can be selected to be pushed again, so as to achieve the purpose of service playback.
Fig. 4 shows a diagram of an apparatus according to an embodiment of the invention. The apparatus comprises a memory 100 and a processor 200, wherein the processor 200 stores a computer program for performing: acquiring events from a plurality of data sources through a plurality of different types of data acquisition modes, and sending the events; event processing, namely receiving an event and executing preprocessing, and further executing cleaning, counting and aggregation processing of the event; and subscribing the event, namely publishing the event to a corresponding subscriber according to subscription configuration and event configuration.
It should be recognized that the method steps in embodiments of the present invention may be embodied or carried out by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer readable memory. The method may use standard programming techniques. Each program may be implemented in a high level procedural or object oriented programming language to communicate with a computer system. However, the program(s) can be implemented in assembly or machine language, if desired. In any case, the language may be a compiled or interpreted language. Furthermore, the program can be run on a programmed application specific integrated circuit for this purpose.
Further, the operations of processes described herein can be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The processes described herein (or variations and/or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) collectively executed on one or more processors, by hardware, or combinations thereof. The computer program includes a plurality of instructions executable by one or more processors.
Further, the method may be implemented in any type of computing platform operatively connected to a suitable interface, including but not limited to a personal computer, mini computer, mainframe, workstation, networked or distributed computing environment, separate or integrated computer platform, or in communication with a charged particle tool or other imaging device, and the like. Aspects of the invention may be embodied in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, optically read and/or write storage medium, RAM, ROM, or the like, such that it may be read by a programmable computer, which when read by the storage medium or device, is operative to configure and operate the computer to perform the procedures described herein. Further, the machine-readable code, or portions thereof, may be transmitted over a wired or wireless network. The invention described herein includes these and other different types of non-transitory computer-readable storage media when such media include instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor. The invention also includes the computer itself when programmed according to the methods and techniques described herein.
A computer program can be applied to input data to perform the functions described herein to transform the input data to generate output data that is stored to non-volatile memory. The output information may also be applied to one or more output devices, such as consumers. In a preferred embodiment of the present invention, the transformed data represents physical and tangible objects, including particular visual depictions of physical and tangible objects produced on the consumer.
The embodiments of the present invention have been described in detail with reference to the accompanying drawings, but the present invention is not limited to the above embodiments, and various changes can be made within the knowledge of those skilled in the art without departing from the gist of the present invention.

Claims (10)

1. An event center management method based on real-time computing is characterized by comprising the following steps:
acquiring events from a plurality of data sources through a plurality of different types of data acquisition modes, and sending the events;
event processing, namely receiving the event and executing preprocessing so as to execute cleaning, counting and aggregation processing of the event;
and subscribing the event, namely publishing the event to a corresponding subscriber according to subscription configuration and event configuration.
2. The real-time computing based event center management method according to claim 1, wherein the event collection comprises:
acquiring the events through a plurality of event acquisition standard interfaces, and preprocessing data based on time configuration of event processing, wherein the preprocessing at least comprises event type verification and event content verification; the event collection mode comprises HTTP interface report, asynchronous collection of message middleware, collection of specific files or memory events by an IOT device or a server through a collection client, and configurable full-volume or incremental collection.
3. The real-time computing based event center management method according to claim 1, wherein the collection standard format of the event comprises an event header and event content, wherein the event header comprises an event identifier, an event occurrence time, an event title, an event filtering tag, an event classification and an event source; the event content comprises an event service message body, and the event service message body is customized according to the service.
4. The real-time computing based event center management method of claim 3, wherein the event collection further comprises:
and filtering one or more items in the collected set standard format of the events according to the event filtering tags, discarding the events with missing items in the collected standard format, and generating an event processing log.
5. The real-time computing based event center management method according to claim 1, wherein the event processing comprises:
executing unified standard and cleaning conversion processing on the events collected and collected by the events to obtain standardized events meeting the event configuration, and distributing the standardized events;
cleaning the received standardized event, and storing the standardized event in a storage medium supporting streaming calculation;
and according to whether the event configuration is matched with the currently stored standardized event or not, or whether an aggregated event exists or not depends on the generation of the currently stored standardized event, and if so, executing the standardized event dynamic configuration.
6. The real-time computing based event center management method according to claim 1, wherein the aggregation process comprises:
acquiring a collected event, searching a unique identifier in the event, and persisting the corresponding event to a streaming computing storage engine;
reading event rule configuration, and triggering event statistics based on a time window if detecting that an aggregation event is dependent on the unique identifier event;
all aggregation-related events in a set time period are counted based on the stream type calculation, and if the number of the aggregation-related events reaches a set value of the aggregation events, the aggregation events are triggered, and no aggregation-related events are generated.
7. The real-time computing based event center management method according to claim 1, wherein the event subscription comprises:
the subscription pushing mode comprises two modes of message middleware pushing and HTTP pushing, multiple times of pushing and delayed pushing are carried out on a subscriber, and all pushing logs are recorded at the same time;
the configuration of the visualization subscribing to the push mode comprises the steps of based on an event type, based on a single event and based on a global event;
and the event subscription further comprises optional re-pushing of the historical push records.
8. The real-time computing based event center management method according to claim 6 or 7, further comprising:
the pushing of the streaming storage or message middleware is realized through the Kafka middleware.
9. An event center management device based on real-time computing, comprising a memory, a processor and a computer program stored in said memory and executable on said processor, characterized in that said processor implements the method steps of any of claims 1-8 when executing said computer program.
10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the method steps of any one of claims 1 to 8.
CN202110910281.4A 2021-08-09 2021-08-09 Event center management method and device based on real-time calculation and readable medium Pending CN113781101A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110910281.4A CN113781101A (en) 2021-08-09 2021-08-09 Event center management method and device based on real-time calculation and readable medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110910281.4A CN113781101A (en) 2021-08-09 2021-08-09 Event center management method and device based on real-time calculation and readable medium

Publications (1)

Publication Number Publication Date
CN113781101A true CN113781101A (en) 2021-12-10

Family

ID=78837183

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110910281.4A Pending CN113781101A (en) 2021-08-09 2021-08-09 Event center management method and device based on real-time calculation and readable medium

Country Status (1)

Country Link
CN (1) CN113781101A (en)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111601291A (en) * 2020-04-07 2020-08-28 吉利汽车研究院(宁波)有限公司 RFID middleware, publish-subscribe system and data transmission method
CN111708645A (en) * 2020-06-12 2020-09-25 北京思特奇信息技术股份有限公司 Event processing method and system based on stream processing
CN112434083A (en) * 2020-11-30 2021-03-02 中国建设银行股份有限公司 Event processing method and device based on big data

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111601291A (en) * 2020-04-07 2020-08-28 吉利汽车研究院(宁波)有限公司 RFID middleware, publish-subscribe system and data transmission method
CN111708645A (en) * 2020-06-12 2020-09-25 北京思特奇信息技术股份有限公司 Event processing method and system based on stream processing
CN112434083A (en) * 2020-11-30 2021-03-02 中国建设银行股份有限公司 Event processing method and device based on big data

Similar Documents

Publication Publication Date Title
CN107992398B (en) Monitoring method and monitoring system of service system
CN110909063B (en) User behavior analysis method and device, application server and storage medium
US20180365085A1 (en) Method and apparatus for monitoring client applications
CN110661659A (en) Alarm method, device and system and electronic equipment
US20030065759A1 (en) Event driven storage resource metering
CN103246735A (en) Abnormal data processing method and abnormal data processing system
CN105099740A (en) Log management system and log collection method
CN103838867A (en) Log processing method and device
CN103942210A (en) Processing method, device and system of mass log information
CN108880934B (en) Data flow statistical method and device based on block chain
CN109871392B (en) Slow sql real-time data acquisition method under distributed application system
CN107979477A (en) A kind of method and system of business monitoring
CN110147470B (en) Cross-machine-room data comparison system and method
CN108509326A (en) A kind of service state statistical method and system based on nginx daily records
CN105989163A (en) Data real-time processing method and system
CN103179211A (en) Cloud-computing-based data processing method and system of comprehensive operation support system
CN113778810A (en) Log collection method, device and system
US8442947B2 (en) Management of performance data
CN113781101A (en) Event center management method and device based on real-time calculation and readable medium
CN116881277A (en) Data aggregation method, apparatus and computer readable medium
CN102930046B (en) Data processing method, computing node and system
US9224128B2 (en) Energy information exchange
CN114493720A (en) Method, device, storage medium and equipment for monitoring Kafka consumers
CN107147542A (en) A kind of information generating method and device
CN111401819A (en) Intersystem data pushing method and system

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination