CN114793245B - Flexible and configurable streaming information processing method and system - Google Patents
Flexible and configurable streaming information processing method and system Download PDFInfo
- Publication number
- CN114793245B CN114793245B CN202210708707.2A CN202210708707A CN114793245B CN 114793245 B CN114793245 B CN 114793245B CN 202210708707 A CN202210708707 A CN 202210708707A CN 114793245 B CN114793245 B CN 114793245B
- Authority
- CN
- China
- Prior art keywords
- configuration
- routing
- module
- message
- processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/08—Configuration management of networks or network elements
- H04L41/0803—Configuration setting
- H04L41/0813—Configuration setting characterised by the conditions triggering a change of settings
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L45/00—Routing or path finding of packets in data switching networks
- H04L45/74—Address processing for routing
- H04L45/742—Route cache; Operation thereof
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
Abstract
The invention discloses a flexible and configurable streaming information processing method and a system, which comprises the following steps: sending the off-line data stored on the server to a message receiving module in batches through HTTP requests, converting the off-line data into streaming information to be processed and storing the streaming information in the message receiving module; the message receiving module transmits the HTTP request to the routing module, and the routing module determines routing configuration through the URI corresponding to the HTTP request; the routing selection module transmits a message processing module to the determined routing configuration, the message processing module performs conversion processing on the streaming information to be processed according to a processing strategy in the routing configuration, if the conversion is successful, the message processing module sends back-end service, and if the conversion is unsuccessful, the message processing module discards the message. The invention can flexibly and dynamically load configuration, execute customized processing logic, avoid service halt updating and reduce the burden of operators.
Description
Technical Field
The invention relates to the field of streaming information processing methods, in particular to a flexibly configurable streaming information processing method and a flexibly configurable streaming information processing system.
Background
As a CDN service provider, there are a large number of requirements for log customization, analysis, and processing. And collecting the logs of a specific client by starting a real-time reporting mode for the client with high log real-time requirement. The existing CDN edge access log reports to a gateway service in real time, and then the gateway routing function distributes data to different back-end processing services, and for request contents in different formats, different servers need to be developed to implement a specific processing logic.
The existing processing scheme has long processing link and the following defects:
(1) flexible customization is not possible;
(2) the machine cost and the development cost are high;
(3) excessive backend services increase the workload of the operation and maintenance personnel.
Disclosure of Invention
The first purpose of the present invention is to provide a flexible and configurable streaming information processing method and system, which can flexibly and dynamically load configuration, execute customized processing logic, avoid service halt and update, and reduce the burden of operators.
A flexibly configurable method of streaming information processing, comprising the steps of:
1) the off-line data stored on the server is sent to a message receiving module in batches through HTTP requests, and the step has the function of converting the off-line data into streaming information to be processed and storing the streaming information in the message receiving module, so that the processing of subsequent modules is facilitated;
2) the message receiving module transmits the HTTP request to the routing module, the routing module determines routing configuration through the URI corresponding to the HTTP request, and the step is used for establishing a one-to-one mapping relation between the specific request URI and the specific message processing strategy;
3) the routing selection module transmits a message processing module to the determined routing configuration, the message processing module performs conversion processing on the streaming information to be processed according to a processing strategy in the routing configuration, if the conversion is successful, the streaming information is sent to a back-end service, and if the conversion is unsuccessful, the message is discarded;
4) the routing selection module checks and synchronizes the routing configuration of the configuration center service every 1-50 seconds;
5) and the message processing module checks the processing strategy of the configuration center service every 1-50 seconds and synchronizes.
In step 2), determining the routing configuration through the URI corresponding to the HTTP request specifically includes:
the routing configuration is determined from the longest prefix tree matching URI.
In step 3), the message processing module performs conversion processing on the streaming information to be processed according to the processing policy in the routing configuration, and specifically includes:
3.1) the message processing module interprets the Domain-specific language (Domain-specific language) of the route configuration into an executable processing strategy;
the specific domain language is formed by combining a JSON text and a Lua script so as to describe a message processing flow; the JSON text format is one of the most widely used general configuration text description forms, and the adoption of the format to describe the processing process is easy to understand and can reduce the starting difficulty of developers; lua is a lightweight programming language, so that developers can quickly get up and write message processing logic; the two are combined to ensure that the customized development work is more flexible and convenient and is more agile when the variable demands are met;
adopting JSON text and Lua script combination, specifically comprising:
combining message processing rules predefined by a message processing module by taking JSON as a configuration description form, wherein the message processing rules comprise: the message processing method comprises the following steps of message segmentation rules, deserialization rules, script executor rules and message output rules, wherein the message processing rules which are not predefined for a message processing module are realized by using Lua script programming;
3.2) according to the message processing rule defined in the step 3.1), executing message processing calculation on the to-be-processed streaming information by a LuaJIT module built in OpenResty;
3.3) serializing the messages after processing and calculation into a target format specified in the configuration, and completing conversion processing.
In step 4), the routing module checks and synchronizes the routing configuration of the configuration center service every 1-50 seconds, and specifically includes:
4.1) the routing module inquires the storage position of the configuration file in the service of the configuration center;
4.2) the route selection module reads and analyzes each route configuration;
4.3) a plurality of route configurations are stored in a key value pair mode, the route selection module constructs the route configurations into a prefix tree data structure and stores the prefix tree data structure in a memory, leaf nodes of each route configuration point to the alias of a unique message processing strategy, and each route configuration is provided with a hash abstract, so that versioning management is facilitated;
4.4) calculating the hash abstract of the analyzed routing configuration, comparing the hash abstract with the hash abstract of the routing configuration in the memory of the routing module, if the hash abstract is different from the hash abstract of the routing configuration in the memory of the routing module, indicating that the routing configuration in the memory is expired, updating the analyzed routing configuration into the memory, and restarting the corresponding working thread to enable the new routing configuration to take effect.
In step 5), the step of checking and synchronizing the processing strategy of the configuration center service by the message processing module every 1-50 seconds specifically comprises:
5.1) the message processing module inquires the storage position of the configuration file in the configuration center service;
5.2) according to the alias of the message processing strategy recorded in the digital data structure constructed by the routing module and the hash abstract information thereof, marking the alias as the required processing strategy;
5.3) the message processing module inquires and reads the required processing strategy, and then analyzes the processing strategy in a specific mode;
and 5.4) calculating a hash abstract of the analyzed processing strategy, comparing the hash abstract with the hash abstract of the processing strategy in the memory of the message processing module, if the hash abstract is different from the hash abstract of the processing strategy in the memory of the message processing module, updating the analyzed processing strategy into the memory, and restarting the corresponding working thread to enable the newly configured processing strategy to take effect.
A flexibly configurable streaming information processing system, comprising:
the message receiving module receives offline data from the server;
the configuration center service is used for dynamically configuring the storage distribution service based on Key/Value (Key Value) storage service;
the routing module selects different routing processing logics for the received streaming information to be processed according to the routing configuration information;
and the message processing module subscribes the routing configuration information from the service of the configuration center and converts the to-be-processed streaming information.
Compared with the prior art, the invention has the following advantages:
the flexible configuration method provided by the invention can avoid service halt and update, and keep the high availability of the whole system;
the configuration file is issued through the configuration center, so that the configuration customized logic is conveniently managed and configured in a centralized manner, the operation and maintenance difficulty of multi-instance deployment is reduced, and the burden of operation and maintenance personnel is reduced;
thirdly, the JSON text and the Lua script combination are used for describing the message processing flow, so that the customization and development work is more flexible and convenient, and is more agile when the variable requirements are met;
the invention directly converts the message processing flow into the byte code which can be executed by the built-in Lua virtual machine of the message processing module, can replace most of back-end services with the same function, and can improve the development efficiency, shorten the online period and reduce the operation and maintenance burden.
Drawings
Fig. 1 is a schematic diagram of a streaming information processing system according to the present invention.
FIG. 2 is a flow chart of the synchronous configuration of the message processing module and the configuration center according to the present invention.
FIG. 3 is a flow chart describing the message processing flow by using JSON text and Lua script combination according to the present invention.
Detailed Description
Example 1
As shown in fig. 1, the present embodiment includes a message sender, a message receiving module, a routing module, a message processing module, a configuration center service, and other backend services.
The message receiving module is a high-performance gateway; the message processing module is a log message processing service developed based on an OpenResty project; the configuration center is a dynamic configuration storage distribution service realized based on Key/Value storage service.
A message sender sends original log messages to a message receiving module in batch through HTTP requests, wherein each log message can be in a standard format or a customized format of NGINX access logs or in a single-row JSON form after space compression, a line feed character is used in a message body of the HTTP requests to separate a plurality of log messages, the number of the log messages of the batch requests is related to the average size of the single-row logs, and the total size of the messages of a single request is generally required to be not more than 5 MB.
The routing module matches the URI in the HTTP request to a corresponding message processing module according to the existing routing configuration, wherein the routing module can synchronize the routing configuration with the service of the configuration center at regular intervals, and the routing configuration can be modified only by the service of the configuration center, so that the routing modules in all program instances can receive the modified routing configuration; the synchronization time interval between the routing module and the configuration center service needs to be a proper value, the too short interval time can unnecessarily increase the pressure of the service end to consume the bandwidth of a machine room, and the too long interval time can cause the configuration to be not effective in time, thereby affecting the publishing efficiency. The reasonable synchronization time interval can be selected according to the actual service requirement and the judgment of experience.
And the message processing module cuts, deserializes and analyzes the received log message, converts the customized format, filters according to the condition and serializes the log message into the message format required by the back-end service according to the existing processing strategy. The message processing module and the configuration center service periodically synchronize processing strategies, the processing strategies are described by Domain-specific language (Domain-specific language) of JSON text and Lua script combination, predefined processing rules are arranged and combined through JSON text, expanded processing rules are realized by Lua script programming, and the expanded processing rules are called through JSON integration.
The message processing module passes the processed streaming message to other back-end services, where the back-end services are typically distributed message middleware clusters.
Example 2
The method of use of example 1, comprising the steps of:
(1) the routing selection module checks and synchronizes the routing configuration of the configuration center service every 15 seconds, and the message processing module checks and synchronizes the processing strategy of the configuration center every 15 seconds;
(2) a message sender generates messages to be processed and sends the messages to a message receiving module in batches through HTTP requests;
(3) the route selection module selects a corresponding route according to the URI in the HTTP request, and transmits the message to be processed sent by the message receiving module to a specific message processing module;
(4) the message processing module receives a message to be processed and further triggers and executes a message processing strategy;
(5) after the message processing is finished, transmitting the message to other back-end services;
as shown in fig. 2, the steps of checking and synchronizing the routing configuration information of the configuration center service include:
(1.1) the current information processing system inquires a configuration file storage position corresponding to the current service;
(1.2) the routing module analyzes each route configuration;
(1.3) if the routing configuration is different from the state stored in the memory of the current message processing module, synchronously configuring the configuration of the central service and restarting the corresponding working thread;
(1.4) analyzing message segmentation rules, deserialization rules, script executor rules and message output rules in the routing configuration one by one;
in the step (1.2), the route configuration is composed of JSON configuration or external Lua script, wherein the JSON data structure and the Lua Table data type can be converted with each other.
As shown in fig. 3, comparing the routing configuration in step (1.4) with the state in the memory of the current message processing module, which may be hashing the configuration item, and if the comparison is different, the configuration is considered to have changed.
The processing flow triggered by the request event in the step (4) to execute the configuration in the route is specifically as follows:
(4.1) the URI in the HTTP request matches to the corresponding route;
(4.2) the message processing module reads and analyzes the processing strategy into byte codes which can be executed by the Lua virtual machine and used for message processing operation;
(4.3) executing message processing operation by a LuaJIT module built in OpenResty;
(4.4) serializing the processed message into a format specified in the configuration.
The comparison of the cost required by the two information processing methods for comparing the new information with the old information of the same type of customized log item on line in terms of manpower, time and the like is shown in table 1.
TABLE 1
The flexibly configurable streaming information processing method and the system can flexibly and dynamically load configuration, execute customized processing logic, avoid service halt updating, reduce the burden of operators, improve the development efficiency, shorten the online period and reduce the operation and maintenance burden.
Claims (6)
1. A flexibly configurable method of processing streaming information, comprising the steps of:
1) sending the offline data stored on the server to a message receiving module in batches through HTTP requests, converting the offline data into streaming information to be processed and storing the streaming information in the message receiving module;
2) the message receiving module transmits the HTTP request to the routing module, and the routing module determines routing configuration through the URI corresponding to the HTTP request;
3) the routing selection module transmits a message processing module to the determined routing configuration, the message processing module performs conversion processing on the streaming information to be processed according to a processing strategy in the routing configuration, if the conversion is successful, the message processing module sends back-end service, and if the conversion is unsuccessful, the message processing module discards the message;
the message processing module performs conversion processing on the streaming information to be processed according to a processing policy in the routing configuration, and specifically includes:
3.1) the message processing module interprets the specific domain language of the route configuration into an executable processing strategy;
3.2) according to the message processing rule defined in the step 3.1), executing message processing calculation on the to-be-processed streaming information by a LuaJIT module built in OpenResty;
3.3) serializing the processed and calculated messages into a format specified in the configuration to complete conversion processing;
in the step 3.1), the specific domain language adopts a combination of JSON text and Lua script, and the specific domain language specifically comprises the following steps:
combining message processing rules predefined by a message processing module by taking JSON as a configuration description form, wherein the message processing rules comprise: message segmentation rules, deserialization rules, script executor rules and message output rules, and Lua scripts are used for message processing rules which are not predefined by the message processing module.
2. The flexibly configurable streaming information processing method according to claim 1, further comprising:
4) the routing selection module checks and synchronizes the routing configuration of the configuration center service every 1-50 seconds;
5) and the message processing module checks the processing strategy of the configuration center service every 1-50 seconds and synchronizes.
3. The flexibly configurable streaming information processing method according to claim 1, wherein in step 2), determining the routing configuration through the URI corresponding to the HTTP request specifically includes:
the routing configuration is determined from the longest prefix tree matching URI.
4. The flexibly configurable streaming information processing method according to claim 2, wherein in step 4), the routing module checks and synchronizes the routing configuration of the configuration center service every 1-50 seconds, and specifically comprises:
4.1) the routing module inquires the storage position of the configuration file in the service of the configuration center;
4.2) the route selection module reads and analyzes each route configuration;
4.3) a plurality of route configurations are stored in a key value pair mode, the route selection module constructs the route configurations into a prefix tree data structure and stores the prefix tree data structure in a memory, a leaf node of each route configuration points to an alias of a unique message processing strategy, and each route configuration is provided with a hash abstract;
4.4) calculating the hash abstract of the analyzed routing configuration, comparing the hash abstract with the hash abstract of the routing configuration in the memory of the routing module, if the hash abstract is different from the hash abstract of the routing configuration in the memory of the routing module, indicating that the routing configuration in the memory is expired, updating the analyzed routing configuration into the memory, and restarting the corresponding working thread to enable the new routing configuration to take effect.
5. The flexibly configurable streaming information processing method according to claim 2, wherein in step 5), the step of checking, by the message processing module, the processing policy of the configuration center service every 1-50 seconds and synchronizing specifically comprises:
5.1) the message processing module inquires the storage position of the configuration file in the configuration center service;
5.2) according to the alias of the message processing strategy and the hash abstract information thereof recorded in the digital data structure constructed by the routing module, marking the alias as the required processing strategy;
5.3) the message processing module inquires and reads the required processing strategy, and then analyzes the processing strategy in a specific mode;
and 5.4) calculating a hash abstract of the analyzed processing strategy, comparing the hash abstract with the hash abstract of the processing strategy in the memory of the message processing module, if the hash abstract is different from the hash abstract of the processing strategy in the memory of the message processing module, updating the analyzed processing strategy into the memory, and restarting the corresponding working thread to enable the newly configured processing strategy to take effect.
6. A flexibly configurable streaming information processing system for implementing the method of any one of claims 1 to 5, comprising:
the message receiving module receives offline data from the server;
the configuration center service is used for dynamically configuring the storage issuing service based on the key value storage service;
the routing module selects different routing processing logics for the received streaming information to be processed according to the routing configuration information;
and the message processing module subscribes the routing configuration information from the configuration center service and converts the to-be-processed streaming information.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210708707.2A CN114793245B (en) | 2022-06-22 | 2022-06-22 | Flexible and configurable streaming information processing method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210708707.2A CN114793245B (en) | 2022-06-22 | 2022-06-22 | Flexible and configurable streaming information processing method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114793245A CN114793245A (en) | 2022-07-26 |
| CN114793245B true CN114793245B (en) | 2022-09-27 |
Family
ID=82462809
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210708707.2A Active CN114793245B (en) | 2022-06-22 | 2022-06-22 | Flexible and configurable streaming information processing method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114793245B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110996372A (en) * | 2019-11-11 | 2020-04-10 | 广州爱浦路网络技术有限公司 | Message routing method, device and system and electronic equipment |
| CN112532425A (en) * | 2020-11-04 | 2021-03-19 | 浙江大学 | Message routing configuration method facing edge calculation |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102035725B (en) * | 2010-08-10 | 2013-01-02 | 国家计算机网络与信息安全管理中心 | Relevant technology system for one-way flow uniform resource identifier (URI) under asymmetric routing and method thereof |
| CN107479878B (en) * | 2017-07-26 | 2021-02-05 | 北京供销科技有限公司 | Lua modularization development method and development framework based on openness |
| US10791056B2 (en) * | 2018-04-16 | 2020-09-29 | Citrix Systems, Inc. | Policy based service routing |
| CN109871502B (en) * | 2019-01-18 | 2020-10-30 | 北京赛思信安技术股份有限公司 | Stream data regular matching method based on Storm |
| CN110035117A (en) * | 2019-03-15 | 2019-07-19 | 启迪云计算有限公司 | One kind is based on configurable monitoring script monitoring system and monitoring method |
| CN110401657B (en) * | 2019-07-24 | 2020-09-25 | 网宿科技股份有限公司 | Processing method and device for access log |
| CN113076107A (en) * | 2021-04-13 | 2021-07-06 | 杭州又拍云科技有限公司 | Method for automatically acquiring and fusing logs through finite state machine |
-
2022
- 2022-06-22 CN CN202210708707.2A patent/CN114793245B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110996372A (en) * | 2019-11-11 | 2020-04-10 | 广州爱浦路网络技术有限公司 | Message routing method, device and system and electronic equipment |
| CN112532425A (en) * | 2020-11-04 | 2021-03-19 | 浙江大学 | Message routing configuration method facing edge calculation |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114793245A (en) | 2022-07-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101827302A (en) | Multi-service unified processing method and unified service platform | |
| CN103546343B (en) | The network traffics methods of exhibiting of network traffic analysis system and system | |
| CN104378234B (en) | Across the data transmission processing method and system of data center | |
| CN112118174B (en) | Software defined data gateway | |
| CN102026228B (en) | Statistical method and equipment for communication network performance data | |
| CN113176948B (en) | Edge gateway, edge computing system and configuration method thereof | |
| CN114268640A (en) | Intelligent routing system of industrial Internet of things with cloud edge cooperation | |
| CN108924007B (en) | Big data acquisition and storage system and method of communication operation information | |
| CN113032379B (en) | Distribution network operation and inspection-oriented multi-source data acquisition method | |
| CN111200523B (en) | Method, device, equipment and storage medium for configuring middle platform system | |
| CN115086379B (en) | Numerical control machine tool data acquisition method based on edge calculation and virtualization technology | |
| CN114968953A (en) | Log storage and retrieval method, system, terminal equipment and medium | |
| WO2016045367A1 (en) | Multi-data-source data fusion method and device | |
| CN113505006A (en) | Mimicry database oriented arbitration device and method | |
| CN114710571B (en) | Data packet processing system | |
| WO2013107143A1 (en) | Method and device for transmitting interface parameter | |
| CN112270600A (en) | Multi-source data processing method, system and related device | |
| CN114817190A (en) | Log synchronization method, device, system, equipment and storage medium | |
| CN114793245B (en) | Flexible and configurable streaming information processing method and system | |
| CN105338107A (en) | Stronghold operation synchronous management system and stronghold operation synchronous management method | |
| CN111405020B (en) | Asynchronous file export method and system based on message queue and fastDFS micro-service framework | |
| CN112182031B (en) | Data query method and device, storage medium and electronic device | |
| CN112131014A (en) | Decision engine system and business processing method thereof | |
| JP2014016758A (en) | Log data decompression device, log data compression device, log data decompression program, and log data compression program | |
| CN115934748A (en) | Switch distribution and metrics collection and summary system and method based on distributed SQL |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |