CN115543445A - Dynamic processing method and device for stream data, electronic equipment and storage medium - Google Patents

Dynamic processing method and device for stream data, electronic equipment and storage medium Download PDF

Info

Publication number
CN115543445A
CN115543445A CN202211117027.XA CN202211117027A CN115543445A CN 115543445 A CN115543445 A CN 115543445A CN 202211117027 A CN202211117027 A CN 202211117027A CN 115543445 A CN115543445 A CN 115543445A
Authority
CN
China
Prior art keywords
task
processing
stream
data
configuration
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202211117027.XA
Other languages
Chinese (zh)
Inventor
戈子根
张强
郗亚静
刘亚维
朱静菠
李春龙
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Jiehui Technology Co Ltd
Original Assignee
Beijing Jiehui Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Jiehui Technology Co Ltd filed Critical Beijing Jiehui Technology Co Ltd
Priority to CN202211117027.XA priority Critical patent/CN115543445A/en
Publication of CN115543445A publication Critical patent/CN115543445A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/30Arrangements for executing machine instructions, e.g. instruction decode
    • G06F9/38Concurrent instruction execution, e.g. pipeline, look ahead
    • G06F9/3867Concurrent instruction execution, e.g. pipeline, look ahead using instruction pipelines

Abstract

The invention relates to the technical field of stream data processing, in particular to a stream data dynamic processing method, a stream data dynamic processing device, electronic equipment and a storage medium, and aims to solve the technical problems that the workload of a stream processing method realized in a hard programming mode is complicated, codes for different tasks are tightly coupled, and once a code for one task is programmed incorrectly, the execution of codes corresponding to other tasks is influenced. To this end, the dynamic processing method of stream data of the present invention comprises: generating a configuration task according to the user requirement, and storing the configuration task in a database; starting a stream processing thread according to the configuration task to dynamically generate a stream processing task; acquiring stream data by the stream processing task and processing the stream data; and carrying out windowing polymerization on the processed stream data to obtain index data. Thus, the stream data processing efficiency is improved.

Description

Dynamic processing method and device for stream data, electronic equipment and storage medium
Technical Field
The invention relates to the technical field of stream data processing, and particularly provides a stream data dynamic processing method, a stream data dynamic processing device, electronic equipment and a storage medium.
Background
An APM (application performance Management) is an application performance monitoring tool, and implements full link performance monitoring of applications by aggregating real-time data (e.g., logs, link data, and service data collected by a probe in fig. 1) of each processing link of a service system, and analyzing transaction paths and processing time of each transaction processing of the service system. The above various data are firstly required to be collected, and then the stream processing is performed to generate various health indexes.
The existing stream processing method is implemented in a hard programming manner, for example, for 50 tasks, 50 program codes similar to yellow highlighting need to be repeated, and in the following example, only 2 tasks are presented, specifically, a keyword statistics task for log analysis and a request amount statistics task for a call chain.
Figure BDA0003845737520000011
However, the above-mentioned stream processing method implemented by the hard programming method has a tedious workload, and codes for different tasks are tightly coupled, and once a code programming error occurs for a certain task, execution of codes corresponding to other tasks is affected, resulting in low stream processing efficiency.
Accordingly, there is a need in the art for a new dynamic processing scheme for streaming data that addresses the above-mentioned problems.
Disclosure of Invention
The present invention has been made to overcome the above-mentioned drawbacks, and aims to provide a solution or at least a partial solution to the above-mentioned technical problem. The invention provides a dynamic processing method and device of stream data, electronic equipment and a storage medium.
In a first aspect, the present invention provides a method for dynamically processing stream data, the method comprising: generating a configuration task according to user requirements, and storing the configuration task in a database; starting a stream processing thread according to the configuration task to dynamically generate a stream processing task; acquiring stream data by the stream processing task and processing the stream data; and performing windowing polymerization on the processed stream data to obtain index data.
In one embodiment, generating configuration tasks according to user requirements includes: acquiring a task template corresponding to the user requirement; and adding corresponding task parameters into the task template to obtain the configuration task, wherein the configuration task comprises a task name and a stream processing windowing parameter.
In one embodiment, the user requirements include log analysis; the generating of the configuration task according to the user requirement comprises the following steps: acquiring a task template corresponding to the log analysis; adding corresponding task parameters into the task template to obtain the configuration task, wherein the task name in the configuration task is a log quantity generated by keyword statistics and application, the stream processing windowing parameters comprise a windowing type, and the windowing type is any one of a sliding window or a rolling window.
In one embodiment, the user requirements include call chain correlation analysis; the generating of the configuration task according to the user requirement comprises the following steps: acquiring a task template corresponding to the call chain correlation analysis; and adding corresponding task parameters into the task template to obtain the configuration task, wherein the task name in the configuration task is at least one of request amount, processing time consumption, failure reason, caller request times and time consumption statistics, the stream processing windowing parameters comprise a windowing type, and the windowing type is any one of a sliding window or a rolling window.
In one embodiment, starting a stream processing thread according to the configuration task to dynamically generate a stream processing task, includes: when the Web container is started, a dynamic function is used for calling a configuration task in the database, wherein the function comprises a variable for representing a task name; initializing the corresponding stream processing thread according to the task name and the stream processing windowing parameter in the called configuration task, and dynamically generating a stream processing task.
In one embodiment, obtaining streaming data by the streaming task comprises: the flow processing task reads flow data from at least one message queue, wherein the flow processing data comprises at least one of traffic data of log, link data and probe acquisition data.
In one embodiment, after generating a configuration task according to a user requirement and storing the configuration task in a database, and before acquiring stream data by the stream processing task and processing the stream data, the method further comprises: and registering a new flow processing thread to the started Web container according to the newly added configuration task.
In a second aspect, the present invention provides a streaming data dynamic processing apparatus, the apparatus comprising:
the generating module is configured to generate a configuration task according to the requirement of a user and store the configuration task in a database;
the starting module is configured to start a stream processing thread according to the configuration task so as to dynamically generate a stream processing task;
a processing module configured to acquire stream data from the stream processing task and process the stream data;
and the aggregation module is configured to perform windowing aggregation on the processed stream data to obtain index data.
In a third aspect, an electronic device is provided, comprising a processor and a storage means adapted to store a plurality of program codes, said program codes being adapted to be loaded and run by said processor to perform the method of dynamic processing of streaming data according to any of the preceding claims.
In a fourth aspect, a computer readable storage medium is provided, having stored therein a plurality of program codes adapted to be loaded and executed by a processor to perform the dynamic processing method of streaming data of any of the preceding claims.
One or more technical schemes of the invention at least have one or more of the following beneficial effects:
the dynamic processing method of the stream data comprises the steps of firstly generating a configuration task according to user requirements, storing the configuration task in a database, then starting a stream processing thread according to the configuration task to dynamically generate a stream processing task, secondly acquiring the stream data by the stream processing task, processing the stream data, and finally windowing and aggregating the processed stream data to obtain index data. Therefore, the configuration tasks are generated in advance and stored in the database, and the configuration tasks are directly called from the database at the later stage to dynamically generate the stream processing tasks, so that the problem of repeatedly executing programs of each task is avoided, and the stream processing efficiency is improved.
Drawings
The disclosure of the present invention will become more readily understood with reference to the accompanying drawings. As is readily understood by those skilled in the art: these drawings are for illustrative purposes only and are not intended to constitute a limitation on the scope of the present invention. Moreover, in the drawings, like numerals are used to indicate like parts, and in which:
FIG. 1 is a schematic flow diagram of a full link performance test of an intelligent monitoring system on an application;
fig. 2 is a flow chart illustrating the main steps of a dynamic processing method of stream data according to an embodiment of the present invention;
fig. 3 is a schematic diagram of a main configuration of a streaming data dynamic processing apparatus according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the invention.
List of reference numerals
11: a generating module; 12: starting a module; 13: a processing module; 14: and (4) a polymerization module.
Detailed Description
Some embodiments of the invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only for explaining the technical principle of the present invention, and are not intended to limit the scope of the present invention.
In the description of the present invention, a "module" or "processor" may include hardware, software, or a combination of both. A module may comprise hardware circuitry, various suitable sensors, communication ports, memory, may comprise software components such as program code, or may be a combination of software and hardware. The processor may be a central processing unit, microprocessor, image processor, digital signal processor, or any other suitable processor. The processor has data and/or signal processing functionality. The processor may be implemented in software, hardware, or a combination thereof. Non-transitory computer readable storage media include any suitable medium that can store program code, such as magnetic disks, hard disks, optical disks, flash memory, read-only memory, random-access memory, and the like. The term "a and/or B" denotes all possible combinations of a and B, such as a alone, B alone or a and B. The term "at least one A or B" or "at least one of A and B" means similar to "A and/or B" and may include only A, only B, or both A and B. The singular forms "a", "an" and "the" may include plural forms as well.
At present, the traditional stream processing method implemented by a hard programming mode has a complex workload, and codes for different tasks are tightly coupled, so that once a code programming error occurs for a certain task, execution of codes corresponding to other tasks is affected, and thus stream processing efficiency is low.
The method comprises the steps of firstly generating a configuration task according to user requirements, storing the configuration task in a database, then starting a stream processing thread according to the configuration task to dynamically generate a stream processing task, then acquiring stream data by the stream processing task, processing the stream data, and finally windowing and aggregating the processed stream data to obtain index data. Therefore, the configuration tasks are generated in advance and stored in the database, and the configuration tasks are directly called from the database at the later stage to dynamically generate the stream processing tasks, so that the problem of repeatedly executing programs of each task is avoided, and the stream processing efficiency is improved.
Referring to fig. 2, fig. 2 is a flow chart illustrating the main steps of a dynamic processing method for stream data according to an embodiment of the present invention.
As shown in fig. 2, the dynamic processing method for stream data in the embodiment of the present invention mainly includes the following steps S101 to S104.
Step S101: and generating a configuration task according to the user requirement, and storing the configuration task in a database.
In one embodiment, generating a configuration task according to a user requirement includes: acquiring a task template corresponding to the user requirement; and adding corresponding task parameters into the task template to obtain the configuration task, wherein the configuration task comprises a task name and a stream processing windowing parameter.
Specifically, various task templates corresponding to user requirements are preset and stored in corresponding positions. Specifically, in the process of generating the configuration task, a task template corresponding to the user requirement is obtained first, and then a corresponding task parameter is added to the task template to obtain the configuration task.
The configuration task includes at least a task name and a stream processing windowing parameter, but is not limited thereto, and may further include task parameters such as a windowing size, an aggregation factor, and a grace period.
Where the grace period is used to control the time waiting for a given window without data recording. A record is discarded if its timestamp indicates that it belongs to a window but the current stream time is greater than the end time of the window plus a grace period.
In one embodiment, the user requirements include log analysis; the generating of the configuration task according to the user requirement comprises the following steps: acquiring a task template corresponding to the log analysis; and adding corresponding task parameters into the task template to obtain the configuration task, wherein the task name in the configuration task is the log amount generated by keyword statistics and application, the stream processing windowing parameter comprises a windowing type, and the windowing type is any one of a sliding window or a rolling window.
Illustratively, when the user requirement is to analyze the log, then the configuration task generated includes the keyword statistics (task name), the amount of log produced by the application (task name). The windowing type is, for example, a sliding window or a scrolling window, etc. The aggregation factor is a specific application name and a specific keyword, for example, the number of times of occurrence of "call upstream exception" of the transaction system, where the call upstream exception in "is the keyword.
In one embodiment, the user requirements include call chain correlation analysis; the generating of the configuration task according to the user requirement comprises the following steps: acquiring a task template corresponding to the call chain correlation analysis; adding corresponding task parameters into the task template to obtain the configuration task, wherein the task name in the configuration task is at least one of request amount, processing time consumption, failure reason, caller request times and time consumption statistics, the stream processing windowing parameters comprise a windowing type, and the windowing type is any one of a sliding window or a rolling window.
Illustratively, when the user requirement is a call chain correlation analysis, then the configuration tasks generated include: the number of requests (task name), the processing time (task name), the failure reason (task name), the number of caller requests (task name), the time consumption statistics (task name), etc. The windowing type is, for example, a sliding window or a scrolling window, etc. The aggregation factor is a specific application name, instance, interface name, service line, etc.
The log shown in fig. 1 is a log generated by various microservices, such as a log generated by a billing microservices in a third party payment scenario. The link data is data generated by calling between services, such as transaction union pay, transaction wind control account, and wind control caller, and forms a calling tree or a calling network. The probe is a point buried at the application side so as to collect customized service data in the application operation.
In practical applications, the configuration task is embodied as a record in a database.
Step S102: and starting the stream processing thread according to the configuration task to dynamically generate the stream processing task.
In a specific embodiment, starting a stream processing thread according to the configuration task to dynamically generate a stream processing task, includes: when the Web container is started, a dynamic function is used for calling a configuration task in the database, wherein the function comprises a variable for representing a task name; initializing the corresponding stream processing thread according to the task name and the stream processing windowing parameter in the called configuration task, and dynamically generating a stream processing task.
Specifically, in practice, each stream processing thread is initialized in accordance with a configuration task when the Web container is started. Wherein the Web container is used to manage the stream processing thread. For example, the start log analyzes each stream processing thread, the call chain relates to each stream processing thread, and the probe relates to each stream processing thread.
For the start of each stream processing thread, a configuration task stored in a database in advance is called by using a dynamic function, and then the stream processing task can be dynamically generated according to a task name, a stream processing windowing type, a windowing size, an aggregation factor, a grace period and the like in the called configuration task. The program code for dynamically generating the stream processing task is, for example:
Figure BDA0003845737520000071
in this way, the configuration tasks pre-stored in the database are called through the dynamic function, wherein the concrete tasks are abstracted into variable tasks in the function statement, so that codes corresponding to the tasks are decoupled, and repeated execution of the program is avoided when a plurality of tasks exist. In addition, when the code programming of a certain task has errors, the execution of codes corresponding to other tasks is not influenced, and the stream processing efficiency is improved.
Step S103: and acquiring stream data by the stream processing task and processing the stream data.
In one embodiment, acquiring streaming data by the streaming task includes: the flow processing task reads flow data from at least one message queue, wherein the flow processing data comprises at least one of traffic data of log, link data and probe acquisition data.
Specifically, the flow data includes logs, link data, traffic class data collected by the probe.
For example, the log is a log generated by various microservices, such as a log generated by a billing microservice in a third party payment scenario. The link data is data generated by calling between services, such as transaction union pay, transaction wind control account, and wind control caller, and forms a calling tree or a calling network. The probe is a point buried at the application side so as to collect customized service data in the application operation.
The streaming data is stored in message middleware such as kafka, rabtmq, etc., with different classes of data (e.g., logs, call chains, etc.) stored in different message queues.
In this step, the stream data is first read from the at least one message queue by the stream processing task, and then the stream data is processed.
For example, the processing may include filtering, cleaning, converting, etc. to unify the data format standards of the streaming data for subsequent further processing.
For example, a log analysis processing task obtains log data from a message queue.
The stream processing task is implemented using, for example, the kafka-streams toolkit.
Step S104: and performing windowing polymerization on the processed stream data to obtain index data.
Specifically, the processed stream data is distributed to different time windows according to the event occurrence time, and aggregation operation is performed on the data in each time window according to an aggregation factor, so that various index data with time stamps are generated. Wherein the plurality of time windows correspond to the plurality of time stamps.
Illustratively, for call chain data: and performing aggregation operation on the collected link data in the window time, wherein specific aggregation tasks comprise request amount (task name), processing time consumption (task name), failure reason (task name), calling party request times (task name), time consumption statistics (task name) and the like.
Illustratively, for log data: and performing aggregation operation on the collected logs in the window time, wherein the specific aggregation tasks comprise keyword statistics, log amount generated by application and the like.
Based on the steps S101 to S104, firstly, a configuration task is generated according to a user requirement, the configuration task is stored in a database, then, a stream processing thread is started according to the configuration task to dynamically generate a stream processing task, secondly, stream data is acquired by the stream processing task, the stream data is processed, and finally, windowing aggregation is performed on the processed stream data to obtain index data. Therefore, the configuration tasks are generated in advance and stored in the database, and the configuration tasks are directly called from the database at the later stage to dynamically generate the stream processing tasks, so that the problem of repeatedly executing programs of each task is avoided, and the stream processing efficiency is improved.
In a specific embodiment, after generating a configuration task according to a user requirement and storing the configuration task in a database, and before acquiring stream data by the stream processing task and processing the stream data, the method further includes: and registering a new flow processing thread to the started Web container according to the newly added configuration task.
Specifically, after a new configuration task is generated according to the user demand, a new stream processing thread may also be registered with the Web container that has been started, and then the execution continues to step S103. Thus, the stream processing task can be dynamically generated according to the newly added configuration task.
It will be understood by those skilled in the art that registering a new stream processing thread with the started Web container belongs to a parallel execution step with the aforementioned step S102.
In a specific embodiment, the index data obtained in step S104 may be stored in a database to be subjected to secondary analysis processing.
In a specific embodiment, various alarm rules may be configured based on the index data obtained after the stream processing, and related personnel may be notified based on the alarm rules.
Illustratively, an index data threshold value may be set based on index data, and when the index data exceeds the threshold value, the relevant person is notified by WeChat, short message, or the like.
In one embodiment, various monitoring charts can be configured based on the index data, and the various monitoring charts are displayed on the UI interface.
It should be noted that, although the foregoing embodiments describe each step in a specific sequence, those skilled in the art will understand that, in order to achieve the effect of the present invention, different steps do not necessarily need to be executed in such a sequence, and they may be executed simultaneously (in parallel) or in other sequences, and these changes are all within the protection scope of the present invention.
Furthermore, the invention also provides a dynamic processing device of the stream data.
Referring to fig. 3, fig. 3 is a main configuration block diagram of a stream data dynamic processing apparatus according to an embodiment of the present invention.
As shown in fig. 3, the dynamic stream data processing apparatus in the embodiment of the present invention mainly includes a generating module 11, a starting module 12, a processing module 13, and an aggregating module 14. In some embodiments, one or more of the generation module 11, the initiation module 12, the processing module 13, and the aggregation module 14 may be combined together into one module.
In some embodiments, the generation module 11 may be configured to generate configuration tasks according to user requirements and store the configuration tasks in a database.
The initiation module 12 may be configured to initiate a stream processing thread in accordance with the configuration task to dynamically generate a stream processing task.
The processing module 13 may be configured to obtain streaming data by a streaming processing task and process the streaming data.
The aggregation module 14 may be configured to perform windowing aggregation on the processed stream data to obtain the index data.
In one embodiment, the description of the specific implementation function may refer to steps S101 to S104.
The technical principles, the solved technical problems, and the generated technical effects of the above-mentioned streaming data dynamic processing apparatus for executing the embodiment of the streaming data dynamic processing method shown in fig. 2 are similar, and it can be clearly understood by those skilled in the art that, for convenience and brevity of description, the contents described in the embodiment of the streaming data dynamic processing method may be referred to for the specific working process and related descriptions of the streaming data dynamic processing apparatus, and are not repeated herein.
It will be understood by those skilled in the art that all or part of the flow of the method according to the above-described embodiment may be implemented by a computer program, which may be stored in a computer-readable storage medium and used to implement the steps of the above-described embodiments of the method when the computer program is executed by a processor. Wherein the computer program comprises computer program code, which may be in the form of source code, object code, an executable file or some intermediate form, etc. The computer-readable storage medium may include: any entity or device capable of carrying said computer program code, medium, U.S. disk, removable hard disk, magnetic diskette, optical disk, computer memory, read-only memory, random access memory, electrical carrier signal, telecommunications signal, software distribution medium, or the like. It should be noted that the computer-readable storage medium may contain content that is subject to appropriate increase or decrease as required by legislation and patent practice in jurisdictions, for example, in some jurisdictions, computer-readable storage media may not include electrical carrier signals and telecommunications signals in accordance with legislation and patent practice.
Furthermore, the invention also provides electronic equipment. In an embodiment of the electronic device according to the present invention, as shown in fig. 4, the electronic device comprises a processor 41 and a storage device 42, the storage device may be configured to store a program for executing the dynamic processing method of streaming data of the above method embodiment, and the processor may be configured to execute the program in the storage device, the program including but not limited to the program for executing the dynamic processing method of streaming data of the above method embodiment. For convenience of explanation, only the parts related to the embodiments of the present invention are shown, and specific technical details are not disclosed.
Further, the invention also provides a computer readable storage medium. In one computer-readable storage medium embodiment according to the present invention, the computer-readable storage medium may be configured to store a program for executing the streaming data dynamic processing method of the above-described method embodiment, and the program may be loaded and executed by a processor to implement the streaming data dynamic processing method. For convenience of explanation, only the parts related to the embodiments of the present invention are shown, and details of the specific techniques are not disclosed. The computer readable storage medium may be a storage device formed by including various electronic devices, and optionally, the computer readable storage medium is a non-transitory computer readable storage medium in the embodiment of the present invention.
Further, it should be understood that, since the configuration of each module is only for explaining the functional units of the apparatus of the present invention, the corresponding physical devices of the modules may be the processor itself, or a part of software, a part of hardware, or a part of a combination of software and hardware in the processor. Thus, the number of individual modules in the figures is merely illustrative.
Those skilled in the art will appreciate that the various modules in the apparatus may be adaptively split or combined. Such splitting or combining of specific modules does not cause the technical solutions to deviate from the principle of the present invention, and therefore, the technical solutions after splitting or combining will fall within the protection scope of the present invention.
So far, the technical solutions of the present invention have been described in connection with the preferred embodiments shown in the drawings, but it is easily understood by those skilled in the art that the scope of the present invention is obviously not limited to these specific embodiments. Equivalent changes or substitutions of related technical features can be made by those skilled in the art without departing from the principle of the invention, and the technical scheme after the changes or substitutions can fall into the protection scope of the invention.

Claims (10)

1. A method for dynamic processing of stream data, the method comprising:
generating a configuration task according to user requirements, and storing the configuration task in a database;
starting a stream processing thread according to the configuration task to dynamically generate a stream processing task;
acquiring stream data by the stream processing task and processing the stream data;
and performing windowing polymerization on the processed stream data to obtain index data.
2. The dynamic streaming data processing method according to claim 1, wherein generating a configuration task according to a user requirement comprises:
acquiring a task template corresponding to the user requirement;
and adding corresponding task parameters into the task template to obtain the configuration task, wherein the configuration task comprises a task name and a stream processing windowing parameter.
3. Dynamic processing method of streaming data according to claim 2,
the user requirements include log analysis;
the generating of the configuration task according to the user requirement comprises the following steps:
acquiring a task template corresponding to the log analysis;
and adding corresponding task parameters into the task template to obtain the configuration task, wherein the task name in the configuration task is the log amount generated by keyword statistics and application, the stream processing windowing parameter comprises a windowing type, and the windowing type is any one of a sliding window or a rolling window.
4. Dynamic processing method of streaming data according to claim 2,
the user requirements include call chain correlation analysis;
the generating of the configuration task according to the user requirement comprises the following steps:
acquiring a task template corresponding to the call chain correlation analysis;
and adding corresponding task parameters into the task template to obtain the configuration task, wherein the task name in the configuration task is at least one of request amount, processing time consumption, failure reason, caller request times and time consumption statistics, the stream processing windowing parameters comprise a windowing type, and the windowing type is any one of a sliding window or a rolling window.
5. The dynamic processing method for stream data according to any one of claims 2 to 4, wherein starting a stream processing thread according to the configuration task to dynamically generate a stream processing task comprises:
when the Web container is started, a dynamic function is used for calling a configuration task in the database, wherein the function comprises a variable for representing a task name;
initializing the corresponding stream processing thread according to the task name and the stream processing windowing parameter in the called configuration task, and dynamically generating the stream processing task.
6. The dynamic processing method of stream data as claimed in claim 1, wherein obtaining stream data by said stream processing task comprises:
the flow processing task reads flow data from at least one message queue, wherein the flow processing data comprises at least one of traffic data of log, link data and probe acquisition data.
7. The dynamic processing method of streaming data as claimed in claim 5, wherein after generating a configuration task according to the user's requirement and storing the configuration task in the database, and before acquiring streaming data by the streaming task and processing the streaming data, the method further comprises: and registering a new flow processing thread to the started Web container according to the newly added configuration task.
8. A dynamic processing apparatus of stream data, the apparatus comprising:
the generating module is configured to generate a configuration task according to the requirement of a user and store the configuration task in a database;
the starting module is configured to start a stream processing thread according to the configuration task so as to dynamically generate a stream processing task;
a processing module configured to acquire stream data by the stream processing task and process the stream data;
and the aggregation module is configured to perform windowing aggregation on the processed stream data to obtain index data.
9. An electronic device comprising a processor and a storage means adapted to store a plurality of program codes, characterized in that said program codes are adapted to be loaded and run by said processor to perform the method of dynamic processing of streaming data according to any of claims 1 to 7.
10. A computer-readable storage medium having stored therein a plurality of program codes, wherein said program codes are adapted to be loaded and run by a processor to execute the dynamic processing method of stream data according to any one of claims 1 to 7.
CN202211117027.XA 2022-09-14 2022-09-14 Dynamic processing method and device for stream data, electronic equipment and storage medium Pending CN115543445A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202211117027.XA CN115543445A (en) 2022-09-14 2022-09-14 Dynamic processing method and device for stream data, electronic equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202211117027.XA CN115543445A (en) 2022-09-14 2022-09-14 Dynamic processing method and device for stream data, electronic equipment and storage medium

Publications (1)

Publication Number Publication Date
CN115543445A true CN115543445A (en) 2022-12-30

Family

ID=84727606

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202211117027.XA Pending CN115543445A (en) 2022-09-14 2022-09-14 Dynamic processing method and device for stream data, electronic equipment and storage medium

Country Status (1)

Country Link
CN (1) CN115543445A (en)

Similar Documents

Publication Publication Date Title
CN110245078B (en) Software pressure testing method and device, storage medium and server
CN106656536B (en) Method and equipment for processing service calling information
CN110046073B (en) Log collection method and device, equipment and storage medium
CN111522711B (en) Data monitoring processing system, method, execution end, monitoring end and electronic equipment
US11188443B2 (en) Method, apparatus and system for processing log data
CN106534242B (en) The processing method and device requested in a kind of distributed system
CN107203464B (en) Method and device for positioning service problem
CN116662875A (en) Interface mapping method and device
CN110647447A (en) Abnormal instance detection method, apparatus, device and medium for distributed system
CN113596078A (en) Service problem positioning method and device
CN112948224A (en) Data processing method, device, terminal and storage medium
CN111666193A (en) Method and system for monitoring and testing terminal function based on real-time log analysis
CN112671878B (en) Block chain information subscription method, device, server and storage medium
CN111767161A (en) Remote calling depth recognition method and device, computer equipment and readable storage medium
CN102547789B (en) Early warning method, device and system for quality of peer-to-peer service
CN115543445A (en) Dynamic processing method and device for stream data, electronic equipment and storage medium
CN113225218A (en) Method and device for checking call ticket quality
CN110928750B (en) Data processing method, device and equipment
CN111158979A (en) Service dial testing method, system, device and storage medium
CN111193631A (en) Information processing method, system, and computer-readable storage medium
CN115309638A (en) Method and device for assisting model optimization
CN111401020A (en) Interface loading method and system and computing equipment
CN114819981A (en) Customer service problem processing method, device, equipment and storage medium
CN113934616B (en) Method for judging abnormal user based on user operation time sequence
CN117132338B (en) Distributed intelligent reconciliation system based on Web3 technology

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination