CN116893952A - Data processing method, probe, acquisition logic processing unit and service - Google Patents

Data processing method, probe, acquisition logic processing unit and service Download PDF

Info

Publication number
CN116893952A
CN116893952A CN202311161159.7A CN202311161159A CN116893952A CN 116893952 A CN116893952 A CN 116893952A CN 202311161159 A CN202311161159 A CN 202311161159A CN 116893952 A CN116893952 A CN 116893952A
Authority
CN
China
Prior art keywords
data
processed
dimension
acquisition logic
processing unit
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202311161159.7A
Other languages
Chinese (zh)
Other versions
CN116893952B (en
Inventor
侯密
周玉祥
刘江帅
徐硕
齐骥
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
China Mobile Communications Group Co Ltd
China Mobile Suzhou Software Technology Co Ltd
Original Assignee
China Mobile Communications Group Co Ltd
China Mobile Suzhou Software Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by China Mobile Communications Group Co Ltd, China Mobile Suzhou Software Technology Co Ltd filed Critical China Mobile Communications Group Co Ltd
Priority to CN202311161159.7A priority Critical patent/CN116893952B/en
Publication of CN116893952A publication Critical patent/CN116893952A/en
Application granted granted Critical
Publication of CN116893952B publication Critical patent/CN116893952B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/30—Monitoring
    • G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
    • G06F11/3466—Performance evaluation by tracing or monitoring
    • G06F11/3476—Data logging
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/901—Indexing; Data structures therefor; Storage structures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/903—Querying
    • G06F16/9035—Filtering based on additional data, e.g. user or group profiles

Abstract

The invention provides a data processing method, a probe, an acquisition logic processing unit and a service, wherein the method comprises the following steps: acquiring data to be processed; the data to be processed comprises: link data and associated index data; a calling instruction is sent to the acquisition logic processing unit and used for indicating the acquisition logic processing unit to mark the dimension identification of the data to be processed according to the preset acquisition logic of the data to be processed on at least one dimension, wherein the link data and the related index data are marked with the same dimension identification; calculating the data to be processed after marking the dimension mark according to the collection logic of the data to be processed in at least one dimension; and sending the calculated data to be processed to a collector, and carrying out fault matching on the calculated data to be processed by the collector to generate a fault alarm notification. In the invention, the link data and the index data are marked with the same dimension identification, and the calculation is carried out according to the dimension identification, so that the dynamic dimension association of the link data and the index data is realized.

Description

Data processing method, probe, acquisition logic processing unit and service
Technical Field
The embodiment of the invention relates to the technical field of computer software, in particular to a data processing method, a probe, an acquisition logic processing unit and a service.
Background
In a distributed system, due to the fact that a plurality of systems with various services and interfaces are complex, faults are difficult to locate generally, and huge amounts of log records need to be checked in a log system in an artificial mode by depending on operation and maintenance and research personnel, so that the efficiency of the mode is low, and how to correlate multi-dimensional index data such as link data, resources and services in a full-link tracking system is faster and more flexible, so that the fault finding and locating speed is further improved, and the method is an important requirement in the observable field.
The monitoring index for fault discovery in the existing call chain tracking system is usually an index based on a fixed dimension, so that fault discovery can only be performed based on a specific dimension, and data analysis, fault prevention, discovery, positioning and improvement cannot be performed on a custom dimension. And even if the custom dimension association exists, the versions of the probes and the analysis service are required to be reissued, the application service of all installed probes is required to be restarted, the difficulty of communication and coordination between the application service development and operation and maintenance personnel of each access is required to be large, and the large-area restarting can influence normal on-line business, so that the influence is wide and inflexible.
Disclosure of Invention
The embodiment of the invention provides a data processing method, a probe, an acquisition logic processing unit and a service, which are used for solving the problem that the existing link data and indexes cannot be associated with dynamic dimensions and lack of flexibility.
In order to solve the technical problems, the invention is realized as follows:
in a first aspect, an embodiment of the present invention provides a data processing method, applied to a probe, including:
acquiring data to be processed; the data to be processed comprises: link data and associated index data;
a calling instruction is sent to an acquisition logic processing unit, and the calling instruction is used for indicating the acquisition logic processing unit to mark a dimension identifier on data to be processed according to preset acquisition logic of the data to be processed on at least one dimension, wherein the link data and the related index data are marked with the same dimension identifier;
receiving the data to be processed after the marking dimension identification sent by the acquisition logic processing unit and the preset acquisition logic of the data to be processed in at least one dimension;
calculating the data to be processed after marking the dimension mark according to the collection logic of the data to be processed in the at least one dimension;
Storing the calculated data to be processed into a database, and sending the calculated data to be processed to a collector, wherein the collector performs fault matching on the calculated data to be processed to generate a fault alarm notification.
Optionally, the dimension includes at least one of: users, businesses, roles, interfaces, resources, anomalies, and services.
Optionally, the method further comprises:
acquiring an acquisition logic update notification sent by an acquisition logic processing service;
sending a calling instruction to the acquisition logic processing unit according to the acquisition logic update notification to send an updated calling instruction; the updated calling instruction is used for indicating the acquisition logic processing unit to mark the dimension identification of the data to be processed according to the acquisition logic of the data to be processed in the at least one newly added dimension, wherein the link data and the related index data are marked with the same dimension identification.
Optionally, the method further comprises:
and sending the calculated data to be processed to an analysis service module, wherein the analysis service module performs analysis processing of different dimensions on the calculated data to be processed according to the preset collection logic of the data to be processed in at least one dimension.
In a second aspect, an embodiment of the present invention provides a data processing method, which is applied to an acquisition logic processing unit, including:
acquiring data to be processed and acquisition logic of the data to be processed in at least one dimension; the data to be processed comprises: link data and associated index data;
receiving and executing a call instruction sent by the probe; the calling instruction is used for indicating a dimension identifier of the data to be processed according to the preset collection logic of the data to be processed in at least one dimension, wherein the link data and the related index data are marked with the same dimension identifier;
and sending the data to be processed after marking the dimension identification and the acquisition logic of the data to be processed in at least one preset dimension to the probe.
Optionally, the method further comprises:
acquiring acquisition logic of data to be processed in at least one newly added dimension, wherein the acquisition logic processing service transmits acquisition logic update notification to the probe;
receiving and executing an updated call instruction sent by the probe; the updated calling instruction is used for indicating that dimension identification is marked on the data to be processed according to the collection logic of the data to be processed on the newly added at least one dimension, wherein the link data and the related index data are marked with the same dimension identification.
In a third aspect, an embodiment of the present invention provides a data processing method, applied to an acquisition logic processing service, including:
acquiring acquisition logic of data to be processed in at least one dimension newly added input by a user;
sending the acquisition logic of the data to be processed in the at least one dimension to an acquisition logic processing unit;
sending a collection logic update notification to the probe.
In a fourth aspect, embodiments of the present invention provide a probe comprising:
the first acquisition module is used for acquiring data to be processed; the data to be processed comprises: link data and associated index data;
the first sending module is used for sending a calling instruction to the acquisition logic processing unit, wherein the calling instruction is used for indicating the acquisition logic processing unit to mark the dimension identification of the data to be processed according to the preset acquisition logic of the data to be processed on at least one dimension, and the link data and the related index data are marked with the same dimension identification;
the first receiving module is used for receiving the data to be processed after the marking dimension identification sent by the acquisition logic processing unit and the acquisition logic of the data to be processed in at least one preset dimension;
The first processing module is used for calculating the data to be processed after marking the dimension mark according to the acquisition logic of the data to be processed in the at least one dimension;
the second processing module is used for storing the calculated data to be processed into a database, sending the calculated data to be processed to a collector, and carrying out fault matching on the calculated data to be processed by the collector to generate a fault alarm notification.
Optionally, the dimension includes at least one of: users, businesses, roles, interfaces, resources, anomalies, and services.
Optionally, the method further comprises:
the fourth acquisition module is used for acquiring an acquisition logic update notification sent by the acquisition logic processing service;
the third processing module is used for sending a calling instruction to the acquisition logic processing unit according to the acquisition logic updating notification to send an updated calling instruction; the updated calling instruction is used for indicating the acquisition logic processing unit to mark the dimension identification of the data to be processed according to the acquisition logic of the data to be processed in the at least one newly added dimension, wherein the link data and the related index data are marked with the same dimension identification.
Optionally, the method further comprises:
and the fifth sending module is used for sending the calculated data to be processed to the analysis service module, and the analysis service module performs analysis processing of different dimensions on the calculated data to be processed according to the preset collection logic of the data to be processed in at least one dimension.
In a fifth aspect, an embodiment of the present invention provides an acquisition logic processing unit, including:
the second acquisition module is used for acquiring the data to be processed and acquisition logic of the data to be processed in at least one dimension; the data to be processed comprises: link data and associated index data;
the second receiving module is used for receiving and executing the calling instruction sent by the probe; the calling instruction is used for indicating a dimension identifier of the data to be processed according to the preset collection logic of the data to be processed in at least one dimension, wherein the link data and the related index data are marked with the same dimension identifier;
and the second sending module is used for sending the data to be processed after the marking dimension identification and the acquisition logic of the data to be processed in at least one preset dimension to the probe.
Optionally, the method further comprises:
a fifth acquisition module, configured to acquire an acquisition logic of data to be processed in at least one newly added dimension sent by an acquisition logic processing service, where the acquisition logic processing service sends an acquisition logic update notification to the probe at the same time;
the third receiving module is used for receiving and executing the updated calling instruction sent by the probe; the updated calling instruction is used for indicating that dimension identification is marked on the data to be processed according to the collection logic of the data to be processed on the newly added at least one dimension, wherein the link data and the related index data are marked with the same dimension identification.
In a sixth aspect, an embodiment of the present invention provides an acquisition logic processing service, including:
the third acquisition module is used for acquiring acquisition logic of the data to be processed in at least one dimension newly added and input by a user;
the third sending module is used for sending the acquisition logic of the data to be processed in the at least one newly added dimension to the acquisition logic processing unit;
and the fourth sending module is used for sending the acquisition logic update notification to the probe.
In a seventh aspect, an embodiment of the present invention provides an electronic device, including a processor, a memory, and a program or an instruction stored on the memory and executable on the processor, where the program or the instruction when executed by the processor implement the steps in the method for processing data according to any one of the first aspect, or implement the method for processing data according to any one of the second aspect, or implement the steps in the method for processing data according to any one of the third aspect.
In an eighth aspect, an embodiment of the present invention provides a readable storage medium, on which a program or an instruction is stored, which when executed by a processor, implements the steps in the method for processing data according to any one of the first aspect, or the method for processing data according to any one of the second aspect, or the method for processing data according to any one of the third aspect.
According to the invention, dynamic association of link index data based on specified dimensions is realized based on the acquisition logic processing unit, when new dimension identification is needed, the link data and the index data are marked with new dimension identification through the new acquisition logic processing unit capable of generating the new dimension identification, and the association of the link data and the index data is realized based on the new dimension identification by the probe, so that the problems that the existing link data and the index cannot be associated in dynamic dimensions and the flexibility is lacking are solved.
Drawings
Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention. Also, like reference numerals are used to designate like parts throughout the figures. In the drawings:
FIG. 1 is a flow chart of a data processing method applied to a probe according to an embodiment of the present invention;
FIG. 2 is a diagram of an overall architecture of a data processing method according to an embodiment of the present invention;
FIG. 3 is a flowchart of a data processing method applied to an acquisition logic processing unit according to an embodiment of the present invention;
FIG. 4 is a flowchart of a data processing method applied to an acquisition logic processing service according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of a probe according to an embodiment of the present invention;
FIG. 6 is a schematic diagram of a logic processing unit for acquisition according to an embodiment of the present invention;
FIG. 7 is a schematic diagram of a structure of an acquisition logic processing service according to an embodiment of the present invention;
fig. 8 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are some, but not all embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Referring to fig. 1, an embodiment of the present invention provides a data processing method, which is applied to a probe, and includes:
step 11: acquiring data to be processed; the data to be processed comprises: link data and associated index data;
in the embodiment of the present invention, the link data is original data for recording call between services, and needs to be specified and configured in advance before the system operates, for example, if call link data between service a and service B needs to be acquired, probes are set on service a and service B before call starts; the index data is data represented when the system operates, for example, a user calls a service 100 times within 1 minute, and the link data corresponding to the index data can be reversely found based on the index data.
Referring to fig. 2, in the embodiment of the present invention, the probe starts with the application service and pulls all the acquisition logic processing units from the acquisition logic processing service; the probe collects call link data from the application service to the application service interior.
Step 12: a calling instruction is sent to an acquisition logic processing unit, and the calling instruction is used for indicating the acquisition logic processing unit to mark a dimension identifier on data to be processed according to preset acquisition logic of the data to be processed on at least one dimension, wherein the link data and the related index data are marked with the same dimension identifier;
In the embodiment of the invention, a probe sends a calling instruction to an acquisition logic processing unit, and the acquisition logic processing unit marks the dimension identification of data to be processed according to the preset acquisition logic of the data to be processed on at least one dimension, wherein the link data and the related index data are marked with the same dimension identification; the collection logic processing unit is configured to perform a custom process on link data and index data collected by the probe from the application service, for example, marking a certain dimension identifier, for example: user dimension identification and identity identification of a specific user; the link data and the index data share the same dimension identification marked by the acquisition logic processing unit through the acquisition logic processing unit, and then the link data and the index data are associated based on the dimension identification through a probe.
Step 13: receiving the data to be processed after the marking dimension identification sent by the acquisition logic processing unit and the preset acquisition logic of the data to be processed in at least one dimension;
step 14: calculating the data to be processed after marking the dimension mark according to the collection logic of the data to be processed in the at least one dimension;
In the embodiment of the invention, the probe performs aggregation calculation of the related indexes at fixed time intervals or at fixed time intervals, and performs calculation of the designated dimension according to the logic provided by the acquisition logic processing unit.
Step 15: storing the calculated data to be processed into a database, and sending the calculated data to be processed to a collector, wherein the collector performs fault matching on the calculated data to be processed to generate a fault alarm notification.
Referring to fig. 2, in the embodiment of the present invention, a probe sends the link data and the related index data after the acquisition and processing to a collector and stores the link data and the related index data in a database; the related operation and maintenance personnel and/or research personnel can conduct regular configuration aiming at index data of specified dimensions, the collector conducts fault matching on the calculated data to be processed, when the calculated data to be processed is matched with an alarm rule, the collector triggers to generate fault alarm and informs related personnel, index data of a certain dimension which is abnormal are obtained firstly in the fault matching positioning process, link data associated with the dimension identification can be inquired according to the dimension identification of the index data, and a specific fault link is obtained according to the link data. Different roles of personnel enter the system from different dimension angles (such as operation and maintenance personnel enter from a resource dimension angle, operation personnel enter from a service dimension angle, and development personnel enter from a service dimension angle), observe analysis scenes of different dimensions and enter detailed link data scenes according to associated dimension identifications to perform fault positioning.
In the embodiment of the invention, the analysis service module can also call the link data and the related index data from the database or the collector and analyze the index data of the appointed dimension by using the acquisition logic of the data to be processed in the acquisition logic processing unit.
In the embodiment of the invention, the dynamic association of the link index data based on the designated dimension is realized based on the acquisition logic processing unit, when the new dimension identification is needed, the new dimension identification is marked on the link data and the index data through the new acquisition logic processing unit capable of generating the new dimension identification, and the association of the link data and the index data is realized based on the new dimension identification by the probe, so that the problems that the existing link data and the index cannot be associated in the dynamic dimension and the flexibility is lacked are solved.
In an embodiment of the present invention, optionally, the dimension includes at least one of: users, businesses, roles, interfaces, resources, anomalies, and services.
In the embodiment of the invention, for example: the dimensions that the acquisition logic processing unit can process include: class a users, class B services, class C roles, class D interfaces, class E resources, class F anomalies, and class G services.
In an embodiment of the present invention, optionally, the method further includes:
Acquiring an acquisition logic update notification sent by an acquisition logic processing service;
sending a calling instruction to the acquisition logic processing unit according to the acquisition logic update notification to send an updated calling instruction; the updated calling instruction is used for indicating the acquisition logic processing unit to mark the dimension identification of the data to be processed according to the acquisition logic of the data to be processed in the at least one newly added dimension, wherein the link data and the related index data are marked with the same dimension identification.
In the embodiment of the invention, if the dimension acquisition and calculation requirement is newly increased, a new acquisition logic processing unit needs to be developed and uploaded to the acquisition logic processing service in the form of a file packet, the acquisition logic processing service tells the probe of the identification of the new acquisition logic processing unit in the form of an acquisition logic update notification, and after receiving the acquisition logic update notification, the probe pulls the corresponding acquisition logic processing unit packet by the identification of the new acquisition logic processing unit and loads the acquisition logic processing unit packet into the probe so as to be convenient for calling, and the acquisition logic of the newly increased acquisition logic processing unit can be used in the subsequent acquisition and calculation.
In an embodiment of the present invention, optionally, the method further includes:
and sending the calculated data to be processed to an analysis service module, wherein the analysis service module performs analysis processing of different dimensions on the calculated data to be processed according to the preset collection logic of the data to be processed in at least one dimension.
In the embodiment of the present invention, the analysis service module is configured to perform analysis of different dimensions on the index data and perform rule configuration based on the analyzed data, for example: alarming rules; and performing other processing on the data conforming to the set rule, such as: early warning; the analysis service module pulls the analysis logic processing unit which is dynamically upgraded according to the new acquisition logic processing unit, and performs index analysis of the dimension according to the data processed by the dimension processing logic of the newly added acquisition logic processing unit.
Referring to fig. 3, an embodiment of the present invention provides a data processing method, which is applied to an acquisition logic processing unit, and includes:
step 31: acquiring data to be processed and acquisition logic of the data to be processed in at least one dimension; the data to be processed comprises: link data and associated index data;
Referring to fig. 2, in the embodiment of the present invention, the collection logic processing unit is configured to perform custom processing on link data and index data from the application service set to the probe, for example: a certain dimension identification is marked.
Step 32: receiving and executing a call instruction sent by the probe; the calling instruction is used for indicating a dimension identifier of the data to be processed according to the preset collection logic of the data to be processed in at least one dimension, wherein the link data and the related index data are marked with the same dimension identifier;
in the embodiment of the invention, a probe sends a calling instruction to an acquisition logic processing unit, and the acquisition logic processing unit marks the dimension identification of data to be processed according to the preset acquisition logic of the data to be processed on at least one dimension, wherein the link data and the related index data are marked with the same dimension identification; the link data and the index data share the same dimension identification marked by the acquisition logic processing unit through the acquisition logic processing unit, and then the link data and the index data are associated based on the dimension identification through a probe.
Step 33: and sending the data to be processed after marking the dimension identification and the acquisition logic of the data to be processed in at least one preset dimension to the probe.
In the embodiment of the invention, the dynamic association of the link index data based on the designated dimension is realized based on the acquisition logic processing unit, when the new dimension identification is needed, the new dimension identification is marked on the link data and the index data through the new acquisition logic processing unit capable of generating the new dimension identification, and the association of the link data and the index data is realized based on the new dimension identification by the probe, so that the problems that the existing link data and the index cannot be associated in the dynamic dimension and the flexibility is lacked are solved.
In an embodiment of the present invention, optionally, the method further includes:
acquiring acquisition logic of data to be processed in at least one newly added dimension, wherein the acquisition logic processing service transmits acquisition logic update notification to the probe;
receiving and executing an updated call instruction sent by the probe; the updated calling instruction is used for indicating that dimension identification is marked on the data to be processed according to the collection logic of the data to be processed on the newly added at least one dimension, wherein the link data and the related index data are marked with the same dimension identification.
In the embodiment of the invention, if the dimension acquisition and calculation requirement is newly increased, a new acquisition logic processing unit needs to be developed and uploaded to the acquisition logic processing service in the form of a file packet, the acquisition logic processing service tells the probe of the identification of the new acquisition logic processing unit in the form of an acquisition logic update notification, and after receiving the acquisition logic update notification, the probe pulls the corresponding acquisition logic processing unit packet by the identification of the new acquisition logic processing unit and loads the acquisition logic processing unit packet into the probe so as to be convenient for calling, and the acquisition logic of the newly increased acquisition logic processing unit can be used in the subsequent acquisition and calculation.
In the embodiment of the present invention, the analysis service module is configured to perform analysis of different dimensions on the index data and perform rule configuration based on the analyzed data, for example: alarming rules; and performing other processing on the data conforming to the set rule, such as: early warning; the analysis service module pulls the analysis logic processing unit which is dynamically upgraded according to the new acquisition logic processing unit, and performs index analysis of the dimension according to the data processed by the dimension processing logic of the newly added acquisition logic processing unit.
Referring to fig. 4, an embodiment of the present invention provides a data processing method applied to an acquisition logic processing service, including:
step 41: acquiring acquisition logic of data to be processed in at least one dimension newly added input by a user;
in the embodiment of the invention, the acquisition logic processing service is used for providing the acquisition logic processing unit with the function operations of dynamic addition, upgrading and the like.
Step 42: sending the acquisition logic of the data to be processed in the at least one dimension to an acquisition logic processing unit;
in the embodiment of the invention, if the dimension acquisition and calculation requirement is newly increased, a new acquisition logic processing unit is developed and uploaded to the acquisition logic processing service in the form of a file package.
Step 43: sending a collection logic update notification to the probe.
In the embodiment of the invention, the acquisition logic processing service tells the probe of the identification of the new acquisition logic processing unit in the form of an acquisition logic update notification, and after the probe receives the acquisition logic update notification, the probe pulls the corresponding acquisition logic processing unit packet by the identification of the new acquisition logic processing unit and loads the acquisition logic processing unit packet into the probe so as to be convenient for calling, and the acquisition logic of the newly added acquisition logic processing unit can be used in the subsequent acquisition and calculation.
In the embodiment of the present invention, the analysis service module is configured to perform analysis of different dimensions on the index data and perform rule configuration based on the analyzed data, for example: alarming rules; and performing other processing on the data conforming to the set rule, such as: early warning; the analysis service module pulls the analysis logic processing unit which is dynamically upgraded according to the new acquisition logic processing unit, and performs index analysis of the dimension according to the data processed by the dimension processing logic of the newly added acquisition logic processing unit.
In the embodiment of the invention, the dynamic association of the link index data based on the designated dimension is realized based on the acquisition logic processing unit, when the new dimension identification is needed, the new dimension identification is marked on the link data and the index data through the new acquisition logic processing unit capable of generating the new dimension identification, and the association of the link data and the index data is realized based on the new dimension identification by the probe, so that the problems that the existing link data and the index cannot be associated in the dynamic dimension and the flexibility is lacked are solved.
Referring to fig. 2, in the embodiment of the present invention, the overall flow is as follows:
the probe starts along with the application service and pulls all the acquisition logic processing units from the acquisition logic processing service; the probe collects call link data from the application service to the inside of the application service;
The probe sends a calling instruction to an acquisition logic processing unit, and marks dimension identification on data to be processed according to preset acquisition logic of the data to be processed on at least one dimension by the acquisition logic processing unit, wherein the link data and the related index data are marked with the same dimension identification; the collection logic processing unit is configured to perform a custom process on link data and index data collected by the probe from the application service, for example, marking a certain dimension identifier, for example: user dimension identification and identity identification of a specific user; the link data and the index data share the same dimension identification marked by the acquisition logic processing unit through the acquisition logic processing unit, and then the association of the link data and the index data is realized based on the dimension identification through a probe;
the probe performs aggregation calculation of related indexes at fixed time intervals or at fixed time intervals, and performs calculation of appointed dimensions according to logic provided by the acquisition logic processing unit;
the probe sends the link data and the related index data after the acquisition and processing to a collector and stores the link data and the related index data in a database;
The related operation and maintenance personnel and/or research personnel can conduct regular configuration aiming at index data of specified dimensions, the collector conducts fault matching on the calculated data to be processed, when the calculated data to be processed is matched with an alarm rule, the collector triggers to generate fault alarm and informs related personnel, index data of a certain dimension which is abnormal are obtained firstly in the fault matching positioning process, link data associated with the dimension identification can be inquired according to the dimension identification of the index data, and a specific fault link is obtained according to the link data. Different roles of personnel enter the system from different dimension angles (such as operation and maintenance personnel enter from a resource dimension angle, operation personnel enter from a service dimension angle, and development personnel enter from a service dimension angle), observe analysis scenes of different dimensions and enter detailed link data scenes according to associated dimension identifications to perform fault positioning;
if the dimension acquisition and calculation requirement is newly increased, a new acquisition logic processing unit needs to be developed and uploaded to an acquisition logic processing service in the form of a file packet, the acquisition logic processing service tells the probe of the identification of the new acquisition logic processing unit in the form of an acquisition logic update notification, the probe receives the acquisition logic update notification and pulls the corresponding acquisition logic processing unit packet by the identification of the new acquisition logic processing unit and loads the acquisition logic processing unit packet into the probe so as to be convenient for calling, and the acquisition logic of the new acquisition logic processing unit can be used in the subsequent acquisition and calculation;
The analysis service module analyzes the index data in different dimensions and performs rule configuration based on the analyzed data, such as: alarming rules; and performing other processing on the data conforming to the set rule, such as: early warning; the analysis service module pulls the analysis logic processing unit which is dynamically upgraded according to the new acquisition logic processing unit, and performs index analysis of the dimension according to the data processed by the dimension processing logic of the newly added acquisition logic processing unit.
Referring to fig. 5, an embodiment of the present invention provides a probe, including:
a first obtaining module 51, configured to obtain data to be processed; the data to be processed comprises: link data and associated index data;
a first sending module 52, configured to send a call instruction to an acquisition logic processing unit, where the call instruction is configured to instruct the acquisition logic processing unit to mark a dimension identifier for data to be processed according to preset acquisition logic of the data to be processed in at least one dimension, where the link data and the related index data are marked with the same dimension identifier;
the first receiving module 53 is configured to receive the data to be processed after the marking dimension identifier sent by the collection logic processing unit and collection logic of the data to be processed in the preset at least one dimension;
The first processing module 54 is configured to calculate, according to the collection logic of the data to be processed in the at least one dimension, the data to be processed after marking the dimension identifier;
the second processing module 55 is configured to store the calculated data to be processed in a database, and send the calculated data to be processed to a collector, where the collector performs fault matching on the calculated data to be processed, so as to generate a fault alarm notification.
In an embodiment of the present invention, optionally, the dimension includes at least one of: users, businesses, roles, interfaces, resources, anomalies, and services.
In an embodiment of the present invention, optionally, the method further includes:
the fourth acquisition module is used for acquiring an acquisition logic update notification sent by the acquisition logic processing service;
the third processing module is used for sending a calling instruction to the acquisition logic processing unit according to the acquisition logic updating notification to send an updated calling instruction; the updated calling instruction is used for indicating the acquisition logic processing unit to mark the dimension identification of the data to be processed according to the acquisition logic of the data to be processed in the at least one newly added dimension, wherein the link data and the related index data are marked with the same dimension identification.
In an embodiment of the present invention, optionally, the method further includes:
and the fifth sending module is used for sending the calculated data to be processed to the analysis service module, and the analysis service module performs analysis processing of different dimensions on the calculated data to be processed according to the preset collection logic of the data to be processed in at least one dimension.
The probe provided by the embodiment of the invention can realize each process realized by the embodiment of the method of fig. 1 and achieve the same technical effect, and in order to avoid repetition, the description is omitted here.
Referring to fig. 6, an embodiment of the present invention provides an acquisition logic processing unit, including:
a second acquiring module 61, configured to acquire data to be processed and preset acquisition logic of the data to be processed in at least one dimension; the data to be processed comprises: link data and associated index data;
a second receiving module 62, configured to receive and execute a call instruction sent by the probe; the calling instruction is used for indicating a dimension identifier of the data to be processed according to the preset collection logic of the data to be processed in at least one dimension, wherein the link data and the related index data are marked with the same dimension identifier;
And the second sending module 63 is configured to send, to the probe, the data to be processed after the marking of the dimension identifier and the logic for acquiring the data to be processed in the preset at least one dimension.
In an embodiment of the present invention, optionally, the method further includes:
a fifth acquisition module, configured to acquire an acquisition logic of data to be processed in at least one newly added dimension sent by an acquisition logic processing service, where the acquisition logic processing service sends an acquisition logic update notification to the probe at the same time;
the third receiving module is used for receiving and executing the updated calling instruction sent by the probe; the updated calling instruction is used for indicating that dimension identification is marked on the data to be processed according to the collection logic of the data to be processed on the newly added at least one dimension, wherein the link data and the related index data are marked with the same dimension identification.
The acquisition logic processing unit provided by the embodiment of the present invention can implement each process implemented by the method embodiment of fig. 3, and achieve the same technical effects, and for avoiding repetition, a detailed description is omitted herein.
Referring to fig. 7, an embodiment of the present invention provides an acquisition logic processing service, including:
A third obtaining module 71, configured to obtain a logic for collecting data to be processed in at least one dimension newly added, which is input by a user;
a third sending module 72, configured to send, to an acquisition logic processing unit, the acquisition logic of the data to be processed in the at least one dimension that is newly added;
a fourth sending module 73, configured to send an acquisition logic update notification to the probe.
The acquisition logic processing service provided by the embodiment of the present invention can implement each process implemented by the method embodiment of fig. 4, and achieve the same technical effects, and for avoiding repetition, a detailed description is omitted herein.
An embodiment of the present invention provides an electronic device 80, referring to fig. 8, and fig. 8 is a schematic block diagram of the electronic device 80 according to an embodiment of the present invention, including a processor 81, a memory 82, and a program or an instruction stored in the memory X2 and executable on the processor 81, where the program or the instruction implements steps in a method for processing any data according to the present invention when executed by the processor.
The embodiment of the invention provides a readable storage medium, on which a program or an instruction is stored, where the program or the instruction, when executed by a processor, implements each process of the embodiment of the data processing method according to any one of the above embodiments, and can achieve the same technical effect, so that repetition is avoided, and no further description is given here.
Computer-readable media include both permanent and non-permanent, removable and non-removable media, and information storage may be implemented by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
The foregoing embodiment numbers of the present application are merely for the purpose of description, and do not represent the advantages or disadvantages of the embodiments.
From the above description of the embodiments, it will be clear to those skilled in the art that the above-described embodiment method may be implemented by means of software plus a necessary general hardware platform, but of course may also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (e.g. ROM/RAM, magnetic disk, optical disk) comprising instructions for causing a service classification device (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to perform the method according to the embodiments of the present application.
The foregoing is merely a preferred embodiment of the present application and it should be noted that modifications and adaptations to those skilled in the art may be made without departing from the principles of the present application, which are intended to be comprehended within the scope of the present application.

Claims (12)

1. A method of processing data applied to a probe, comprising:
Acquiring data to be processed; the data to be processed comprises: link data and associated index data;
a calling instruction is sent to an acquisition logic processing unit, and the calling instruction is used for indicating the acquisition logic processing unit to mark a dimension identifier on data to be processed according to preset acquisition logic of the data to be processed on at least one dimension, wherein the link data and the related index data are marked with the same dimension identifier;
receiving the data to be processed after the marking dimension identification sent by the acquisition logic processing unit and the preset acquisition logic of the data to be processed in at least one dimension;
calculating the data to be processed after marking the dimension mark according to the collection logic of the data to be processed in the at least one dimension;
storing the calculated data to be processed into a database, and sending the calculated data to be processed to a collector, wherein the collector performs fault matching on the calculated data to be processed to generate a fault alarm notification.
2. The method for processing data according to claim 1, wherein,
the dimensions include at least one of: users, businesses, roles, interfaces, resources, anomalies, and services.
3. The method for processing data according to claim 1, further comprising:
acquiring an acquisition logic update notification sent by an acquisition logic processing service;
sending a calling instruction to the acquisition logic processing unit according to the acquisition logic update notification to send an updated calling instruction; the updated calling instruction is used for indicating the acquisition logic processing unit to mark the dimension identification of the data to be processed according to the acquisition logic of the data to be processed in at least one newly added dimension, wherein the link data and the related index data are marked with the same dimension identification.
4. The method for processing data according to claim 1, further comprising:
and sending the calculated data to be processed to an analysis service module, wherein the analysis service module performs analysis processing of different dimensions on the calculated data to be processed according to the preset collection logic of the data to be processed in at least one dimension.
5. A method for processing data, applied to an acquisition logic processing unit, comprising:
acquiring data to be processed and acquisition logic of the data to be processed in at least one dimension; the data to be processed comprises: link data and associated index data;
Receiving and executing a call instruction sent by the probe; the calling instruction is used for indicating a dimension identifier of the data to be processed according to the preset collection logic of the data to be processed in at least one dimension, wherein the link data and the related index data are marked with the same dimension identifier;
and sending the data to be processed after marking the dimension identification and the acquisition logic of the data to be processed in at least one preset dimension to the probe.
6. The method for processing data according to claim 5, further comprising:
acquiring acquisition logic of data to be processed in at least one newly added dimension, wherein the acquisition logic processing service transmits acquisition logic update notification to the probe;
receiving and executing an updated call instruction sent by the probe; the updated calling instruction is used for indicating that dimension identification is marked on the data to be processed according to the collection logic of the data to be processed on the newly added at least one dimension, wherein the link data and the related index data are marked with the same dimension identification.
7. A method for processing data, applied to an acquisition logic processing service, comprising:
Acquiring acquisition logic of data to be processed in at least one dimension newly added input by a user;
sending the acquisition logic of the data to be processed in the at least one dimension to an acquisition logic processing unit;
sending a collection logic update notification to the probe.
8. A probe, comprising:
the first acquisition module is used for acquiring data to be processed; the data to be processed comprises: link data and associated index data;
the first sending module is used for sending a calling instruction to the acquisition logic processing unit, wherein the calling instruction is used for indicating the acquisition logic processing unit to mark the dimension identification of the data to be processed according to the preset acquisition logic of the data to be processed on at least one dimension, and the link data and the related index data are marked with the same dimension identification;
the first receiving module is used for receiving the data to be processed after the marking dimension identification sent by the acquisition logic processing unit and the acquisition logic of the data to be processed in at least one preset dimension;
the first processing module is used for calculating the data to be processed after marking the dimension mark according to the acquisition logic of the data to be processed in the at least one dimension;
The second processing module is used for storing the calculated data to be processed into a database, sending the calculated data to be processed to a collector, and carrying out fault matching on the calculated data to be processed by the collector to generate a fault alarm notification.
9. An acquisition logic processing unit, comprising:
the second acquisition module is used for acquiring the data to be processed and acquisition logic of the data to be processed in at least one dimension; the data to be processed comprises: link data and associated index data;
the second receiving module is used for receiving and executing the calling instruction sent by the probe; the calling instruction is used for indicating a dimension identifier of the data to be processed according to the preset collection logic of the data to be processed in at least one dimension, wherein the link data and the related index data are marked with the same dimension identifier;
and the second sending module is used for sending the data to be processed after the marking dimension identification and the acquisition logic of the data to be processed in at least one preset dimension to the probe.
10. An acquisition logic processing service, comprising:
The third acquisition module is used for acquiring acquisition logic of the data to be processed in at least one dimension newly added and input by a user;
the third sending module is used for sending the acquisition logic of the data to be processed in the at least one newly added dimension to the acquisition logic processing unit;
and the fourth sending module is used for sending the acquisition logic update notification to the probe.
11. An electronic device, characterized in that: comprising a processor, a memory and a program or instruction stored on the memory and executable on the processor, which program or instruction when executed by the processor performs the steps in the method of processing data according to any one of claims 1 to 4, or the method of processing data according to any one of claims 5 or 6, or the method of processing data according to any one of claim 7.
12. A readable storage medium, characterized by: the readable storage medium has stored thereon a program or instructions which, when executed by a processor, implement the method of processing data according to any one of claims 1 to 4, or implement the method of processing data according to any one of claims 5 or 6, or implement the steps of the method of processing data according to any one of claim 7.
CN202311161159.7A 2023-09-11 2023-09-11 Data processing method, probe, acquisition logic processing unit and service Active CN116893952B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202311161159.7A CN116893952B (en) 2023-09-11 2023-09-11 Data processing method, probe, acquisition logic processing unit and service

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202311161159.7A CN116893952B (en) 2023-09-11 2023-09-11 Data processing method, probe, acquisition logic processing unit and service

Publications (2)

Publication Number Publication Date
CN116893952A true CN116893952A (en) 2023-10-17
CN116893952B CN116893952B (en) 2024-01-26

Family

ID=88315200

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202311161159.7A Active CN116893952B (en) 2023-09-11 2023-09-11 Data processing method, probe, acquisition logic processing unit and service

Country Status (1)

Country Link
CN (1) CN116893952B (en)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030061232A1 (en) * 2001-09-21 2003-03-27 Dun & Bradstreet Inc. Method and system for processing business data
US20120041938A1 (en) * 2010-08-12 2012-02-16 BrightEdge Technologies Operationalizing search engine optimization
CN113282623A (en) * 2021-05-20 2021-08-20 浙江网商银行股份有限公司 Data processing method and device
CN114780328A (en) * 2021-12-22 2022-07-22 中国建设银行股份有限公司 Data processing method and service management and control system

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030061232A1 (en) * 2001-09-21 2003-03-27 Dun & Bradstreet Inc. Method and system for processing business data
US20120041938A1 (en) * 2010-08-12 2012-02-16 BrightEdge Technologies Operationalizing search engine optimization
CN113282623A (en) * 2021-05-20 2021-08-20 浙江网商银行股份有限公司 Data processing method and device
CN114780328A (en) * 2021-12-22 2022-07-22 中国建设银行股份有限公司 Data processing method and service management and control system

Also Published As

Publication number Publication date
CN116893952B (en) 2024-01-26

Similar Documents

Publication Publication Date Title
WO2019179248A1 (en) Anomaly detection method and device
CN111478889B (en) Alarm method and device
CN111897705B (en) Service state processing and model training method, device, equipment and storage medium
CN110572764A (en) Remote fault diagnosis method and device for vehicle-mounted audio equipment
CN111314137A (en) Information communication network automation operation and maintenance method, device, storage medium and processor
CN111092752A (en) Fault positioning method and device spanning multiple network slices
CN112102137A (en) Policy data processing method, device and storage medium
CN116893952B (en) Data processing method, probe, acquisition logic processing unit and service
CN111143151B (en) Service monitoring method and device and electronic equipment
CN112087320A (en) Abnormity positioning method and device, electronic equipment and readable storage medium
CN116719750A (en) Software testing method and device, server equipment and storage medium
CN116566873A (en) ELK-based automatic log analysis method, system and storage medium
CN114500249B (en) Root cause positioning method and device
CN116527553A (en) Processing method, system and storage medium for automatic test report of switch
CN115658441A (en) Method, equipment and medium for monitoring abnormality of household service system based on log
CN112966056B (en) Information processing method, device, equipment, system and readable storage medium
CN111045915A (en) Safety test method and device based on product function test case
CN112286792A (en) Interface testing method, device, equipment and storage medium
CN114092246A (en) Problem positioning method and device for financial transaction link
CN116302095A (en) Instruction jump judging method and device, electronic equipment and readable storage medium
CN115801557A (en) Fault root cause positioning method and device and readable storage medium
CN112231185A (en) Knowledge acquisition method and device based on alarm information of application system
CN117592950B (en) Inspection detection management platform and data processing method thereof
CN117632666B (en) Alarm method, equipment and storage medium
CN113778800B (en) Error information processing method, device, system, equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant