CN109861838B - Method and system for generating service topology and monitoring service based on stream computing - Google Patents
Method and system for generating service topology and monitoring service based on stream computing Download PDFInfo
- Publication number
- CN109861838B CN109861838B CN201711238278.2A CN201711238278A CN109861838B CN 109861838 B CN109861838 B CN 109861838B CN 201711238278 A CN201711238278 A CN 201711238278A CN 109861838 B CN109861838 B CN 109861838B
- Authority
- CN
- China
- Prior art keywords
- service
- index value
- minute
- physical
- time
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Landscapes
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention discloses a method and a system for generating service topology and monitoring service based on stream computing, wherein the method comprises the following steps: receiving a source data stream, and generating incidence relations of all service levels according to the source data stream; receiving the logical and physical names and the association relation of all service levels, storing the logical and physical names and the association relation in the RDBMS, and generating logical and physical service data of all service levels; and receiving logical and physical service data of all service levels, calculating three time index values of year, day and minute and a modulo result value of the minute index value, and calculating a monitoring index value in a corresponding minute slot. According to the scheme, the monitoring index value in the time slot is calculated in a quasi-real-time mode through stream type calculation, and a service topological relation facing to a service and technology view is established by defining the corresponding relation between the logic/physical service and the service instance, so that operation and maintenance personnel can monitor the running state of the complex telecommunication system from the service and technology view.
Description
Technical Field
The invention relates to the field of telecommunication service monitoring, in particular to a method and a system for generating service topology and service monitoring based on stream computing.
Background
Because the telecommunication system has large traffic and a large number of application services, and the application services mostly adopt a cluster deployment scheme, the prior art mostly uses a mode of independent monitoring of each service, and the operation state of the whole system is difficult to be monitored in a global quasi-real time manner from the perspective of business and technology.
Disclosure of Invention
In order to solve the above-mentioned drawbacks of the prior art, an object of the present invention is to provide a method and a system for generating a service topology and service monitoring based on streaming computing.
In order to achieve the purpose, the technical scheme of the invention is as follows:
a method for generating service topology and service monitoring based on stream computing comprises the following steps:
receiving a source data stream, and generating incidence relations of all service levels according to the source data stream;
receiving the logical and physical names and the association relation of all service levels, storing the logical and physical names and the association relation in the RDBMS, and generating logical and physical service data of all service levels;
and receiving logical and physical service data of all service levels, calculating three time index values of year, day and minute and a modulo result value of the minute index value, and calculating a monitoring index value in a corresponding minute slot.
Further, the step of receiving the source data stream and generating the association relationship of all service levels according to the source data stream includes,
extracting Span data items from a source data stream;
identifying whether the Span data item contains the logical service names and the physical service names of all service levels;
if yes, generating the association relation of all service levels;
if not, the logical service name and the physical service name of the missing service level are configured, and the association relationship of all the service levels is generated.
Further, the step of receiving the logical and physical names and the association relation of all the service levels and storing the same in the RDBMS, and generating the logical and physical service data of all the service levels includes,
setting the updating time of the logic and physical service data, and judging whether the updating time is up;
and if the update time is up, storing the logical and physical names and the association relation of the service hierarchy into the RDBMS.
Further, the step of receiving logical and physical service data of all service levels, calculating three time index values of year, day and minute and a modulo result value of the minute index value, calculating a monitoring index value in a corresponding minute slot includes,
calculating three time index values of year, day and minute and a modulus result value of the minute index value according to the received time stamps of the logical service data and the physical service data;
comparing the calculated time index values with the time index values of the three time slots respectively, and judging whether the time index values are the same or not;
if the calculated time index value is equal to the time index value corresponding to any time slot, taking a module by the minute index value;
and if the calculated time index value is larger than the index value of the maximum time slot, storing the monitoring index value of the minute slot corresponding to the serial number into the database, and reinitializing the minute slot of the serial number.
Further, the step of modulo the minute index value includes,
and identifying a modulus result value, and accumulating and calculating the monitoring index value.
A system for generating service topology and service monitoring based on streaming computing, comprising:
the service analysis generator is used for receiving the source data stream and generating the incidence relation of all service levels according to the source data stream;
the service data memory is used for receiving the logical and physical names and the association relation of all the service levels, storing the logical and physical names and the association relation into the RDBMS and generating the logical and physical service data of all the service levels;
and the monitoring index value calculator and the memory are used for receiving the logic and physical service data of all the service levels, calculating three time index values of year, day and minute and a modulus result value of the minute index value, and calculating the monitoring index value in the corresponding minute slot.
Further, the service analysis generator comprises an extraction module, a recognition module and a generation module,
the extraction module is used for extracting Span data items from the source data stream;
the identification module is used for identifying whether the Span data item contains the logical service names and the physical service names of all service levels;
and the generating module generates the incidence relation of all service levels if the identifying module judges that the service levels contain the logical service names and the physical service names of the missing service levels, and generates the incidence relation of all service levels if the identifying module judges that the service levels do not contain the logical service names and the physical service names of the missing service levels.
Furthermore, the service data memory comprises a timing updating module, which is used for setting the updating time of the logic and physical service data and judging whether the updating time is up; and if the update time is up, storing the logical and physical names and the association relation of the service hierarchy into the RDBMS.
Further, the monitoring index value calculator and the storage comprise a calculation module, a comparison module, a modulus taking module and an initialization module,
the calculation module is used for calculating three time index values of year, day and minute and a modulus result value of the minute index value according to the received timestamp of the logic and physical service data;
the comparison module is used for comparing the calculated time index values with the time index values of the three time slots respectively and judging whether the time index values are the same or not;
the module taking module is used for taking the module by the minute index value after the calculated time index value is equal to the time index value corresponding to any time slot;
and the initialization module is used for storing the monitoring index value in the minute slot corresponding to the serial number into the database and re-initializing the minute slot with the serial number after the calculated time index value is larger than the index value of the maximum time slot.
Further, the module taking module is configured to identify a module taking result value, and accumulate and calculate the monitoring index value.
The invention has the beneficial effects that: the scheme calculates the monitoring index value in a time slot in a quasi-real-time manner through stream type calculation, and establishes a service topological relation facing to a service and technical view angle by defining the corresponding relation between the logic/physical service and the service instance, so that operation and maintenance personnel can globally monitor the running state of the complex telecommunication system from the service and technical view angle; and the monitoring index value is calculated in a time slot mode, so that the accuracy of the monitoring index value is higher, and the monitoring index value is limited in a small storage space.
Drawings
Fig. 1 is a flowchart of a method for generating a service topology and monitoring a service based on streaming computing according to an embodiment of the present invention;
FIG. 2 is a flowchart of a method for generating associations of all service levels according to an embodiment of the present invention;
FIG. 3 is a flowchart of a method for updating logical and physical service data periodically according to an embodiment of the present invention;
FIG. 4 is a flowchart illustrating a method for monitoring an indicator calculation method according to an embodiment of the present invention;
FIG. 5 is a flowchart illustrating a method for monitoring an indicator calculation method according to another embodiment of the present invention;
FIG. 6 is a diagram illustrating exemplary monitoring indicator values in a time slot according to an embodiment of the present invention;
FIG. 7 is an architecture diagram of a system for generating service topology and service monitoring based on streaming computing according to an embodiment of the present invention;
FIG. 8 is a logic flow diagram of a system for generating service topology and service monitoring based on streaming computing in accordance with an embodiment of the present invention;
FIG. 9 is a block diagram of a service analyzer according to an embodiment of the present invention;
FIG. 10 is a block diagram of a service data storage according to an embodiment of the present invention;
FIG. 11 is a block diagram of a monitoring indicator calculator and a memory according to an embodiment of the invention.
Detailed Description
For the purpose of illustrating the spirit and objects of the present invention, the present invention will be further described with reference to the accompanying drawings and specific embodiments.
RDBMS, english full name: relative database management system, Chinese name: a relational database management system. The RDBMS stores and manages data through a data model consisting of data, relationships, and constraints on the data.
Real-time calculation, emphasis is on real-time. For example, if you want to see his total consumption in the last year, then when you hit the statistics button, the server cluster will snap to the score and must get the result within the time that you can tolerate. The calculation is realized behind the background, redundancy and various high-performance components are generally used as supports, the algorithm is optimized for real-time performance, but the real-time calculation does not call the algorithm by force, so long as the high real-time performance can be ensured. The biggest difference between real-time calculation and offline calculation is that the offline calculation is calculation within unbearable time of a person, so that the person does not need to wait and the task is lost to a computer.
Streaming computing is somewhat more sluggish than real-time computing, but is more real-time than off-line computing, and the main emphasis is on the computational method. For example, the server side has a value of recording the number of the small order. When buying every item, the server end immediately sends out a successful transaction event, and the value is added with 1 immediately after receiving the event. If the calculation is performed in an off-line mode, the estimation is that all orders which are small and clear are taken out from the low-speed storage and the number is counted only in a slow-flight mode when the calculation is carried out. Streaming computing has triggers in the field of point databases, and also has some computing modes like event buses, middleware, etc.
Referring to fig. 1 to 6, a method for generating a service topology and service monitoring based on streaming computation according to an embodiment of the present invention includes the following steps:
and S10, receiving the source data stream, and generating the association relation of all service levels according to the source data stream.
And S20, receiving the logical and physical names and the association relation of all the service levels, storing the logical and physical names and the association relation in the RDBMS, and generating the logical and physical service data of all the service levels.
And S30, receiving logical and physical service data of all service levels, calculating the time index values of year, day and minute and the modulus result value of the minute index value, and calculating the monitoring index value in the corresponding minute slot.
For step S10, after the source data stream is received, the physical and logical service names of each level are obtained from the source data stream, wherein part of the physical and logical service names may be directly obtained from the source data stream, and part of the physical and logical service names do not directly exist in the source data stream and need to be obtained through a certain configuration.
Referring to fig. 2, step S10 includes the steps of:
and S11, extracting Span data items from the source data stream.
And S12, identifying whether the Span data item contains the logical service names and the physical service names of all service levels.
And S13, if the service hierarchy contains the association relation, generating the association relation of all the service hierarchies.
S14, if not, arranging the logical service name and the physical service name of the missing service level, and generating the association relationship of all service levels.
For step S11, after receiving the source data stream, extracting a Span data item from the source data stream, where the Span data item at least contains fields such as TraceId, SpanId, ParentId, hostpip, InstanceName, CreatTime, CallTime, IsEntry, ErrCode, and the like, where InstanceName is a service instance name, which is a 2-level logical service name and a 2-level physical service name, and updating InstantceName into the database.
For steps S12-S14, it is identified whether the Span data item contains the logical service names and physical service names of all service levels, and there are corresponding logical service names and physical service names, and the logical service names and physical service names of the missing service levels are called directly or obtained by configuration.
For example, if the level 1 logical service name ComponentName can be extracted from the Span data item, the association relationship data of the ComponentName and the InstanceName is generated; otherwise, obtaining the ComponentName of the class 1 logical service name through configuration, and generating the association relation data of the ComponentName and the instanceName. In addition, the source data stream contains HostIp, which has an association relationship with instanceName as a level 1 physical service name.
Meanwhile, if the 0-level logical service name AppName can be extracted from the source data stream, the association relationship data of the AppName and the ComponentName is generated. Otherwise, obtaining the 0-level logic service name AppName through configuration, generating the association relation data of the AppName and the ComponentName, and storing the association relation data in the database. Wherein, AppName can be used as the 0-level physical service name at the same time.
And finally, obtaining the logical/physical service names and the incidence relations of the 2-level, the 1-level and the 0-level, wherein the logical/physical service names and the incidence relations are used as incidence relation data of all service levels for subsequent use.
Specifically, the instanceName is used as a common basic level of the physical topology and the logical topology (in the invention, the basic level is used as a default level 2 level), so that the physical topology and the logical topology establish an association relationship from the bottom-layer service; establishing a service topological relation facing to the service by defining the corresponding relation between the logic service and one or more physical services; meanwhile, instanceName (used as a default level 2 level in the invention) and Hostip (used as a default level 1 level in the invention) are used as two-level physical service levels to establish a service topological relation facing the technology, so that operation and maintenance personnel can monitor the running state of the complex telecommunication system from the perspective of business and technology.
For step S20, the logical and physical names and association relationships of all service levels are received and stored in the RDBMS, and logical and physical service data of all service levels are generated. For example, the above-mentioned logical/physical service names and association relations of level 2, level 1, and level 0 are received, and further the logical/physical service names and association relations of level 2, level 1, and level 0 are stored in the RDBMS to form three-level logical and physical service data, specifically, the logical and physical service data is logical/physical service topological graph data.
The logical service is not limited to three levels of 2, 1 and 0, and a more multi-level logical topology structure can be expanded according to actual services, and the logical service can be obtained by acquiring or configuring from a source data stream. The physical service is defined except for the level 0, the level 1 and the level 2 are both really existed, as long as the physical service level actually existed exists, more physical topological structures can be expanded according to the reality, only the level 1 and the level 2 are physically really existed at present, the level 0 is defined, if the definition is continued on the basis of the level 0, the level-1 level and the level-2 level can be continuously defined. Specifically, in order to avoid the occurrence of negative number hierarchy, the top level service is defined as level 0, and other levels are incremented to levels of 1, 2, 3, and so on.
Referring to fig. 3, step S20 includes the steps of:
and S21, setting the update time of the logic and physical service data, and judging whether the update time is reached.
And S22, if the update time is up, updating the logic and physical service data.
For step S21, a specific update time, for example, 10 minutes, 5 minutes, or 3 minutes, is set as the time interval for updating the logical and physical service data, and the logical and physical service data are automatically updated when the update time arrives. It is possible to determine whether to update the logical and physical service data by determining whether the update time is reached.
Whether the update time is reached or not can be realized by judging whether the time point is the corresponding update time point or not. For example, if the update time interval is 10 minutes and the last update time is 15 minutes, it is determined whether to update the data by determining whether the current time is 25 minutes, 35 minutes, 45 minutes, 55 minutes, or 5 minutes, and if it is any of the above time points, it is determined that the update time has been reached, and the logical and physical service data are automatically updated.
With respect to step S22, when it is determined at step S21 that the update time has come, the logical and physical service data are updated, and new service logical and physical service data are generated.
For step S30, after each time the logical and physical service data is received, the logical and physical service data includes time stamp information, three time index values of year, day, and minute and a modulo result value of the minute index value are calculated according to the time stamp of the data, and a monitoring index value is calculated in the corresponding minute slot.
The time slot is used for storing a year index value, a day index value, a minute index value and a monitoring index value. The year index value, the day index value and the minute index value in the time slot jointly form slot identification.
Referring to fig. 4, step S30 includes the steps of:
and S31, calculating the time index values of year, day and minute and the modulus result value of the minute index value according to the received time stamp of the logical service data and the physical service data.
S32, comparing the calculated time index values with the time index values of the three time slots respectively, and judging whether the time index values are the same; and if the time index value is smaller than the time index value of the minimum time slot, ending the process.
And S33, if the calculated time index value is equal to the time index value corresponding to any time slot, taking the modulus of the minute index value.
And S34, if the calculated time index value is larger than the index value of the maximum time slot, storing the monitoring index value of the minute slot corresponding to the serial number into the database, and reinitializing the minute slot of the serial number.
For step S31, the time slot is used to store the year index value, the day index value, the minute index value, the monitoring index value. The year index value, the day index value and the minute index value in the time slot jointly form slot identification. And the monitoring index value is calculated in a time slot mode, so that the accuracy of the monitoring index value is higher, and the monitoring index value is limited in a small storage space.
The year index value refers to the year value, the day index value refers to the day of the year, and the minute index value refers to the minute number of the day. For example, in the morning of 1 point 18 points, 6 and 23 days in 2017, the year index is 2017, the day index is 174, the minute index is 78, the result of modulo 3 of the minute slot index 78 is 0, and the data indicating the minute is stored in the 0 th minute slot. And obtaining a year index value, a day index value and a minute index value corresponding to the timestamp time information through stream type calculation for subsequent use.
In step S32, the calculated time index values are compared with the time index values of the three time slots, respectively, to determine whether or not the calculated time index values match the time index values of the time slots. And only if the three index values of the year, the day and the minute of the new incoming logic and physical service data are equal to the three index values of the year, the day and the minute corresponding to the time slot, the monitoring value of the new incoming logic and physical service data is superposed to the existing monitoring index value of the minute slot.
For step S33, when it is determined in step S32 that the calculated time index value is equal to the time index value corresponding to any time slot, the modulo minute index value is taken. And when the calculated time index value is judged not to be equal to the time index value corresponding to any time slot, the monitoring value of the new logic and physical service data is not superposed into the existing monitoring value of the minute slot.
Step S33, including the steps of: s330, identifying the modulus result value, and accumulating and calculating the monitoring index value.
In step S330, the monitoring index values in the corresponding minute slots are accumulated and calculated according to the identified magnitude of the modulo result value. The number of the time slots is modulus and is also the sending period of the monitoring index value, and in a quasi-real-time scene, N minutes can be taken as one sending period, namely N time slots are correspondingly arranged. And taking the operation result of the minute index value% N as the serial number of the time slot, and calculating the monitoring value of each data in the corresponding time slot according to the minute index value. Where N is a natural number, and may be modulo by other numbers as desired, such as 1, 2, 3, 4, etc.
In step S34, when the calculated time index value is greater than the index value of the largest time slot, the monitoring index value of the minute slot corresponding to the index number (the index number is the modulo result value of the minute index value) is stored in the database, and the minute slot of the index number is reinitialized. The original monitoring index value existing in the minute slot is removed.
Referring to fig. 6, an example of the monitoring index value in the time slot is shown, specifically, the monitoring index value includes TPS, error number, and average response time.
TPS, chinese, is throughput, increasing by 1 for each service data received.
The error number, Err, is increased by 1 each time an abnormal service data is received.
The average response time, which is called CallTimeAvg in english, is subtracted from the average response time in the minute slot after receiving the response time of each piece of service data, the obtained difference is divided by TPS, TPS is the TPS containing the current piece of service data, that is, the service data is received and then increased by 1, the obtained quotient value is added to the original average response time in the minute slot, and the result value is the average response time. The calculation formula is as follows:
CallTimeAvgNew = (CallTime - CallTimeAvg)/TPS + CallTimeAvg
such as: the current throughput TPS in the time slot is 2, the average response time CallTimeAvg is 2.5 ms, 1 new piece of data is received, the response time CallTime is 7 ms, the TPS increments 1 by 3, the new average response time is CallTimeAvgNew = (7 ms-2.5 ms)/3 + 2.5 ms, and the result is 4 ms, i.e., the new average response time value is 4 ms.
In FIG. 6, TPS of the leftmost time slot is 10, error number is 3, and average response time is 95; the TPS of the middle time slot is 8, the error number is 1, and the average response time is 90; the TPS for the rightmost time slot is 6, the number of errors is 2, and the average response time is 100.
Referring to fig. 5, in an embodiment of the present invention, 3 minutes is taken as one transmission period, i.e. there are three time slots. And taking the operation result of the minute index value% 3 as the serial number of the time slot, and calculating the monitoring index value in the corresponding time slot according to the minute index value for each data. The method comprises the following specific steps:
and S31, calculating the time index values of year, day and minute and the modulus result value of the minute index value according to the received time stamp of the logical service data and the physical service data.
And S32, comparing the calculated time index values with the time index values of the three time slots respectively, and judging whether the calculated time index values are the same.
And S33, if the calculated time index value is equal to the time index value corresponding to any time slot, taking the modulus of the minute index value.
And S331, if the result is equal to 0, accumulating and calculating the monitoring index value of the minute slot 0.
And S332, if the result is equal to 1, accumulating and calculating the monitoring index value of the minute slot 1.
And S333, if the result is equal to 2, accumulating and calculating the monitoring index value of the minute slot 2.
S341, if the calculated time index value is greater than the index value of the largest time slot, the monitoring index value of the minute slot ( slot number 0, 1, or 2) corresponding to the number is binned and the minute slot ( slot number 0, 1, or 2) corresponding to the number is reinitialized.
When the data of the second period arrives, the module result value of the minute index value can be any time slot serial number, and the monitoring index value of the time slot is sent corresponding to the time slot serial number, the time slot data is cleared, and the monitoring index value of the new data is stored. Wherein the sequence number is a modulo result value of the minute index value.
And if no new data still arrives in more than two periods, in order to avoid untimely updating of the monitoring index value, sending out the time slot monitoring data in more than two periods, and clearing the time slot data.
According to the scheme, the monitoring index value in the time slot is calculated in a quasi-real-time mode through stream type calculation, and a service topological relation facing to a service and technology view is established by defining the corresponding relation between the logic/physical service and the service instance, so that operation and maintenance personnel can monitor the running state of the complex telecommunication system from the service and technology view globally; and the monitoring index value is calculated in a time slot mode, so that the accuracy of the monitoring index value is higher, and the monitoring index value is limited in a small storage space.
Referring to fig. 7-11, the present invention further provides a system for generating service topology and service monitoring based on streaming computation, including:
the service analysis generator 10 is configured to receive a source data stream, and generate association relationships of all service hierarchies according to the source data stream.
And the service data storage 20 is used for receiving the logical and physical names and the association relation of all the service levels, storing the logical and physical names and the association relation into the RDBMS and generating the logical and physical service data of all the service levels.
And the monitoring index value calculator and memory 30 is used for receiving the logical and physical service data of all the service levels, calculating three time index values of year, day and minute and a modulo result value of the minute index value, and calculating the monitoring index value in the corresponding minute slot.
Referring to fig. 7, the source data stream belongs to a source data stream layer, the service analysis generator 10, the service data storage 20, the supervision value calculator and storage 30 are located at a stream calculation layer, and the RDBMS and the time series database are located at a data storage layer. Referring to fig. 8, the source data stream is sent to the service analysis generator 10, and then sent to the service data storage 20 and the monitoring value calculator and storage 30 in service packets, and finally the service topology is stored in the RDBMS, and the monitoring index value is stored in the time series database.
For the service analysis generator 10, after receiving the source data stream, a Span data item is extracted from the source data stream, where the data item at least contains fields such as TraceId, SpanId, ParentId, hostpip, InstanceName, CreatTime, CallTime, IsEntry, ErrCode, and the like, where InstanceName is a service instance name, which is a 2-level logical service name and a 2-level physical service name, and InstantceName is updated into the database.
Referring to fig. 9, the service analysis generator 10 includes an extraction module 11, a recognition module 12, and a generation module 13.
And the extracting module 11 is configured to extract a Span data item from the source data stream.
And the identification module 12 is used for identifying whether the Span data item contains the logical service names and the physical service names of all service levels.
The generation module 13 generates an association relationship between all service levels if the identification module 12 determines that the service level is included, and generates an association relationship between all service levels by arranging a logical service name and a physical service name of a missing service level if the identification module determines that the service level is not included.
The extraction module 11, the identification module 12 and the generation module 13 cooperate to identify whether the Span data item contains logical service names and physical service names of all service levels, and if the logical service names and the physical service names are consistent, the logical service names and the physical service names are directly called, otherwise, the logical service names and the physical service names of the missing service levels are obtained through configuration.
For example, if the level 1 logical service name ComponentName can be extracted from the Span data item, the association relationship data of the ComponentName and the InstanceName is generated; otherwise, obtaining the ComponentName of the class 1 logical service name through configuration, and generating the association relation data of the ComponentName and the instanceName. In addition, the source data stream contains HostIp, which has an association relationship with instanceName as a level 1 physical service name.
Meanwhile, if the 0-level logical service name AppName can be extracted from the source data stream, the association relationship data of the AppName and the ComponentName is generated. Otherwise, obtaining the 0-level logic service name AppName through configuration, generating the association relation data of the AppName and the ComponentName, and storing the association relation data in the database. Wherein, AppName can be used as the 0-level physical service name at the same time.
And finally, obtaining the logical/physical service names and the incidence relations of the 2-level, the 1-level and the 0-level, wherein the logical/physical service names and the incidence relations are used as incidence relation data of all service levels for subsequent use.
For the service data storage 20, the logical and physical names and the association relation of all the service levels are received and stored in the RDBMS, and the logical and physical service data of all the service levels are generated. For example, the above-mentioned logical/physical service names and association relations of level 2, level 1, and level 0 are received, and further the logical/physical service names and association relations of level 2, level 1, and level 0 are stored in the RDBMS to form three-level logical and physical service data, specifically, the logical and physical service data is logical/physical service topological graph data.
The logical service is not limited to three levels of 2, 1 and 0, and a more multi-level logical topology structure can be expanded according to actual services, and the logical service can be obtained by acquiring or configuring from a source data stream. The physical service is defined except for the level 0, the level 1 and the level 2 are both really existed, as long as the physical service level actually existed exists, more physical topological structures can be expanded according to the reality, only the level 1 and the level 2 are physically really existed at present, the level 0 is defined, if the definition is continued on the basis of the level 0, the level-1 level and the level-2 level can be continuously defined. Specifically, in order to avoid the occurrence of negative number hierarchy, the top level service is defined as level 0, and other levels are incremented to levels of 1, 2, 3, and so on.
Referring to fig. 10, the service data storage 20 includes a timing update module 21 for setting the update time of the logical and physical service data and determining whether the update time is reached; and if the update time is up, storing the logical and physical names and the association relation of the service hierarchy into the RDBMS.
For the timing update module 21, a specific update time, for example, 10 minutes, 5 minutes, or 3 minutes, is set as a time interval for updating the logical and physical service data, and the logical and physical service data are automatically updated when the update time arrives. It is possible to determine whether to update the logical and physical service data by determining whether the update time is reached.
Whether the update time is reached or not can be realized by judging whether the time point is the corresponding update time point or not. For example, if the update time interval is 10 minutes and the last update time is 15 minutes, it is determined whether to update the data by determining whether the current time is 25 minutes, 35 minutes, 45 minutes, 55 minutes, or 5 minutes, and if it is any of the above time points, it is determined that the update time has been reached, and the logical and physical service data are automatically updated.
And if the update time is judged to be up, updating the logical and physical service data to generate new service logical and physical service data.
For the monitoring index value calculator and memory 30, after each time of receiving the logical and physical service data, the logical and physical service data includes time stamp information, three time index values of year, day and minute and a modulo result value of the minute index value are calculated according to the time stamp of the data, and the monitoring index value is calculated in the corresponding minute slot.
The time slot is used for storing a year index value, a day index value, a minute index value and a monitoring index value. The year index value, the day index value and the minute index value in the time slot jointly form slot identification.
Referring to fig. 11, the monitoring metric value calculator and memory 30 includes a calculation module 31, a comparison module 32, a modulus extraction module 33 and an initialization module 34,
the calculating module 31 is configured to calculate three time index values of year, day, and minute and a modulo result value of the minute index value according to the received time stamps of the logical and physical service data;
a comparison module 32, configured to compare the calculated time index values with the time index values of the three time slots, respectively, and determine whether the calculated time index values are the same;
a module-taking module 33, configured to take a module with the minute index value after the calculated time index value is equal to the time index value corresponding to any time slot;
and an initialization module 34, configured to store the monitoring index value in the minute slot corresponding to the sequence number (the sequence number is a modulo result value of the minute index value) into the database after the calculated time index value is greater than the index value of the largest time slot, and reinitialize the minute slot corresponding to the sequence number.
For the calculation module 31, the time slot is used to store the year index value, the day index value, the minute index value, the monitoring index value. The year index value, the day index value and the minute index value in the time slot jointly form slot identification.
The year index value refers to the year value, the day index value refers to the day of the year, and the minute index value refers to the minute number of the day. For example, in the morning of 1 point 18 points, 6 and 23 days in 2017, the year index is 2017, the day index is 174, the minute index is 78, the result of modulo 3 of the minute slot index 78 is 0, and the data indicating the minute is stored in the 0 th minute slot. And obtaining a year index value, a day index value and a minute index value corresponding to the timestamp time information through stream type calculation for subsequent use.
For the comparison module 32, it can be determined whether the calculated time index value is consistent with the time index values of the time slots by comparing the calculated time index values with the time index values of the three time slots, respectively. And only if the three index values of the year, the day and the minute of the new incoming logic and physical service data are equal to the three index values of the year, the day and the minute corresponding to the time slot, the monitoring value of the new incoming logic and physical service data is superposed to the existing monitoring index value of the minute slot.
For the module 33, when the comparison module 32 determines that the calculated time index value is equal to the time index value corresponding to any time slot, the module is taken by the minute index value. And when the calculated time index value is judged not to be equal to the time index value corresponding to any time slot, the monitoring value of the new logic and physical service data is not superposed into the existing monitoring value of the minute slot.
Specifically, the module 33 is configured to identify a module result value, and accumulate and calculate a monitoring index value.
And accumulating and calculating the monitoring index values in the corresponding minute slots according to the identified magnitude of the modulus result value. The number of the time slots is modulus and is also the sending period of the monitoring index value, and in a quasi-real-time scene, N minutes can be taken as one sending period, namely N time slots are correspondingly arranged. And taking the operation result of the minute index value% N as the serial number of the time slot, and calculating the monitoring value of each data in the corresponding time slot according to the minute index value. Where N is a natural number, and may be modulo by other numbers as desired, such as 1, 2, 3, 4, etc.
For the initialization module 34, when the calculated time index value is greater than the index value of the largest time slot, the monitoring index value of the minute slot corresponding to the sequence number (the sequence number is the modulo result value of the minute index value) is stored in the database, and the minute slot of the sequence number is reinitialized. The original monitoring index value existing in the minute slot is removed.
According to the scheme, the monitoring index value in the time slot is calculated in a quasi-real-time mode through stream type calculation, and a service topological relation facing to a service and technology view is established by defining the corresponding relation between the logic/physical service and the service instance, so that operation and maintenance personnel can monitor the running state of the complex telecommunication system from the service and technology view globally; and the monitoring index value is calculated in a time slot mode, so that the accuracy of the monitoring index value is higher, and the monitoring index value is limited in a small storage space.
The above description is only a preferred embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes, which are made by using the contents of the present specification and the accompanying drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.
Claims (8)
1. A method for generating service topology and service monitoring based on stream computing is characterized by comprising the following steps:
receiving a source data stream, and generating incidence relations of all service levels according to the source data stream;
receiving the logical service names, the physical service names and the association relations of all the service levels, storing the logical service names, the physical service names and the association relations in the RDBMS, and generating logical service data and physical service data of all the service levels;
receiving logic service data and physical service data of all service levels, calculating three time index values of year, day and minute and a modulus result value of the minute index value, and calculating a monitoring index value in a corresponding minute slot;
the InstanceName is both a level 2 logical service name and a level 2 physical service name; the 1-level logic service name is ComponentName, and the incidence relation data of the ComponentName and the instanceName is generated; the 1-level physical service name is HostIP, and the source data stream already has the incidence relation between the HostIP and the instanceName; the 0-level logic service name is AppName, and the incidence relation data of the AppName and the ComponentName is generated;
the step of receiving the logic service data and the physical service data of all the service levels, calculating the time index values of year, day and minute and the modulus result value of the minute index value, and calculating the monitoring index value in the corresponding minute slot comprises,
calculating three time index values of year, day and minute and a modulus result value of the minute index value according to the received timestamps of the logical service data and the physical service data;
comparing the calculated time index values with the time index values of the three time slots respectively, and judging whether the time index values are the same or not;
if the calculated time index value is equal to the time index value corresponding to any time slot, taking a module by the minute index value;
if the calculated time index value is larger than the index value of the maximum time slot, storing the monitoring index value of the minute slot corresponding to the serial number into a database, and reinitializing the minute slot of the serial number; wherein the sequence number is a modulo result value of the minute index value.
2. The method for generating service topology and service monitoring based on streaming calculation as claimed in claim 1, wherein the step of receiving the source data stream and generating the association relationship of all service layers according to the source data stream comprises,
extracting Span data items from a source data stream;
identifying whether the Span data item contains the logical service names and the physical service names of all service levels;
if yes, generating the association relation of all service levels;
if not, the logical service name and the physical service name of the missing service level are configured, and the association relationship of all the service levels is generated.
3. The method for generating service topology and service monitoring based on streaming calculation according to claim 1, wherein the steps of receiving the logical service names, the physical service names and the association relations of all the service levels and storing the same in the RDBMS, and generating the logical service data and the physical service data of all the service levels comprise,
setting the updating time of the logic service data and the physical service data, and judging whether the updating time is reached;
and if the updating time is up, storing the logical service name, the physical service name and the association relation of the service level into the RDBMS.
4. The method for generating service topology and service monitoring based on streaming computation of claim 1, wherein the step of receiving logical service data and physical service data of all service levels, computing three time index values of year, day, and minute and a modulo result value of the minute index value, computing the monitoring index value in the corresponding minute slot comprises,
and identifying a modulus result value, and accumulating and calculating the monitoring index value.
5. A system for generating service topology and service monitoring based on streaming computing, comprising:
the service analysis generator is used for receiving the source data stream and generating the incidence relation of all service levels according to the source data stream;
the service data memory is used for receiving the logical service names, the physical service names and the association relations of all the service levels, storing the logical service names, the physical service names and the association relations into the RDBMS and generating the logical service data and the physical service data of all the service levels;
the monitoring index value calculator and the memory are used for receiving the logic service data and the physical service data of all the service levels, calculating three time index values of year, day and minute and a modulo result value of the minute index value, and calculating the monitoring index value in the corresponding minute slot;
the InstanceName is both a level 2 logical service name and a level 2 physical service name; the 1-level logic service name is ComponentName, and the incidence relation data of the ComponentName and the instanceName is generated; the 1-level physical service name is HostIP, and the source data stream already has the incidence relation between the HostIP and the instanceName; the 0-level logic service name is AppName, and the incidence relation data of the AppName and the ComponentName is generated;
the monitoring index value calculator and the memory comprise a calculating module, a comparing module, a modulus taking module and an initializing module,
the calculation module is used for calculating three time index values of year, day and minute and a modulus result value of the minute index value according to the received timestamps of the logic service data and the physical service data;
the comparison module is used for comparing the calculated time index values with the time index values of the three time slots respectively and judging whether the time index values are the same or not;
the module taking module is used for taking the module by the minute index value after the calculated time index value is equal to the time index value corresponding to any time slot;
and the initialization module is used for storing the monitoring index value in the minute slot corresponding to the sequence number into the database and re-initializing the minute slot of the sequence number after the calculated time index value is larger than the index value of the maximum time slot, wherein the sequence number is a modulo result value of the minute index value.
6. The system for generating service topology and service monitoring based on streaming computation of claim 5, wherein the service analysis generator comprises an extraction module, a recognition module and a generation module,
the extraction module is used for extracting Span data items from the source data stream;
the identification module is used for identifying whether the Span data item contains the logical service names and the physical service names of all service levels;
and the generating module generates the incidence relation of all service levels if the identifying module judges that the service levels contain the logical service names and the physical service names of the missing service levels, and generates the incidence relation of all service levels if the identifying module judges that the service levels do not contain the logical service names and the physical service names of the missing service levels.
7. The system for generating service topology and service monitoring based on streaming computation of claim 5, wherein the service data storage comprises a timing update module for setting update time of logical service data and physical service data and judging whether the update time is reached; and if the updating time is up, storing the logical service name, the physical service name and the association relation of the service level into the RDBMS.
8. The system for generating service topology and service monitoring based on streaming computation of claim 5, wherein the modulo module is configured to identify a modulo result value and accumulate and compute a monitoring index value.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711238278.2A CN109861838B (en) | 2017-11-30 | 2017-11-30 | Method and system for generating service topology and monitoring service based on stream computing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711238278.2A CN109861838B (en) | 2017-11-30 | 2017-11-30 | Method and system for generating service topology and monitoring service based on stream computing |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109861838A CN109861838A (en) | 2019-06-07 |
| CN109861838B true CN109861838B (en) | 2022-02-22 |
Family
ID=66888101
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711238278.2A Active CN109861838B (en) | 2017-11-30 | 2017-11-30 | Method and system for generating service topology and monitoring service based on stream computing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109861838B (en) |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102143220A (en) * | 2010-02-02 | 2011-08-03 | 国际商业机器公司 | Method and system for discovering physical server location by correlating external and internal server information |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101552739A (en) * | 2008-04-03 | 2009-10-07 | 北京邮电大学 | Method and system for managing life cycle of network service process example |
| US9219684B2 (en) * | 2013-09-09 | 2015-12-22 | Grigori Dzekon | Methods and systems for packet delivery based upon embedded instructions |
| US20180035153A1 (en) * | 2015-03-01 | 2018-02-01 | Lg Electronics Inc. | Broadcasting signal transmission apparatus, broadcasting signal reception apparatus, broadcasting signal transmission method, and broadcasting signal reception method |
| CN107305518A (en) * | 2016-04-20 | 2017-10-31 | 中兴通讯股份有限公司 | The display methods and device of warning information |
-
2017
- 2017-11-30 CN CN201711238278.2A patent/CN109861838B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102143220A (en) * | 2010-02-02 | 2011-08-03 | 国际商业机器公司 | Method and system for discovering physical server location by correlating external and internal server information |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109861838A (en) | 2019-06-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110399347B (en) | Alarm log compression method, device and system and storage medium | |
| US6658367B2 (en) | System for time-bucketing of baselined data collector data | |
| US10318366B2 (en) | System and method for relationship based root cause recommendation | |
| WO2022257423A1 (en) | Warning information association method and apparatus, and electronic device and readable storage medium | |
| US10909018B2 (en) | System and method for end-to-end application root cause recommendation | |
| CN106790718A (en) | Service call link analysis method and system | |
| CN110535713B (en) | Monitoring management system and monitoring management method | |
| CN113497726B (en) | Alarm monitoring method, alarm monitoring system, computer readable storage medium and electronic equipment | |
| US20090222472A1 (en) | Method and Apparatus for Aggregation in Uncertain Data | |
| US8484212B2 (en) | Providing reconstructed data based on stored aggregate data in response to queries for unavailable data | |
| Lange et al. | Remote real-time trajectory simplification | |
| CN103595569A (en) | Method for handling database storage of alarm information of network management system | |
| CN110825545A (en) | Cloud service platform anomaly detection method and system | |
| US20070226216A1 (en) | Methods and Apparatus for Dynamic Classification of Data in Evolving Data Stream | |
| CN113986595A (en) | Abnormity positioning method and device | |
| CN107346270A (en) | Method and system based on the sets cardinal calculated in real time | |
| EP3204848B1 (en) | Real-time reporting based on instrumentation of software | |
| CN110324211A (en) | A kind of data capture method and device | |
| CN109861838B (en) | Method and system for generating service topology and monitoring service based on stream computing | |
| US20090177953A1 (en) | Method and system for updating topology changes of a computer network | |
| CN117421188A (en) | Alarm grading method, device, equipment and readable storage medium | |
| CN105446707B (en) | Data conversion method | |
| CN112685261B (en) | Micro-service operation state monitoring method based on observer mode | |
| CN116264541A (en) | Multi-dimension-based database disaster recovery method and device | |
| CN111771195A (en) | Stream processing apparatus and data stream processing method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |