CN113268398A - Service monitoring method and system based on cloud native data warehouse - Google Patents
Service monitoring method and system based on cloud native data warehouse Download PDFInfo
- Publication number
- CN113268398A CN113268398A CN202110612042.0A CN202110612042A CN113268398A CN 113268398 A CN113268398 A CN 113268398A CN 202110612042 A CN202110612042 A CN 202110612042A CN 113268398 A CN113268398 A CN 113268398A
- Authority
- CN
- China
- Prior art keywords
- data
- index
- cloud
- index calculation
- native
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000012544 monitoring process Methods 0.000 title claims abstract description 81
- 238000000034 method Methods 0.000 title claims abstract description 44
- 238000004364 calculation method Methods 0.000 claims abstract description 89
- 238000004519 manufacturing process Methods 0.000 claims description 24
- 238000003860 storage Methods 0.000 claims description 17
- 238000004140 cleaning Methods 0.000 claims description 8
- 230000008569 process Effects 0.000 claims description 7
- 238000010586 diagram Methods 0.000 description 9
- 238000011156 evaluation Methods 0.000 description 5
- 230000009286 beneficial effect Effects 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 230000009467 reduction Effects 0.000 description 4
- 238000004220 aggregation Methods 0.000 description 3
- 230000002776 aggregation Effects 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 3
- 238000012423 maintenance Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 230000001360 synchronised effect Effects 0.000 description 3
- 238000006243 chemical reaction Methods 0.000 description 2
- 238000004590 computer program Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000009826 distribution Methods 0.000 description 2
- 230000010354 integration Effects 0.000 description 2
- 238000002955 isolation Methods 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 238000010223 real-time analysis Methods 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 238000004891 communication Methods 0.000 description 1
- 230000008602 contraction Effects 0.000 description 1
- 238000005520 cutting process Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000000737 periodic effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/302—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system component is a software system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/32—Monitoring with visual or acoustical indication of the functioning of the machine
- G06F11/324—Display of status information
- G06F11/327—Alarm or error message display
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/275—Synchronous replication
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Quality & Reliability (AREA)
- Data Mining & Analysis (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention discloses a service monitoring method and a monitoring system based on a cloud primary digital warehouse, which relate to the technical field of Internet, wherein the method comprises the following steps: synchronizing data generated in each business field into a cloud primary data warehouse, inquiring corresponding data from the cloud primary data warehouse based on preset index calculation configuration data, and performing index calculation to generate index data; and calculating the index data according to a preset alarm rule to obtain alarm information, and informing an alarm receiver when the alarm information reaches a preset alarm threshold value. Compared with the traditional method, the cloud-native-data-bin-based service monitoring method and the cloud-native-data-bin-based service monitoring system solve the problems that the iteration period of the incremental indexes is long, the index data are scattered, and the instantaneity of index calculation cannot be guaranteed.
Description
Technical Field
The invention relates to the technical field of internet, in particular to a service monitoring method and system based on a cloud native data warehouse.
Background
The rapid development of the internet makes the use of the internet more and more common, and the internet not only becomes a communication bridge inside an enterprise, but also is an important pipeline for the enterprise and the outside to carry out various business transactions. With the continuous innovation and improvement of the service mode and the product structure of the internet, the internet service is continuously developed. The internet service fully plays the optimization and integration roles of the internet in production element configuration, the innovation achievements of the internet are deeply integrated in various fields of the economic society, a wider new economic development state taking the internet as infrastructure and a realization tool is formed, and meanwhile, the monitoring technology for the internet service is timely and timely.
However, the total amount of index data in each service field is huge, and the index data is difficult to be aggregated and integrated in real time, and cannot provide uniform and real-time query analysis capability, so that the index data is dispersed to form a data island to a certain extent, meanwhile, data sources are diversified, processing and conversion logic needs to be customized, the index data is reported by self, and is tedious, a uniform index production path does not exist, index iteration needs to be developed periodically, so that the incremental index iteration period is long, and meanwhile, the index data production is limited by data volume and computing capability, and the instantaneity of the index data production cannot be ensured.
Based on the problems, how to realize real-time service index data production, real-time alarm and real-time service report forms, further cannot find internet service problems in time, provides real-time data support for service pain points to duplicate and analyze, and becomes a problem which needs to be solved in the internet service monitoring process.
Disclosure of Invention
The invention aims to provide a cloud-native data warehouse-based service monitoring method and system, which can realize real-time aggregation, calculation and production of service indexes based on the cloud-native data warehouse, further discover the internet service problems in time, and provide real-time data support for replying and analyzing service pain points.
In order to achieve the above purpose, the invention provides the following technical scheme:
a service monitoring method based on a cloud native data warehouse is characterized by comprising the following steps:
synchronizing data generated in each business field into a cloud native data warehouse;
inquiring corresponding data from the cloud native data warehouse based on preset index calculation configuration data, and performing index calculation to generate index data;
and calculating the index data according to a preset alarm rule to obtain alarm information, and informing an alarm receiver when the alarm information reaches a preset alarm threshold value.
Preferably, the index calculation configuration data includes an index basic attribute configuration and an index production scheduling configuration, where the index production scheduling configuration is used to define data query logic and index calculation logic.
Further, the data generated by each business domain includes but is not limited to log data and data in a business database;
cleaning and synchronizing the data generated in each business field to the cloud native digital warehouse by using a data synchronization tool built in the cloud native digital warehouse;
wherein the data synchronization process includes synchronization of data values and synchronization of data structures.
Further, the method for querying corresponding data from the cloud native data warehouse based on the index calculation configuration data and performing index calculation comprises the following steps:
according to data query logic defined by index production scheduling configuration, querying data meeting the index basic attribute configuration from a cloud native data warehouse;
and performing index calculation on the data meeting the index basic attribute configuration according to index calculation logic defined by the index production scheduling configuration to generate index data.
Preferably, the index data is calculated according to preset alarm rules and service field classifications, and alarm information corresponding to different service fields is respectively obtained;
the alert recipients include individuals, virtual ad hoc groups, organizational structures, universal dynamic resources, and customized recipients.
Further, the cloud native bin-based business monitoring method further includes:
and displaying the index data in real time, and generating a report in real time based on the index data by using a report tool.
A service monitoring system based on a cloud-native data warehouse comprises a configuration module, the cloud-native data warehouse, an index calculation module and a monitoring module,
the configuration module is used for setting index calculation configuration data;
a plurality of data synchronization tools are arranged in the cloud native number, and the data synchronization tools are used for cleaning and synchronizing data generated in each business field into the cloud native number bin;
the index calculation module includes a plurality of index calculation units, each of which is configured to perform one or more index calculation tasks including: inquiring corresponding data from the cloud native data warehouse based on the index calculation configuration data, and performing index calculation to generate index data;
the monitoring module is used for calculating the index data according to a preset alarm rule, acquiring alarm information and notifying an alarm receiver when the alarm information reaches a preset alarm threshold value.
Preferably, different index calculation units are configured to perform different index calculation tasks, and the index calculation module supports lateral expansion.
Preferably, the monitoring module comprises a plurality of monitoring unit groups, and each monitoring unit group comprises one or more monitoring units;
the monitoring unit group is respectively used for calculating index data of different service fields and respectively acquiring alarm information corresponding to the different service fields.
A computer readable storage medium having computer readable program instructions stored thereon for performing the cloud-native bin-based traffic monitoring method described above.
Compared with the prior art, the service monitoring method and system based on the cloud native data warehouse have the following beneficial effects:
according to the service monitoring method based on the cloud native digital warehouse, provided by the invention, data generated in each service field is synchronized into the cloud native digital warehouse, the cloud native digital warehouse adopts a cloud native technical architecture, the storage and calculation separation is realized, the calculation resources and the storage resources can be dynamically expanded and contracted as required, the bottleneck problem of calculation and storage resources caused by service growth and fluctuation can be solved, and meanwhile, the cost is reduced to the maximum extent. And then, inquiring corresponding data from the cloud native data warehouse based on preset index calculation configuration data, performing index calculation, generating index data, and generating the index calculation configuration data in real time, so that the running time can be saved, the format, the content and the standard of the inquiry data can be unified, and the subsequent processing is facilitated. And finally, calculating the index data according to a preset alarm rule to obtain alarm information, and informing an alarm receiver when the alarm information reaches a preset alarm threshold value, setting the preset alarm rule to monitor data related to normal operation of the system, setting the alarm threshold value to filter some non-serious alarm information, concentrating on serious alarm information which can have great influence on the operation of the system, improving the maintenance efficiency and reducing manpower and material resources required by maintenance.
The service monitoring system based on the cloud primary data warehouse provided by the invention adopts the service monitoring method based on the cloud primary data warehouse, so that the iteration period of incremental indexes is shortened, the index data is managed in a centralized manner, a data isolated island is avoided, the real-time performance of the whole service monitoring system is ensured, the problem of internet service is found in time, and real-time data support is provided for the service pain point to be duplicated and analyzed.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the invention and not to limit the invention. In the drawings:
fig. 1 is a schematic overall flow chart of a cloud-based data warehouse-based business monitoring method in an embodiment of the present invention;
FIG. 2 is a schematic diagram illustrating a process of synchronizing data generated in each business domain into a cloud native data bin according to an embodiment of the present invention;
fig. 3 is a schematic diagram of a process of querying corresponding data from a cloud native data warehouse based on index calculation configuration data and performing index calculation according to an embodiment of the present invention;
FIG. 4 is a diagram illustrating an expansion of an index calculation unit according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of a monitoring module according to an embodiment of the present invention;
fig. 6 is a diagram of system module relationships proposed in an embodiment of the present invention.
Detailed Description
In order to make the aforementioned objects, features and advantages of the present invention comprehensible, embodiments accompanied with figures are described in detail below. It is to be understood that the described embodiments are merely exemplary of the invention, and not restrictive of the full scope of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Example one
Referring to fig. 1, an embodiment of the present invention provides a service monitoring method based on a cloud-native data warehouse, including:
synchronizing data generated in each business field into a cloud native data warehouse;
inquiring corresponding data from the cloud native data warehouse based on preset index calculation configuration data, and performing index calculation to generate index data;
and calculating the index data according to a preset alarm rule to obtain alarm information, and informing an alarm receiver when the alarm information reaches a preset alarm threshold value.
The monitoring method disclosed by the invention technically ensures the real-time performance of index production based on the mass and real-time analysis capability of the cloud primary data warehouse; based on the computing resource isolation and the elastic capacity expansion and reduction capability of the cloud and native digital warehouse, the capacity expansion and reduction capability of the whole index production is realized so as to easily cope with the rapid business growth; based on the calculation and analysis capabilities of the bins, real-time data services such as monitoring and report forms are provided, and important alarm information is timely notified to an alarm receiver for processing.
Referring to fig. 2, in the service monitoring method based on the cloud-native digital warehouse, data generated in each service domain includes, but is not limited to, log data and data in a service database; cleaning and synchronizing data generated in each business field into the cloud native digital warehouse by using a data synchronization tool built in the cloud native digital warehouse; wherein the data synchronization process includes synchronization of data values and synchronization of data structures.
In specific implementation, the cloud native digital warehouse is implemented by using databases such as AnalyticDB, MySQL, and the like, the data synchronization tool is a tool in the cloud native digital warehouse ecology, and has a data cleaning function, and the data cleaning mainly includes cutting of original business data, aggregation of original database-based and table-divided data, and the like. The cloud primary data warehouse can be compatible with various data sources, has massive and high-performance real-time analysis capability, retains an original data structure after data are synchronized to the cloud primary data warehouse, and stores the data in a classified manner according to different service fields and provides a uniform and standard path, and each heterogeneous data is processed and converged to the cloud primary data warehouse, so that a downstream uniform index production path is realized, and the data can be rapidly inquired and called.
Referring to fig. 3, the service monitoring method based on the cloud-native data warehouse according to the embodiment further includes: setting index calculation configuration data. Specifically, the index calculation configuration data is set through a configuration interface provided by the system, and includes index basic attribute configuration and index production scheduling configuration, where the index production scheduling configuration is used to define data query logic and index calculation logic, and the data query logic specifically includes information such as a scheduling period and database query. On this basis, the method for inquiring corresponding data from the cloud native data warehouse based on the index calculation configuration data and carrying out index calculation comprises the following steps:
according to data query logic defined by index production scheduling configuration, querying data meeting the index basic attribute configuration from a cloud native data warehouse;
and performing index calculation on the data meeting the basic attribute configuration of the index according to index calculation logic defined by the index production scheduling configuration to generate index data.
The process of inquiring the data meeting the basic attribute configuration of the index from the cloud primary data warehouse is an sql inquiry task of the index, namely, the data to be processed is inquired from the cloud primary data warehouse according to the basic attribute of the index and a uniform index production path provided by the cloud primary data warehouse, the elastic expansion and contraction capacity and the isolated allocation of computing resources of the cloud primary data warehouse provide guarantee for the index computing of the overall service monitoring, and simultaneously provide the aggregation, uniform index conversion and reporting capabilities.
The index production scheduling configuration supports a cron expression, and the cron expression is a group of character strings consisting of a plurality of numbers, spaces and symbols according to a certain rule, so that the time information is expressed. The expression supports a general periodic form, such as 1 minute, 5 minutes and the like, and then a database query task, such as an sql task, is executed at regular time according to a configured frequency, and data is queried from a plurality of bins by a specific sql, wherein the data is index result data at the time point.
It is to be noted that, in the service monitoring method based on the cloud-native data warehouse provided in this embodiment, the index computing capability supports stateless lateral expansion, and the computing resources are stateless, and the expansion manner is the same in different fields. Referring to fig. 4, when the index calculation capacity reaches the bottleneck, the index calculation module resources are simply expanded, the index calculation tasks are re-allocated according to the ID modulo of the index calculation tasks, so as to expand the overall index calculation capacity, thereby adapting to the real-time operation of mass data, and when the data volume rapidly increases, the index production capacity of the whole device can be rapidly and transversely expanded, thereby ensuring the production efficiency of the index data.
Further, referring to fig. 5, the monitoring module collects the index data generated by each index calculation module, and sends the index data to the monitoring unit groups corresponding to each service field, so as to calculate the index data according to the preset alarm rule according to the service field classification, and respectively obtain the alarm information corresponding to different service fields, and notify the alarm receiver when the alarm information reaches the preset alarm threshold.
The alarm rule in the embodiment is formulated based on specific services, for example, when the operation index of the CPU is monitored, if the occupancy rate of the CPU exceeds 50%, an alarm signal is generated and an alarm receiver is notified; when the indexes of the user system are monitored, if the number of the user registration is less than 10000, an alarm signal is generated and an alarm receiver is informed. The alarm rule is configured on the alarm module and is synchronized to the monitoring module; the monitoring module performs calculation in real time, and the calculated object is the time sequence index data acquired by the monitoring module in real time. In addition, the monitoring tool built in the monitoring module in this embodiment can be implemented by using open-source Grafana, and the alarm receivers include individuals, virtual temporary groups, organizational structures, general dynamic resources, and customized receivers, so that the alarm signals can be processed in time, and the influence on the service can be avoided.
In addition, the service monitoring method based on the cloud native data warehouse further comprises the following steps: and displaying the index data in real time through the monitoring panel, and generating a report in real time based on the index data by using a report tool. The report related to the scheme is realized by adopting the existing report tools such as Sail software, and the capability of generating the report in real time can be achieved based on the real-time index data generated by the index module.
In summary, the cloud-native-data-bin-based service monitoring method provided by the embodiment of the invention utilizes the computing resource isolation and elastic capacity expansion and reduction capabilities of the cloud-native data bin to realize the capacity expansion and reduction capability of the overall index production so as to easily cope with faster service growth, and meanwhile, by relying on the computing and analyzing capabilities of the cloud-native data bin, real-time data services such as monitoring and reporting are provided.
Example two
Referring to fig. 6, an embodiment of the present invention provides a cloud-based service monitoring system, which includes a configuration module, a cloud-based data warehouse, an index calculation module, and a monitoring module,
the configuration module is used for setting index calculation configuration data;
a plurality of data synchronization tools are arranged in the cloud primary data, and the data synchronization tools are used for cleaning and synchronizing data generated in each business field into the cloud primary data warehouse;
the index calculation module comprises a plurality of index calculation units, each index calculation unit is used for executing one or more index calculation tasks, and each index calculation task comprises: inquiring corresponding data from the cloud native data warehouse based on the index calculation configuration data, and performing index calculation to generate index data;
the monitoring module is used for calculating the index data according to a preset alarm rule, acquiring alarm information and notifying an alarm receiver when the alarm information reaches a preset alarm threshold value.
In practical application, the system maintenance cost can be reduced by adopting a modular design, only a target module needs to be maintained when a certain module makes a mistake or needs to be updated, the whole system does not need to be stopped, and the operation efficiency and the customer experience are influenced. For example: when more configurable options are added to the configuration module, the off-line configuration module is only needed to be upgraded, and the modules such as the cloud native digital warehouse and the like can be continuously on line to meet the downstream query requirement.
Different index calculation units are configured to perform different index calculation tasks, and the index calculation module supports lateral expansion.
Different index calculation tasks are distributed to different index calculation units, so that the operation logic of each index calculation unit can be simplified, and faults are reduced while calculation is accelerated. In addition, the index calculation module can be easily transversely expanded, and the calculation power requirement can be met at any time only by adding more index calculation units when higher calculation capacity is required.
Further, in the cloud-based data warehouse-based business monitoring system provided by the embodiment of the present invention, the monitoring module includes a plurality of monitoring unit groups, and each monitoring unit group includes one or more monitoring units;
the monitoring unit group is respectively used for calculating index data of different service fields and respectively acquiring alarm information corresponding to the different service fields.
Compatible with the transformation of various heterogeneous basic data, and uniformly providing a friendly use interface in the form of SQL
The monitoring module adopts a hierarchical design, can macroscopically judge whether the whole system has alarm information or not, and can also conveniently find the accurate source of the alarm information when analyzing specific problems. Index data of each service field are brought into the monitoring range, so that the monitoring module can completely reflect the running state of the whole system and information cannot be lost. The heterogeneous basic data are converted into the same display mode, so that the user experience is improved, and the time wasted due to non-uniform data formats in the use process is reduced.
By adopting the method provided by the first embodiment of the invention, the service monitoring system based on the cloud-native digital warehouse can find the problem of the internet service in time, provide real-time data support for duplication and analysis of service pain points and better meet the requirements on real-time performance and data integration. Compared with the prior art, the beneficial effects of the cloud-based data warehouse service monitoring system provided by the embodiment of the invention are the same as those of the cloud-based data warehouse service monitoring method provided by the first embodiment, and are not repeated herein.
EXAMPLE III
A computer-readable storage medium having computer-readable program instructions stored thereon for performing the cloud-based traffic monitoring method of the first embodiment, comprising: synchronizing data generated in each business field into a cloud primary data warehouse, inquiring corresponding data from the cloud primary data warehouse based on preset index calculation configuration data, and performing index calculation to generate index data; and calculating the index data according to a preset alarm rule to obtain alarm information, and informing an alarm receiver when the alarm information reaches a preset alarm threshold value.
The computer readable storage medium provided by the embodiments of the present invention may be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, or device, or any combination thereof. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present embodiment, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, or device. Program code embodied on a computer readable storage medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, optical cables, RF (radio frequency), etc., or any suitable combination of the foregoing.
The computer-readable storage medium may be embodied in an electronic device; or may be present alone without being incorporated into the electronic device.
The computer readable storage medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: acquiring at least two internet protocol addresses; sending a node evaluation request comprising the at least two internet protocol addresses to node evaluation equipment, wherein the node evaluation equipment selects the internet protocol addresses from the at least two internet protocol addresses and returns the internet protocol addresses; receiving an internet protocol address returned by the node evaluation equipment; wherein the obtained internet protocol address indicates an edge node in the content distribution network.
Alternatively, the computer readable storage medium carries one or more programs which, when executed by an electronic device, cause the electronic device to: receiving a node evaluation request comprising at least two internet protocol addresses; selecting an internet protocol address from the at least two internet protocol addresses; returning the selected internet protocol address; wherein the received internet protocol address indicates an edge node in the content distribution network.
Computer program code for carrying out operations for aspects of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The modules described in the embodiments of the present disclosure may be implemented by software or hardware. Where the name of a module does not in some cases constitute a limitation on the unit itself, for example, an index calculation module may also be described as a "module for performing one or more index calculation tasks".
The computer readable storage medium provided by the invention stores the computer readable program instruction for executing the management method of the distributed system configuration item, and solves the problems that the existing service monitoring system has insufficient index calculation real-time performance and cannot find the internet service fault in time. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the embodiment of the present invention are the same as the beneficial effects of the cloud-based data warehouse-based service monitoring method provided by the first embodiment, and are not described herein again.
It should be understood that portions of the present disclosure may be implemented in hardware, software, firmware, or a combination thereof. In the foregoing description of embodiments, the particular features, structures, materials, or characteristics may be combined in any suitable manner in any one or more embodiments or examples.
The above description is only for the specific embodiments of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present invention, and all the changes or substitutions should be covered within the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the appended claims.
Claims (10)
1. A service monitoring method based on a cloud native data warehouse is characterized by comprising the following steps:
synchronizing data generated in each business field into a cloud native data warehouse;
inquiring corresponding data from the cloud native data warehouse based on preset index calculation configuration data, and performing index calculation to generate index data;
and calculating the index data according to a preset alarm rule to obtain alarm information, and informing an alarm receiver when the alarm information reaches a preset alarm threshold value.
2. The cloud-native-data-bin-based traffic monitoring method of claim 1, wherein the metric-computation-configuration data comprises a metric base attribute configuration and a metric production-scheduling configuration, wherein the metric production-scheduling configuration is used to define data-query logic and metric-computation logic.
3. The cloud-based data warehouse business monitoring method of claim 2, wherein the data generated by each business domain includes but is not limited to log data and data in a business database;
cleaning and synchronizing the data generated in each business field to the cloud native digital warehouse by using a data synchronization tool built in the cloud native digital warehouse;
wherein the data synchronization process includes synchronization of data values and synchronization of data structures.
4. The cloud-based business monitoring method according to claim 2 or 3, wherein the method for querying corresponding data from the cloud-based data warehouse based on the index calculation configuration data and performing index calculation comprises:
according to data query logic defined by index production scheduling configuration, querying data meeting the index basic attribute configuration from a cloud native data warehouse;
and performing index calculation on the data meeting the index basic attribute configuration according to index calculation logic defined by the index production scheduling configuration to generate index data.
5. The cloud-native-digital-bin-based business monitoring method according to claim 4, wherein the index data is calculated according to a preset alarm rule according to business field classification, and alarm information corresponding to different business fields is respectively obtained;
the alert recipients include individuals, virtual ad hoc groups, organizational structures, universal dynamic resources, and customized recipients.
6. The cloud-native-bin-based traffic monitoring method of claim 5, further comprising:
and displaying the index data in real time, and generating a report in real time based on the index data by using a report tool.
7. A service monitoring system based on a cloud original data warehouse is characterized by comprising a configuration module, the cloud original data warehouse, an index calculation module and a monitoring module, wherein,
the configuration module is used for setting index calculation configuration data;
a plurality of data synchronization tools are arranged in the cloud native number, and the data synchronization tools are used for cleaning and synchronizing data generated in each business field into the cloud native number bin;
the index calculation module includes a plurality of index calculation units, each of which is configured to perform one or more index calculation tasks including: inquiring corresponding data from the cloud native data warehouse based on the index calculation configuration data, and performing index calculation to generate index data;
the monitoring module is used for calculating the index data according to a preset alarm rule, acquiring alarm information and notifying an alarm receiver when the alarm information reaches a preset alarm threshold value.
8. The cloud-based data bin business monitoring system of claim 7, wherein different index calculation units are configured to perform different index calculation tasks and the index calculation module supports lateral scaling.
9. The cloud-based data bin traffic monitoring system of claim 7, wherein the monitoring module comprises a plurality of monitoring unit groups, each monitoring unit group comprising one or more monitoring units;
the monitoring unit group is respectively used for calculating index data of different service fields and respectively acquiring alarm information corresponding to the different service fields.
10. A computer readable storage medium having computer readable program instructions stored thereon for performing the cloud-native bin-based traffic monitoring method of any of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110612042.0A CN113268398B (en) | 2021-06-02 | 2021-06-02 | Business monitoring method and system based on cloud primary number bin |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110612042.0A CN113268398B (en) | 2021-06-02 | 2021-06-02 | Business monitoring method and system based on cloud primary number bin |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113268398A true CN113268398A (en) | 2021-08-17 |
| CN113268398B CN113268398B (en) | 2024-02-27 |
Family
ID=77233954
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110612042.0A Active CN113268398B (en) | 2021-06-02 | 2021-06-02 | Business monitoring method and system based on cloud primary number bin |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113268398B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113672473A (en) * | 2021-08-27 | 2021-11-19 | 工银科技有限公司 | Monitoring data processing method, device, equipment, medium and program product of cloud database |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108471366A (en) * | 2018-02-02 | 2018-08-31 | 深圳市康拓普信息技术有限公司 | A kind of stereoscopic monitoring system of facing cloud native applications |
| CN109660407A (en) * | 2019-01-18 | 2019-04-19 | 鑫涌算力信息科技（上海）有限公司 | Distributed system monitoring system and method |
| US20190362014A1 (en) * | 2018-05-28 | 2019-11-28 | Wipro Limited | Method and device for managing big-data in cloud environment |
-
2021
- 2021-06-02 CN CN202110612042.0A patent/CN113268398B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108471366A (en) * | 2018-02-02 | 2018-08-31 | 深圳市康拓普信息技术有限公司 | A kind of stereoscopic monitoring system of facing cloud native applications |
| US20190362014A1 (en) * | 2018-05-28 | 2019-11-28 | Wipro Limited | Method and device for managing big-data in cloud environment |
| CN109660407A (en) * | 2019-01-18 | 2019-04-19 | 鑫涌算力信息科技（上海）有限公司 | Distributed system monitoring system and method |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113672473A (en) * | 2021-08-27 | 2021-11-19 | 工银科技有限公司 | Monitoring data processing method, device, equipment, medium and program product of cloud database |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113268398B (en) | 2024-02-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107402976B (en) | Power grid multi-source data fusion method and system based on multi-element heterogeneous model | |
| CN108039959B (en) | Data situation perception method, system and related device | |
| US10394821B2 (en) | Providing reconstructed data based on stored aggregate data in response to queries for unavailable data | |
| US11945337B2 (en) | System and method for vehicle battery management, storage medium, and server system | |
| JP2015146183A (en) | Managing big data in process control systems | |
| CN106210124B (en) | Unified cloud data center monitoring system | |
| CN108268529B (en) | Data summarization method and system based on business abstraction and multi-engine scheduling | |
| CN108170832B (en) | Monitoring system and monitoring method for heterogeneous database of industrial big data | |
| CN113595761A (en) | Micro-service component optimization method of power system information and communication integrated scheduling platform | |
| CN107944036B (en) | Method for acquiring map change difference | |
| US9009533B2 (en) | Home/building fault analysis system using resource connection map log and method thereof | |
| CN111523004B (en) | Storage method and system for edge computing gateway data | |
| CN114090366A (en) | Method, device and system for monitoring data | |
| CN115344207A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN113268398A (en) | Service monitoring method and system based on cloud native data warehouse | |
| CN113810234B (en) | Method and device for processing micro-service link topology and readable storage medium | |
| CN112865311B (en) | Method and device for monitoring message bus of power system | |
| CN117762968A (en) | Data aggregation method, system, equipment and storage medium | |
| Zhang et al. | The modeling of big traffic data processing based on cloud computing | |
| CN115439015B (en) | Local area power grid data management method, device and equipment based on data middleboxes | |
| CN103226572A (en) | Expandable monitoring method and monitoring system based on data compression | |
| CN116260702A (en) | Method, device, computer equipment and storage medium for data monitoring | |
| CN115514618A (en) | Alarm event processing method and device, electronic equipment and medium | |
| CN114610798A (en) | Resource allocation management method, system, device, storage medium and electronic equipment | |
| AU2017269259A1 (en) | Data driven invocation of real time wind market forecasting analytics |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |