CN115766498A - Big data link full link tracking monitoring method and system - Google Patents
Big data link full link tracking monitoring method and system Download PDFInfo
- Publication number
- CN115766498A CN115766498A CN202211284793.5A CN202211284793A CN115766498A CN 115766498 A CN115766498 A CN 115766498A CN 202211284793 A CN202211284793 A CN 202211284793A CN 115766498 A CN115766498 A CN 115766498A
- Authority
- CN
- China
- Prior art keywords
- component
- monitoring
- target
- task
- big data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000012544 monitoring process Methods 0.000 title claims abstract description 150
- 238000000034 method Methods 0.000 title claims abstract description 55
- 238000012545 processing Methods 0.000 claims abstract description 101
- 238000011144 upstream manufacturing Methods 0.000 claims description 42
- 238000004088 simulation Methods 0.000 claims description 15
- 230000006870 function Effects 0.000 claims description 9
- 238000005516 engineering process Methods 0.000 abstract description 5
- 230000000694 effects Effects 0.000 abstract description 3
- 230000008569 process Effects 0.000 description 16
- 238000013075 data extraction Methods 0.000 description 13
- 238000001514 detection method Methods 0.000 description 6
- 238000003780 insertion Methods 0.000 description 4
- 230000037431 insertion Effects 0.000 description 4
- 230000002159 abnormal effect Effects 0.000 description 3
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000005034 decoration Methods 0.000 description 2
- 230000036541 health Effects 0.000 description 2
- 230000002452 interceptive effect Effects 0.000 description 2
- 238000012423 maintenance Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000005192 partition Methods 0.000 description 2
- 230000000737 periodic effect Effects 0.000 description 2
- 238000011084 recovery Methods 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- 238000013024 troubleshooting Methods 0.000 description 2
- 239000002699 waste material Substances 0.000 description 2
- 238000012098 association analyses Methods 0.000 description 1
- 235000000332 black box Nutrition 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000002950 deficient Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Debugging And Monitoring (AREA)
Abstract
The application discloses a big data link full link tracking monitoring method and a system, wherein the method comprises the following steps: after receiving a monitoring task for a target big data link, acquiring a monitoring strategy for the target big data link, wherein the monitoring strategy comprises the following steps: sending a processing task corresponding to the component to a target component on the target big data link, and executing the processing task corresponding to the component in the component; and determining a monitoring result according to the execution result of the monitoring strategy. According to the method and the device, the technical problem that an effective monitoring mode for the stability of the whole big data link is lacked in the related technology is solved, and the technical effect of providing an effective method for tracking and monitoring the whole big data link is achieved.
Description
Technical Field
The application relates to the field of big data, in particular to a big data link full link tracking monitoring method and system.
Background
Big data related technologies are very common at present, the big data technologies involve a large amount of data relocation and distributed computing, after the big data technologies are operated, the whole data link is very long, the number of components involved in the data link is large, and the network among the components is also complex.
However, for monitoring the stability of the whole link of the big data, the related art has not proposed an effective solution.
Disclosure of Invention
The embodiment of the application provides a method and a system for tracking and monitoring a big data link and a full link, which at least solve the technical problem that an effective monitoring mode for the stability of the whole big data link is lacked in the prior art.
According to one aspect of the application, a large data link full link tracking monitoring method is provided, which includes: after receiving a monitoring task for a target big data link, acquiring a monitoring strategy for the target big data link, wherein the monitoring strategy comprises the following steps: sending a processing task corresponding to the component to a target component on the target big data link, and executing the processing task corresponding to the component in the component; and determining a monitoring result according to the execution result of the monitoring strategy.
Optionally, the monitoring policy further comprises: and executing the processing task corresponding to the monitoring component in the component to be monitored under the condition that the monitoring result of the upstream component of the component to be monitored is normal.
Optionally, there is an intersection between the execution process of the processing task corresponding to the component in the component and the process of the component processing data.
Optionally, the processing task includes a task of simulating data writing, and a task of acquiring target data of the simulated data processed by the upstream component from the upstream component.
Optionally, when the processing task is the task for writing the simulation data, determining the monitoring result according to the execution result of the monitoring policy includes: establishing a target table in a component corresponding to the processing task; writing the simulation data into the target table, and feeding back the execution time of the processing task after the writing is finished; and comparing the execution time with a first preset threshold value, and determining the monitoring result of the component corresponding to the processing task according to the comparison result.
Optionally, when the processing task is a task of acquiring target data of the simulation data processed by the upstream component from the upstream component, determining a monitoring result according to an execution result of the monitoring policy includes: establishing a target task in a component corresponding to the processing task, wherein the target task is determined by the processing task and corresponds to the function of the component; executing the target task, and determining the acquisition delay of the target data according to the execution result of the target task; and comparing the acquired delay with a second preset threshold, and determining a monitoring result between the component corresponding to the processing task and the upstream component according to a comparison result.
Optionally, the monitoring result includes network delay data of the component of the processing task and the upstream component and availability data of the component of the processing task itself.
Optionally, the target component is determined according to a function of a component included in the big data link.
Optionally, the target big data link is obtained by combining a plurality of components according to a big data task.
According to an aspect of the present application, there is also provided a large data link full link tracking monitoring system, including: the acquisition module is used for acquiring a monitoring strategy of a target big data link after receiving a monitoring task of the target big data link, wherein the monitoring strategy comprises the following steps: sending a processing task corresponding to the component to a target component on the target big data link, and executing the processing task corresponding to the component in the component; and the determining module is used for determining a monitoring result according to the execution result of the monitoring strategy.
According to another aspect of the present application, there is also provided an electronic device comprising a memory and a processor; wherein the memory is configured to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the above-described method steps.
According to another aspect of the present application, there is also provided a readable storage medium having stored thereon computer instructions, wherein the computer instructions, when executed by a processor, implement the above-mentioned method steps.
In the embodiment of the present application, after receiving a monitoring task for a target big data link, a monitoring policy for the target big data link is obtained, where the monitoring policy includes: sending a processing task corresponding to the component to a target component on the target big data link, and executing the processing task corresponding to the component in the component; and determining a monitoring result according to the execution result of the monitoring strategy. That is to say, in the embodiment of the present application, the corresponding task is sent to the component on the big data link from the outside, the monitoring result is determined according to the fed back task execution result, and the big data link external monitoring is implemented, so that the technical problem that an effective monitoring mode for the stability of the whole big data link is lacked in the related art is solved, and the technical effect of providing an effective method for the big data link full link tracking monitoring is achieved.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this application, illustrate embodiments of the application and, together with the description, serve to explain the application and are not intended to limit the application. In the drawings:
FIG. 1 is a flowchart of a method for monitoring a large data link full link trace according to an embodiment of the present application;
FIG. 2 is a schematic diagram of a flowchart of another big data link full link tracking monitoring method according to an embodiment of the present application;
fig. 3 is a schematic structural diagram of a large data link full link tracking monitoring system according to an embodiment of the present application.
Detailed Description
In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that the terms "first," "second," and the like in the description and claims of this application and in the accompanying drawings are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It should be understood that the data so used may be interchanged under appropriate circumstances such that embodiments of the application described herein may be implemented in sequences other than those illustrated or described herein. Moreover, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
In accordance with an embodiment of the present application, there is provided an embodiment of a large data link full link trace monitoring method, it should be noted that the steps illustrated in the flowchart of the drawings may be performed in a computer system such as a set of computer executable instructions, and that while a logical order is illustrated in the flowchart, in some cases the steps illustrated or described may be performed in an order different than here.
Big data is a data set mainly characterized by large capacity, multiple types, high storage speed and high application value, and is rapidly developed into a new generation of information technology and service state for acquiring, storing and performing association analysis on data with huge quantity, scattered sources and various formats, discovering new knowledge, creating new value and improving new capability. The big data component means that in order to achieve the business target of the big data, in the landing process, by means of a plurality of big data engineering software, different software in a big data landing link needs to truly play the business effect, a plurality of big data components need to be matched together to complete the landing, the data of the business can be calculated and analyzed through a plurality of different types of data components, and finally the business result is generated, and the link is called as a big data link.
The full link tracking capability under the micro-service architecture is relatively perfect at present, the whole link of each request can be tracked in a full chain, but under a big data scene, the construction of the tracking capability of a complex link is relatively deficient, and because the architecture and the technical stack of a big data product are very complex, the stack condition can not be tracked just like a simple JAVA application. In addition, similar to a link detection function of a micro service application, the flow direction and the operation condition of each piece of data can be monitored and detected, but when the method is applied to the detection of a big data link, business logic must be invaded, specifically, each big data component is required to be capable of actively outputting the operation log of each piece of data, and for an open source component, the log of each piece of data can be output by injecting in a plug-in manner, but the operation log cannot be obtained by black-box cloud products at present. Therefore, the tracking method under the micro service architecture is not suitable for the big data link, and based on this, the embodiment of the present application proposes the big data link full link tracking monitoring method as shown in fig. 1, which includes:
s102, after receiving a monitoring task of a target big data link, acquiring a monitoring strategy of the target big data link, wherein the monitoring strategy comprises the following steps: sending a processing task corresponding to the component to a target component on the target big data link, and executing the processing task corresponding to the component in the component;
it should be noted that the target big data link may be determined from known big data links, and the known big data links may be determined by combining big data components according to specific services. For example, for traffic 1, its big data link may be component 1 → component 2 → component 3, and for traffic 2, its big data link may be component 1 → component 2 → component 3 → component 4. The monitoring task may be sent periodically, for example, the monitoring task is sent every 10 minutes, and the period value is determined according to the traffic stability requirement. The relationship between the big data link and the monitoring strategies may be one-to-one correspondence, or one big data link may correspond to a plurality of monitoring strategies, and one monitoring strategy is selected from the plurality of monitoring strategies for use according to specific service requirements during monitoring. The target component on the target big data link may be all components on the target big data link, or may be a part of components on the target big data link, and may be determined according to functions of components included in the big data link. For example, if the target big data link includes component 1 → component 2 → component 3 → component 4, the target component may be component 1, component 2, component 3, or component 4, or the target component may be component 1, component 3, or component 4, and when monitoring is performed, the corresponding processing task is sent to the component 1, component 2, component 3, or component 4, or the corresponding processing task is sent to the component 1, component 3, or component 4. The processing task is used for acquiring monitoring data of the components or monitoring data among the components, such as component availability and network stability data among the components.
And S104, determining a monitoring result according to the execution result of the monitoring strategy.
Through the above S102 to S104, after receiving a monitoring task for a target big data link, obtaining a monitoring policy for the target big data link, where the monitoring policy includes: sending a processing task corresponding to the component to a target component on the target big data link, and executing the processing task corresponding to the component in the component; and determining a monitoring result according to the execution result of the monitoring strategy. That is to say, in the embodiment of the present application, a corresponding task is sent to a component on a big data link from the outside, a monitoring result is determined according to a fed-back task execution result, and external monitoring of the big data link is achieved.
For a large data link, if a problem occurs in the first component, it is not necessary to continue monitoring the downstream component, because data cannot flow to the downstream component, if the downstream node is continuously monitored, resource waste inevitably occurs, and monitoring efficiency is reduced, so the embodiment of the present application provides that the monitoring strategy further includes: and executing the processing task corresponding to the monitoring component in the component to be monitored under the condition that the monitoring result of the upstream component of the component to be monitored is normal, so that the monitoring efficiency is improved. For example, assuming that the target big data link includes component 1 → component 2 → component 3 → component 4, when the condition that component 1 is unavailable is detected, an alarm is issued directly to indicate that the link is unstable without continuing to monitor component 2, component 3 and component 4. For another example, assuming that the target big data link includes component 1 → component 2 → component 3 → component 4, when component 2 or the network between component 1 and component 2 is monitored to be unstable, an alarm is issued directly to indicate that the link is unstable without continuing to monitor component 3, component 4. The description is given here by way of example only and is not intended as a limitation on the present application.
In the embodiment of the application, a piece of data is simulated to flow on a big data link, and external monitoring is performed outside a big data component system in the flowing process, so that the processing task issued to the component is basically consistent with the process of the component in normal data processing when the processing task is executed, and the monitoring effectiveness can be realized. For example, if the component 1 executes the operation 1 when processing the normal data 1, the processing procedure of the processing task issued to the component 1 in the embodiment of the present application in the component 1 also includes the above-mentioned operation 1, and for example, if the component 1 executes the operations 1 and 2 when processing the normal data 1, but the operation 2 does not much use the monitoring analysis, the processing procedure of the processing task issued to the component 1 in the embodiment of the present application in the component 1 may include only the operation 1. The description is given here by way of example only and is not intended as a limitation on the present application.
In an alternative embodiment, the processing task may include a task of simulating data writing, and a task of acquiring target data of the simulated data processed by the upstream component from the upstream component.
It should be noted that the type of the analog data may be determined according to a specific service, for example, for service 1, the analog data may be data 1, for service 2, the analog data may be data 2, and for example, for service 1 and service 2, the analog data may also be only data 1. In addition, it should be noted that the component corresponding to the task of simulating data writing may be a database, for example, a service database/end service database. The components corresponding to the task of acquiring the target data of the simulation data processed by the upstream component from the upstream component can be a database data extraction component, a data message queue component and a big data computing platform component. The database data extraction component is a tool for uploading/downloading data and can meet most common data cloud scenes. The database data extraction component is also a log-based structured data backup tool, and is generally used for master-slave backup between business databases and synchronization of the business databases to other databases. The database data extraction component is also provided with a log reading module and a log playback module which are highly available, and once the disaster recovery system of the database data extraction component detects that the link is abnormal, the link can be restarted at a breakpoint on the health service node, so that the high availability of the synchronous link is effectively ensured. The data message queue component is a component which supports interactive message content retrieval in the field of message queues first, provides message retrieval service capability based on data dump table storage, supports free combination of retrieval messages according to any conditions such as values and partitions, supports value-taking full-text retrieval messages, has the characteristics of development-free property, operation-free property and high elasticity, can retrieve messages directly through table storage indexes or SQL (structured query language), and greatly improves the speed of daily troubleshooting of the existence or the correctness of the messages. For example, suppose that an operation and maintenance team needs to monitor the operation condition of an on-line cluster, a log at the process level is collected and imported into a data message queue component, and a downstream uses a big data computing platform component for consumption, so that the resource consumption condition of each process is computed in real time. When log data of a certain time period of a certain process is found to be lost in a big data computing platform assembly, a data message queue assembly is required to be used for retrieving message data based on a message value and a time range, and whether the log is successfully pushed into a message queue of the data message queue assembly is judged.
The processing tasks include a task of simulating data writing and a task of acquiring target data of the simulated data processed by the upstream component from the upstream component, and how to monitor the two situations is described in detail below.
When the processing task is the task for writing the simulation data, determining the monitoring result according to the execution result of the monitoring policy includes: s11, establishing a target table in the component corresponding to the processing task; s12, writing the simulation data into the target table, and feeding back the execution time of the processing task after the writing is finished; and S13, comparing the execution time with a first preset threshold value, and determining the monitoring result of the component corresponding to the processing task according to the comparison result. For example, assuming that a component corresponding to a processing task is a service database component, because a normal use of a service is to be simulated, for the service database component, a table is created first, and then data is written into the surface and the back surface, in the embodiment of the present application, a target table is created in the service database component, and a single insertion statement is executed to write simulated data into the target table, and a corresponding time is returned when the execution of a general insertion statement is completed, therefore, the execution time of the processing task may be determined by the time of writing and the time of completing writing, and then the execution time is compared with a first preset threshold (for example, 1 second), and when the first preset threshold is exceeded, it is determined that the service database component is unavailable and abnormal.
When the processing task is a task of acquiring target data of the simulation data processed by the upstream component from the upstream component, determining a monitoring result according to an execution result of the monitoring policy includes: s21, establishing a target task in a component corresponding to the processing task, wherein the target task is determined by the processing task and corresponds to the function of the component; s22, executing the target task, and determining the acquisition delay of the target data according to the execution result of the target task; and S23, comparing the acquired delay with a second preset threshold, and determining a monitoring result between the component corresponding to the processing task and the upstream component according to a comparison result. For example, assuming that the component corresponding to the processing task is a data message queue component, because normal use of the service is to be simulated, a Topic is established for the data message queue component, and then data in the upstream node is automatically consumed and written into a target message queue of the data message queue component, so that the acquisition delay of the data in the upstream node can be compared with a second preset threshold (for example, 0.5 second), and when the second preset threshold is exceeded, it is determined that a problem occurs in a network between the data message queue component and the upstream node or a problem occurs in the data message queue component itself. Certainly, an end-to-end connection ping operation can be executed on the connection address of the data message queue component, whether the data message queue component has a problem or not is determined according to the return delay of the ping, a final result is further determined from the problem of the network between the data message queue component and the upstream node or the problem of the data message queue component, and the network stability of the whole link of the large data link and the availability of the component are effectively monitored.
Optionally, in this embodiment, after the detection result is obtained, a periodic comparison may be performed to obtain a state update change of the link condition, so as to improve the user experience.
The embodiments of the present application will be described below by way of examples in conjunction with specific examples.
The big data link shown in fig. 2 includes a service database component, a database data extraction component, a data message queue component, a big data computing platform component, and a terminal service database component, finds a first service database component of the whole big data link, automatically builds a table in the service database component, writes prepared analog data in the service database component, and records a delay index according to the time for writing the analog data and the time for completing writing. After the analog data enters the service database component, the changed log is pulled by the architecture database data extraction component according to fig. 2 and written into a message queue, at this time, a Topic is automatically created on the message queue, the data in the service database component is automatically consumed and written into a target message queue, so that the message queue can be monitored to wait for receiving the message, and the delay index from the database data extraction component to the data message queue component is obtained according to the time for acquiring the corresponding data in the message queue. The simulation data enters the big data computing platform assembly, a consumption task is automatically created and started, the aim is to acquire data from the message queue of the big data computing platform assembly, the event of acquiring the data from the message queue is monitored, and the time delay of the data from the message queue to the big data computing platform assembly is determined. After the simulation data enters the big data computing platform assembly, a consumption task can be created, the data is only obtained from the message queue and is written into the business database of the target end, the data writing time of the target business database is monitored, and the delay index from the message queue to the business database is judged.
In summary, the embodiments of the present application achieve the objectives of delay detection, availability detection, and the like of a full-link big data component by simulating a data stream transfer and monitoring the transfer process outside the big data component system.
According to an embodiment of the present invention, an embodiment of a large data link full link tracking and monitoring system for implementing the method embodiment is further provided, and in an application scenario of the embodiment, the embodiment provides a schematic structural diagram of the large data link full link tracking and monitoring system shown in fig. 3. As shown in fig. 3, the large data link full link tracking monitoring system includes:
an obtaining module 32, configured to obtain a monitoring policy of a target big data link after receiving a monitoring task for the target big data link, where the monitoring policy includes: sending a processing task corresponding to the component to a target component on the target big data link, and executing the processing task corresponding to the component in the component;
it should be noted that the target big data link may be determined from known big data links, and the known big data links may be determined by combining big data components according to specific services. For example, for business 1, its big data link may be component 1 → component 2 → component 3, and for business 2, its big data link may be component 1 → component 2 → component 3 → component 4. The monitoring task may be sent periodically, for example, the monitoring task is sent every 10 minutes, and the period value is determined according to the traffic stability requirement. The relationship between the big data link and the monitoring strategies can be one-to-one correspondence, or one big data link corresponds to a plurality of monitoring strategies, and one monitoring strategy is selected from the plurality of monitoring strategies for use according to specific service requirements during monitoring. The target component on the target big data link may be all components on the target big data link, or may be a part of components on the target big data link, and may be determined according to functions of components included in the big data link. For example, assuming that the target big data link includes component 1 → component 2 → component 3 → component 4, the target component may be component 1, component 2, component 3, or component 4, or the target component may be component 1, component 3, or component 4, and when monitoring is performed, the corresponding processing task is sent to the component 1, component 2, component 3, or component 4, or the corresponding processing task is sent to the component 1, component 3, or component 4. The processing task is used for acquiring monitoring data of the components or monitoring data among the components, such as component availability and network stability data among the components.
And the determining module 34 is configured to determine a monitoring result according to an execution result of the monitoring policy.
Through the system, after receiving a monitoring task for a target big data link, a monitoring strategy of the target big data link is obtained, wherein the monitoring strategy comprises the following steps: sending a processing task corresponding to the component to a target component on the target big data link, and executing the processing task corresponding to the component in the component; and determining a monitoring result according to the execution result of the monitoring strategy. That is to say, in the embodiment of the present application, a corresponding task is sent to a component on a big data link from the outside, a monitoring result is determined according to a fed-back task execution result, and external monitoring of the big data link is achieved.
For a large data link, if a problem occurs in the first component, it is not necessary to continue monitoring the downstream component, because data cannot flow to the downstream component, if the downstream node is continuously monitored, resource waste inevitably occurs, and monitoring efficiency is reduced, so the embodiment of the present application provides that the monitoring policy further includes: and executing the processing task corresponding to the monitoring component in the component to be monitored under the condition that the monitoring result of the upstream component of the component to be monitored is normal, so that the monitoring efficiency is improved. For example, assuming that the target big data link includes component 1 → component 2 → component 3 → component 4, when the condition that component 1 is unavailable is detected, an alarm is issued directly to indicate the link is unstable without continuing to monitor component 2, component 3 and component 4. For another example, assuming that the target big data link includes component 1 → component 2 → component 3 → component 4, when component 2 or the network between component 1 and component 2 is monitored to be unstable, an alarm is issued directly to indicate that the link is unstable without continuing to monitor component 3, component 4. The description is given here by way of example only and is not intended as a limitation on the present application.
In the embodiment of the application, a piece of data is simulated to flow on a big data link, and external monitoring is performed outside a big data component system in the flowing process, so that the processing task issued to the component is basically consistent with the process of the component in normal data processing when the processing task is executed, and the monitoring effectiveness can be realized. For example, if the component 1 executes the operation 1 when processing the normal data 1, the processing procedure of the processing task issued to the component 1 in the embodiment of the present application in the component 1 also includes the above-mentioned operation 1, and for example, if the component 1 executes the operations 1 and 2 when processing the normal data 1, but the operation 2 has little use for monitoring and analysis, the processing procedure of the processing task issued to the component 1 in the embodiment of the present application in the component 1 may only include the operation 1. The description is given here by way of example only and is not intended as a limitation on the present application.
In an optional embodiment, the processing task may include a task of simulating data writing, and a task of acquiring target data of the simulated data processed by the upstream component from the upstream component.
It should be noted that the type of the analog data may be determined according to a specific service, for example, for service 1, the analog data may be data 1, for service 2, the analog data may be data 2, and for example, for service 1, service 2, the analog data may also be only data 1. In addition, it should be noted that the component corresponding to the task of simulating data writing may be a database, for example, a service database/end service database. The component corresponding to the task of acquiring the target data of the simulation data processed by the upstream component from the upstream component can be a database data extraction component, a data message queue component and a big data computing platform component. The database data extraction component is a tool for uploading/downloading data and can meet most common data cloud scenes. The database data extraction component is also a log-based structured data backup tool, and is generally used for master-slave backup between business databases and synchronization of the business databases to other databases. The database data extraction component is also provided with a log reading module and a log playback module which are highly available, and once the disaster recovery system of the database data extraction component detects that the link is abnormal, the link can be restarted at a breakpoint on the health service node, so that the high availability of the synchronous link is effectively ensured. The data message queue component is a component which supports interactive message content retrieval in the field of message queues first, provides message retrieval service capability based on data dump table storage, supports free combination of retrieval messages according to any conditions such as values and partitions, supports value-taking full-text retrieval messages, has the characteristics of development-free property, operation-free property and high elasticity, can retrieve messages directly through table storage indexes or SQL (structured query language), and greatly improves the speed of daily troubleshooting of the existence or the correctness of the messages. For example, suppose an operation and maintenance team needs to monitor the operation condition of the on-line cluster, the log of the collected process level is imported into the data message queue component, and the downstream uses the big data computing platform component for consumption, and calculates the resource consumption condition of each process in real time. When log data of a certain time period of a certain process is found to be lost in a big data computing platform assembly, a data message queue assembly is required to be used, message data is retrieved based on a message value and a time range, and whether the log is successfully pushed into a message queue of the data message queue assembly is judged.
The above describes the tasks of processing tasks including the task of simulating data writing and the task of acquiring target data of the simulated data processed by the upstream component from the upstream component, and the following explains how to monitor these two situations specifically.
When the processing task is the task for writing the simulation data, determining the monitoring result according to the execution result of the monitoring policy includes: s11, establishing a target table in the component corresponding to the processing task; s12, writing the simulation data into the target table, and feeding back the execution time of the processing task after the writing is finished; and S13, comparing the execution time with a first preset threshold value, and determining the monitoring result of the component corresponding to the processing task according to the comparison result. For example, assuming that a component corresponding to a processing task is a service database component, because normal use of a service is to be simulated, for the service database component, a table is created first, and then data is written into the table and the back, in this embodiment of the present application, a target table is created in the service database component, a single insertion statement is executed to write simulated data into the target table, and a corresponding time is returned when execution of a general insertion statement is completed.
When the processing task is a task of acquiring target data of the simulation data processed by the upstream component from the upstream component, determining a monitoring result according to an execution result of the monitoring policy includes: s21, establishing a target task in a component corresponding to the processing task, wherein the target task is determined by the processing task and corresponds to the function of the component; s22, executing the target task, and determining the acquisition delay of the target data according to the execution result of the target task; and S23, comparing the acquired delay with a second preset threshold, and determining a monitoring result between the component corresponding to the processing task and the upstream component according to the comparison result. For example, assuming that the component corresponding to the processing task is a data message queue component, because normal use of the service is to be simulated, a Topic is established for the data message queue component, and then data in the upstream node is automatically consumed and written into a target message queue of the data message queue component, so that the acquisition delay of the data in the upstream node can be compared with a second preset threshold (for example, 0.5 second), and when the second preset threshold is exceeded, it is determined that a problem occurs in a network between the data message queue component and the upstream node or a problem occurs in the data message queue component itself. Certainly, an operation of end-to-end connection ping can be executed on the connection address of the data message queue component, whether the data message queue component has a problem or not is determined according to the return delay of the ping, a final result is further determined from the network problem between the data message queue component and the upstream node or the data message queue component, and the network stability of the whole link of the large data link and the availability of the component are effectively monitored.
Optionally, in this embodiment, after the detection result is obtained, a periodic comparison may be performed to obtain a state update change of the link condition, so as to improve the user experience.
An embodiment of the present invention further provides an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to perform the steps of any of the methods described above.
Embodiments of the present invention further provide a computer-readable storage medium, on which instructions are stored, and when executed by a processor, the instructions implement the steps of any one of the above methods.
In the above embodiments of the present invention, the description of each embodiment has its own emphasis, and reference may be made to the related description of other embodiments for parts that are not described in detail in a certain embodiment.
In the embodiments provided in the present application, it should be understood that the disclosed technical content can be implemented in other manners. The above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one type of logical functional division, and other divisions may be implemented in practice, for example, multiple units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed coupling or direct coupling or communication connection between each other may be an indirect coupling or communication connection through some interfaces, units or modules, and may be electrical or in other forms.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one position, or may be distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention, which is substantially or partly contributed by the prior art, or all or part of the technical solution may be embodied in a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic disk, or an optical disk, and various media capable of storing program codes.
The foregoing is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, various modifications and decorations can be made without departing from the principle of the present invention, and these modifications and decorations should also be regarded as the protection scope of the present invention.
Claims (12)
1. A big data link full link tracking monitoring method is characterized by comprising the following steps:
after receiving a monitoring task for a target big data link, acquiring a monitoring strategy for the target big data link, wherein the monitoring strategy comprises the following steps: sending a processing task corresponding to the component to a target component on the target big data link, and executing the processing task corresponding to the component in the component;
and determining a monitoring result according to the execution result of the monitoring strategy.
2. The method of claim 1, wherein the monitoring strategy further comprises: and executing the processing task corresponding to the monitoring component in the component to be monitored under the condition that the monitoring result of the upstream component of the component to be monitored is normal.
3. The method of claim 1, wherein the processing tasks corresponding to the component intersect between the execution of the processing tasks within the component and the processing of the data by the component.
4. The method according to claim 1, wherein the processing tasks include a task of simulating data writing, a task of acquiring target data of the simulated data processed by an upstream component from the upstream component.
5. The method according to claim 4, wherein when the processing task is the task simulating data writing, determining a monitoring result according to the execution result of the monitoring policy comprises:
establishing a target table in the component corresponding to the processing task;
writing the simulation data into the target table, and feeding back the execution time of the processing task after the writing is finished;
and comparing the execution time with a first preset threshold value, and determining the monitoring result of the component corresponding to the processing task according to the comparison result.
6. The method of claim 4, wherein when the processing task is a task of acquiring target data of the simulation data processed by an upstream component from the upstream component, determining a monitoring result according to an execution result of the monitoring policy comprises:
establishing a target task in a component corresponding to the processing task, wherein the target task is determined by the processing task and corresponds to the function of the component;
executing the target task, and determining the acquisition delay of the target data according to the execution result of the target task;
and comparing the acquired delay with a second preset threshold, and determining a monitoring result between the component corresponding to the processing task and the upstream component according to a comparison result.
7. The method according to claim 5 or 6, characterized in that the monitoring results comprise network latency data of components and upstream components of the processing task and availability data of the components of the processing task themselves.
8. The method of claim 2, wherein the target component is determined based on a function of a component included in the big data link.
9. The method of claim 1, wherein the target big data link is obtained by combining multiple components according to big data tasks.
10. A big data link full link tracking monitoring system, comprising:
the acquisition module is used for acquiring a monitoring strategy of a target big data link after receiving a monitoring task of the target big data link, wherein the monitoring strategy comprises the following steps: sending a processing task corresponding to the component to a target component on the target big data link, and executing the processing task corresponding to the component in the component;
and the determining module is used for determining a monitoring result according to the execution result of the monitoring strategy.
11. An electronic device comprising a memory and a processor; wherein the memory is configured to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method steps of any of claims 1 to 9.
12. A readable storage medium having stored thereon computer instructions, wherein the computer instructions, when executed by a processor, implement the method steps of any one of claims 1 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211284793.5A CN115766498B (en) | 2022-10-20 | Big data link full-link tracking monitoring method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211284793.5A CN115766498B (en) | 2022-10-20 | Big data link full-link tracking monitoring method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115766498A true CN115766498A (en) | 2023-03-07 |
| CN115766498B CN115766498B (en) | 2024-07-02 |
Family
ID=
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170124272A1 (en) * | 2015-10-30 | 2017-05-04 | Dexcom, Inc. | Data backfilling for continuous glucose monitoring |
| US20200405223A1 (en) * | 2015-07-17 | 2020-12-31 | Chao-Lun Mai | Method, apparatus, and system for automatic and adaptive wireless monitoring and tracking |
| CN112653586A (en) * | 2019-10-12 | 2021-04-13 | 苏州工业园区测绘地理信息有限公司 | Time-space big data platform application performance management method based on full link monitoring |
| CN114124743A (en) * | 2021-11-16 | 2022-03-01 | 广东电网有限责任公司 | Method and system for executing data application full link check rule |
| CN114116396A (en) * | 2021-11-29 | 2022-03-01 | 重庆富民银行股份有限公司 | Full link tracking method, system, storage medium and equipment |
| CN114490268A (en) * | 2022-02-09 | 2022-05-13 | 中国工商银行股份有限公司 | Full link monitoring method, device, equipment, storage medium and program product |
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20200405223A1 (en) * | 2015-07-17 | 2020-12-31 | Chao-Lun Mai | Method, apparatus, and system for automatic and adaptive wireless monitoring and tracking |
| US20170124272A1 (en) * | 2015-10-30 | 2017-05-04 | Dexcom, Inc. | Data backfilling for continuous glucose monitoring |
| CN112653586A (en) * | 2019-10-12 | 2021-04-13 | 苏州工业园区测绘地理信息有限公司 | Time-space big data platform application performance management method based on full link monitoring |
| CN114124743A (en) * | 2021-11-16 | 2022-03-01 | 广东电网有限责任公司 | Method and system for executing data application full link check rule |
| CN114116396A (en) * | 2021-11-29 | 2022-03-01 | 重庆富民银行股份有限公司 | Full link tracking method, system, storage medium and equipment |
| CN114490268A (en) * | 2022-02-09 | 2022-05-13 | 中国工商银行股份有限公司 | Full link monitoring method, device, equipment, storage medium and program product |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9773015B2 (en) | Dynamically varying the number of database replicas | |
| US20150213100A1 (en) | Data synchronization method and system | |
| JP5308403B2 (en) | Data processing failure recovery method, system and program | |
| CN103001796A (en) | Method and device for processing weblog data by server | |
| CN111177165B (en) | Method, device and equipment for detecting data consistency | |
| CN102833281B (en) | It is a kind of distributed from the implementation method counted up, apparatus and system | |
| CN109298978B (en) | Recovery method and system for database cluster of specified position | |
| CN104657497A (en) | Mass electricity information concurrent computation system and method based on distributed computation | |
| US20070220481A1 (en) | Limited source code regeneration based on model modification | |
| JP2020057416A (en) | Method and device for processing data blocks in distributed database | |
| CN114048217A (en) | Incremental data synchronization method and device, electronic equipment and storage medium | |
| CN111240936A (en) | Data integrity checking method and equipment | |
| CN105574026A (en) | Method and device for service supporting by using non-relational database | |
| CN114416868A (en) | Data synchronization method, device, equipment and storage medium | |
| CN115766498A (en) | Big data link full link tracking monitoring method and system | |
| CN115766498B (en) | Big data link full-link tracking monitoring method and system | |
| CN115993932A (en) | Data processing method, device, storage medium and electronic equipment | |
| CN109254880A (en) | A kind of method and device handling database delay machine | |
| KR101656011B1 (en) | System and method for fault monitoring based on big-data | |
| CN113407629A (en) | Data synchronization method and device, electronic equipment and storage medium | |
| CN117349384B (en) | Database synchronization method, system and equipment | |
| CN108268662B (en) | Social graph generation method based on H5 page, electronic device and storage medium | |
| US8214846B1 (en) | Method and system for threshold management | |
| CN113760923B (en) | Data heterogeneous method, device, system and storage medium | |
| EP2833300A1 (en) | Power management of electronic devices configured to generate analytical reports |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |