CN112286757A - Data synchronization monitoring method and device, electronic equipment and storage medium - Google Patents
Data synchronization monitoring method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN112286757A CN112286757A CN202011086521.5A CN202011086521A CN112286757A CN 112286757 A CN112286757 A CN 112286757A CN 202011086521 A CN202011086521 A CN 202011086521A CN 112286757 A CN112286757 A CN 112286757A
- Authority
- CN
- China
- Prior art keywords
- log
- warning information
- service
- logs
- data synchronization
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
- G06F11/3072—Monitoring arrangements determined by the means or processing involved in reporting the monitored data where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/32—Monitoring with visual or acoustical indication of the functioning of the machine
- G06F11/324—Display of status information
- G06F11/327—Alarm or error message display
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/178—Techniques for file synchronisation in file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
Abstract
The invention discloses a data synchronization monitoring method, a data synchronization monitoring device, electronic equipment and a computer storage medium, wherein the monitoring method comprises the following steps: the method comprises the steps of collecting a plurality of service logs, wherein the service logs are provided with log type identifiers, the log type identifiers are used for distinguishing sources of the service logs, the service logs are filtered based on the log type identifiers to obtain first logs, the first logs are stored in an Elasticissearch, the first logs are inquired according to log indexes in the Elasticissearch, and early warning information or warning information is sent according to error codes in the first logs, wherein the early warning information comprises abnormal fluctuation of a network, and the warning information comprises abnormal data transmission. Through the method and the device, the problem of low efficiency in monitoring the data synchronization of the heterogeneous system is solved, and the efficiency in monitoring the data synchronization of the heterogeneous system is improved.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a method and an apparatus for monitoring data synchronization, an electronic device, and a storage medium.
Background
The most sales modes in the tourism industry are modes of performing online e-commerce sales and offline physical window sales synchronously, and orders of an online platform often come from different offline scenic spots, so that order data of the online platform and ticket business clients of each offline entity need to be synchronized, and therefore the orders can be guaranteed to be completed smoothly. In practice, order data synchronization is abnormal, and the online platform or the ticket client can be quickly repaired only by determining the error type. In the related art, interfaces and data of different ticketing clients are not uniform, so that existing services on the different ticketing clients need to be modified, data synchronization can be performed on heterogeneous systems on the different ticketing clients, and a data synchronization process is monitored, and the efficiency is low.
Aiming at the problem of low efficiency in monitoring data synchronization of a heterogeneous system in the related technology, no effective solution is provided at present.
Disclosure of Invention
The embodiment of the application provides a data synchronization monitoring method and device, electronic equipment and a storage medium, so as to at least solve the problem of low efficiency in monitoring data synchronization of a heterogeneous system in the related art.
In a first aspect, an embodiment of the present application provides a method for monitoring data synchronization, where the method includes:
the method comprises the steps of collecting a plurality of service logs, wherein the service logs are configured with log type identifiers, and the log type identifiers are used for distinguishing sources of the service logs;
filtering the plurality of service logs based on the log type identifier to obtain a first log, and storing the first log in an Elasticissearch;
inquiring the first log according to a log index in an elastic search, and sending out early warning information or warning information according to an error code in the first log, wherein the early warning information comprises that the network has abnormal fluctuation, and the warning information comprises that the data transmission has abnormality.
In some embodiments, the filtering the plurality of service logs based on the log type identifier to obtain a first log includes: and configuring a grok expression, and filtering the service log according to a field corresponding to the log type identifier to obtain the first log.
In some embodiments, the filtering the plurality of service logs based on the log type identifier to obtain a first log, and storing the first log in an Elasticsearch includes:
matching a plurality of service logs with client service codes under the condition that the log type identifier is a client log, and capturing order numbers, order time, abnormal codes and operation time consumption of data synchronization in the plurality of service logs to obtain the first log, wherein the client service code is the unique identifier of the client to which the service log belongs;
creating a log index, and storing the first log to an Elasticissearch based on the log index.
In some embodiments, the filtering the plurality of service logs based on the log type identifier to obtain a first log includes:
and matching component service codes for a plurality of service logs under the condition that the log type identifier is an interactive weblog, and capturing component abnormal logs in the service logs to obtain the first log.
In some embodiments, the filtering the plurality of service logs based on the log type identifier to obtain a first log includes:
and matching the serial numbers of the equipment to the plurality of service logs under the condition that the log type identification is the equipment running log, and capturing the equipment running state abnormal log in the plurality of service logs to obtain the first log.
In some embodiments, the issuing warning information or warning information according to the error code in the first log includes:
and sending out early warning information or warning information under the condition that the number of at least one error code in the first log reaches a preset abnormal threshold value.
In some embodiments, the sending out warning information or warning information according to the error code in the first log further includes:
and sending out early warning information or warning information according to the error type corresponding to the error code in the first log, wherein the first log comprises a plurality of error codes, and each error code corresponds to one error type.
In a second aspect, an embodiment of the present application provides a monitoring apparatus for data synchronization, where the monitoring apparatus includes: the system comprises a log acquisition module, a log processing module and a query prompt module;
the log collection module is used for collecting a plurality of service logs, wherein the service logs are configured with log type identifiers, and the log type identifiers are used for distinguishing sources of the service logs;
the log processing module is used for filtering the plurality of service logs based on the log type identifier to obtain a first log, and storing the first log into an elastic search;
the query prompt module is used for querying the first log according to a log index in an elastic search, and sending out early warning information or warning information according to an error code in the first log, wherein the early warning information comprises that the network has abnormal fluctuation, and the warning information comprises that the data transmission has abnormality.
In a third aspect, an embodiment of the present application provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and running on the processor, where the processor implements the monitoring method for data synchronization according to the first aspect when executing the computer program.
In a fourth aspect, the present application provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the monitoring method for data synchronization according to the first aspect.
Compared with the related art, the data synchronization monitoring method provided by the embodiment of the application acquires a plurality of service logs, wherein the service logs are configured with log type identifiers, the log type identifiers are used for distinguishing the sources of the service logs, the service logs are filtered based on the log type identifiers to obtain first logs, the first logs are stored in an elastic search, the first logs are inquired according to log indexes in the elastic search, and early warning information or warning information is sent according to error codes in the first logs, wherein the early warning information comprises abnormal fluctuation of a network, the warning information comprises abnormal data transmission, the problem of low efficiency in monitoring data synchronization of a heterogeneous system is solved, and the efficiency in monitoring the data synchronization of the heterogeneous system is improved.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the application and together with the description serve to explain the application and not to limit the application. In the drawings:
FIG. 1 is a schematic diagram of an application environment of a monitoring method for data synchronization according to an embodiment of the present application;
FIG. 2 is a flow chart of a monitoring method of data synchronization according to an embodiment of the application;
FIG. 3 is a schematic flow chart diagram of a monitoring method for data synchronization according to a preferred embodiment of the present application;
FIG. 4 is a schematic diagram of a monitoring device for data synchronization according to an embodiment of the present application;
fig. 5 is an internal structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments provided in the present application without any inventive step are within the scope of protection of the present application.
It is obvious that the drawings in the following description are only examples or embodiments of the present application, and that it is also possible for a person skilled in the art to apply the present application to other similar contexts on the basis of these drawings without inventive effort. Moreover, it should be appreciated that in the development of any such actual implementation, as in any engineering or design project, numerous implementation-specific decisions must be made to achieve the developers' specific goals, such as compliance with system-related and business-related constraints, which may vary from one implementation to another.
Reference in the specification to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the specification. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those of ordinary skill in the art will explicitly and implicitly appreciate that the embodiments described herein may be combined with other embodiments without conflict.
Unless defined otherwise, technical or scientific terms referred to herein shall have the ordinary meaning as understood by those of ordinary skill in the art to which this application belongs. Reference to "a," "an," "the," and similar words throughout this application are not to be construed as limiting in number, and may refer to the singular or the plural. The present application is directed to the use of the terms "including," "comprising," "having," and any variations thereof, which are intended to cover non-exclusive inclusions; for example, a process, method, system, article, or apparatus that comprises a list of steps or modules (elements) is not limited to the listed steps or elements, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus. Reference to "connected," "coupled," and the like in this application is not intended to be limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The term "plurality" as referred to herein means two or more. "and/or" describes an association relationship of associated objects, meaning that three relationships may exist, for example, "A and/or B" may mean: a exists alone, A and B exist simultaneously, and B exists alone. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship. Reference herein to the terms "first," "second," "third," and the like, are merely to distinguish similar objects and do not denote a particular ordering for the objects.
Fig. 1 is a schematic application environment diagram of a monitoring method for data synchronization according to an embodiment of the present application, and the monitoring method for data synchronization provided by the present application may be applied to the application environment shown in fig. 1. The operating states of the ticketing systems on different scenic spot clients can be monitored through the scenic spot heartbeat monitoring service, and then, scenic spot heartbeat logs are generated. The ticketing system on each scenic spot client generates an order synchronization log. And acquiring a scenic spot heartbeat log and an order synchronization log through the Filebeat, and pushing the scenic spot heartbeat log and the order synchronization log to a Logstash. And filtering the scene area heartbeat log and the order synchronization log on the Logstash, and storing the filtered scene area heartbeat log and the filtered order synchronization log into an elastic search. By inquiring the log index, under the condition that the number of at least one error code in the first log reaches a preset abnormal threshold value, sending out early warning information or warning information according to the error codes in the first log, wherein the early warning information comprises that the network has abnormal fluctuation, and the warning information comprises that the data transmission has abnormality. For example, the error code indicates that the operation is successful on the ticketing system of the client, but the operation time is too long, the early warning information can be sent out in the form of a mail to inform the user that the network has abnormal fluctuation, and the warning information can be sent out in the form of a short message or/and a mail to inform the user that the data transmission has a problem when the error code such as no return value or ordering failure continuously appears. Therefore, the problem of low efficiency in monitoring the data synchronization of the heterogeneous system is solved, and the efficiency of monitoring the data synchronization of the heterogeneous system is improved.
The embodiment provides a monitoring method for data synchronization. Fig. 2 is a flowchart of a monitoring method for data synchronization according to an embodiment of the present application, and as shown in fig. 2, the flowchart includes the following steps:
s210, collecting a plurality of service logs, wherein the service logs are configured with log type identifiers, and the log type identifiers are used for distinguishing sources of the service logs. The service log comprises a heartbeat log, an order log and the like. Each business log is provided with a unique service registration identifier based on different application services in different scenic spots, the sources of the business logs are distinguished through different identifiers, and error types are distinguished through specific abnormal codes of the logs. The service log may originate from clients at various scenic spots, systems interacting with the clients, and off-site equipment components. In the embodiment, a plurality of service logs are collected through the Filebeat, the Filebeat is developed based on the go language without other dependence, and the Filebeat has the biggest characteristics of stable performance, simple configuration, less occupied system resources and very simple installation and use, and is beneficial to improving the efficiency of monitoring data synchronization.
S220, filtering the plurality of service logs based on the log type identifier to obtain a first log, and storing the first log in an elastic search. The plurality of service logs are input into the Logstash to be filtered based on the log type identifier, so that operation logs with the following behaviors, such as ordering, data synchronization, pushing and the like can be filtered, and a first log can be obtained. The Logstash is an open-source data collection engine and has real-time pipeline processing capacity. The Logstash is used as a bridge between a data source and a data storage analysis tool, can dynamically unify data from different data sources, standardizes the data to a selected destination, and can greatly facilitate the processing and analysis of the data. With multiple plug-ins, Logstash can accept a wide variety of data, including logs, network requests, relational databases, sensors or the internet of things, and so on. The Logstash can be connected to the Elasticsearch by configuration using the http protocol. And warehousing and storing the first log into an elastic search, creating a log index record according to the configured index template by the first log, and recording the log index of the first log according to a preset time interval, such as recording the log index by days. The Elasticissearch is a distributed, high-expansion and high-real-time search and data analysis engine.
Preferably, the filtering the plurality of service logs based on the log type identifier to obtain a first log comprises: and configuring a grok expression, and filtering the service log according to a field corresponding to the log type identifier to obtain a first log. The grok filter plug-in is one of several plug-ins available by default in the Logstash. The grok filter plug-in can parse unstructured log data into structured and queryable data. And the efficiency of monitoring the data synchronization of the heterogeneous system is further improved.
S230, inquiring the first log according to the log index in the elastic search, and sending out early warning information or warning information according to the error code in the first log, wherein the early warning information comprises that the network has abnormal fluctuation, and the warning information comprises that the data transmission has abnormality. According to a preset query period, regularly querying a log index of an elastic search, for example, querying a log index record of the last 10 minutes, thereby actively discovering an abnormal log, and sending out early warning information or warning information according to an error code in a first log to inform a user that abnormal fluctuation exists in a network or abnormal data transmission exists.
Preferably, the sending out the early warning information or the warning information according to the error code in the first log comprises: and sending out early warning information or warning information under the condition that the number of at least one error code in the first log reaches a preset abnormal threshold value, so that technicians are informed to perform corresponding repair, and the stability of monitoring data synchronization is improved.
Preferably, when the number of at least one error code in the first log reaches a preset abnormal threshold, the first log sends out early warning information or warning information according to the error type corresponding to the error code in the first log, so that a technician can perform corresponding repair according to different information in a targeted manner, and the stability of monitoring data synchronization is improved. Each error code corresponds to an error type. The error code shows that the operation is successful on the ticketing system of the client, but the operation time is too long, the early warning information can be sent out in the form of a mail to inform a user that abnormal fluctuation exists in the network, and for the continuous error code return without a return value or ordering failure and the like, the warning information can be sent out in the form of a short message or/and a mail to inform the user that a problem occurs in data transmission, so that the user experience is improved.
Through the steps, the heartbeat condition of each offline ticketing client system service can be actively discovered, and the synchronization condition of the online order to the offline ticketing client system can be actively discovered. The method has the advantages that the abnormal condition can be actively found and notified without changing the existing service of the offline ticketing client system, the online platform and the offline ticketing client system in scenic spots can be enabled to be in data synchronization, seamlessly connected and completely decoupled, the user experience can be improved, the problem of low efficiency in monitoring data synchronization between heterogeneous systems is solved, the function of actively finding the abnormal early warning is achieved, and the efficiency of monitoring the data synchronization of the heterogeneous systems is improved.
In some embodiments, in the case that the log type identifier is a client log, the service log includes order information, ticket checking information, order operation, operation time consumption and the like. And matching the service codes of the client side for the plurality of service logs, and capturing order numbers, order time, abnormal codes and operation time consumption of data synchronization in the plurality of service logs to obtain a first log. The client service code is the unique identification of the client to which the service log belongs. All collected service logs are identified through the client service codes, log time, operation contents, operation states (operation success or operation failure) and error codes are identified, then filtering is carried out on the basis of the client service codes, logs under different client service codes are filtered, and meanwhile different logs are filtered according to the operation contents. And then, a log index is created, the first log is stored in an elastic search based on the log index, the first log is inquired according to the log index in the elastic search, when the number of error codes reaches a preset abnormal threshold or the time consumed for operation reaches a preset time-consuming threshold, an early warning or warning operation is triggered, warning information is pushed to a designated user by a message service, and the active abnormal discovery function of data synchronization on the heterogeneous system can be realized.
In some embodiments, in the case that the log type identifier is an interactive weblog, the service log includes external request information, system access throughput information, and a system component operation log. Matching the service codes of the components to the multiple service logs, capturing abnormal component logs in the multiple service logs to obtain a first log, creating an index and storing the index into an Elasticissearch. And regularly inquiring the log index of the Elasticissearch according to a preset inquiry period, and triggering early warning or warning operation when an early warning triggering condition is reached.
In some embodiments, when the log type identifier is an equipment running log, matching the equipment numbers to the multiple service logs, capturing an equipment running state abnormal log in the multiple service logs, obtaining a first log, creating an index, and storing the index into an Elasticsearch. And regularly inquiring the log index of the Elasticissearch according to a preset inquiry period, and triggering early warning or warning operation when an early warning triggering condition is reached.
The embodiments of the present application are described and illustrated below by means of preferred embodiments. Fig. 3 is a flow chart of a monitoring method for data synchronization according to the preferred embodiment of the present application. As shown in fig. 3, data synchronization between the online platform and the ticketing clients in a plurality of heterogeneous offline scenic spots is monitored, and in a log collection stage, service logs of each ticketing client, device and network are collected through Filebeat, for example, heartbeat status logs of each scenic spot client and online orders are synchronized to offline order logs and the like. In the case that the log type identifier of the service log is a client log, the service log may include order information, ticket checking information, order operation, operation time consumption, and the like. In the case that the log type identifier of the service log is an interactive weblog, the service log may include external request information, system access throughput information, a system component operation log, and the like. In the case that the log type identifier of the service log is the device component log, the service log may include a registered application service heartbeat, a gate running state, an application port state, a device disk running state, a CPU use state, a memory state, a database state, and the like. Each log is provided with unique service code identifications based on different application services in different scenic spots, the source of the log is distinguished through different identifications, and the error type is distinguished through specific error codes of the log.
And inputting the collected service logs into a Logstash, and filtering the logs by the Logstash according to different service configurations. And filtering different service logs by different grok expressions, and collecting a first log to an elastic search according to a formatted field. And creating a log index record according to the acquired log and the configured index template. And setting a customizer according to a preset query period (such as 10 minutes), and querying the log index record at regular time, thereby actively discovering the abnormal log. And recording abnormal failure times (namely the number of error codes) according to the inquired first log, and when the abnormal failure times reaches an abnormal failure times threshold configured according to each log type identifier, sending early warning information or warning information to a specified user according to the error type corresponding to the error code.
When the data synchronization monitoring method is applied to a plurality of heterogeneous offline ticketing systems in an offline scenic region, the heartbeat condition of each offline scenic region system service can be actively discovered, and the synchronization condition of online orders to the offline ticketing systems in the scenic region can be actively discovered. The existing business does not need to be changed, the monitoring program configures a business abnormal frequency threshold value through a business log, actively finds abnormal conditions and notifies the abnormal conditions, and the monitoring service, the scenic spot offline ticket business service, the entrance ticket synchronous service, seamless connection and complete decoupling are realized.
The embodiment of the application provides a monitoring device for data synchronization. Fig. 4 is a schematic structural diagram of a monitoring apparatus for data synchronization according to an embodiment of the present application, and as shown in fig. 4, the monitoring apparatus includes a log collection module 410, a log processing module 420, and a query prompt module 430: the log collection module 410 is configured to collect a plurality of service logs, where the service logs are configured with log type identifiers, and the log type identifiers are used to distinguish sources of the service logs; the log processing module 420 is configured to filter the multiple service logs based on the log type identifier to obtain a first log, and store the first log in an Elasticsearch; the query prompt module 430 is configured to query the first log according to the log index in the Elasticsearch, and send out early warning information or warning information according to an error code in the first log, where the early warning information includes that there is an abnormal fluctuation in the network, and the warning information includes that there is an abnormality in data transmission.
For specific limitations of the monitoring device for data synchronization, reference may be made to the above limitations of the monitoring method for data synchronization, which are not described herein again. The modules in the monitoring device for data synchronization can be wholly or partially realized by software, hardware and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
An embodiment of the present application further provides an electronic device, which includes a memory and a processor, where the memory stores a computer program, and the processor is configured to execute the computer program to perform the steps in any of the method embodiments described above.
Optionally, the electronic device may further include a transmission device and an input/output device, wherein the transmission device is connected to the processor, and the input/output device is connected to the processor.
It should be noted that, for specific examples in this embodiment, reference may be made to examples described in the foregoing embodiments and optional implementations, and details of this embodiment are not described herein again.
In addition, in combination with the monitoring method for data synchronization in the foregoing embodiments, the embodiments of the present application may provide a storage medium to implement. The storage medium having stored thereon a computer program; the computer program, when executed by a processor, implements any of the monitoring methods for data synchronization in the above embodiments.
In one embodiment, fig. 5 is a schematic diagram of an internal structure of an electronic device according to an embodiment of the present application, and as shown in fig. 5, an electronic device is provided, where the electronic device may be a server, and the internal structure diagram may be as shown in fig. 5. The electronic device includes a processor, a memory, a network interface, and a database connected by a system bus. Wherein the processor of the electronic device is configured to provide computing and control capabilities. The memory of the electronic equipment comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The database of the electronic device is used for storing data. The network interface of the electronic device is used for connecting and communicating with an external terminal through a network. The computer program is executed by a processor to implement a method of monitoring data synchronization.
Those skilled in the art will appreciate that the configuration shown in fig. 5 is a block diagram of only a portion of the configuration associated with the present application, and does not constitute a limitation on the electronic device to which the present application is applied, and a particular electronic device may include more or less components than those shown in the drawings, or may combine certain components, or have a different arrangement of components.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
It should be understood by those skilled in the art that various features of the above-described embodiments can be combined in any combination, and for the sake of brevity, all possible combinations of features in the above-described embodiments are not described in detail, but rather, all combinations of features which are not inconsistent with each other should be construed as being within the scope of the present disclosure.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (10)
1. A method for monitoring data synchronization, the method comprising:
the method comprises the steps of collecting a plurality of service logs, wherein the service logs are configured with log type identifiers, and the log type identifiers are used for distinguishing sources of the service logs;
filtering the plurality of service logs based on the log type identifier to obtain a first log, and storing the first log in an Elasticissearch;
inquiring the first log according to a log index in an elastic search, and sending out early warning information or warning information according to an error code in the first log, wherein the early warning information comprises that the network has abnormal fluctuation, and the warning information comprises that the data transmission has abnormality.
2. The method of claim 1, wherein filtering the plurality of traffic logs based on the log type identifier to obtain a first log comprises: and configuring a grok expression, and filtering the service log according to a field corresponding to the log type identifier to obtain the first log.
3. The method of claim 1, wherein filtering the plurality of traffic logs based on the log type identifier to obtain a first log, and storing the first log in an Elasticsearch comprises:
matching a plurality of service logs with client service codes under the condition that the log type identifier is a client log, and capturing order numbers, order time, abnormal codes and operation time consumption of data synchronization in the plurality of service logs to obtain the first log, wherein the client service code is the unique identifier of the client to which the service log belongs;
creating a log index, and storing the first log to an Elasticissearch based on the log index.
4. The method of claim 2, wherein filtering the plurality of traffic logs based on the log type identifier to obtain a first log comprises:
and matching component service codes for a plurality of service logs under the condition that the log type identifier is an interactive weblog, and capturing component abnormal logs in the service logs to obtain the first log.
5. The method of claim 3, wherein filtering the plurality of traffic logs based on the log type identifier to obtain a first log comprises:
and matching the serial numbers of the equipment to the plurality of service logs under the condition that the log type identification is the equipment running log, and capturing the equipment running state abnormal log in the plurality of service logs to obtain the first log.
6. The method according to any one of claims 1 to 5, wherein the sending out warning information or alarm information according to the error code in the first log comprises:
and sending out early warning information or warning information under the condition that the number of at least one error code in the first log reaches a preset abnormal threshold value.
7. The method of claim 6, wherein the sending out warning information or warning information according to the error code in the first log further comprises:
and sending out early warning information or warning information according to the error type corresponding to the error code in the first log, wherein the first log comprises a plurality of error codes, and each error code corresponds to one error type.
8. A monitoring device for data synchronization, the monitoring device comprising: the system comprises a log acquisition module, a log processing module and a query prompt module;
the log collection module is used for collecting a plurality of service logs, wherein the service logs are configured with log type identifiers, and the log type identifiers are used for distinguishing sources of the service logs;
the log processing module is used for filtering the plurality of service logs based on the log type identifier to obtain a first log, and storing the first log into an elastic search;
the query prompt module is used for querying the first log according to a log index in an elastic search, and sending out early warning information or warning information according to an error code in the first log, wherein the early warning information comprises that the network has abnormal fluctuation, and the warning information comprises that the data transmission has abnormality.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and running on the processor, characterized in that the processor implements the monitoring method of data synchronization according to any of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium having stored thereon a computer program, characterized in that: the computer program, when executed by a processor, implements the method of monitoring data synchronization of any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011086521.5A CN112286757A (en) | 2020-10-12 | 2020-10-12 | Data synchronization monitoring method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011086521.5A CN112286757A (en) | 2020-10-12 | 2020-10-12 | Data synchronization monitoring method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112286757A true CN112286757A (en) | 2021-01-29 |
Family
ID=74496852
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011086521.5A Pending CN112286757A (en) | 2020-10-12 | 2020-10-12 | Data synchronization monitoring method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112286757A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113157545A (en) * | 2021-05-20 | 2021-07-23 | 京东方科技集团股份有限公司 | Method, device and equipment for processing service log and storage medium |
| CN113239007A (en) * | 2021-05-14 | 2021-08-10 | 广州广电运通金融电子股份有限公司 | Multi-level heterogeneous log transaction analysis method, system, equipment and storage medium |
| CN113660356A (en) * | 2021-08-16 | 2021-11-16 | 迈普通信技术股份有限公司 | Network access method, system, electronic device and computer readable storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103412893A (en) * | 2013-07-24 | 2013-11-27 | 广东电子工业研究院有限公司 | Collecting system and collecting method of logs |
| CN110245158A (en) * | 2019-06-10 | 2019-09-17 | 上海理想信息产业（集团）有限公司 | A kind of multi-source heterogeneous generating date system and method based on Flink stream calculation technology |
| CN110750426A (en) * | 2019-10-30 | 2020-02-04 | 北京明朝万达科技股份有限公司 | Service state monitoring method and device, electronic equipment and readable storage medium |
| CN111274095A (en) * | 2020-02-24 | 2020-06-12 | 深圳前海微众银行股份有限公司 | Log data processing method, device, equipment and computer readable storage medium |
| CN111400127A (en) * | 2020-02-28 | 2020-07-10 | 平安医疗健康管理股份有限公司 | Service log monitoring method and device, storage medium and computer equipment |
-
2020
- 2020-10-12 CN CN202011086521.5A patent/CN112286757A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103412893A (en) * | 2013-07-24 | 2013-11-27 | 广东电子工业研究院有限公司 | Collecting system and collecting method of logs |
| CN110245158A (en) * | 2019-06-10 | 2019-09-17 | 上海理想信息产业（集团）有限公司 | A kind of multi-source heterogeneous generating date system and method based on Flink stream calculation technology |
| CN110750426A (en) * | 2019-10-30 | 2020-02-04 | 北京明朝万达科技股份有限公司 | Service state monitoring method and device, electronic equipment and readable storage medium |
| CN111274095A (en) * | 2020-02-24 | 2020-06-12 | 深圳前海微众银行股份有限公司 | Log data processing method, device, equipment and computer readable storage medium |
| CN111400127A (en) * | 2020-02-28 | 2020-07-10 | 平安医疗健康管理股份有限公司 | Service log monitoring method and device, storage medium and computer equipment |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113239007A (en) * | 2021-05-14 | 2021-08-10 | 广州广电运通金融电子股份有限公司 | Multi-level heterogeneous log transaction analysis method, system, equipment and storage medium |
| CN113239007B (en) * | 2021-05-14 | 2023-09-29 | 广州广电运通金融电子股份有限公司 | Multi-level heterogeneous log transaction analysis method, system, equipment and storage medium |
| CN113157545A (en) * | 2021-05-20 | 2021-07-23 | 京东方科技集团股份有限公司 | Method, device and equipment for processing service log and storage medium |
| CN113660356A (en) * | 2021-08-16 | 2021-11-16 | 迈普通信技术股份有限公司 | Network access method, system, electronic device and computer readable storage medium |
| CN113660356B (en) * | 2021-08-16 | 2024-01-23 | 迈普通信技术股份有限公司 | Network access method, system, electronic device and computer readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112286757A (en) | Data synchronization monitoring method and device, electronic equipment and storage medium | |
| US7644408B2 (en) | System for assigning and monitoring grid jobs on a computing grid | |
| WO2021169064A1 (en) | Edge network-based anomaly processing method and apparatus | |
| CN110995468A (en) | System fault processing method, device, equipment and storage medium of system to be analyzed | |
| US20070130330A1 (en) | System for inventing computer systems and alerting users of faults to systems for monitoring | |
| CN112035404A (en) | Medical data monitoring and early warning method, device, equipment and storage medium | |
| GB2456619A (en) | Managing errors generated in an apparatus | |
| CN112506870B (en) | Data warehouse increment updating method and device and computer equipment | |
| CN112506755B (en) | Log acquisition method, device, computer equipment and storage medium | |
| CN108322495B (en) | Method, device and system for processing resource access request | |
| CN112015752A (en) | Method and device for dynamically querying data, electronic equipment and storage medium | |
| CN115622906A (en) | Application log capturing system and method | |
| CN111049882B (en) | Cache state processing system, method, device and computer readable storage medium | |
| CN105893150B (en) | Interface calling frequency control method and device and interface calling request processing method and device | |
| CN108880913B (en) | traffic characteristic management method and device and central node server | |
| CN114422576A (en) | Session cleaning method and device, computer equipment and readable storage medium | |
| CN114844771A (en) | Monitoring method, device, storage medium and program product for micro-service system | |
| CN110011845B (en) | Log collection method and system | |
| CN116431677B (en) | Data routing method, system and readable storage medium based on memory database | |
| US20180324036A1 (en) | System for cloud-based service outage detection and verification | |
| CN113448747B (en) | Data transmission method, device, computer equipment and storage medium | |
| CN112637068B (en) | Network data forwarding method, computer device, computer network and storage medium | |
| CN113472808B (en) | Log processing method and device, storage medium and electronic device | |
| US11611513B2 (en) | System and method for discovering interfaces in a network | |
| CN108062327B (en) | Matching method and device for client |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210129 |
|
| RJ01 | Rejection of invention patent application after publication |