CN117290123A - Data monitoring processing method, device, equipment and storage medium - Google Patents

Abstract

The application is applicable to the technical field of data monitoring, and provides a data monitoring processing method, a device, equipment and a storage medium, which comprise the following steps: acquiring at least one piece of data change information; transmitting the at least one piece of data change information to a card message queue by using a card module; processing the at least one piece of data change information in the kaff card message queue; and when the data processing process of the at least one piece of data change information triggers a preset fault threshold value, fault processing is carried out. According to the scheme, the data change information is monitored in real time through the Canale component, so that the information of service data change can be timely obtained, and the real-time performance of service data processing is ensured; meanwhile, stability and accuracy of the business data processing process can be guaranteed by utilizing the characteristics of high reliability and high throughput of the Kaff card message queue.

Description

Data monitoring processing method, device, equipment and storage medium

Technical Field

The present disclosure relates to the field of data monitoring technologies, and in particular, to a data monitoring processing method, device, equipment, and storage medium.

Background

In the working process of the service system, the service system needs to collect and store a large amount of service data, including structured data (such as tables in a database) and unstructured data (such as log files, documents, etc.), and the service system needs to process and analyze the collected data to extract corresponding information.

In order to be able to process the collected data more efficiently, some business systems in related schemes take the form of timed batch processing. However, this approach has certain drawbacks that limit the performance and efficiency of the business system. For example, a timed batch process requires waiting a period of time for data processing, which means that data delays can occur in the business system, thereby affecting the accuracy and timeliness of business decisions.

Disclosure of Invention

The embodiment of the application provides a data monitoring processing method, a device, equipment and a storage medium, which can solve the technical problem of how to monitor and stably process service data in real time.

In a first aspect, an embodiment of the present application provides a data monitoring processing method, including:

at least one piece of data change information is acquired.

At least one piece of data change information is sent to a card message queue by using a card ner component.

By utilizing the Canale component to monitor the data change operation of the service table, the change event of the database can be acquired and processed in real time, and a more flexible and reliable data synchronization and processing mechanism is provided.

And processing at least one piece of data change information in the Kaff card message queue.

The message queue of the card is utilized for further processing, the message queue can be used as a middleware, and a sender and a receiver of the message are decoupled, so that the message queue and the receiver can be independently developed and deployed, and the flexibility of a data processing process is improved.

And when the data processing process of at least one piece of data change information triggers a preset fault threshold value, performing fault processing.

The emergency is handled by setting the fault threshold, the reliability of the data processing process is ensured,

in a second aspect, embodiments of the present application provide a data monitoring processing apparatus, where the apparatus has a function of implementing the method in the first aspect or any possible implementation manner thereof. In particular, the apparatus comprises means for implementing the method of the first aspect or any possible implementation thereof.

In one embodiment thereof, the apparatus comprises:

and the acquisition unit is used for acquiring at least one piece of data change information.

And the receiving and transmitting unit is used for transmitting the data change information to the card message queue by using the card Navigator assembly.

And the processing unit is used for processing the data change information in the card message queue.

The processing unit is also used for performing fault processing when the data processing process of the data change information triggers a preset fault threshold value.

In a third aspect, embodiments of the present application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the computer program to cause the computer device to implement a method according to any one of the implementation manners of the first aspect.

In a fourth aspect, embodiments of the present application provide a computer readable storage medium, where a computer program is stored, where the computer program when executed by a computer device causes the computer device to implement a method according to any implementation manner of the first aspect.

In a fifth aspect, embodiments of the present application provide a computer program product for, when run on a computer device, causing the computer device to perform the method of any one of the implementations of the first aspect described above.

Compared with the prior art, the embodiment of the application has the beneficial effects that: the data change information data is monitored in real time through the Canal assembly, so that the information of service data change can be timely obtained, and the real-time performance of service data processing is ensured; meanwhile, the stability and the accuracy of the business data processing process can be ensured by utilizing the characteristics of high reliability and high throughput of the Kaff card message queue; by utilizing the preset fault threshold value and performing fault processing, the method can realize the effective processing of sudden faults in the data change information processing process, further improve the stability and reliability of the data processing process and reduce the service interruption and data loss risks caused by the data processing faults.

Drawings

Fig. 1 is a data monitoring processing method provided in an embodiment of the present application.

Fig. 2 is a method for processing data change information in a kaff card message queue according to an embodiment of the present application.

Fig. 3 is a flow chart of another data monitoring processing method according to an embodiment of the present application.

Fig. 4 is a schematic structural diagram of an apparatus according to an embodiment of the present application.

Fig. 5 is a schematic structural diagram of a computer device according to an embodiment of the present application.

Detailed Description

In daily life, the behavior of people in life, work and the like can be represented by data, for example, the data corresponding to the transaction behavior of depositing at a bank can comprise account information, deposit amount, transaction time, transaction place and the like. To manage such data more easily and efficiently, the corresponding business system is typically used by the service provider or enterprise to perform the processing.

With the development of the internet, the data volume of service data has proliferated, and at the same time, the requirement for real-time performance of service system processing data has also increased. How to efficiently process huge data volume and provide real-time related business services for users in time is a urgent problem to be solved.

Aiming at the problems, the application provides a data monitoring processing method which can monitor service data in real time, thereby solving the technical problems of service data delay and poor instantaneity.

In order to further explain the technical solution of the present application, the following description is given by specific examples.

For ease of understanding, related terms and concepts related to the embodiments of the present application are described below.

(1)Binlog

The binary log file (Binlog) is a binary log in MySQL database for recording the change operation of the database. It contains all SQL statements that modify the database, such as insert, update, delete, etc.

The principle of Binlog is by recording the change operation of the database in binary form in a log file. When the database executes an SQL statement that modifies the data, the operation is written into Binlog. The Binlog files are named and segmented according to a time sequence, and each Binlog file contains database change operation in a period of time.

(2)Canal

Canal (Canal) is a MySQL based incremental data subscription and consumption component of the Alaba open source. As an open source database change data capture solution, canal is used to monitor and analyze Binlog events of the database and acquire the database change operation in real time.

(3)Kafka

A Kafka (Kafka) is a message queue, which can also be understood as a distributed stream processing platform, for high throughput, scalable real-time data stream processing.

The principle of Kafka is based on a publish-subscribe model by publishing data in the form of messages into the topic (topic) of Kafka, from which subscribers can then consume the messages. The core components of Kafka include Producer, proxy server and Consumer.

The Producer is responsible for publishing the data in the form of messages into the topic of Kafka. The Producer may send messages into multiple partitions, with the messages in each partition being stored in sequence.

The Broker is a middleware of Kafka and is responsible for receiving and storing messages sent by the Producer and distributing the messages to the corresponding Consumers. A Broker is a distributed message queue, and high availability and load balancing can be achieved by adding multiple brokers.

Consumer subscribes to and consumes messages in Kafka. Consumer may be organized in Consumer groups, each of which may have multiple Consumer instances, each instance consuming messages in one or more partitions.

(4)RabbitMQ

Rabbit message queues (Rabbit MQ) allow applications to communicate via messages rather than directly relying on network connections. It provides a reliable messaging mechanism that ensures that messages can be securely delivered to a target application. The method supports a publish-subscribe mode and a point-to-point mode, and can meet the requirements of different application scenes. The core concepts of RabbitMQ include: producer, consumer, queue, exchange, binding, and Routing Key.

Wherein, the producer refers to an application program for sending messages; consumer refers to an application that receives and processes messages; the queue is used for storing the messages, the producer sends the messages to the queue, and the consumer receives the messages from the queue; the switch may receive the message sent by the producer and route it to the queue; the binding function is to associate the queue with the switch, defining the routing rule of the message; the routing key is a key specified by the producer when sending a message for routing the message to a specified queue.

When the system works, a producer sends a message to a switch of the RabbitMQ, the switch routes the message to a corresponding queue according to a binding rule, a consumer acquires the message from the queue for processing, and if the consumer cannot process the message in time, the message is kept in the queue for processing by the consumer.

Fig. 1 is a data monitoring processing method provided in an embodiment of the present application.

As shown in fig. 1, the above method includes the following steps S101 to S104.

S101, at least one piece of data change information is acquired.

The data change information refers to information corresponding to data change operation on the data information in the service table.

It can be understood that the data change operation includes a series of business operations such as a new operation, a delete operation, or a modify operation, and the number of corresponding data change information is at least one.

Each industry has its corresponding business data and business processes. The following describes related business data by taking a bank as an example.

Assume that a bank's business table includes the following data information: customer information, account information, transaction information, loan information, interest rate information, risk assessment information, and the like.

The customer information includes basic information of the customer, such as name, ID card number, contact information, etc. This information is used to identify and manage the customer.

The account information includes account information of the customer, such as account type (savings account, checking account, etc.), account number, account date of opening, account balance, etc. This information is used to manage the customer's account and funds.

The transaction information includes a customer's transaction records such as deposit, withdrawal, transfer, payment, etc. This information is used to record the customer's transaction activity and funds movement.

The loan information includes the customer's loan records, such as loan type, loan amount, loan duration, repayment plan, etc. This information is used to manage the loan business and repayment profiles of the customers.

The interest rate information includes interest rate information of banks, such as deposit interest rate, loan interest rate, exchange rate, etc. This information is used to guide the loan deposit decisions and foreign exchange transactions of the customer.

The risk assessment information includes risk assessment results of the customer, such as credit rating, risk level, and the like. This information is used to assess the credit status and risk tolerance of the customer.

The specific service table contents vary according to the service type and requirements, and are merely examples and are not limiting.

By combining the contents in the banking table, the specific meaning of the data change information can be more intuitively understood. For example, when a contact of a certain user is changed, the contact of the certain user before the change is "18888888888", the contact of the certain user after the change is "16666666666", and the data change information refers to information for changing the contact of the certain user. It will also be appreciated that the data change information may indicate that the user's contact is transitioning from "18888888888" to "16666666666".

It will be appreciated that different data change operations correspond to different data change information, and the principle is the same as that of changing the contact manner of a user, and is not listed here.

After knowing the meaning of the data change information, a way of obtaining the data change information is described below.

In one implementation, a binary log file may be utilized to obtain data change information.

Taking MySQL database as an example, when the data of the business table is changed, mySQL records these change operations in Binlog. The specific manner of recording depends on the configuration of MySQL, which generally includes two ways.

The first is statement-based replication, which records the SQL statement itself that is executed. For example, if an UPDATE statement is executed, binlog records the contents of the UPDATE statement.

The second is a line-based copy, which records changes at the line level. For example, if an UPDATE statement is executed, binlog records the specific content of the modified row.

It is to be understood that the content of the data change information corresponding to different recording modes is also different, and the specific recording mode may be selected according to the actual situation, which is not limited herein.

By utilizing the Binlog to record the data change operation, the functions of database backup, data recovery, data synchronization and the like can be realized. Meanwhile, operations such as data audit and data analysis can be performed by analyzing and analyzing the Binlog file.

S102, at least one piece of data change information is sent to a Kaff card message queue by using a Karl component.

As can be seen in the technical terms above, the kanal component (cana) can be used to monitor changes to the Binlog file in real time and forward the change operation to other systems or applications, such as the Kafka message queue (Kafka).

The following describes the operation flow of monitoring data change information by means of the Canal and transmitting it to Kafka.

And carrying out corresponding configuration on a Canal Server (Canal Server), wherein the configuration comprises corresponding service database connection information, binlog analysis rules, connection information of Kafka and the like. And starting the Canal Server through a command line or a script to enable the Canal Server to start monitoring Binlog events of the corresponding service database.

The Canal Server analyzes the monitored Binlog event, including operations such as insertion, update, deletion, etc., and converts the analyzed event into a specific data format. The Canal Server will send the transformed Binlog event to the designated Topic of Kafka. The configuration file of the Canal may specify which Kafka cluster, which Topic, and other relevant configuration information to send to, and is specifically selected according to the actual situation, which is not limited herein.

By utilizing the data change operation of the Canal monitoring service table, the change event of the database can be acquired and processed in real time, and a more flexible and reliable data synchronization and processing mechanism is provided. Meanwhile, canal also supports various databases, such as MySQL, oracle and the like, and can be suitable for different database environments and business scenes. Meanwhile, by utilizing the open source characteristic, the high research and development cost is avoided, and the development and maintenance cost of the system is reduced.

Through the steps, the Canal can send the data change information (Binlog event) of the corresponding database to the Kafka in real time, so that the real-time data subscription and distribution of the database are realized. In this way, other systems or applications can acquire the update operation of the database in real time by consuming the data in Kafka, so as to facilitate corresponding processing and analysis.

S103, processing at least one piece of data change information in the Kaff card message queue.

In Kafka, consumer clients can be used to subscribe to and consume data change information (Binlog events) sent by canals. The consumer may perform data processing, storage, or further distribution as desired.

In connection with the above technical terms, an operation method for processing data change information using Kafka is described below.

A topic of Kafka is created for storing the operation of the data change information.

In the business system, a Kafka Producer (Kafka Producer) component is added. When the data of the service table is changed, the operation of changing the data is sent to the topic of Kafka in the form of a message through the Producer.

At the other end of the business system, a Kafka Consumer (Kafka Consumer) component is added. Consumer subscribes to topic of Kafka, from which data change messages are consumed.

Asynchronous message transmission and decoupling can be realized through the Kafka message queue, and a service system can monitor and consume data change information in the Kafka in real time, so that real-time synchronization and processing of service data are realized.

S104, when the data processing process of at least one piece of data change information triggers a preset fault threshold, fault processing is carried out.

The preset fault threshold value represents a value of fault risk existing in a certain value in the data processing process of the preset data change information.

By way of example and not limitation, the preset fault threshold may include a preset time threshold, a preset error proportion threshold, and the like.

For example, in the process of processing the data change information, if the data processing time exceeds the preset time threshold, the preset fault threshold may be determined to be triggered, so that fault processing is performed.

For another example, if the error rate during the data processing exceeds the preset error rate threshold, the trigger of the preset fault threshold may be determined to perform fault processing.

It will be appreciated that, as long as the value can represent a fault in the data processing process, such as abnormal operation, login time, transaction location, etc., can be set as a preset fault threshold, and specifically may be selected according to practical situations, which are not listed here.

By the method, real-time monitoring of data is realized by using Binlog, information of service data change can be timely obtained, and real-time performance of service is ensured; the Canal is used for monitoring the data change information and sending the data change information to the Kafka, so that the stability and the accuracy of the service data can be ensured; by presetting a fault threshold and performing corresponding fault processing, the stability and reliability of the system can be further improved, and the service interruption and data loss risks caused by data processing faults are reduced.

A method of processing data change information in a card message queue is described below.

In one embodiment, the at least one piece of data change information includes data change information from a plurality of service data tables, and processing the at least one piece of data change information in the card message queue includes: and processing the data change information of the plurality of service data tables by using the information processing platform.

It will be appreciated that the source of the data change information may be the same service data table or may be a different data table. The meaning of the service data tables is the same as that of the service data tables, that is, the service data tables are not described in detail in order to indicate the source of the data change information.

The information processing platform refers to a system or application program for consuming data change information (such as Binlog event) of a message queue and analyzing business table change data. The main function of the method is to acquire data change information from a message queue (such as Kafka), analyze the data change operation in the information queue and perform corresponding business processing.

A method of handling data change information in a card message queue is described in detail below with reference to fig. 2.

Fig. 2 is a method for processing data change information in a kaff card message queue according to an embodiment of the present application. As shown in fig. 2, fig. 2 includes the following steps S201 to S203.

S201, acquiring data change information of a plurality of service data tables from the Kaff card message queue.

It will be appreciated in light of the foregoing that in an information processing platform, a corresponding consumer program is written to consume data change information in a message queue. The consumer program may subscribe to and consume messages using the interface provided by the message queue.

S202, analyzing data changing operations corresponding to the data changing information of the plurality of business data tables.

Taking the example that the data change information is a Binlog event as an example, if the information processing platform consumes the Binlog event, the information processing platform needs to analyze the data change operation in the Binlog event. The corresponding parsing tool or library can be selected to parse the Binlog event according to the type and the Binlog format of the database, and the business table change data in the Binlog event can be extracted.

S203, corresponding data processing is performed according to the data changing operation.

After analyzing the business table change data, the information processing platform can perform corresponding processing according to specific business requirements.

For example, the message processing platform may store the data change information in other databases or data warehouses for data analysis, report generation, and the like. For another example, the message processing platform may query other databases for data change information, etc., and specific data processing operations may be processed according to actual requirements, which are not listed here.

The message processing platform acquires the data change information from the Kafka message queue and further processes the data change information, so that the message queue can be used as middleware, and a sender and a receiver of the message can be decoupled, so that the sender and the receiver can be independently developed and deployed. The sender only needs to send the message to the message queue, and the receiver can acquire the message from the message queue for processing, so that direct interaction is not needed, and the dependency between systems is reduced. It can also be understood that the message processing platform can implement the capability of asynchronously processing the message, and after the message is sent to the message queue, the sender can immediately return without waiting for the processing result of the receiver, thereby improving the response speed and throughput of the system.

In addition, the message processing platform can support a plurality of consumers to acquire messages from the message queue for processing, and can dynamically increase or decrease the consumers according to the load condition of the system, so that the elastic expansion of the system is realized. The message processing platform generally provides rich monitoring and management functions, can monitor the sending and processing conditions of the message in real time, performs performance analysis and fault detection, and improves the stability and reliability of the system.

Another method of processing the data change information in the kaff card message queue is described below.

In one embodiment, the service system corresponding to the first data change information is reversely checked, and the first data change information is any one of at least one piece of data change information.

And acquiring first association information corresponding to the first data change information by utilizing the service system, and processing data corresponding to the first data change information by combining the first association information.

The service system corresponding to the first data change information may be a database of synchronous data change information, or may be a database having a business association relationship with the data change information.

The first association information is information having a business association with the first data change information. The data corresponding to the first association information and the data corresponding to the first data change information may be the same type of data, or may be different types of data having a business supporting relationship with each other.

For example, assume that there is an e-commerce platform, and when a user places an order, a message of an order change is generated, including an order number, a user ID, a commodity ID, and the like. At this time, the association information of the order change message may include: commodity information, payment information, and the like.

In one implementation, when the first data change information is acquired, the first data change information is synchronized to other databases while the first data change information is sent to the kafu card message queue by the kanal component.

For example, the first data change information is synchronized to the line analysis processing (Online Analytical Processing, OLAP) library while the first data change information is processed, and the synchronized data change information (i.e., the first association information) in the OLAP library can be directly queried by the information processing platform. At this time, the information processing platform may implement the checking function by querying the synchronized data change information in the OLAP library while consuming the Kafka message queue, for example, checking whether the transaction amount information is identical or not, and the like (i.e., processing the data corresponding to the first data change information in combination with the first association information). That is, if the data in the Kafka message queue has errors, the real-time correction function can be realized by reversely checking the synchronous data in the OLAP library, and the accuracy of real-time data processing is improved.

In one implementation, when the first data change information is acquired, the first association information of another database having an association relationship with the first data change information may be acquired while the first data change information is sent to the message queue by using the kanal component.

In combination with the above, it is assumed that there is an e-commerce platform, and when a user places an order, a message of order change is generated. By subscribing to the order change message (i.e., the first data change information) in Kafka, basic information of the order, such as an order number, a user ID, a commodity ID, and the like, can be acquired. But if more in-depth analysis is desired, it may be necessary to obtain order-related user information, merchandise information, payment information (i.e., first-related information), etc. At this time, the user table, the commodity table and the payment system which have the association relation with the order information are reversely checked through the information processing platform, so that the user information, the commodity information, the payment information and the like related to the order can be obtained, and more comprehensive order data can be obtained. Thus, more accurate statistical analysis, personalized recommendation, risk control and other business processes can be performed.

The data in other business tables or business systems are reversely checked to acquire more comprehensive business data and perform deeper analysis and processing. It will be appreciated that by subscribing to the message in Kafka, information about the change of the service data may be obtained, but these information may be only key information for the change operation and may not provide the complete service context and related data.

By looking back up data in other business tables or systems, more information about the business data changes can be obtained, including data of other tables, associated data, historical data, and the like. These data may help us better understand the context and impact of business data changes, further analyzing and processing the data.

In one embodiment, network abnormality or system abnormality detection is performed, and when network abnormality and/or system abnormality is detected, the information processing platform sends data change information to a rabbit message queue and processes the data change information.

Network anomalies refer to anomalies that occur during computer network communications, resulting in unstable or non-functioning network connections. Network anomalies may be caused by a variety of reasons, such as network failures, network congestion, network equipment failures, and the like. For example, a router in the network fails, resulting in increased data transmission delay or packet loss.

The system abnormality refers to an abnormal condition of the computer system in the running process, which causes the system to fail to work or generate errors. System anomalies may be caused by a variety of causes, such as software failures, hardware failures, resource exhaustion, and the like. For example, the application program has memory overflow during the execution process, which results in insufficient system resources and failure to continue normal operation.

When data change information in Kafka is consumed, possible anomalies such as network anomalies or system anomalies are captured.

As can be appreciated in connection with the above technical terms, in the exception handling, data change information in which an exception occurs is sent to the delay queue of the rabkitmq. A delay queue plug-in to RabbitMQ, such as a delay message plug-in (RabbitMQ Delayed Message Plugin), may be used herein, which may enable the sending and consumption of delay messages.

In the delay queue, the consumer can listen for and consume delay messages. Once the delay time has arrived, the message will be re-delivered to the consumer for a retry operation.

By sending the abnormal data change information to the delay queue of the RabbitMQ, the delay processing and retry operation of the message can be realized. Thus, the reliability and the processing consistency of the message can be ensured, and meanwhile, the message loss or processing failure caused by network abnormality or other system abnormality is avoided.

The fault handling methods in the two data processing procedures are described below.

In one embodiment, the preset fault threshold comprises a preset alarm threshold, and when the data processing time is greater than the preset processing time, the preset alarm threshold is determined to be triggered; and responding to the action of triggering the preset alarm threshold value, and carrying out alarm processing.

Alarms are a monitoring mechanism for timely discovery and notification of anomalies or faults in the system. When a certain key index or monitoring item of the system exceeds a preset threshold value, an alarm operation is triggered.

The alarm threshold may be a preset processing time, i.e. a time for measuring whether the data processing time is within a normal range. For example, 10 minutes are required to process data in a certain service table under normal conditions, and if the information processing platform processes the data in the service table for more than 10 minutes, the data processing time is considered to be longer than the preset processing time, and an alarm is triggered.

It will be appreciated that the alarm threshold may also be any other system performance indicator exceeding a threshold, such as important data or transaction anomalies or loss rates, etc., which are not listed here.

Once the alarm is triggered, the system may take some action, such as: sending mail, short message or instant notice to related personnel, etc.

In one embodiment, the preset fault threshold includes a preset fusing threshold, and when the system resource utilization is greater than the preset utilization, the preset fusing threshold is determined to be triggered; and responding to the action of triggering a preset fusing threshold value, and performing fusing processing.

Fusing is a fault handling mechanism for protecting a system from fault services.

The system resource utilization rate refers to utilization rates of a CPU, a memory, a disk, and the like. The system resource utilization may be calculated by monitoring the resource consumption of the system.

For example, when the memory utilization rate of the system exceeds 90%, the system resource utilization rate is considered to be greater than the preset utilization rate, and the fusing operation is performed.

It will be appreciated that the fusing threshold may also be any threshold exceeded by other system performance indicators, such as network interface anomalies, data batch errors, etc., which are not listed here.

Once triggered by a blow, the system may take some action, such as: returning cached data or default values, rather than invoking a failed service; returning an error code or error information to prompt that the service is not available; downgrade operation, use the spare scheme or tactics; logging, sending notifications, or performing other fault handling operations, etc., are examples and not limiting.

Fusing can improve the stability and reliability of the system, prevent the spread of faults, and reduce the consumption of system resources.

It will be appreciated that both fusing and alerting are to protect the stability and reliability of the system, to discover and handle anomalies or faults in time, and to reduce impact on the system and the user.

The data monitoring process in the overall context is described below in conjunction with fig. 3.

Fig. 3 is a flow chart of another data monitoring processing method according to an embodiment of the present application.

As shown in fig. 3, fig. 3 includes the following steps S301 to S308.

S301, acquiring data change information.

In connection with the above, the data change information is obtained here by listening to Binlog events of the business table data changes. The service table may be any one service table, may be all service tables, or may be a key service table of a plurality of service tables, which is not limited herein.

S302A, data change information is transmitted to Kafka.

In connection with the above, the Binlog event is sent to Kafka using Canal here. It will be appreciated that the message queues herein may also be other message queues having similar functionality, such as rocket message queues (Rocketmq), rabbitmq, and the like.

S302B, synchronizing data change information.

In combination with the above, the data change information can be synchronized into the OLAP database.

S303, processing data change information.

In connection with the above, the information processing platform herein may consume data change information in Kafka. At this time, an analysis operation may be performed.

S304, checking the data change information.

The information processing platform can reversely check the content in the OLAP database in the process of processing the data, and process the service change data to realize the service functions such as checking.

S305, triggering an alarm.

In combination with the above, when the alarm threshold is triggered, an alarm operation is performed.

S306, triggering fusing.

In combination with the above, when the fusing threshold is triggered, a fusing operation is performed.

S307, the data change information is sent to a delay queue.

The information processing platform sends the data change information to a delay queue, such as RabbitMQ, if an abnormal situation such as network abnormality occurs in the processing process.

S308, consuming the data change information in the delay queue.

The information processing platform may repeat the steps from step S303 to step S307 to change the data in the consumption delay queue.

The method utilizes Binlog to realize real-time monitoring and inverse checking of the data, avoids modification and damage to the original service data, ensures the reliability and integrity of the service data, can acquire the information of service data change in time, and ensures the real-time performance of the service; meanwhile, the stability and the accuracy of service data can be ensured by utilizing the characteristics of high reliability and high throughput of Kafka; and the mode of reversely checking other service tables can be freely added or modified according to different service requirements, so that the expansibility is high.

In addition, the technology of Canal, rabbitMq, kafka and the like with open sources is utilized, so that high research and development cost is avoided, and development and maintenance cost of the system is reduced.

By utilizing the delay queue function of RabbitMq, the message retry under the condition of network abnormality or other system abnormality is realized, the service interruption caused by the network abnormality and other reasons is avoided, and the reliability and stability of the service are improved.

It can be understood that the method supports monitoring and inverse checking of various data sources and service tables, is applicable to the requirements of different industries and different service scenes, and has strong flexibility.

The foregoing description of the method of the embodiments of the present application is provided primarily with reference to the accompanying drawings. It should be understood that, although the steps in the flowcharts related to the above embodiments are shown in order, these steps are not necessarily performed in the order shown in the drawings. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in the flowcharts described in the above embodiments may include a plurality of steps or a plurality of stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of the steps or stages is not necessarily performed sequentially, but may be performed alternately or alternately with at least some of the other steps or stages. An apparatus according to an embodiment of the present application is described below with reference to the accompanying drawings. For brevity, the description of the apparatus will be omitted appropriately, and the relevant content may be referred to the relevant description in the above method, and the description will not be repeated.

Fig. 4 is a schematic structural diagram of a data monitoring processing device according to an embodiment of the present application.

As shown in fig. 4, the apparatus 1000 includes the following units.

An acquiring unit 1001 is configured to acquire at least one piece of data change information.

The transceiver 1002 is configured to send at least one piece of data change information to the kava message queue by using the kanal component.

A processing unit 1003, configured to process at least one piece of data change information in the kaff card message queue.

The processing unit 1003 is further configured to perform fault processing when the data processing procedure of the at least one piece of data change information triggers a preset fault threshold.

In one implementation, the obtaining unit 1001 may also be configured to perform the methods in steps S201 and S301.

In one implementation, the transceiver unit 1002 may also be configured to perform the method in step S302A, S B.

In one implementation, the processing unit 1003 may also be configured to perform the methods in steps S202, S203, S303 to S308.

In one implementation, the apparatus 1000 further includes a storage unit, where the storage unit may be configured to store instructions and/or data, thereby implementing the method in the above embodiment.

It should be noted that, because the content of information interaction and execution process between the above units is based on the same concept as the method embodiment of the present application, specific functions and technical effects thereof may be referred to in the method embodiment section, and will not be described herein again.

Fig. 5 is a schematic structural diagram of a computer device according to an embodiment of the present application. As shown in fig. 5, the computer device 3000 of this embodiment includes: at least one processor 3100 (only one is shown in fig. 5), a memory 3200, and a computer program 3210 stored in the memory 3200 and executable on the at least one processor 3100, the processor 3100, when executing the computer program 3210, causing the computer apparatus to carry out the steps in the embodiments described above.

The processor 3100 may be a central processing unit (Central Processing Unit, CPU), but the processor 3100 may also be other general purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), off-the-shelf programmable gate arrays (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

Memory 3200 may in some embodiments be an internal storage unit of computer device 3000, such as a hard disk or memory of computer device 3000. Memory 3200 may also be an external storage device of computer device 3000 in other embodiments, such as a plug-in hard disk, smart Media Card (SMC), secure Digital (SD) Card, flash Card (Flash Card) or the like, which are provided on computer device 3000. Further, memory 3200 may also include both internal and external storage units of computer device 3000. The memory 3200 is used to store an operating system, application programs, boot Loader (Boot Loader) data, other programs, and the like, such as program codes of computer programs, and the like. The memory 3200 may also be used to temporarily store data that has been output or is to be output.

It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-described division of the functional units is illustrated, and in practical application, the above-described functional distribution may be performed by different functional units or modules according to needs, i.e. the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-described functions. The functional units in the embodiment may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit, where the integrated units may be implemented in a form of hardware or a form of a software functional unit. In addition, the specific names of the functional units are also only for distinguishing from each other, and are not used to limit the protection scope of the present application. The specific working process of the units in the above system may refer to the corresponding process in the foregoing method embodiment, which is not described herein again.

The embodiments of the present application also provide a computer readable storage medium, where a computer program is stored, where the computer program when executed by a computer device causes the computer device to implement the steps in the embodiments of the method described above.

Embodiments of the present application provide a computer program product enabling a computer device to carry out the above-mentioned methods when the computer program product is run on the computer device.

The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the present application implements all or part of the flow of the method of the above embodiments, and may be implemented by a computer program to instruct related hardware, where the computer program may be stored in a computer readable storage medium, where the computer program when executed by a processor causes a computer device to implement the steps of each of the method embodiments described above. Wherein the computer program comprises computer program code which may be in source code form, object code form, executable file or some intermediate form etc. The computer readable medium may include at least: any entity or device capable of carrying computer program code to a photographing device/terminal apparatus, recording medium, computer Memory, read-Only Memory (ROM), random access Memory (Random Access Memory, RAM), electrical carrier signals, telecommunications signals, and software distribution media. Such as a U-disk, removable hard disk, magnetic or optical disk, etc. In some jurisdictions, computer readable media may not be electrical carrier signals and telecommunications signals in accordance with legislation and patent practice.

It should be understood that the sequence number of each step in the foregoing embodiment does not mean that the execution sequence of each process should be determined by the function and the internal logic of each process, and should not limit the implementation process of the embodiment of the present application in any way. In the description, for purposes of explanation and not limitation, specific details are set forth such as the particular system architecture, techniques, etc., in order to provide a thorough understanding of the embodiments of the present application. It will be apparent, however, to one skilled in the art that the present application may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.

It should be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

It should also be understood that the term "and/or" as used in this specification and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes such combinations.

Furthermore, in the description of the present application and the claims, the terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically noted.

In the foregoing embodiments, the descriptions of the embodiments are emphasized, and in part, not described or illustrated in any particular embodiment, reference is made to the related descriptions of other embodiments.

Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

In the embodiments provided in the present application, it should be understood that the disclosed apparatus, computer device, and method may be implemented in other manners. For example, the apparatus, computer device embodiments described above are merely illustrative, e.g., the partitioning of elements is merely a logical functional partitioning, and there may be additional partitioning in actual implementation, e.g., multiple elements or components may be combined or integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection via interfaces, devices or units, which may be in electrical, mechanical or other forms.

The above embodiments are only for illustrating the technical solution of the present application, and are not limiting; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present application, and are intended to be included in the scope of the present application.

Claims (10)

1. A data monitoring processing method, characterized by comprising:

acquiring at least one piece of data change information;

transmitting the at least one piece of data change information to a card message queue by using a card module;

processing the at least one piece of data change information in the kaff card message queue;

and when the data processing process of the at least one piece of data change information triggers a preset fault threshold value, fault processing is carried out.

2. The method of claim 1, wherein the at least one piece of data change information comprises data change information from a plurality of service data tables, and wherein the processing the at least one piece of data change information in the kaff message queue comprises:

And processing the data change information of the plurality of service data tables by using an information processing platform.

3. The method of claim 2, wherein processing the data change information of the plurality of service data tables using the information processing platform comprises:

acquiring data change information of the plurality of service data tables from the Kaff card message queue;

analyzing data changing operations corresponding to the data changing information of the plurality of service data tables;

and carrying out corresponding data processing according to the data changing operation.

4. The method of claim 1, wherein said processing said at least one piece of data change information in said kaff message queue comprises:

checking the business system corresponding to the first data change information, wherein the first data change information is any piece of data change information in the at least one piece of data change information;

acquiring first association information corresponding to the first data change information by using the service system;

and processing the data corresponding to the first data change information by combining the first association information.

5. The method according to claim 3 or 4, characterized in that the method further comprises:

Detecting network abnormality or system abnormality;

and when the network abnormality and/or the system abnormality are detected, the information processing platform sends the data change information to a rabbit message queue and processes the data change information.

6. The method according to claim 1, wherein the preset fault threshold includes a preset alarm threshold, and the performing fault handling when the processing of the data change information triggers the preset fault threshold includes:

when the data processing time is longer than the preset processing time, the preset alarm threshold value is determined to be triggered;

and responding to the action of triggering a preset alarm threshold value, and carrying out alarm processing.

7. The method according to claim 1, wherein the preset fault threshold includes a preset fusing threshold, and the performing fault handling when the processing of the data change information triggers the preset fault threshold includes:

when the utilization rate of the system resources is larger than the preset utilization rate, the preset fusing threshold value is determined to be triggered;

and responding to the action of triggering a preset fusing threshold value, and performing fusing processing.

8. A data monitoring processing apparatus, comprising:

An acquisition unit configured to acquire at least one piece of data change information;

the receiving and transmitting unit is used for transmitting the at least one piece of data change information to the card message queue by using the card Navigator;

the processing unit is used for processing the at least one piece of data change information in the Kaff card message queue;

the processing unit is further used for performing fault processing when the data processing process of the at least one piece of data change information triggers a preset fault threshold.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, when executing the computer program, causing the computer device to implement the method of any one of claims 1 to 7.

10. A computer readable storage medium, characterized in that the computer readable storage medium stores a computer program which, when executed by a computer device, implements the method according to any of claims 1-7.