CN116545949A - Data processing method, device, computer equipment and storage medium - Google Patents

Data processing method, device, computer equipment and storage medium Download PDF

Info

Publication number
CN116545949A
CN116545949A CN202310577186.6A CN202310577186A CN116545949A CN 116545949 A CN116545949 A CN 116545949A CN 202310577186 A CN202310577186 A CN 202310577186A CN 116545949 A CN116545949 A CN 116545949A
Authority
CN
China
Prior art keywords
message
local queue
queue
local
party system
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202310577186.6A
Other languages
Chinese (zh)
Inventor
张宏根
胡凯乐
刘金凤
雷琼
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Industrial and Commercial Bank of China Ltd ICBC
Original Assignee
Industrial and Commercial Bank of China Ltd ICBC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Industrial and Commercial Bank of China Ltd ICBC filed Critical Industrial and Commercial Bank of China Ltd ICBC
Priority to CN202310577186.6A priority Critical patent/CN116545949A/en
Publication of CN116545949A publication Critical patent/CN116545949A/en
Pending legal-status Critical Current

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L47/00—Traffic control in data switching networks
    • H04L47/50—Queue scheduling
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Computer And Data Communications (AREA)

Abstract

The present application relates to a data processing method, apparatus, computer device, storage medium and computer program product, and relates to the technical field of big data, and may be used in the financial and technological field or other fields. The method comprises the following steps: responding to the data transmission instruction, reading a plurality of first communication messages from a target distributed message queue according to the message sorting parameter, and distributing the read first communication messages to local queues corresponding to the third party systems; for each local queue, reading the second communication message from the local queue according to the data transmission parameters corresponding to the local queue, and sending the read second communication message to a third party system corresponding to the local queue; the data transmission parameters corresponding to the local queue are determined according to the processing capacity parameters of the third party system corresponding to the local queue. The method can improve the data processing efficiency.

Description

Data processing method, device, computer equipment and storage medium
Technical Field
The present application relates to the field of big data technology, and in particular, to a data processing method, apparatus, computer device, storage medium, and computer program product.
Background
In some scenarios, the bank operating system needs to send a large number of communication messages to the third party systems that are in butt joint. If the bank operation system issues the interests of the third party platform or the virtual resources (such as shopping platform coupons) and the user obtains the interests of the third party platform through the application program (such as mobile phone banking) of the bank operation system, the bank operation system needs to send communication messages corresponding to the users to the third party systems to instruct the third party systems to process (such as recording the virtual resources corresponding to the user so that the user can smoothly use the third party platform). With the application of the distributed technology in the bank operation system, the current bank operation system has higher concurrent throughput capacity.
In the related art, a bank operation system can store communication messages corresponding to each user in a distributed message queue, then read the communication messages from the distributed message queue at a certain data transmission speed, and transmit the communication messages to a corresponding third party system for processing.
However, because the processing capacities of the third party systems are different, if the data transmission speed adopted by the bank operation system is greater than the processing capacity of part of the third party systems, the third party systems are easy to avalanche, and cannot process data in time, so that the data processing efficiency is lower.
Disclosure of Invention
In view of the foregoing, it is desirable to provide a data processing method, apparatus, computer device, computer-readable storage medium, and computer program product that can improve data processing efficiency.
In a first aspect, the present application provides a data processing method. The method comprises the following steps:
responding to the data transmission instruction, reading a plurality of first communication messages from a target distributed message queue according to the message sorting parameter, and distributing the read first communication messages to local queues corresponding to the third party systems;
for each local queue, reading a second communication message from the local queue according to the data transmission parameters corresponding to the local queue, and sending the read second communication message to a third party system corresponding to the local queue; and the data transmission parameters corresponding to the local queue are determined according to the processing capacity parameters of the third party system corresponding to the local queue.
In one embodiment, the processing capability parameter of the third party system comprises a transaction amount per unit time, and the message sorting parameter comprises a first time window and a target read number within the first time window; the method further comprises the steps of:
Acquiring the transaction processing amount in unit time of each third party system, and calculating the sum of the transaction processing amounts in unit time;
and determining the target reading quantity in a first time window based on the sum value, the quantity of the processing nodes of the distributed system and the preset first time window.
In one embodiment, the reading a plurality of first communication messages from the target distributed message queue according to the message sorting parameter, and distributing each read first communication message to a local queue corresponding to each third party system includes:
reading the target read number of first communication messages from a target distributed message queue under the condition that the current time reaches the starting time of the current first time window;
and determining a third party system corresponding to each first communication message according to the characteristic information contained in each read first communication message, and distributing each first communication message to a local queue corresponding to each third party system based on the third party system corresponding to each first communication message.
In one embodiment, the processing capability parameter of the third party system includes a transaction amount per unit time, and the data transmission parameter includes a second time window and a target transmission amount within the second time window; the method further comprises the steps of:
For each local queue, acquiring the transaction processing amount in unit time of a third party system corresponding to the local queue;
determining the transaction processing amount in unit time corresponding to the local queue according to the transaction processing amount in unit time of the third party system and the number of processing nodes of the distributed system;
and determining the target transmission quantity in a second time window corresponding to the local queue based on the transaction processing quantity in the unit time corresponding to the local queue and the preset second time window.
In one embodiment, for each local queue, according to the data transmission parameter corresponding to the local queue, reading a second communication message from the local queue, and sending the read second communication message to a third party system corresponding to the local queue, where the third party system includes:
and under the condition that the current time reaches the starting time of the current second time window corresponding to the local queue, reading the target transmission number of second communication messages from the local queue, and sending the read second communication messages to a third party system corresponding to the local queue.
In one embodiment, the data transmission parameters include a second time window and a target number of transmissions within the second time window; the step of reading the second communication message from the local queues according to the data transmission parameters corresponding to the local queues and sending the read second communication message to the third party system corresponding to the local queues, includes:
for each local queue, judging whether an abnormal message exists in an abnormal message queue corresponding to the local queue under the condition that the current time reaches the starting time of a current second time window corresponding to the local queue; the abnormal message queue is used for storing communication messages with abnormal processing results;
reading an abnormal message from the abnormal message queue under the condition that the abnormal message exists in the abnormal message queue;
under the condition that the first number of the read abnormal messages is not smaller than the target transmission number, the read abnormal messages are sent to a third party system corresponding to the local queue;
determining a second number according to the first number and the target transmission number and reading the second number of second communication messages from the local queue when the first number is smaller than the target transmission number;
And sending the read abnormal message and the second communication message to a third party system corresponding to the local queue.
In one embodiment, after the sending the read second communication message to the third party system corresponding to the local queue, the method further includes:
and determining the processing result of each communication message according to the feedback information of the third party system corresponding to the local queue for each local queue, and storing the communication message with the processing result of processing abnormality as an abnormal message into an abnormal message queue corresponding to the local queue.
In one embodiment, after determining the processing result of each communication message according to the feedback information of the third party system corresponding to the local queue, the method further includes:
and determining the processing result as the ratio of the number of the communication messages with abnormal processing to the total number of the communication messages, and outputting first alarm information under the condition that the ratio is larger than a preset threshold value.
In one embodiment, the method further comprises:
and outputting second alarm information for each local queue under the condition that the number of communication messages in the local queue exceeds a preset threshold value.
In a second aspect, the present application also provides a data processing apparatus. The device comprises:
the distribution module is used for responding to the data transmission instruction, reading a plurality of first communication messages from the target distributed message queue according to the message sorting parameters, and distributing the read first communication messages to the local queues corresponding to the third party systems;
the sending module is used for reading the second communication message from the local queues according to the data transmission parameters corresponding to the local queues and sending the read second communication message to a third party system corresponding to the local queues; and the data transmission parameters corresponding to the local queue are determined according to the processing capacity parameters of the third party system corresponding to the local queue.
In one embodiment, the processing capability parameter of the third party system comprises a transaction amount per unit time, and the message sorting parameter comprises a first time window and a target read number within the first time window; the apparatus further comprises:
the first acquisition module is used for acquiring the transaction processing amount in unit time of each third party system and calculating the sum of the transaction processing amounts in each unit time;
The first determining module is used for determining the target reading quantity in a first time window based on the sum value, the quantity of the processing nodes of the distributed system and the preset first time window.
In one embodiment, the allocation module is specifically configured to:
reading the target read number of first communication messages from a target distributed message queue under the condition that the current time reaches the starting time of the current first time window; and determining a third party system corresponding to each first communication message according to the characteristic information contained in each read first communication message, and distributing each first communication message to a local queue corresponding to each third party system based on the third party system corresponding to each first communication message.
In one embodiment, the processing capability parameter of the third party system includes a transaction amount per unit time, and the data transmission parameter includes a second time window and a target transmission amount within the second time window; the apparatus further comprises:
the second acquisition module is used for acquiring the transaction processing amount in unit time of the third party system corresponding to each local queue;
The second determining module is used for determining the transaction processing amount in the unit time corresponding to the local queue according to the transaction processing amount in the unit time of the third party system and the number of processing nodes of the distributed system;
and the third determining module is used for determining the target transmission quantity in the second time window corresponding to the local queue based on the transaction processing quantity in the unit time corresponding to the local queue and the preset second time window.
In one embodiment, the sending module is specifically configured to:
and under the condition that the current time reaches the starting time of the current second time window corresponding to the local queue, reading the target transmission number of second communication messages from the local queue, and sending the read second communication messages to a third party system corresponding to the local queue.
In one embodiment, the data transmission parameters include a second time window and a target number of transmissions within the second time window; the sending module is specifically configured to:
for each local queue, judging whether an abnormal message exists in an abnormal message queue corresponding to the local queue under the condition that the current time reaches the starting time of a current second time window corresponding to the local queue; the abnormal message queue is used for storing communication messages with abnormal processing results; reading an abnormal message from the abnormal message queue under the condition that the abnormal message exists in the abnormal message queue; under the condition that the first number of the read abnormal messages is not smaller than the target transmission number, the read abnormal messages are sent to a third party system corresponding to the local queue; determining a second number according to the first number and the target transmission number and reading the second number of second communication messages from the local queue when the first number is smaller than the target transmission number; and sending the read abnormal message and the second communication message to a third party system corresponding to the local queue.
In one embodiment, the apparatus further comprises:
the storage module is used for determining the processing result of each communication message according to the feedback information of the third party system corresponding to the local queue aiming at each local queue, and storing the communication message with the processing result of processing abnormality as an abnormal message into an abnormal message queue corresponding to the local queue.
In one embodiment, the apparatus further comprises:
the first alarm module is used for determining that the processing result is the ratio of the number of the communication messages with abnormal processing to the total number of the communication messages, and outputting first alarm information when the ratio is larger than a preset threshold value.
In one embodiment, the apparatus further comprises:
and the second alarm module is used for outputting second alarm information for each local queue under the condition that the number of communication messages in the local queue exceeds a preset threshold value.
In a third aspect, the present application also provides a computer device. The computer device comprises a memory storing a computer program and a processor implementing the steps of the method of the first aspect when the processor executes the computer program.
In a fourth aspect, the present application also provides a computer-readable storage medium. The computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the method of the first aspect.
In a fifth aspect, the present application also provides a computer program product. The computer program product comprising a computer program which, when executed by a processor, implements the steps of the method of the first aspect.
The data processing method, the device, the computer equipment, the storage medium and the computer program product read a plurality of communication messages from the distributed message queues storing the communication messages according to the message sorting parameters (which can reflect the message sorting speed), distribute the communication messages to the local queues corresponding to the third party systems, read the second communication messages from the local queues based on the data transmission parameters (which can reflect the data transmission speed) corresponding to the local queues, and send the second communication messages to the corresponding third party systems for processing. The data transmission parameters corresponding to the local queues are determined according to the processing capacity parameters of the corresponding third-party system and are matched with the processing capacity of the third-party system. Therefore, the method can avoid the condition that the system is crashed due to the fact that the data transmission speed is larger than the processing capacity of part of the third-party systems, namely the method can improve the stability of the system and the overall data processing efficiency.
Drawings
FIG. 1 is a schematic diagram of a distributed system in one example;
FIG. 2 is a flow diagram of a data processing method in one embodiment;
FIG. 3 is a flow chart of a data processing method according to another embodiment;
FIG. 4 is a schematic diagram of a distributed system in another example;
FIG. 5 is a flow chart of a data processing method according to another embodiment;
FIG. 6 is a flow diagram of sending a read message to a third party system in one embodiment;
FIG. 7 is a block diagram of a data processing apparatus in one embodiment;
fig. 8 is an internal structural diagram of a computer device in one embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the present application.
First, before the technical solution of the embodiments of the present application is specifically described, a description is first given of a technical background or a technical evolution context on which the embodiments of the present application are based. In some situations, such as a situation that the bank operation system issues rights or interests of a third party shopping platform or virtual resources (such as shopping platform coupons), the bank operation system needs to send communication messages corresponding to each user to the third party system to instruct the third party system to process (e.g. record virtual resources corresponding to the user so that the user can smoothly use the virtual resources on the third party platform).
For equity acquisition or other activities, during which traffic bursts may occur, distributed technology has been currently employed in banking systems in order to improve concurrency throughput and system stability of the system. That is, the bank operation system may adopt a distributed architecture, which may include a plurality of distributed nodes, such as N distributed nodes (e.g. sub-service layers set1 to setN) in the bank operation system of the distributed architecture shown in fig. 1. The service access layer of the bank operation system can flow the rights and interests acquisition request or other transaction request of each user to different distributed nodes for processing through consistent hash routing based on certain rules, such as information of user areas, guest groups, labels and the like. The distributed node can obtain a successful user aiming at the rights, generate a communication message corresponding to the user and send the communication message to a third party system corresponding to the rights for further processing.
In the related art, each distributed node of the bank operation system may store the generated communication message corresponding to each user in the message queue of the node. Each distributed node can read the communication message from the message queue according to a certain data transmission speed (a preset fixed value), call an interface of a third party system corresponding to the communication message, and transmit the communication message to the corresponding third party system for processing.
However, because the processing capacities of the third party systems are different, if the data transmission speed adopted by the banking system is greater than the processing capacity of a certain third party system, the third party system is easy to avalanche, and cannot process data in time, so that the data processing efficiency is lower. If a lower data transmission speed is adopted, the overall efficiency is still lower, and because the concurrency capability of the banking system is higher and the processing capability of part of third party systems is stronger, the resources of the systems with stronger capability cannot be fully utilized by adopting the lower data transmission speed so as to exert the system advantage. Based on the background, the applicant provides the data processing method through long-term research and development and experimental verification, the data transmission threads of the third party systems can be decoupled, each thread respectively adopts the data transmission speed matched with the processing capacity of the third party systems, and the communication message to be processed is sent to the third party systems, so that the influence on the overall data processing efficiency due to system breakdown caused by the fact that the data transmission speed is larger than the processing capacity of the third party systems can be avoided. And the data transmission speed can be reasonably set according to the processing capacity of the third party system, so that the resources of the system are fully utilized, and the utilization rate of the resources is improved. In addition, the applicant has made a great deal of creative effort to find out the technical problems of the present application and to introduce the technical solutions of the following embodiments.
The data processing method provided in the embodiment of the present application may be applied to the distributed system 102 shown in fig. 1, where the distributed system may be a bank operation system or other systems. The distributed system 102 may include a plurality of distributed nodes (e.g., sub-service layers set1 through setN in the figure), each of which may communicate with a plurality of third party systems 104 via a network. It is understood that multiple processing nodes may be included in each distributed node. The distributed system 102 and the third party system 104 may be implemented by a terminal, a server, and/or a server cluster, among other computer devices.
In one embodiment, as shown in fig. 2, a data processing method is provided, and the method is applied to the distributed system in fig. 1 for illustration, and includes the following steps:
step 201, in response to the data transmission instruction, according to the message sorting parameter, a plurality of first communication messages are read from the target distributed message queue, and each read first communication message is distributed to a local queue corresponding to each third party system.
In implementations, the service access layer of the distributed system may stream a user's request (e.g., a rights acquisition request or other request) to each distributed node (sub-service layer) for processing. After each distributed node performs related processing based on the request of the user, a communication message to be sent to a corresponding third party system can be generated according to a processing result (such as success of rights acquisition), and the communication message can be stored in a distributed message queue (such as a card message queue). Each distributed node of the distributed system may respond to the data transmission instruction, read a plurality of communication messages from a target distributed message queue storing communication messages to be sent to the third party according to the message sorting parameter, and determine the third party system corresponding to each communication message according to characteristic information (such as identification information of a target field) included in each read communication message, so as to distribute each communication message to a local queue corresponding to the corresponding third party system. The message sorting parameter may reflect a message sorting speed, such as a communication message reading amount in a unit time.
Step 202, for each local queue, according to the data transmission parameters corresponding to the local queue, reading the second communication message from the local queue, and sending the read second communication message to the third party system corresponding to the local queue.
In an implementation, for each local queue, the distributed node of the distributed system may read, according to the data transmission parameter corresponding to the local queue, the communication message from the local queue through the message processing thread, and send the read communication message to the third party system corresponding to the local queue. The data transmission parameters corresponding to the local queues are determined according to the processing capacity parameters of the third party system corresponding to the local queues, and can reflect the data transmission speed of transmitting data to the third party system. For example, the data transmission parameter may be a transaction amount per unit time (such as transaction amount tps per second, transactionPer Second), and the processing result of reading a communication message from the local queue, transmitting the communication message to the third party system, and determining the communication message according to feedback information (if any) of the third party system may be used as a transaction.
In the data processing method, according to the message sorting parameters reflecting the message sorting speed, a plurality of communication messages are read from the distributed message queues storing the communication messages and distributed to the local queues corresponding to the third party systems, and based on the data transmission parameters corresponding to the local queues and capable of reflecting the data transmission speed, a second communication message is read from the local queues and sent to the corresponding third party systems for processing. The data transmission parameters corresponding to the local queues are determined according to the processing capacity parameters of the corresponding third-party system and are matched with the processing capacity of the third-party system. Therefore, the method can avoid the condition that the system is crashed due to the fact that the data transmission speed is larger than the processing capacity of part of the third-party systems, namely the method can improve the stability of the system and the overall data processing efficiency. And the data transmission speed can be reasonably set according to the processing capacity of the third party system, so that the resources of the system are fully utilized, and the utilization rate of the resources is improved. In addition, two-layer flow control is carried out through the message sorting parameters and the data transmission parameters, so that peak clipping and valley filling of the third-party system docking flow can be realized, and the overall operation stability and the resource utilization efficiency of the system are improved.
In one embodiment, the processing capability parameter of the third party system includes a transaction amount per unit time, and the message sort parameter includes a first time window and a target read number within the first time window. As shown in fig. 3, the method may further include a message sorting parameter determining process, specifically including the following steps:
step 301, obtaining the transaction amount in unit time of each third party system, and calculating the sum of the transaction amounts in unit time.
In practice, the transaction amount per unit time (e.g., transaction amount tps per second) of the third party system may be stored as an attribute parameter of the third party system. The distributed system may invoke a related interface to send a request to the docked third party system to obtain the transaction amount per unit time of the three party system and calculate the sum of the transaction amounts per unit time (which may be denoted as P). The third party system can receive a communication message of the distributed system, process the communication message and feed back corresponding information to be used as a transaction.
Step 302, determining a target read number in a first time window based on the sum, the number of processing nodes of the distributed system, and the preset first time window.
In implementations, each distributed node of the distributed system may include a plurality of processing nodes, which may be a device (physical device or virtual device). The distributed system shown in fig. 4 includes N distributed nodes (set 1 to setN), where each distributed node may include I devices (in this example, set1 includes 6 devices), and each device is a processing node. Thus, the distributed system will contain Q processing nodes, q=i×n. The first time window (which may be denoted as T1) may be set manually as required or automatically according to other rules, e.g. set to 200ms. The distributed system can determine the target reading quantity M1 of each processing node in the first time window T1 according to the sum value P of the transaction processing quantity in unit time of each third party system, the quantity Q of the processing nodes contained in the distributed system and the preset first time window T1, so that the total quantity of the communication messages sorted in unit time by the distributed system is larger than or equal to the sum value of the transaction processing quantity in unit time of the third party system. For example, if the first time window T1 is measured in milliseconds and the unit time is measured in seconds, the determined target read number M1 may satisfy the following formula:
P≤M1×Q×(1000/T1)
It will be appreciated that each processing node may be provided with a plurality of consumer threads, each of which may read the first communication message from the target distributed message queue in parallel. Thus, the number of reads for each consumer thread may be determined based on the target number of reads M1 and the number of consumer threads C such that the sum of the number of communication messages read by each consumer thread T1 within the first time window within one processing node corresponds to the target number of reads M1.
Alternatively, the target read number M1 may be set so that the total number of communication messages sorted by the distribution system per unit time is greater than 5% to 20% of the sum of the transaction amounts per unit time of the respective third party systems.
In this embodiment, the message sorting parameters of the distributed system may be determined based on the sum of the processing capacities of the third party systems, so that the number of messages stored in the local queue may be matched with the processing capacity of each third party system, that is, the flow may be reasonably controlled by the message sorting parameters, so as to improve the system stability and the overall data processing efficiency, and in addition, the capacity of the local queue may be reasonably set, so as to improve the memory resource utilization.
In one embodiment, the process of message sorting in step 201 specifically includes the steps of: reading a target read number of first communication messages from a target distributed message queue under the condition that the current time reaches the starting time of a current first time window; and determining a third party system corresponding to each first communication message according to the characteristic information contained in each read first communication message, and distributing each first communication message to a local queue corresponding to each third party system based on the third party system corresponding to each first communication message.
In implementations, each processing node of the distributed system may perform message sorting based on the message sorting parameters (i.e., the first time window T1 and the target read number M1 within the first time window). The processing nodes may process in parallel. Specifically, each processing node may read the target read number of communication messages from the target distributed message queue under the condition that the current time reaches the start time of the current first time window, and determine the third party system corresponding to each communication message based on the feature information of each communication message, so as to allocate each read communication message to the local queue corresponding to the third party system. The end time of the current first time window may be taken as the start time of the next time window.
In this embodiment, each processing node of the distributed system may periodically perform the message sorting step based on the first time window and the target read number, and because the first time window (period) and the message sorting number in the time window are matched with the processing capacity of each third party system, on one hand, the system stability may be ensured by sorting flow control, and on the other hand, the number of messages stored in the local queue and the number of messages consumed from the local queue may be kept in dynamic balance, so that the capacity of the local queue may be reasonably set, so that the capacity of the local queue may be increased while the storage of the expected number of communication messages matched with the processing capacity of the third party system may be satisfied.
In one embodiment, the processing capability parameter of the third party system comprises a transaction amount per unit time, and the data transfer parameter comprises a second time window and a target transfer amount within the second time window. As shown in fig. 5, another data processing method is provided, which further includes the steps of:
step 501, for each local queue, obtain the transaction amount in unit time of the third party system corresponding to the local queue.
In implementations, the distributed system may invoke the relevant interface to obtain the transaction amount per unit time, such as transaction amount tps per second, for each third party system from the third party systems.
Step 502, determining the transaction processing amount in unit time corresponding to the local queue according to the transaction processing amount in unit time of the third party system and the number of processing nodes of the distributed system.
In implementation, the distributed system may calculate, according to the transaction amount (Pi) in a unit time of the third party system and the number Q of processing nodes of the distributed system, the transaction amount (Si) in a unit time of a local queue corresponding to the third party system, so that the sum of the transaction amounts in unit time of local queues corresponding to the same third party system in each processing node is greater than or equal to the transaction amount in unit time of the third party system. For example, if the processing nodes use the same data transmission parameters, the calculation formula of the transaction amount Si in unit time of the local queue may be as follows:
Si＝Pi/Q
It will be appreciated that different data transmission parameters may be used by each processing node, for example, by setting different weights for each processing node, a weighted sum of transaction amounts in a unit time of a local queue corresponding to the same third party system in each processing node is greater than or equal to the transaction amount in the unit time of the third party system.
In step 503, the target transmission number in the second time window corresponding to the local queue is determined based on the transaction amount in the unit time corresponding to the local queue and the preset second time window.
In implementation, each processing node in the distributed system may determine, based on the transaction amount Si in a unit time corresponding to the local queue and a preset second time window T2, the target transmission number M2 in the second time window corresponding to the local queue, so that the data transmission speed corresponding to the transmission target transmission number M2 of communication messages in the second time window T2 is consistent with the data transmission speed reflected by the transaction amount Si in the unit time. For example, if the second time window T2 is measured in milliseconds and the unit time is measured in seconds, the determined target transmission number M2 may satisfy the following formula:
Si＝M2×(1000/T2)
in this embodiment, each processing node in the distributed system may perform processing flow control by using the second time window of the local queue and the target transmission number in the time window corresponding to the third party system as the data transmission parameter. The processing nodes can process in parallel, so that the sum of the quantity of communication messages transmitted to the same third party system by the processing nodes in the distributed system in a second time window is matched with the processing capacity of the third party system, the system stability can be improved, the overall data processing efficiency is improved, the data transmission parameters can be reasonably set based on the processing capacity of the third party system, and the concurrent performance advantage of the system is fully exerted.
In one embodiment, the process of data transmission in step 202 specifically includes the following steps: and under the condition that the current time reaches the starting time of the current second time window corresponding to the local queue, reading the target transmission number of second communication messages from the local queue, and sending the read second communication messages to a third party system corresponding to the local queue.
In implementation, each processing node of the distributed system may be provided with a processing thread corresponding to a third party system, and as shown in fig. 4, the communication message may be read from the corresponding local queue by the processing thread of each third party system. And under the condition that the current time reaches the starting time of the current second time window corresponding to the local queue, reading the target transmission number M2 second communication messages from the local queue through a processing thread, and sending the read second communication messages to a third party system corresponding to the local queue. It may be appreciated that the processing node may employ one thread or multiple threads to send the read second communication message to the third party system corresponding to the local queue.
In this embodiment, each processing node in the distributed system may include a processing thread corresponding to each third-party system, so as to periodically execute the data transmission step based on the second time window of the local queue corresponding to the third-party system and the target transmission number in the time window, so as to implement decoupling of the docking threads of each third-party system, so that the data transmission speed corresponding to each third-party system can be reasonably set, the stability of the system and the performance utilization rate of the system are ensured, and the overall data processing efficiency is improved.
In one embodiment, the data transmission parameters include a second time window and a target number of transmissions within the second time window. As shown in fig. 6, the process of data transmission in step 202 specifically includes the following steps:
step 601, for each local queue, determining whether an abnormal message exists in an abnormal message queue corresponding to the local queue when the current time reaches the start time of the current second time window corresponding to the local queue.
The abnormal message queue is used for storing communication messages with abnormal processing results. After the processing node sends the communication messages to the third party system, the processing result of each communication message can be determined based on the feedback information of the third party system, and the processing result comprises normal processing and abnormal processing. The processing node may store the communication message handling the exception in an exception message queue.
In an implementation, for each local queue, each processing node in the distributed system may determine whether an exception message exists in an exception message queue corresponding to the local queue when a current time reaches a start time of a current second time window corresponding to the local queue.
In step 602, in the case that an abnormal message exists in the abnormal message queue, the abnormal message is read from the abnormal message queue.
In implementations, if an exception message exists in the exception message queue, the processing node may read all or a portion of the exception message from the exception message queue.
Step 603, sending the read abnormal message to a third party system corresponding to the local queue under the condition that the first number of the read abnormal messages is not smaller than the target transmission number.
In implementation, if the number of the abnormal messages read by the processing node is not less than (greater than or equal to) the target transmission number M2 in the second time window, the processing node may send the read abnormal messages to the corresponding third party system.
In step 604, in the case that the first number is smaller than the target transmission number, a second number is determined according to the first number and the target transmission number, and a second number of second communication messages is read from the local queue.
In implementation, if the number of the abnormal messages read by the processing node is smaller than the target transmission number M2 in the second time window, the difference between the number of the abnormal messages and the target transmission number M2 may be calculated, that is, the second number. The processing node may then read a second number of communication messages from the local queue. That is, the sum of the number of communication messages read from the local queue and the number of exception messages read is consistent with the target number of transmissions within the second time window.
Step 605, the read abnormal message and the second communication message are sent to a third party system corresponding to the local queue.
In implementation, the processing node may send the communication message read from the local queue and the read abnormal message to a third party system corresponding to the local queue, so as to complete the data transmission task in the current time window.
In this embodiment, the communication message with the history processing result of processing abnormality is stored as the abnormality message in the abnormality message queue, so that in the current time window, the abnormality message can be obtained from the abnormality message queue and sent to the third party system again for processing, thereby realizing message retry, avoiding the abnormality processing result caused by the network problem or other reasons, and improving the success rate of data processing after multiple retries.
In one embodiment, the method further comprises the steps of: and determining the processing result of each communication message according to the feedback information of the third party system corresponding to the local queue aiming at each local queue, and storing the communication message with the processing result of processing abnormality as an abnormal message into an abnormal message queue corresponding to the local queue.
In an implementation, after the processing node sends the communication message to the third party system, feedback information of the third party system may be received, and a processing result may be determined based on the feedback information of the third party. Then, the processing node can store the communication message with the processing result of processing abnormality as an abnormal message into an abnormal message queue corresponding to the local queue, so as to retry the message and improve the success rate of data processing. Optionally, the number of retries may be set, and if the same communication message retries for multiple times and still processes the exception, the communication message may be stored in the database for query and review processing by service personnel.
In one embodiment, the method may further include an abnormal data out-of-standard alarm step, specifically including: and determining the processing result as the ratio of the number of the communication messages with abnormal processing to the total number of the communication messages, and outputting first alarm information under the condition that the ratio is larger than a preset threshold value.
In implementations, after the distributed system sends the communication message to the third party system, the processing results may be determined based on feedback information from the third party system. If the ratio of the number of abnormal communication messages processed by a certain third party system corresponding to the same period (within the second time window) to the total number of communication messages sent to the third party system for processing within the period is greater than a preset threshold (for example, greater than 20%, namely, the success rate is lower than 80%), the distributed system can output first alarm information for reminding service personnel to timely troubleshoot. It will be appreciated that the proportion or success rate of processing anomalies may also be calculated based on the processing results over multiple cycles.
In this embodiment, the processing result of the communication message may be monitored, and if the proportion of the processing abnormality of the communication message corresponding to a certain third party system is high (the success rate is low), the third party system may crash or other problems occur, so that the alarm information may be output to remind the service personnel to intervene in time to troubleshoot.
In one embodiment, the method may further include a queue length early warning step, specifically including: and outputting second alarm information for each local queue under the condition that the number of communication messages in the local queue exceeds a preset threshold value.
In implementation, the distributed system may monitor the number of messages stored in each local queue, i.e. the length of the queue, and if the length is greater than a threshold (e.g. the length reaches 90% of the maximum capacity, or the length reaches 1000), output second alarm information for reminding service personnel to check for faults.
In this embodiment, the length of the local queue may be monitored, and if the length of the local queue exceeds the threshold, it is indicated that the data transmission thread does not read the message from the local queue as expected and send the message to the third party system, and the local system or the third party system may fail, so that service personnel may be timely reminded of troubleshooting the failure.
It should be understood that, although the steps in the flowcharts related to the embodiments described above are sequentially shown as indicated by arrows, these steps are not necessarily sequentially performed in the order indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in the flowcharts described in the above embodiments may include a plurality of steps or a plurality of stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of the steps or stages is not necessarily performed sequentially, but may be performed alternately or alternately with at least some of the other steps or stages.
Based on the same inventive concept, the embodiment of the application also provides a data processing device for realizing the above related data processing method. The implementation of the solution provided by the device is similar to the implementation described in the above method, so the specific limitation of one or more embodiments of the data processing device provided below may refer to the limitation of the data processing method hereinabove, and will not be repeated herein.
In one embodiment, as shown in FIG. 7, there is provided a data processing apparatus 700 comprising: an allocation module 701 and a transmission module 702, wherein:
the allocation module 701 is configured to read a plurality of first communication messages from the target distributed message queue according to the message sorting parameter in response to the data transmission instruction, and allocate each read first communication message to a local queue corresponding to each third party system.
The sending module 702 is configured to, for each local queue, read a second communication message from the local queue according to a data transmission parameter corresponding to the local queue, and send the read second communication message to a third party system corresponding to the local queue; the data transmission parameters corresponding to the local queue are determined according to the processing capacity parameters of the third party system corresponding to the local queue.
In one embodiment, the processing capability parameter of the third party system comprises a transaction amount per unit time, and the message sort parameter comprises a first time window and a target read number within the first time window. The device further comprises a first acquisition module and a first determination module, wherein:
the first acquisition module is used for acquiring the transaction processing amount in unit time of each third party system and calculating the sum of the transaction processing amounts in each unit time.
The first determining module is used for determining the target reading quantity in a first time window based on the sum value, the quantity of the processing nodes of the distributed system and the preset first time window.
In one embodiment, the allocation module 701 is specifically configured to: reading a target read number of first communication messages from a target distributed message queue under the condition that the current time reaches the starting time of a current first time window; and determining a third party system corresponding to each first communication message according to the characteristic information contained in each read first communication message, and distributing each first communication message to a local queue corresponding to each third party system based on the third party system corresponding to each first communication message.
In one embodiment, the processing capability parameter of the third party system comprises a transaction amount per unit time, and the data transmission parameter comprises a second time window and a target transmission amount within the second time window. The device further comprises a second acquisition module, a second determination module and a third determination module, wherein:
and the second acquisition module is used for acquiring the transaction processing amount in unit time of the third party system corresponding to the local queues aiming at each local queue.
And the second determining module is used for determining the transaction processing amount in the unit time corresponding to the local queue according to the transaction processing amount in the unit time of the third party system and the number of the processing nodes of the distributed system.
And the third determining module is used for determining the target transmission quantity in the second time window corresponding to the local queue based on the transaction processing quantity in the unit time corresponding to the local queue and the preset second time window.
In one embodiment, the sending module 702 is specifically configured to: and under the condition that the current time reaches the starting time of the current second time window corresponding to the local queue, reading the target transmission number of second communication messages from the local queue, and sending the read second communication messages to a third party system corresponding to the local queue.
In one embodiment, the data transmission parameters include a second time window and a target number of transmissions within the second time window. The sending module 702 is specifically configured to: for each local queue, judging whether an abnormal message exists in an abnormal message queue corresponding to the local queue under the condition that the current time reaches the starting time of a current second time window corresponding to the local queue; the abnormal message queue is used for storing communication messages with processing results of processing abnormality; reading the abnormal message from the abnormal message queue under the condition that the abnormal message exists in the abnormal message queue; under the condition that the first number of the read abnormal messages is not smaller than the target transmission number, the read abnormal messages are sent to a third party system corresponding to the local queue; determining a second number according to the first number and the target transmission number under the condition that the first number is smaller than the target transmission number, and reading a second number of second communication messages from the local queue; and sending the read abnormal message and the second communication message to a third party system corresponding to the local queue.
In one embodiment, the device further includes a storage module, configured to determine, for each local queue, a processing result of each communication message according to feedback information of a third party system corresponding to the local queue, and store, as an exception message, a communication message with the processing result of processing exception in an exception message queue corresponding to the local queue.
In one embodiment, the device further includes a first alarm module, configured to determine a ratio of a number of communication messages that are processed abnormally to a total number of communication messages that are processed, and output first alarm information if the ratio is greater than a preset threshold.
In one embodiment, the apparatus further includes a second alarm module configured to output, for each local queue, second alarm information if the number of communication messages in the local queue exceeds a preset threshold.
Each of the modules in the above-described data processing apparatus may be implemented in whole or in part by software, hardware, and combinations thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.
In one embodiment, a computer device is provided, which may be a server, and the internal structure of which may be as shown in fig. 8. The computer device includes a processor, a memory, and a network interface connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of the computer device is used for storing data required or generated for executing the data processing method. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a data processing method.
It will be appreciated by those skilled in the art that the structure shown in fig. 8 is merely a block diagram of some of the structures associated with the present application and is not limiting of the computer device to which the present application may be applied, and that a particular computer device may include more or fewer components than shown, or may combine certain components, or have a different arrangement of components.
In one embodiment, a computer device is provided, comprising a memory and a processor, the memory having stored therein a computer program, the processor implementing the steps of the method embodiments described above when the computer program is executed.
In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored which, when executed by a processor, implements the steps of the method embodiments described above.
In an embodiment, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the steps of the method embodiments described above.
The data processing method, apparatus, computer device, storage medium and computer program product provided in the present application relate to the technical field of big data, and may be used in the financial and technological field or other fields, and the application field is not limited in this application.
It should be noted that, user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, presented data, etc.) referred to in the present application are information and data authorized by the user or sufficiently authorized by each party.
Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, database, or other medium used in the various embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high density embedded nonvolatile Memory, resistive random access Memory (ReRAM), magnetic random access Memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric Memory (Ferroelectric Random Access Memory, FRAM), phase change Memory (Phase Change Memory, PCM), graphene Memory, and the like. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, and the like. By way of illustration, and not limitation, RAM can be in the form of a variety of forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), and the like. The databases referred to in the various embodiments provided herein may include at least one of relational databases and non-relational databases. The non-relational database may include, but is not limited to, a blockchain-based distributed database, and the like. The processors referred to in the embodiments provided herein may be general purpose processors, central processing units, graphics processors, digital signal processors, programmable logic units, quantum computing-based data processing logic units, etc., without being limited thereto.
The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The above examples only represent a few embodiments of the present application, which are described in more detail and are not to be construed as limiting the scope of the present application. It should be noted that it would be apparent to those skilled in the art that various modifications and improvements could be made without departing from the spirit of the present application, which would be within the scope of the present application. Accordingly, the scope of protection of the present application shall be subject to the appended claims.

Claims (13)

1. A method of data processing, the method comprising:
responding to the data transmission instruction, reading a plurality of first communication messages from a target distributed message queue according to the message sorting parameter, and distributing the read first communication messages to local queues corresponding to the third party systems;
for each local queue, reading a second communication message from the local queue according to the data transmission parameters corresponding to the local queue, and sending the read second communication message to a third party system corresponding to the local queue; and the data transmission parameters corresponding to the local queue are determined according to the processing capacity parameters of the third party system corresponding to the local queue.
2. The method of claim 1, wherein the processing capability parameter of the third party system comprises a transaction amount per unit time, and the message sort parameter comprises a first time window and a target read number within the first time window; the method further comprises the steps of:
acquiring the transaction processing amount in unit time of each third party system, and calculating the sum of the transaction processing amounts in unit time;
and determining the target reading quantity in a first time window based on the sum value, the quantity of the processing nodes of the distributed system and the preset first time window.
3. The method according to claim 2, wherein the reading the plurality of first communication messages from the target distributed message queue according to the message sorting parameter, and distributing each read first communication message to a local queue corresponding to each third party system, includes:
reading the target read number of first communication messages from a target distributed message queue under the condition that the current time reaches the starting time of the current first time window;
and determining a third party system corresponding to each first communication message according to the characteristic information contained in each read first communication message, and distributing each first communication message to a local queue corresponding to each third party system based on the third party system corresponding to each first communication message.
4. The method of claim 1, wherein the processing capability parameter of the third party system comprises a transaction amount per unit time, and wherein the data transmission parameter comprises a second time window and a target transmission number within the second time window; the method further comprises the steps of:
for each local queue, acquiring the transaction processing amount in unit time of a third party system corresponding to the local queue;
determining the transaction processing amount in unit time corresponding to the local queue according to the transaction processing amount in unit time of the third party system and the number of processing nodes of the distributed system;
and determining the target transmission quantity in a second time window corresponding to the local queue based on the transaction processing quantity in the unit time corresponding to the local queue and the preset second time window.
5. The method according to claim 4, wherein for each local queue, according to the data transmission parameters corresponding to the local queue, reading the second communication message from the local queue, and sending the read second communication message to the third party system corresponding to the local queue, including:
And under the condition that the current time reaches the starting time of the current second time window corresponding to the local queue, reading the target transmission number of second communication messages from the local queue, and sending the read second communication messages to a third party system corresponding to the local queue.
6. The method according to any of claims 1 to 5, wherein the data transmission parameters comprise a second time window and a target number of transmissions within the second time window; the step of reading the second communication message from the local queues according to the data transmission parameters corresponding to the local queues and sending the read second communication message to the third party system corresponding to the local queues, includes:
for each local queue, judging whether an abnormal message exists in an abnormal message queue corresponding to the local queue under the condition that the current time reaches the starting time of a current second time window corresponding to the local queue; the abnormal message queue is used for storing communication messages with abnormal processing results;
reading an abnormal message from the abnormal message queue under the condition that the abnormal message exists in the abnormal message queue;
Under the condition that the first number of the read abnormal messages is not smaller than the target transmission number, the read abnormal messages are sent to a third party system corresponding to the local queue;
determining a second number according to the first number and the target transmission number and reading the second number of second communication messages from the local queue when the first number is smaller than the target transmission number;
and sending the read abnormal message and the second communication message to a third party system corresponding to the local queue.
7. The method of claim 6, wherein after the sending the read second communication message to the third party system corresponding to the local queue, further comprises:
and determining the processing result of each communication message according to the feedback information of the third party system corresponding to the local queue for each local queue, and storing the communication message with the processing result of processing abnormality as an abnormal message into an abnormal message queue corresponding to the local queue.
8. The method of claim 7, wherein after determining the processing result of each communication message according to the feedback information of the third party system corresponding to the local queue, further comprises:
And determining the processing result as the ratio of the number of the communication messages with abnormal processing to the total number of the communication messages, and outputting first alarm information under the condition that the ratio is larger than a preset threshold value.
9. The method according to claim 1, wherein the method further comprises:
and outputting second alarm information for each local queue under the condition that the number of communication messages in the local queue exceeds a preset threshold value.
10. A data processing apparatus, the apparatus comprising:
the distribution module is used for responding to the data transmission instruction, reading a plurality of first communication messages from the target distributed message queue according to the message sorting parameters, and distributing the read first communication messages to the local queues corresponding to the third party systems;
the sending module is used for reading the second communication message from the local queues according to the data transmission parameters corresponding to the local queues and sending the read second communication message to a third party system corresponding to the local queues; and the data transmission parameters corresponding to the local queue are determined according to the processing capacity parameters of the third party system corresponding to the local queue.
11. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any one of claims 1 to 9 when the computer program is executed.
12. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 9.
13. A computer program product comprising a computer program, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any one of claims 1 to 9.
CN202310577186.6A 2023-05-22 2023-05-22 Data processing method, device, computer equipment and storage medium Pending CN116545949A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310577186.6A CN116545949A (en) 2023-05-22 2023-05-22 Data processing method, device, computer equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310577186.6A CN116545949A (en) 2023-05-22 2023-05-22 Data processing method, device, computer equipment and storage medium

Publications (1)

Publication Number Publication Date
CN116545949A true CN116545949A (en) 2023-08-04

Family

ID=87450489

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310577186.6A Pending CN116545949A (en) 2023-05-22 2023-05-22 Data processing method, device, computer equipment and storage medium

Country Status (1)

Country Link
CN (1) CN116545949A (en)

Similar Documents

Publication Publication Date Title
US9805140B2 (en) Striping of directed graphs and nodes with improved functionality
CN111835562B (en) Log uploading method, log inquiring method and system
CN115334082A (en) Load balancing method, device, computer equipment, storage medium and product
CN116302580B (en) Method and device for scheduling calculation force resources of nano relay
CN116545949A (en) Data processing method, device, computer equipment and storage medium
CN116737373A (en) Load balancing method, device, computer equipment and storage medium
CN115757398A (en) Data storage method and device, computer equipment and storage medium
CN114756451A (en) Safety testing method and device for power grid system, computer equipment and storage medium
CN113347238A (en) Message partitioning method, system, device and storage medium based on block chain
CN114428704A (en) Method and device for full-link distributed monitoring, computer equipment and storage medium
CN112115119B (en) Graph database service performance monitoring method and device and storage medium
CN117271082A (en) Task processing method, device, server, system, medium and program product
CN117081961A (en) Method, system, device and server for monitoring quantity of accumulated messages
CN117648310A (en) Data degradation processing method, device, equipment and medium
CN117271277A (en) Alarming method, device, equipment and storage medium for abnormal service
CN112131058A (en) Cloud system health detection method and system
CN116578455A (en) Change risk monitoring method, device, computer equipment and storage medium
CN118071502A (en) Transaction fault identification method, device, equipment, storage medium and product
CN117651026A (en) Method and device for acquiring service resource data of external system and computer equipment
CN118034885A (en) Task processing method, device, computer equipment and storage medium
CN115061778A (en) Container starting method and device, computer equipment and storage medium
CN118034884A (en) Resource scheduling method, device, equipment, medium and program product
CN115964188A (en) Data processing method, system, device, computer equipment and storage medium
CN117193958A (en) Method, device, computer equipment and storage medium for determining timeout duration
CN117149404A (en) Method, device, computer equipment and storage medium for allocating detection task resources

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination