CN115695587A

CN115695587A - Service data processing system, method, device and storage medium

Info

Publication number: CN115695587A
Application number: CN202110858692.3A
Authority: CN
Inventors: 张伦聪; 秦政; 禤欢子
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2021-07-28
Filing date: 2021-07-28
Publication date: 2023-02-03

Abstract

The embodiment of the invention provides a service data processing system, a method, a device and a storage medium, wherein the method comprises the following steps: receiving a second service data set obtained by processing the first processing link, wherein the second service data set is data obtained by processing the first service data set, and the first service data set is obtained from a database node on the first processing link through a service intermediate node on the first processing link; and receiving service updating data processed by the second processing link, wherein the service updating data is obtained by processing target service supplementary data, the target service supplementary data is determined according to difference comparison between a first service data set and a second service data set, the first service data set is obtained from a database node, and the second service data set is obtained from a target search node on the first processing link, so that the writing efficiency and the accuracy of data transmission can be improved.

Description

Service data processing system, method, device and storage medium

Technical Field

The present invention relates to the field of internet technologies, and in particular, to a system, a method, an apparatus, and a storage medium for processing service data.

Background

With the development of internet technology, a large amount of service data is generated in various service processes, and the service data needs to be transmitted to a corresponding place for data storage, so that subsequent operations such as query on the service data are facilitated. For example, in the case of mobile payment service, with the rapid growth of commercial payment and the diversification of payment scenarios, the payment service, whether it is data capacity or response delay, is faced with higher requirements.

At present, the payment service data is stored in a relational database by adopting synchronous double writing, and simultaneously written into an elastic search engine, so that the service data can be inquired through the elastic search engine subsequently. However, in this case, the synchronous double writing requires strong consistency of data, and requires that the service data be successfully written into the relational database and the flexible search engine. However, the consumption of various resources and resources between the devices where the database is located and the devices where the flexible search engine is located may not be consistent, for example, bandwidth resources, processor resources, memory resources, and the like, which causes a problem of low writing efficiency when the synchronous double writing is performed.

Disclosure of Invention

The embodiment of the invention provides a service data processing system, a method, a device and a storage medium, which can improve the writing efficiency and the accuracy of data transmission.

In a first aspect, an embodiment of the present application provides a method for processing service data, where the method includes:

acquiring a first service data set stored by a database node through a service intermediate node, wherein the first service data set is data which is stored by the database node based on a data processing request and aims at a target service, and the service intermediate node is a node which is configured according to a database protocol of the database node and is used as slave equipment of the database node;

performing data processing on data in the first service data set to obtain a second service data set of the target service;

and sending the second service data set to the target search node, wherein the data in the second service data set belongs to the first service data set.

In one embodiment, the step of obtaining, by the service intermediate node, the first service data set from the database node includes:

performing data analysis on the plurality of service data stored in the service log node through a service intermediate node to obtain a plurality of analyzed service data;

and acquiring a first service data set from the analyzed plurality of service data.

In one embodiment, the parsed service data are stored in a message queue of a service message queue node, and the first service data set is read from the message queue of the service message queue node by using a read thread.

In one embodiment, the sending the second set of traffic data to the target search node includes:

acquiring connection resources for accessing the target search node from a Transmission Control Protocol (TCP) long connection pool; the connection resource comprises one or two of an available port number or an available communication address;

establishing a TCP connection according to connection resources for accessing the target search node;

and sending the second service data set to the target searching node by utilizing a write thread and through the established TCP connection.

In one embodiment, a service agent node is configured for the target search node; the establishing of the TCP connection according to the connection resource for accessing the target search node comprises the following steps:

establishing TCP connection with the service agent node according to the connection resource for accessing the target search node;

the sending the second service data set to the target search node by using the write thread and through the established TCP connection includes:

based on the established TCP connection, writing the second service data set into the service proxy node by using a write thread so as to trigger the service proxy node to process the second service data set, and storing the processed second service data set in the target search node.

In one embodiment, the method further comprises:

when the second service data set fails to be sent to the target search node, carrying out service rewriting processing on the second service data set at a first time interval;

acquiring the failure times of sending the second service data set to the target searching node in the service rewriting processing process;

if the failure times exceed a time threshold, adding the second service data set into a service polling queue;

traversing the service data in the service polling queue according to a second time interval, and rewriting the currently traversed service data into the target search node by using a write thread;

the second time interval is greater than the first time interval.

In one embodiment, the method further comprises:

and if the service data in the service polling queue comprises target service data with survival time length exceeding preset time length, deleting the target service data from the service polling queue.

In a second aspect, an embodiment of the present application provides a service data processing method, where the method includes:

acquiring target service supplementary data from a database supplementary node through a service supplementary intermediate node;

performing data processing on the target service supplementary data to obtain service updating data;

sending the service updating data to a target searching node;

the target service supplementary data is data which is stored in the database supplementary node and is determined by comparing differences of a first service data set and a second service data set through a data comparison node, wherein the first service data set is acquired from the database node, and the second service data set is acquired from the target search node.

In one embodiment, the step of obtaining target service supplementary data from the database supplementary node by the service supplementary intermediate node includes:

performing data analysis on the plurality of service supplementary data stored by the service supplementary log node through a service supplementary intermediate node to obtain a plurality of analyzed service supplementary data;

and acquiring target service supplementary data from the analyzed plurality of service supplementary data.

In one embodiment, the parsed service supplementary data are stored in a message queue of a service supplementary message queue node, and the target service supplementary data are read from the message queue of the service supplementary message queue node by using a read thread.

In one embodiment, the sending the traffic update data to the target search node includes:

acquiring connection resources for accessing the target search node from a Transmission Control Protocol (TCP) long connection pool; the connection resources include: one or both of available port numbers or available communication addresses;

and sending the service updating data to the target searching node by utilizing a write thread and through the established TCP connection.

In one embodiment, a service supplementary proxy node is configured for the target search node; the establishing of the TCP connection according to the connection resource for accessing the target search node includes:

establishing TCP connection with the service supplement proxy node according to the connection resource for accessing the target search node;

the sending the service update data to the target search node by using the write thread and through the established TCP connection includes:

based on the established TCP connection, writing the service updating data into the service supplement proxy node by using a write thread so as to trigger the service supplement proxy node to process the service updating data, and storing the processed service updating data in the target search node.

In one embodiment, the method further comprises:

when the service updating data is failed to be sent to the target searching node, performing supplementary service rewriting processing on the service updating data at a first time interval;

acquiring the failure times of sending the service updating data to a target searching node in the process of rewriting the supplementary service;

if the failure times exceed a time threshold, adding the service updating data into a supplementary service polling queue;

traversing the service updating data in the supplementary service polling queue according to a second time interval, and rewriting the service updating data traversed before to the target search node by using a write thread;

the second time interval is greater than the first time interval.

In one embodiment, the method further comprises:

and if the service updating data in the supplementary service polling queue comprises target service updating data with survival time exceeding preset time, deleting the target service updating data from the supplementary service polling queue.

In a third aspect, an embodiment of the present application provides a method for processing service data, where the method includes:

receiving a second service data set obtained by processing a first service data set, wherein the second service data set is data obtained by processing the first service data set, the first service data set is obtained from a database node on a first processing link through a service intermediate node on the first processing link, and the service intermediate node is a node which is configured according to a database protocol of the database node and is used as slave equipment of the database node;

receiving service updating data processed by a second processing link, wherein the service updating data is obtained by processing target service supplementary data, the target service supplementary data is determined according to difference comparison between a first service data set and a second service data set, the first service data set is obtained from the database node, and the second service data set is obtained from a target search node on the first processing link.

In one embodiment, the plurality of service data stored in the database node are allowed to be recorded in a service log node, and the first service data set is read from a message queue of a service message queue node by a service transmission node by using a reading thread; a plurality of analyzed service data are stored in a message queue of the service message queue node, and the plurality of analyzed service data are obtained by performing data analysis on the plurality of service data stored in the service log node by the service intermediate node; the service log node, the service transmission node and the service message queue node belong to the first processing link;

a plurality of service supplementary data stored in a database supplementary node are allowed to be recorded into a service supplementary log node, and the target service supplementary data are read from a message queue of a service supplementary message queue node by a supplementary service transmission node by using a reading thread; the message queue of the service supplementary message queue node stores a plurality of analyzed service supplementary data, and the plurality of analyzed service supplementary data are obtained by performing data analysis on the plurality of service supplementary data stored by the service supplementary log node by the service supplementary intermediate node; the database supplementary node, the service supplementary log node, the supplementary service transmission node, the service message queue node and the service supplementary intermediate node belong to the second processing link.

In one embodiment, the first processing link comprises a TCP connection; the TCP connection is established according to connection resources of a target search node accessed to the first processing link; the connection resource comprises one of an available port number or an available communication address; the connection resource of the target search node accessed to the first processing link is obtained from a Transmission Control Protocol (TCP) long connection pool; the second set of traffic data is sent by a traffic transport node on the first processing link using a write thread and over the TCP connection.

In one embodiment, a service agent node is configured for the target search node; the TCP connection is established with the service agent node according to the connection resource for accessing the target search node; the second service data set is sent by the service transmission node to the service proxy node by using a write thread and through the TCP connection.

In one embodiment, the second processing link comprises a TCP connection; the TCP connection is established according to the connection resource accessed to the target searching node; the connection resource accessed to the target search node is obtained from a TCP long connection pool; the traffic update data is sent by a supplementary traffic transfer node on the second processing link using a write thread and over the TCP connection.

In one embodiment, a service supplementary proxy node is configured for the target search node; the TCP connection is established with the service supplement proxy node according to the connection resource for accessing the target search node; and the service updating data is sent to the service supplementary agent node by the supplementary service transmission node through the TCP connection by using a write thread.

In a fourth aspect, an embodiment of the present application provides a service data processing apparatus, where the apparatus includes:

a service processing unit, configured to obtain, by a service intermediate node, a first service data set stored by a database node, where the first service data set is data, stored by the database node based on a data processing request, for a target service, and the service intermediate node is a node configured according to a database protocol of the database node and serving as a slave device of the database node;

the service processing unit is further configured to perform data processing on the data in the first service data set to obtain a second service data set of the target service;

and the service communication unit is used for sending the second service data set to the target search node, wherein the data in the second service data set belongs to the first service data set.

In a fifth aspect, an embodiment of the present application provides a service data processing apparatus, where the apparatus includes:

the service supplement processing unit is used for acquiring target service supplement data from the database supplement node through the service supplement intermediate node;

the service supplement processing unit is also used for carrying out data processing on the target service supplement data to obtain service update data;

the service supplementary communication unit is used for sending the service updating data to the target searching node;

the target service supplementary data are data which are stored in the database supplementary node and are determined after a first service data set and a second service data set are subjected to difference comparison through a data comparison node, the first service data set is obtained from the database node, and the second service data set is obtained from the target search node.

In a sixth aspect, an embodiment of the present application provides a service data processing apparatus, where the apparatus includes:

a communication unit for transceiving data;

a processing unit, configured to receive, through the communication unit, a second service data set obtained by processing a first service data set, where the second service data set is data obtained by processing the first service data set, the first service data set is obtained from a database node on the first processing link through a service intermediate node on the first processing link, and the service intermediate node is a node configured according to a database protocol of the database node and serving as a slave device of the database node;

the processing unit is further configured to receive, through the communication unit, service update data obtained by processing a second processing link, where the service update data is obtained by processing target service supplementary data, the target service supplementary data is data determined according to difference comparison between the first service data set and the second service data set, the first service data set is obtained from the database node, and the second service data set is obtained from a target search node on the first processing link.

In a seventh aspect, an embodiment of the present application provides a service data processing system, where the system includes:

the database node is used for storing service data aiming at the target service based on the data processing request, wherein the stored service data aiming at the target service comprises a first service data set;

the service intermediate node is used for acquiring the first service data set from the database node;

the service transmission node is configured to acquire the first service data set from the database node through the service intermediate node, and perform data processing on data in the first service data set to obtain a second service data set of the target service; and sending the second service data set to a target search node, wherein the data in the second service data set belongs to the first service data set.

In an eighth aspect, an embodiment of the present application provides an intelligent device, where the intelligent device includes: a storage device and a processor; the storage device is used for storing a computer program; the processor is configured to execute the computer program to implement the method of the first aspect, or to implement the method of the second aspect, or to implement the method of the third aspect.

In a ninth aspect, the present invention further provides a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed, the method of the first aspect or the method of the second aspect or the method of the third aspect will be implemented.

In a tenth aspect, the present application also provides a computer program product or a computer program, where the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the smart device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to cause the smart device to perform the method described above.

In the embodiment of the application, a second service data set obtained by processing a first processing link is received, where the second service data set is data obtained by processing a first service data set, and the first service data set is obtained from a database node on the first processing link through a service intermediate node on the first processing link; the data in the database node can be directly stored in the target searching node through the first processing link without the participation of other links, so that the accuracy of data transmission is improved; and receiving service updating data processed by the second processing link, wherein the service updating data is obtained by processing target service supplementary data, and the target service supplementary data is determined by performing difference comparison on the first service data set and the second service data set. Through the second processing link, data difference comparison is carried out on the data in the database node and the data in the target searching node, and when the data difference is determined, the service updating data corresponding to the difference data can be stored in the target searching node again, so that the data in the database node and the data in the searching equipment can be enabled to be consistent well, and the accuracy of subsequent data query is facilitated.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.

Fig. 1 is an architecture diagram of a service data processing system according to an embodiment of the present invention;

fig. 2 is a schematic flowchart of a service data processing method according to an embodiment of the present invention;

fig. 3 is a schematic structural diagram of a service transmission node according to an embodiment of the present invention;

fig. 4 is a schematic flowchart of a service data processing method according to an embodiment of the present invention;

fig. 5 is a schematic flowchart of a service data processing method according to an embodiment of the present invention;

fig. 6a is a schematic flow chart of a service data processing scheme according to an embodiment of the present invention;

fig. 6b is a schematic flow chart of a service data processing scheme according to an embodiment of the present invention;

fig. 6c is a schematic flow chart of a service data processing scheme according to an embodiment of the present invention;

fig. 7 is a schematic structural diagram of a service data processing apparatus according to an embodiment of the present invention;

fig. 8 is a schematic structural diagram of a service data processing apparatus according to an embodiment of the present invention;

fig. 9 is a schematic structural diagram of a service data processing apparatus according to an embodiment of the present invention;

fig. 10 is a schematic structural diagram of an intelligent node according to an embodiment of the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

The embodiment of the application provides a business data processing scheme, which mainly realizes data transmission and data storage of business data through two processing links, and the accuracy of data storage and subsequent data query is better ensured while the efficiency and the accuracy of data writing and data transmission are improved through the two processing links. The first processing link is mainly that a plurality of nodes are arranged between the database node and the target searching node, and the service data stored in the database can be directly and accurately transmitted to the target searching node. The plurality of nodes in the first processing link comprise a database node, a service intermediate node, a service transmission node and a service message queue node. The service intermediate node can be disguised as a node of slave equipment in the database node to acquire service data in the database node, then the service intermediate node can analyze the acquired service data and push the analyzed service data to the service message queue node, then the service transmission node reads the analyzed service data in the service message queue node, performs service processing on the read analyzed service data and sends the processed service data to the target search node, and when the service data is directly written into the database node through the first processing link, the service data in the database node is directly transmitted to the target search node, so that the data transmission is simple and efficient, and meanwhile, the accuracy of the data transmission and the efficiency of the data writing are improved better.

The second processing link is mainly used for checking data in the target searching node and data in the database node, if the data in the target searching node is found to be missing compared with the data in the database node or if the data in the target searching node is found to be inconsistent with the data in the database node, the data in the database node is taken as the standard, target service supplementary data are obtained from a supplementary database node of the second processing link, the target service supplementary data are initiated through the second processing link for remediation, then the target service supplementary data are transmitted to the target searching node through each node on the second processing link, and the data are corrected, so that the data in the database node and the searching equipment are better consistent, and the accuracy of service data query is improved. In one embodiment, a Socket pool (Socket pool) (hereinafter referred to as a TCP (Transmission Control Protocol) long connection pool) and a delay failure queue are constructed in advance in a service Transmission node and a supplementary service Transmission node, so as to reduce the delay problem and improve the efficiency of data writing and the efficiency of data Transmission.

Referring to fig. 1, based on the service data processing scheme provided above, fig. 1 is an architecture diagram of a service data processing system according to an embodiment of the present invention. The service data processing system may comprise a service system, a database node 101, a service intermediate node 103, a service transport node 105 and a target search node 106. In one embodiment, the service data processing system may further include a data comparison node 107, a database supplement node 108, a service supplement intermediate node 110, and a supplementary service transmission node 112, where the service intermediate node 103 is a node configured according to a database protocol of the database node 101 and serving as a slave of the database node 101; in one embodiment, the service processing system may further include a service log node 102 and a service supplementary log node 109, a service message queue node 104 and a service supplementary message queue node 111. In one embodiment, the business data processing system further includes a service broker node 1061 and a service replenishment broker node 1062 configured for the target search node 106. The service intermediate node 103 and the service supplementary intermediate node 110 may be the same intermediate node or different intermediate nodes; the service transmission node 105 and the supplementary service transmission node 112 may be the same transmission node or different transmission nodes; the service log node 102 and the service supplementary log node 109 may be the same log node or different log nodes; the service message queue node 104 and the service supplement message queue node 110 may be the same message queue node, or may be different message queue nodes, and the service agent node 1061 and the service supplement agent node 1062 may be the same agent node, or may be different agent nodes; the embodiments of the present application do not limit this.

The aforementioned database node 101, the service log node 102, the service intermediate node 103, the service message queue node 104, the service transmission node 105, and the target search node 106 may be communicatively connected through a mobile communication network, a computer network, or the like, and form a first processing link; the database node 101, the data comparison node 107, the database supplement node 108, the service supplement log node 109, the service supplement intermediate node 110, the service supplement message queue node 111, the supplementary service transmission node 112, and the target search node 106 are also in communication connection through a mobile communication network, a computer network, and the like, and form a second processing link. It should be noted that: the nodes related to the embodiments of the present application may be components (middleware), that is, the related nodes mainly refer to some middleware that performs corresponding processing, and the middleware is deployed into corresponding intelligent devices to execute corresponding steps to implement corresponding functions. The intelligent device can be a terminal or a server; the terminal may be, but is not limited to, a smart phone, a tablet computer, a notebook computer, a desktop computer, and the like. The server may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing basic cloud computing services such as cloud service, a cloud database, cloud computing, a cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN (Content Delivery Network), big data, an artificial intelligence platform, and the like.

The service system may send a data processing request to the database node 101, where the data processing request carries service data of the target service. The data processing request may be a data storage request, a data modification request, etc. for the target service. The target business can be a payment business, a refund business and the like, and can be other businesses, such as a business for recording commodity information in an e-commerce platform, a business for recording services provided by stores and the like.

The database node 101 may be configured to receive a data processing request sent by the service system, and store service data for a target service based on the data processing request. The service data for the target service stored in the database 101 includes a first service data set. Wherein, the database node 101 may be a relational database MySQL; the database node 101 is configured to process and store the service data for the target service according to the data processing request. The processing here may be, for example: and converting the format of the service data of the target service, so that the service data set of the target service can be stored in the database node 101.

The service log node 102 may be a binary log node (e.g., binlog node of MySQL), and the service log node 102 is configured to record a plurality of service data stored by the database node 101. Specifically, the service log node is configured to store a plurality of service data in the first log information, where the plurality of service data may be part of or all of the data stored in the database node 101. The partial data may refer to some traffic data in the database node 101 that meet a preset time range. The preset time can be set according to requirements. For example, when the database node 101 is configured to store the service data for the target service based on the data processing request, the first log information of the service log node 102 is configured to record the service data for the target service stored by the database node 101.

The service intermediate node 103 is configured to obtain a plurality of service data recorded by the service log node 102, analyze the plurality of recorded service data to obtain a plurality of analyzed service data, and send the plurality of analyzed service data to the service message queue node 104. The service intermediate node 103 may be a data synchronization backup tool DBSync (a database synchronization backup tool). The DBSync may be used to acquire service data of the database node 101 change recorded by the service log node 102. This can be understood as follows: the plurality of service data are service data that have changed in the database node 101. In a specific implementation, the service intermediate node 103 may be configured to send a dump command (a command for backup) to the service logging node, so as to obtain a plurality of service data with changes recorded by the service logging node. In one embodiment, the service data recorded in the service log all correspond to a time stamp; therefore, the service intermediate node 103 may be further configured to obtain, according to the timestamp of each service data, a plurality of service data meeting the preset time range from the service log node 102, perform data analysis on the plurality of service data meeting the preset time range, and store the plurality of analyzed service data meeting the preset time range in the message queue of the service message queue.

The service message queue node 104 is configured to store the analyzed multiple service data, where the service message queue node 104 may be a HIPPO, and the HIPPO is a new generation message system (or a message queue), and may meet service requirements of a high-reliability and high-availability application scenario, and be used to support services of high-value data such as advertisement charging and transaction pipelining. The analyzed plurality of service data may be obtained by analyzing a plurality of service data meeting a preset time range.

The service transmission node 105 is configured to acquire the first service data set from the database node 101 through the service intermediate node 103, perform data processing on the first service data set to obtain a second service data set of the target service, and then send the second service data set to the target search node 106. The data in the second service data set belongs to the first service data set; the data processing here may be, for example, format conversion of data, so that the format of the service data in the database node is converted into a data format that the target search node can store. The data processing may be performed in other processing manners, and the embodiment of the present application is not limited thereto. The traffic transfer node 105 may be a DTS (Data Transmission Service) device, and may be for Data migration between homogeneous or heterogeneous storage nodes.

In one embodiment, when the first service data set is obtained from the database node 101 through the service intermediate node 103, the service transmission node 105 is configured to perform data analysis on the plurality of service data stored in the service log node 102 through the service intermediate node 103 to obtain a plurality of analyzed service data, and then the service transmission node 105 obtains the first service data set from the plurality of analyzed service data. In one embodiment, the first service data set may include all or part of the parsed service data in the plurality of service data. In one embodiment, the service transmission node 105 may obtain the first service data set within the target time range from the parsed plurality of service data. In specific implementation, the analyzed service data carries a timestamp, and the analyzed service data with the time indicated by the timestamp within the target time range is added to the first service data set. For example, the analyzed service data are service data 1 and service data 2, respectively, the time indicated by the timestamp of the service data 1 is 11 points, and the time indicated by the timestamp of the service data 2 is 11 points 20; the preset time range is 10-12 points; the service transmission node 105 then determines that both service data 1 and service data 2 are added to the first set of service data.

In an embodiment, the plurality of parsed service data are stored in a message queue of the service message queue node 104, and the service transmission node 105 may be configured to obtain the first service data set from the message queue of the service message queue node 104 by using the read thread.

In one embodiment, the traffic transport node 105 is configured to obtain a connection resource for accessing the target searching node from the TCP long connection pool when sending the second traffic data set to the target searching node 106, wherein the connection resource includes one of an available port number or an available communication address. The traffic transport node 105 then establishes a TCP connection based on the connection resources for accessing the target search node, and sends a second set of traffic data to the target search node 106 using the write thread in the traffic transport node 105 and through the established TCP connection. The TCP long connection pool includes many connection resources of the access target search node 106, and in actual application, any connection resource of the access target search node 106 may be selected from the TCP connection pool. The available communication Address may be an IP Address (Internet Protocol Address), a MAC Address (Media Access Control Address), and the like.

In one embodiment, as can be seen from the foregoing, the service proxy node 1601 is configured for the target search node, and when the service transport node 105 establishes a TCP connection according to the connection resource for accessing the target search node, the service proxy node may specifically be configured to: a TCP connection is established with the service proxy node 1601, based on connection resources for accessing the target search node. Then, by using a write thread and through the established TCP connection, a second service data set is sent to the target search node, specifically configured to: based on the established TCP connection, a second service data set is written into the service proxy node 1601 by using a write thread, so as to trigger the service proxy node 1601 to process the second service data set, and the processed second service data set is stored in the target search node 106.

In one embodiment, the service transmission node 105 is configured to, when sending the second service data set to the target search node 106 fails, perform service rewriting processing on the second service data set at a first time interval, and obtain the number of times of failure of sending the second service data set to the target search node 106 in the process of service rewriting processing; then, the service transmission node 105 may determine whether the number of failures exceeds a number threshold, and if it is determined that the number of failures exceeds the number threshold, add the second service data set to the service polling queue, traverse the service data in the service polling queue according to a second time interval, and rewrite the currently traversed service data to the target search node by using the write thread. Wherein the second time interval is greater than the first time interval; the first time interval may be understood as an extremely short time interval, which may be 1S, 4S, etc. The second time interval may be set as desired, and the second time interval may be 30 minutes, 40 minutes, and so on. In one embodiment, the service transmission node 105 is configured to remove the currently traversed service data from the service polling queue if the service data currently traversed is successfully rewritten to the target search node 106 by using the write thread.

In an embodiment, the service transmission node 105 is further configured to determine whether the service data in the service polling queue includes target service data whose survival duration exceeds the preset duration, and delete the target service data from the service polling queue if the service data in the service polling queue includes the target service data whose survival duration exceeds the preset duration. The preset time period may be set according to a requirement, for example, the preset time period may be 1 day, 2 days, and the like. The time-to-live may be understood as an effective time of the service data in the service polling queue, and the time-to-live of any service data in the service polling queue refers to a time from when any service data is added to the service polling queue to the current time. For example, any service data is data a; the time for adding the data a to the service polling queue is 11, and the current time is 12:00; the survival time for data a was 40 minutes.

In one embodiment, if it is determined that the service data in the service polling queue does not include the target service data whose survival time exceeds the preset time, the service transmission node 105 continuously traverses the service data in the service polling queue, so that the service data in the service polling queue can be rewritten into the target search node within the validity period; of course, there may be a case where some or all of the service data in the service polling queue fails to be rewritten in the validity period, and the data that fails to be rewritten in the validity period will be deleted (that is, the validity period is within the preset time length).

The service proxy node 1061 is configured to receive, based on the TCP connection, the second service data set sent by the service transmission node 105, process the service data in the received second service data set, and store the processed service data set in the target search node 106. The processing here may be, for example, slicing, tabulating, and so on of the service data in the second service data set according to the timestamp of the service data in the second service data set. The TCP connection means: a long-time communication connection is established with the service transmission node 105 based on the connection resources of the access target search node.

The target search node 106 is configured to receive the second service data set sent by the service transmission node 105, or the target search node 106 is configured to store the second service data set processed by the service agent node 1061; in one embodiment, the target search node 106 is configured to receive the service update data sent by the supplementary service transmission node, or the target search node 106 is configured to store the service update data processed by the service supplementary proxy node 1062. The target search node 106 may be an ES (distributed full text search engine).

A data comparison node 107 for checking whether the data in the target search node 106 and the data in the database node 101 are consistent; the data comparison node 107 may be an ESCheck (a data comparison node), and in one embodiment, the data comparison node 107 is configured to obtain the first service data set and the second service data set according to the timestamp; and comparing the data difference between the first service data set and the second service data set to obtain the target service supplementary data. The first service data set is obtained from the database node 101, and the second service data set is obtained from the target search node 106. The difference of the data has two reasons: (1) The data in the second service data set is inconsistent with the data in the first service data set, and it is possible that the data in the second service data set is abnormal when being transmitted; (2) The data in the first service data set is not stored in the second service data set. In this case, the target service supplementary data is subject to the service data in the first service data set; for example, the first service data set includes service data as payment a; the second service data set includes service data as payment B, and the data comparison node 107 is configured to perform data difference comparison between the payment a and the payment B to obtain target service supplementary data as payment a. As another example, the first service data set includes service data as payment a; the second service data set does not include service data as payment a, and the data comparison node 107 is configured to perform data difference comparison on the first service data set and the second service data set to obtain target service supplementary data as payment a.

In one embodiment, since the format of the service data in the first service data set is different from the format of the service data in the second service data set, it is difficult to perform data difference comparison; therefore, when the data comparison node 107 performs data difference comparison on the first service data set and the second service data set to obtain target service supplementary data, the data comparison node 107 is configured to convert the format of the first service data set into a Json format, convert the format of the second service data set into the Json format, perform MD5 (Message-Digest Algorithm) calculation on the first service data set converted into the Json format to obtain a first MD5 value, and perform MD5 calculation on the second service data set converted into the Json format to obtain a second MD5 value; and then the data comparison node 107 compares whether the first MD5 value is consistent with the second MD5 value, and if the first MD5 value is inconsistent with the second MD5 value, performs data difference comparison on the first service data set and the second service data set to determine the target service supplementary data.

The database supplementary node 108 is configured to store a plurality of service supplementary data, so that the subsequent supplementary service transmission node 112 may obtain the service data to be supplemented from the database supplementary node 108, where the database supplementary node 108 may be MySQL. The service data to be supplemented is consistent with the corresponding service data in the database node; in one embodiment, the database supplemental node 108 is configured to store the target service supplemental data. In the example above, database supplemental node 108 is used to store payment A.

The service supplementary log node 109 is configured to record a plurality of service supplementary data stored by the database supplementary node 108. The traffic supplementary log node 109 may be a binary log node (e.g., binlog node). Specifically, the service supplementary log node is configured to store a plurality of service supplementary data as the second log information, where the plurality of service supplementary data may be part of or all of the data in the database supplementary node 108, for example, the plurality of service supplementary data may be some of the service supplementary data in the database supplementary node 108 that meet a preset time range.

The service supplement intermediate node 110 is configured to obtain a plurality of service supplement data recorded by the service supplement log node 109, analyze the plurality of recorded service supplement data to obtain a plurality of analyzed service supplement data, and send the plurality of analyzed service supplement data to the service supplement message queue node 111. Wherein the service supplementation intermediate node 110 may be a DBSync (database synchronization tool). DBSync may be used to acquire the service supplementary data recorded by the service supplementary log node 109 and changed in the database supplementary node 108. This can be understood as follows: the plurality of service supplementary data is the service supplementary data that has changed in the database supplementary node 108. In a specific implementation, the service supplementation intermediate node 110 may send a dump command to the service supplementation logging node 109, so as to obtain a plurality of changed service supplementation data recorded by the service supplementation logging node 109. Wherein the plurality of service supplementary data may be part of the data in the service supplementary log node 109. For example, the service supplementing intermediate node 110 may obtain, according to the timestamp, a plurality of service supplementing data that satisfy the preset time range from the service supplementing log node 109, where a specific implementation manner in which the service supplementing intermediate node 110 may obtain, according to the timestamp, a plurality of service supplementing data that satisfy the preset time range from the service supplementing log 109 may refer to a specific implementation manner in which the service intermediate node 103 obtains, according to the timestamp of each service data, a plurality of service data that satisfy the preset time range from the service log node 102, and details of the implementation manner are not described herein again.

The service supplementary message queue node 111 is configured to store the parsed multiple service supplementary data, and the service supplementary message queue node 111 may be a HIPPO.

The supplementary service transmission node 112 is configured to perform data analysis on the multiple pieces of service supplementary data stored by the service supplementary log node 109 through the service supplementary intermediate node 110, so as to obtain multiple pieces of analyzed service supplementary data; and acquiring target service supplementary data from the analyzed plurality of service supplementary data. The target service supplementary data may be one or more. In one embodiment, the supplementary service transmission node 112 may obtain the target service supplementary data within the target time range from the parsed plurality of service supplementary data. Specifically, each analyzed service supplementary data corresponds to a time stamp, and then, according to the time indicated by the time stamp of each analyzed service supplementary data, target service supplementary data within a target time range is acquired from the plurality of analyzed service supplementary data. The target time range can be set according to actual requirements; it should be noted that: if the plurality of analyzed service supplementary data are obtained by analyzing the acquired service supplementary data meeting the preset time range, the target time range is smaller than or equal to the preset time range. The supplementary service transmission node 112 may also be a DTS device. The supplementary service transmission node 112 and the service transmission node may be the same node or different nodes.

In one embodiment, the parsed multiple service supplementary data are stored in a message queue of the service supplementary message queue 111, and the supplementary service transmission node 112 is configured to read the target service supplementary data from the message queue of the service supplementary message queue 111 by using a reading thread.

In one embodiment, the supplementary service transport node 112 is configured to obtain connection resources for accessing the target search node 106 from a TCP long connection pool; the connection resource comprises one of an available port number or an available communication address; establishing a TCP connection according to connection resources for accessing a target search node; the traffic update data is then sent to the target search node 106 using the write thread and over the established TCP connection.

In one embodiment, the supplementary service transmitting node 112, when establishing a TCP connection based on the connection resources of the access target searching node, is configured to: and establishing TCP connection with the service supplement proxy node according to the connection resource for accessing the target search node. The supplementary service transmitting node 112, when sending the service update data to the target searching node by using the write thread and through the established TCP connection, is configured to: based on the established TCP connection, the write thread is used to write the service update data into the service supplement proxy node, so that the service supplement proxy node processes the service update data, and stores the processed service update data in the target search node 106.

In one embodiment, the supplementary service transmission node 112 is further configured to: when the service updating data is failed to be sent to the target searching node, performing supplementary service rewriting processing on the service updating data at a first time interval; acquiring the failure times of sending service updating data to the target search node 106 in the process of rewriting the supplementary service; judging whether the failure times exceed a failure threshold value; if the supplementary service transmission node 112 determines that the failure times exceed the time threshold, adding the service update data into a supplementary service polling queue; and traversing the service update data in the supplementary service polling queue according to a second time interval, and rewriting the currently traversed service update data to the target search node 106 by using a write thread. Wherein the second time interval is greater than the first time interval; the first time interval may be understood as an extremely short time interval, which may be 1 second, 4 seconds, etc. The second time interval may be set as desired, and the second time interval may be 30 minutes, 40 minutes, and so on. If the rewriting of the currently traversed service update data into the target search node 106 exceeds a certain time threshold, for example, 24 hours, the currently traversed service update data may be directly discarded, different preset durations may be set according to attribute information corresponding to the currently traversed service update data, when attribute information corresponding to the currently traversed service update data indicates that a service to which the currently traversed service update data belongs is marked as an important service, the set preset duration is a first duration, and when attribute information corresponding to the currently traversed service update data indicates that a service to which the currently traversed service update data belongs is marked as a normal service, the set preset duration is a second duration, where the first duration is longer than the second duration.

In one embodiment, the supplementary service transmission node 112 is further configured to: judging whether the service updating data in the supplementary service polling queue comprises target service updating data of which the survival time length exceeds the preset time length or not; if the supplementary service transmission node 112 determines that the service update data in the supplementary service polling queue includes target service update data whose survival time exceeds the preset time, the target service update data is deleted from the supplementary service polling queue. The time-to-live duration here can be understood as the effective time duration of the service update data in the supplementary service polling queue, and the time-to-live duration of the service update data in the supplementary service polling queue refers to the time duration from the time when the service update data is added to the supplementary service polling queue to the current time. For example, the service update data is data a; the time for adding the data a to the service polling queue is 11, and the current time is 12:00; the survival time for data a was 40 minutes. If the survival time of the target service update data exceeds a certain time threshold, for example, 24 hours, the target service update data may be directly discarded, different preset times may be set according to attribute information corresponding to the target service update data, when the attribute information corresponding to the target service update data indicates that the service to which the target service update data belongs is marked as an important service, the set preset time is a first time, and when the attribute information corresponding to the target service update data indicates that the service to which the target service update data belongs is marked as a normal service, the set preset time is a second time, and the first time is longer than the second time. At this time, it can be understood that the survival time of the target service update data exceeds the preset time, and the target service update data can be discarded in advance.

In one embodiment, if the supplementary service transmission node 112 determines that the service update data in the supplementary service polling queue does not include the target service update data whose survival time exceeds the preset time, the supplementary service transmission node 112 may continue polling the service update data in the supplementary service polling queue according to the second time interval, so that the service update data in the supplementary service polling queue can be rewritten into the target search node within the validity period; of course, there may be a case where some or all of the service update data in the supplementary service polling queue fails to be rewritten within the validity period, and the data that fails to be rewritten will be deleted.

And the service supplementary proxy node 1062 is configured to receive, based on the TCP connection, service update data sent by the supplementary service transmission node, process the received service update data, and store the processed service update data in the target search node. The processing here may be, for example, slicing, sub-listing, etc. the service update data according to the timestamp of the service update data. The TCP connection means: a long-term communication connection is established with the supplementary service transmission node 112 based on the connection resources for accessing the target search node.

It should be noted that, in the embodiment of the present application, a service log node and a service supplementary log node; a service intermediate node and a service supplement intermediate node; a service message queue node and a service supplement message queue node; the service transmission node and the supplementary service transmission node have similar functions and can be referred to each other.

In one embodiment, because a large amount of service data is generated for a target service, for example, the target service is a mobile payment service, a large number of users carry out mobile payment every day, which easily causes a problem in system performance; therefore, in order to save service data acquisition time and solve the bottleneck of system performance, the embodiment of the application adopts a concurrent clock at each node, a copy can be created for a thread local variable in each thread through a thread local variable in Java, and each thread can only access the copy variable of the thread when accessing, so that the concurrent clock is realized; the asynchronous log means that all service data are saved as much as possible when service requirements are met when the service data are written into the service log of the corresponding node, and meanwhile, data irrelevant to the service requirements can not be saved.

In the embodiment of the application, a first service data set stored by a database node 101 is acquired through a service intermediate node 103, and data processing is performed on data in the first service data set to obtain a second service data set of a target service; then, the second service data set is sent to the target search node 106, the data of the database node is directly written into the target search node 106 through the service intermediate node 103 and the service transmission node 105 after being written into the database node 101 without being simultaneously written into the target search node 106 when being written into the database node 101, so that the data writing efficiency is improved, and the accuracy of data transmission is better ensured; in addition, the data difference comparison is carried out on the first service data set and the second service data set through the data comparison node, the target service supplementary data are obtained from the database supplementary node according to the data difference, then the target service supplementary data are processed to obtain service updating data, the service updating data are sent to the target search node, and the service updating data can be stored into the target search node again when the data difference exists through the data difference comparison, so that the consistency and the accuracy of the data in the database node and the data in the target search node can be realized, and the accuracy of subsequent data query is facilitated.

Referring to fig. 2, fig. 2 is a schematic flowchart of a service data processing method according to an embodiment of the present invention. The service data processing method may be executed by the service transmission node 105 in the service data processing system, where the service transmission node 105 may be a DTS device. The service data processing method described in this embodiment includes the following steps S201 to S203:

s201: the method comprises the steps that a first service data set stored by a database node is obtained through a service intermediate node, the first service data set is data which are stored by the database node based on a data processing request and aim at a target service, and the service intermediate node is a node which is configured according to a database protocol of the database node and is used as slave equipment of the database node. Here, the service intermediate node may be understood as a node masquerading as a slave device of the database node to obtain a plurality of service data stored in the database node. From the perspective of the database node master device, the service intermediate node is only a node of a slave device of the database node.

In one embodiment, a plurality of traffic data stored in the database node is allowed to be recorded in the traffic log node. The specific implementation manner of step S201 is: the service transmission node can perform data analysis on the plurality of service data stored in the service log node through the service intermediate node to obtain a plurality of analyzed service data. And then acquiring a first service data set from the analyzed plurality of service data. Specifically, the service intermediate node may pull a plurality of service data recorded in the service log node, and then perform data analysis on the plurality of service data stored in the service log node through a database log protocol. And if the database node is MySQL, the database log protocol is the binlog protocol of the MySQL. Then, the service transmission node may obtain the analyzed multiple service data, and obtain a first service data set from the analyzed multiple service data.

In the embodiment of the present application, the same thread is generally used for processing read/write operations for the conventional message queue reading, which has a disadvantage that the read/write operations are mutually pinned, so that a bottleneck is easily generated in a scenario where the read/write performance is inconsistent, and in the implementation of the present application, the write time is far longer than the read time, because the target search node (e.g. bill ES) has a current limiting measure and the bottom layer server does not make the SSD (Solid State Disk or Solid State Drive) be a mechanical Disk used, a phenomenon of more burrs occurs, and the upstream needs to retry continuously. Therefore, in the application embodiment, the architecture of the service transmission node is optimized, the structure of the service transmission node is as shown in fig. 3, and a read-write separation design is used in the service transmission node, so that a read thread and a write thread are separated. The read thread is only responsible for reading data, and the write thread is only responsible for writing data. The design greatly improves the efficiency and can achieve quasi-real-time aging.

In one embodiment, the parsed service data sets are stored in a message queue of a service message queue node, and the service transmission node may read the first service data set from the message queue of the service message queue node by using a reading thread. For example, in fig. 3, the message queues of the service message queue node include queue 1, queue 2, queue 3 \8230; queue n; the read thread is then used to read a first set of traffic data from the message queue of the traffic message queue node (e.g., the first set of traffic data includes queue 1 and queue 2).

S202: and carrying out data processing on the data in the first service data set to obtain a second service data set of the target service. The data processing may be, for example, format conversion processing, which mainly refers to converting a service data format in a database node into a data format that can be stored by a target search node. The first set of traffic data may be one or more pieces of traffic data for the target traffic. The data in the second service data set belongs to the first service data set.

S203: and sending the second service data set to the target searching node. In one embodiment, in fig. 3, the service transfer node stores a TCP long connection pool, where the TCP long connection pool stores a connection resource for accessing the target search node, where the connection resource may be one of an available port number and an available communication address, and the service transfer node may obtain the connection resource for accessing the target search node from the TCP long connection pool, establish a TCP connection according to the connection resource of the target search node, and then send the second service data set to the target search node through the established TCP connection by using a write thread in the service transfer node. The available communication address may be an IP address, a MAC address, or the like, among others. In a specific implementation, a service transmission node may obtain connection resources accessed to a target search node from a TCP long connection pool, then initiate a TCP request to the target search node, where the TCP request carries an Svrkit (an application layer protocol) protocol package, and then establish a TCP connection based on the TCP request.

When the system is started, the TCP connection pool interacts with cl5 (a device for realizing load balancing) to obtain an available communication address and an available port of an access target search node, so that a TCP (or Socket) long connection pool with a maximum limit is cached, and meanwhile, a method for verifying TCP (or Socket) validity is set. The method for verifying the validity of the TCP (or Socket) may be: when each machine accesses, recording access information of each machine, wherein the access information comprises the current state of the machine, an available port and an available communication address; then determining an available port and an available communication address of each machine based on each machine access information; and generates a maximum number of limited long connection pools of TCP (or Socket) according to the available ports and the available communication addresses of each machine. And then continuously and periodically verifying the validity of the available port or the available communication address in the TCP (or Socket) long connection pool, if a certain machine is detected to have a fault or the flow of the certain machine is relatively large, the service transmission node determines that the available port and the available communication address corresponding to the machine in the TCP (or Socket) long connection pool are invalid, and timely eliminates the port and the communication address corresponding to the machine. In one embodiment, whether the TCP (or Socket) long connection pool is healthy or not may also be periodically polled, and if the TCP (or Socket) long connection pool is unhealthy, which indicates that there is an unavailable port or communication address, the unavailable port or communication address is rejected, so as to ensure that the established TCP connection is valid. Wherein, the health here means: whether a port or communication address in the TCP long connection pool is available. The machine related to the above may be a device corresponding to the service agent node or a device corresponding to the service supplementary agent node in the embodiment of the present application; but also a device corresponding to the target search node.

In one embodiment, for a target service as a payment service, a target search node (e.g., a bill ES) in the embodiment of the present application is slightly different from an external open source search node (e.g., an open source ES), and a service Proxy node (ES Proxy) adapted for payment exists, and a port of a service transport node needs to be opened to establish a TCP connection with the service Proxy node ES Proxy when communicating with the bill ES. In this case, a service proxy node may be configured for the target search node, and the specific implementation manner of establishing the TCP connection by the service transmission node according to the connection resource used for accessing the target search node is as follows: and establishing TCP connection with the service agent node according to the connection resources for accessing the target search node.

In one embodiment, a specific implementation manner that the service transmission node may send the second service data set to the target search node by using the write thread and through the established TCP connection is as follows: and writing a second service data set into the service agent node by using the write thread based on the established TCP connection so as to trigger the service agent node to process the second service data set, and storing the processed second service data set in the target search node. The TCP connection refers to a long-time communication connection between the service transmission node and the service proxy node.

In one embodiment, various problems always occur due to the continuous writing scenario of high concurrent data volume. For example, downstream disk failures, current limits, glitches, etc. may all cause data write failures. Therefore, in order to prevent data write failure and reduce data transmission delay, an exception failure queue and an exception handling thread are introduced into the architecture of the service transmission node provided in fig. 3. The abnormal failure queue can comprise business data which fails to be written into the target search node by using a write thread; the exception handling thread is used for handling the service data which fails to be written into the target searching node by the write thread. In this case, when sending the second service data set to the target search node fails, that is, when it indicates that the service transfer node fails to write the second service data set to the target search node by using the write thread based on the TCP connection, the service transfer node may add the second service data set to the abnormal failure queue and retry the first layer in real time. In a specific implementation, when sending the second service data set to the target search node fails, the service transmission node may perform service rewriting processing on the second service data set at a first time interval.

Then, the service transmission node can obtain the failure times of sending the second service data set to the target search node in the process of service rewriting processing; and judging whether the failure times exceed a time threshold, if the failure times exceed the time threshold, degrading the second service data set, namely retrying the second-layer queue, and traversing the whole second-layer queue at intervals to sequentially retrying the service data in the second-layer queue. In one embodiment, if the service transmission node determines that the failure times exceed the time threshold, the service transmission node may add a second service data set to the service polling queue, traverse the service data in the service polling queue according to a second time interval, and rewrite the currently traversed service data into the target search node by using the write thread; wherein the second time interval can be set according to requirements. For example, the number threshold may be 3, 5, etc.; the second time interval may be 1 hour, 30 minutes, etc. It should be noted that the obtained failure times are failure times for sending the second service data set to the target search node. It should be understood that the service polling queue belongs to the abnormal failure queue, and the second layer queue is the service polling queue mentioned in the embodiments of the present application. In one embodiment, when the rewriting of the currently traversed service data to the target search node by using the write thread is successful, the service transmission node may delete the currently traversed service data from the service polling queue.

For example, taking the second service data set as a bill data set, the target search node as a bill ES, and the time threshold value as 3; when the bill data set is failed to be written into the bill ES through the write thread based on the TCP connection, the service transmission node performs service rewriting processing on the bill data set in real time; acquiring the failure times of sending the bill data set to the ES in the process of service rewriting processing; if the failure times of sending the bill data to the ES exceed the time threshold value 3, degrading the bill data set and adding the bill data set to a service polling queue; and then traversing the bill data in the service polling queue according to 30 minutes, and rewriting the current traversed bill data into the target search node by using a write thread.

In an embodiment, the service transmission node may determine whether the service data in the service polling queue includes target service data whose survival duration exceeds the preset duration, and delete the target service data from the service polling queue if the service data in the service polling queue includes the target service data whose survival duration exceeds the preset duration. And if the service data in the service polling queue does not comprise the target service data of which the survival time length exceeds the preset time length, continuously traversing the service data in the service polling queue. The survival time of the service data in the service polling queue refers to the time from the time when the service data is added to the service polling queue to the current time; the preset time period may be set according to requirements, for example, the preset time period may be 12 hours, 24 hours, and the like.

In the embodiment of the application, a service transmission node acquires a first service data set stored by a database node through a service intermediate node, and performs data processing on data in the first service data set to obtain a second service data set of the target service; and sending a second service data set to the target searching node, wherein the data in the second service data set belongs to the first service data set. The data of the database node is directly written into the target search node through the service intermediate node and the service transmission node instead of being simultaneously written into the target search node when the data is written into the database node, so that the data writing efficiency can be improved, and the accuracy of data transmission is better ensured.

Based on the foregoing service data processing system, please refer to fig. 4, where fig. 4 is a schematic flowchart of a service data processing method provided in an embodiment of the present application. The service data processing method may be performed by a supplementary service transmission node 112 in the service data processing system, where the supplementary service transmission node 112 may be a DTS device. The structure of the supplementary service transmission node 112 may be as described above in fig. 3. In the embodiment of the present application, the service data processing method includes the following steps S401 to S403:

s401: and acquiring target service supplementary data from the database supplementary node through the service supplementary intermediate node. The target service supplementary data are data which are stored in the database supplementary node and are determined by comparing the difference of the first service data set and the second service data set through the data comparison node. The first service data set is acquired from a database node, and the second service data set is acquired from a target search node.

In an embodiment, a plurality of service supplementary data stored in the database supplementary node are allowed to be recorded in the service supplementary log node, and the specific implementation manner of step S401 is as follows: and performing data analysis on the plurality of service supplementary data stored by the service supplementary log node through the service supplementary intermediate node to obtain a plurality of analyzed service supplementary data. In a specific implementation, the service supplementation intermediate node may perform data analysis on a plurality of service supplementation data stored by the service supplementation log node to obtain a plurality of analyzed service supplementation data, then the service supplementation intermediate node sends the plurality of analyzed service supplementation data to the supplementation service transmission node, and the supplementation service transmission node acquires target service supplementation data from the plurality of analyzed service supplementation data. The plurality of service supplementary data are all difference data between service data in the database node and service data in the target searching node, and the plurality of service supplementary data are all the same as corresponding service data in the database node.

In one embodiment, the parsed plurality of service supplementary data may be deposited in a message queue of a service supplementary message queue node. The service supplement intermediate node sends the analyzed plurality of service supplement data to the service supplement message queue node, and then the supplement service transmission node reads the target service supplement data from the message queue of the service supplement message queue node by using the reading thread.

S402: and carrying out data processing on the target service supplementary data to obtain service updating data. The data processing may be format conversion processing, for example.

S403: and sending the service updating data to the target searching node.

In one embodiment, connection resources for accessing a target search node are acquired from a long connection pool of a Transmission Control Protocol (TCP); the connection resource comprises one of an available port number or an available communication address; then, establishing TCP connection according to connection resources for accessing the target search node; and sending the service updating data to the target searching node by utilizing the writing thread and through the established TCP connection. The specific implementation mode of establishing the TCP connection according to the connection resource used for accessing the target searching node is as follows: and establishing TCP connection with the service supplement proxy node according to the connection resources for accessing the target search node. Wherein, the TCP connection means: and a long-time communication connection is established between the supplementary service transmission node and the service supplementary proxy node.

In one embodiment, the specific implementation manner of the supplementary service transmission node sending the service update data to the target search node by using the write thread and the established TCP connection is as follows: and writing the service updating data into the service supplement proxy node by using the write thread based on the established TCP connection so as to trigger the service supplement proxy node to process the service updating data, and storing the processed service updating data in the target search node.

In one embodiment, when the service updating data is failed to be sent to the target searching node, supplementary service rewriting processing is carried out on the service updating data at a first time interval; acquiring the failure times of writing the service updating data into the target searching node in the process of rewriting the supplementary service; if the failure times exceed the time threshold, adding the service updating data into a supplementary service polling queue; and traversing the service updating data in the supplementary service polling queue according to a second time interval, and rewriting the currently traversed service updating data into the target search node by using the write thread. In one embodiment, when the rewriting of the currently traversed traffic update data to the target search node using the write thread is successful, the currently traversed traffic update data may be deleted from the supplementary traffic polling queue.

In one embodiment, the supplementary service transmission node determines whether the service update data in the supplementary service polling queue includes target service update data with a survival time length exceeding a preset time length. The survival time of the service updating data in the supplementary service polling queue refers to the time from the time when the supplementary service polling queue is added to the time to the current time; and if the supplementary service transmission node determines that the service updating data in the supplementary service polling queue comprises target service updating data with survival time exceeding preset time, deleting the currently traversed target service updating data from the supplementary service polling queue. If the supplementary service transmission node determines that the service update data in the supplementary service polling queue does not include the target service update data with the survival time exceeding the preset time, the service data in the supplementary service polling queue can be continuously traversed.

It should be noted that, for the embodiments related to the present application, all the related contents can be referred to above, and are not described herein again.

In the embodiment of the application, target service supplementary data is acquired from a database supplementary node through a service supplementary intermediate node; performing data processing on the target service supplementary data to obtain service updating data; sending the service updating data to a target searching node; the target service supplementary data are data which are stored in a database supplementary node and are determined by comparing differences of a first service data set and a second service data set through a data comparison node, the first service data set is acquired from the database node, the second service data set is acquired from a target search node, and through data difference comparison, when the data are determined to have differences, the target supplementary data corresponding to the data differences can be acquired from the database supplementary node again and stored in the target search node, so that the data in the database node and the data in the search device can be well consistent and accurate, and the accuracy of data query can be improved.

Referring to fig. 5, fig. 5 is a schematic flowchart of a service data processing method according to an embodiment of the present application. The service data processing method may be executed by the target search node 106 in the service data processing system, where the target search node 106 may be a full text search engine ES device, and the service data processing method in this embodiment of the present application may include the following steps S501 to S503:

s501: and receiving a second service data set obtained by processing the first processing link, wherein the second service data set is data obtained by processing the first service data set, the first service data set is obtained from a database node on the first processing link through a service intermediate node on the first processing link, and the service intermediate node is a node which is configured according to a database protocol of the database node and is used as slave equipment of the database node.

In one embodiment, a plurality of service data stored in the database node are allowed to be recorded in the service log node, and the first service data set is read from a message queue of the service message queue node by using a reading thread by the service transmission node; a plurality of analyzed service data are stored in a message queue of the service message queue node, and the plurality of analyzed service data are obtained by analyzing the plurality of service data stored in the service log node by the service intermediate node; the service log node, the service transmission node and the service message queue node belong to a first processing link.

In one embodiment, the first processing link comprises a TCP connection; the TCP connection is established according to the connection resource of the target search node accessed to the first processing link; the connection resource comprises one of an available port number or an available communication address; the connection resource of the target search node accessed to the first processing link is obtained from a long connection pool of a Transmission Control Protocol (TCP); the second set of traffic data is sent by the traffic transfer node on the first processing link using the write thread and over the TCP connection.

In one embodiment, a service agent node is configured for a target search node; the TCP connection is established with the service agent node according to the connection resource for accessing the target search node; the second set of traffic data is sent by the traffic transport node to the traffic proxy node using a write thread and over a TCP connection.

In one embodiment, the second set of business data may include currently traversed business data; the current traversal service data is obtained by traversing service data in a service polling queue by the service transmission node according to a second time interval, wherein the service polling queue comprises service data of which the failure times of a second service data set exceed a threshold value and which is sent to the target search node in the process of service rewriting processing.

S502: and receiving service updating data processed by the second processing link, wherein the service updating data is obtained by processing target service supplementary data, the target service supplementary data is determined according to difference comparison between a first service data set and a second service data set, the first service data set is obtained from a database node, and the second service data set is obtained from a target search node on the first processing link.

A plurality of service supplementary data stored in the database supplementary node are allowed to be recorded into a service supplementary log node, and target service supplementary data are read from a message queue of a service supplementary message queue node by a supplementary service transmission node by using a reading thread; the message queue of the service supplementary message queue node stores a plurality of analyzed service supplementary data, and the plurality of analyzed service supplementary data are obtained by performing data analysis on the plurality of service supplementary data stored by the service supplementary log node by the service supplementary intermediate node. And the database supplementary node, the service supplementary log node, the supplementary service transmission node and the service message queue node belong to a second processing link.

The second processing link comprises a TCP connection; the TCP connection is established according to the connection resource of the target search node accessed to the second processing link; the connection resource of the target search node accessed to the second processing link is obtained from the TCP long connection pool; the traffic update data is sent by the supplementary traffic transmitting node on the second processing link using the write thread and over the TCP connection.

In one embodiment, a service supplement proxy node is configured for a target search node; the TCP connection is established with the service supplement proxy node according to the connection resource used for accessing the target search node; the second set of traffic data is sent by the supplementary traffic transmitting node to the traffic supplementary proxy node using a write thread and over a TCP connection.

In one embodiment, the service update data comprises currently traversed service update data; the current traversal service updating data is obtained by traversing a supplementary service polling queue by the supplementary service transmission node according to a second time interval, wherein the supplementary service polling queue comprises service updating data of which the failure times of the service updating data are more than a threshold value and which are sent to the target search node in the process of supplementary service rewriting processing.

It should be noted that, for the embodiments related to the above steps S501 to S502, reference may be made to the foregoing related contents, and details are not described herein again.

In the embodiment of the application, a second service data set obtained by processing a first processing link is received, where the second service data set is data obtained by processing a first service data set, and the first service data set is obtained from a database node on the first processing link through a service intermediate node on the first processing link; and receiving service updating data processed by the second processing link, wherein the service updating data is obtained by processing target service supplementary data, the target service supplementary data is determined according to difference comparison between a first service data set and a second service data set, the first service data set is obtained from a database node on the second processing link, and the second service data set is obtained from a target search node on the second processing link. The data in the database node can be directly stored in the target search node through the first processing link without the participation of other links, so that the accuracy of data transmission is improved; through the second processing link, data difference comparison is carried out on the data in the database node and the data in the target searching node, and when the data difference is determined, the service updating data corresponding to the difference data can be stored in the target searching node again, so that the data in the database node and the data in the searching equipment can be enabled to be consistent well, and the accuracy of subsequent data query is facilitated.

The business data processing method provided by the embodiment of the application can be applied to business refund bills, can meet the vertical search requirement of the business refund bills, can solve the problem of long-distance second-level asynchronous data transmission in the intra-domain and out-of-domain scenes from the database node MySQL to the target search node ES, and is favorable for the accuracy of data transmission. Wherein, the out-of-domain means that the user can access the service data processing system by using an external network; the intra-domain means that the service data processing system can be directly accessed without an external network. Wherein, in the service refund bill scene, the service database node is service MySQL; the service log node is a service MySQL binlog; the service intermediate node is a service DBSYNC; the service message queue node is a service Hippo; the service transmission node is a first DTS device and the target searching node is a bill ES device; the service supplement log node supplements the MySQLbrinlog for the service; the service supplement intermediate node supplements DBSYNC for the service; the service supplement message queue node supplements Hippo for the service; the supplementary service transmission node is a second DTS device and the target search node is a bill ES device; the data comparison node is ESCheck, and the database supplement node is complementary MySQL; the business data processing method comprises the following steps:

(1) The business system writes bill data into the business MySQL, the business DBSYNC pretends to be a slave node of the business MySQL master node, and the bill data in the business MySQL Binlog is drawn by analyzing a MySQL Binlog protocol; analyzing the pulled bill data; and pushing the analyzed bill data to the service Hippo.

(2) The first DTS equipment reads bill data (corresponding to the first service data set) in a service Hippo message queue, performs data processing on the read bill data, and then sends the processed bill data (corresponding to the second service data set) to bill ES equipment;

(3) And the ESCheck checks the data in the pull service MySQL with the data of the bill ES device, and if the data of the bill ES device is found to be missing/inconsistent, a complement number is initiated. In the specific implementation, the bill data in the ESCheck pull service MySQL is subjected to data check (data check, namely data difference comparison) with the bill data of the bill ES equipment, so as to determine target service supplementary data;

(4) And a link which is the same as the business MySQL and the bill RS device exists between the supplementary mySQL and the bill ES device, the link is specially used for processing the supplementary data, and finally the bill supplementary data flow is consistent to the bill ES device as the target. In a specific implementation, a slave node of a service supplementary MySQL master node disguised as a complement by DBSYNC draws bill supplementary data in the service supplementary MySQL Binlog by analyzing a MySQL Binlog protocol; analyzing the pulled bill supplementary data; and pushing the parsed bill supplementary data to the service supplementary Hippo.

(5) And the second DTS equipment reads the target bill supplementary data (corresponding to the target service supplementary data) in the message queue of the service supplementary Hippo, performs data processing on the target bill supplementary data, and then sends the processed target bill supplementary data (corresponding to the service updating data) to the bill ES equipment.

The business data processing method realized by various nodes can solve the problem of complex construction of a binlog system, improve the efficiency of data writing and better ensure the accuracy of the data. Each node in the business data processing system generates a simple, efficient and accurate componentization program, and the program is already used in financial payment communication, and the currently used business comprises orders (hundred-library ten-table orders, date orders and historical orders), recharging orders and the like. By the service data processing method, the migration of data volume of 15 hundred million/day level can be realized, the system throughput TPS is close to 2w/s, the payment of a refund bill is supported, and the current query request is switched from a starting database HBase to ES. Future KV merchant bill new services are also being staged for access on the order of approximately 30 billion/day.

An embodiment of the present application provides a service data processing scheme, which may be as shown in fig. 6a, where the service data processing scheme mainly uses asynchronous double writing to perform service data processing, and introduces an MQ (Message Queue), and the general principle of the scheme is as follows: two links implement an asynchronous double write process, one is to write the service data into a database node (such as the service relational database in fig. 6 a), and the other is to store the service data in an MQ (Message Queue) first, and then store the service data in a target search node (such as the full-text search engine in fig. 6 a) by performing consumption processing on the Message Queue MQ.

In an embodiment, the present application further provides a service data processing scheme, which may be as shown in fig. 6b, where the service data processing scheme mainly uses asynchronous double write-worker to perform service data processing; the general principle is as follows: adding a field as a timestamp field in a correlation table of a database node, wherein any add, delete, update and other cropping operations (a delete, add, modify and check operation) can cause the time of the field to change, while the cropping operations in the original program do not change; adding a timer program, scanning the appointed correlation table by the timer program according to a certain time period, and then extracting the data changed in the time period; and writes the changed data into the ES one by one.

In an embodiment, the present application further provides a service data processing scheme, where the data processing scheme may be as shown in fig. 6c, and the general principle of the service data processing scheme is as follows: and the service data processing is realized by using a service log node (such as a binary log binlog of the mysql) in the database node. The method comprises the following specific steps: 1) Reading a binlog log of the mysql, and acquiring log information of an appointed table; 2) Converting the read service data into a message queue MQ; 3) Writing a message queue MQ consumption program; 4) The MQ is read continuously, and each time a message is read, the message is written into the ES.

Referring to fig. 7 again, fig. 7 is a schematic structural diagram of a service data processing apparatus according to an embodiment of the present application, where the service data processing apparatus according to the embodiment of the present application may be disposed on an intelligent device, and the intelligent device may be the service transmission node in fig. 1 or fig. 2. The business data processing device comprises the following units:

a service processing unit 701, configured to obtain, by a service intermediate node, a first service data set stored by a database node, where the first service data set is data, stored by the database node based on a data processing request, for a target service, and the service intermediate node is a node configured according to a database protocol of the database node and serving as a slave device of the database node;

the service processing unit 701 is further configured to perform data processing on data in the first service data set to obtain a second service data set of the target service;

a communication unit 702, configured to send the second service data set to the target search node, where data in the second service data set belongs to the first service data set.

In an embodiment, the plurality of service data stored in the database node are allowed to be recorded in a service log node, and the service processing unit 701, when obtaining the first service data set from the database node through a service intermediate node, may specifically be configured to:

In an embodiment, the parsed service data sets are stored in a message queue of a service message queue node, and the first service data set is read from the message queue of the service message queue node by using a read thread.

In one embodiment, the service processing unit 701 is configured to obtain a connection resource for accessing the target search node from a TCP long connection pool; the connection resource comprises one or two of an available port number or an available communication address; establishing a TCP connection according to connection resources for accessing the target search node;

the communication unit 702 is configured to send the second service data set to the target search node by using a write thread and through the established TCP connection.

In one embodiment, a service agent node is configured for the target search node; the service processing unit 701 establishes a TCP connection according to a connection resource used for accessing the target search node, and may specifically be configured to: establishing TCP connection with the service agent node according to the connection resource for accessing the target search node;

the service processing unit 701 is configured to write the second service data set into the service proxy node by using a write thread based on the established TCP connection, so as to trigger the service proxy node to process the second service data set, and store the processed second service data set in the target search node.

In an embodiment, the service processing unit 701 is further configured to:

when the second service data set fails to be sent to the target search node, performing service rewriting processing on the second service data set at a first time interval;

acquiring the failure times of sending the second service data set to the target searching node in the service rewriting process;

the second time interval is greater than the first time interval.

In an embodiment, the service processing unit 701 is further configured to:

and if the service data in the service polling queue comprises target service data with survival time exceeding preset time, deleting the target service data from the service polling queue.

It can be understood that the functions of each functional unit of the service data processing apparatus in this embodiment may be specifically implemented according to the method in the foregoing method embodiment, and the specific implementation process may refer to the description related to fig. 1 or fig. 2 in the foregoing method embodiment, which is not described herein again.

Referring to fig. 8 again, fig. 8 is a schematic structural diagram of a service data processing apparatus according to an embodiment of the present application, where the service data processing apparatus according to the embodiment of the present application may be disposed on an intelligent device, and the intelligent device may be the supplementary service transmission node in fig. 1 or fig. 4. The service data processing device comprises the following units:

a service supplement processing unit 801, configured to obtain target service supplement data from a database supplement node through a service supplement intermediate node;

the service supplementary processing unit 801 is further configured to perform data processing on the target service supplementary data to obtain service update data;

a service supplementary communication unit 802, configured to send the service update data to a target search node;

sending the service updating data to a target searching node;

In one embodiment, a plurality of service supplementary data stored in the database supplementary node are allowed to be recorded in a service supplementary log node, and the service supplementary processing unit 801 is configured to, after obtaining target service supplementary data from the database supplementary node through a service supplementary intermediate node, specifically:

In an embodiment, when sending the service update data to the target search node, the service supplementary processing unit 801 is configured to obtain a connection resource for accessing the target search node from a TCP long connection pool; the connection resources include: one or both of available port numbers or available communication addresses; establishing a TCP connection according to connection resources for accessing the target search node;

the service supplementary communication unit 801 is configured to send the service update data to the target search node through the established TCP connection by using a write thread.

In one embodiment, a service supplementary proxy node is configured for the target search node; the service supplementary processing unit 801, when establishing a TCP connection according to a connection resource used for accessing the target search node, is specifically configured to: establishing TCP connection with the service supplement proxy node according to connection resources for accessing the target search node;

the service supplementary processing unit 801, using a write thread and through the established TCP connection, sends the service update data to the target search node, which may be specifically configured to:

In an embodiment, the service supplementary processing unit 801 is further configured to:

the second time interval is greater than the first time interval.

It can be understood that the functions of each functional unit of the service data processing apparatus in this embodiment may be specifically implemented according to the method in the foregoing method embodiment, and the specific implementation process may refer to the description related to fig. 1 or fig. 4 in the foregoing method embodiment, which is not described herein again.

In the embodiment of the application, target service supplementary data is acquired from a database supplementary node through a service supplementary intermediate node; performing data processing on the target service supplementary data to obtain service updating data; sending the service updating data to a target searching node; the target service supplementary data are data which are stored in a database supplementary node and are determined by comparing differences of a first service data set and a second service data set through a data comparison node, the first service data set is acquired from the database node, the second service data set is acquired from a target search node, and when the data are determined to be different, the target supplementary data corresponding to the data differences can be acquired from the database supplementary node again through data difference comparison and are stored in the target search node, so that the data in the database node and the data in the search device can be well consistent and accurate, and the accuracy of data query is improved.

Referring to fig. 9 again, fig. 9 is a schematic structural diagram of a service data processing apparatus according to an embodiment of the present application, where the service data processing apparatus according to the embodiment of the present application may be disposed on an intelligent device, and the intelligent device may be the target search device in fig. 1 or fig. 5. The business data processing device comprises the following units:

a communication unit 901 for transmitting and receiving data;

a processing unit 902, configured to receive, by the communication unit, a second service data set obtained by processing a first service data set, where the second service data set is data obtained by processing the first service data set, the first service data set is obtained from a database node on the first processing link through a service intermediate node on the first processing link, and the service intermediate node is a node configured according to a database protocol of the database node and serving as a slave device of the database node;

the processing unit 902 is further configured to receive, through the communication unit, service update data obtained by processing a second processing link, where the service update data is obtained by processing target service supplementary data, the target service supplementary data is data determined according to difference comparison between the first service data set and the second service data set, the first service data set is obtained from the database node, and the second service data set is obtained from a target search node on the first processing link.

In one embodiment, a service agent node is configured for the target search node; the TCP connection is established with the service agent node according to the connection resource used for accessing the target search node; the second service data set is sent by the service transmission node to the service proxy node by using a write thread and through the TCP connection.

In one embodiment, a service supplementary proxy node is configured for the target search node; the TCP connection is established with the service supplement proxy node according to the connection resource used for accessing the target search node; and the service updating data is sent to the service supplementary proxy node by the supplementary service transmission node through the TCP connection by using a write thread.

It can be understood that the functions of each functional unit of the service data processing apparatus in this embodiment may be specifically implemented according to the method in the foregoing method embodiment, and the specific implementation process may refer to the description related to fig. 1 or fig. 5 in the foregoing method embodiment, which is not described herein again.

In the embodiment of the application, a second service data set obtained by processing a first processing link is received, where the second service data set is data obtained by processing a first service data set, and the first service data set is obtained from a database node on the first processing link through a service intermediate node on the first processing link; receiving service updating data processed by the second processing link, wherein the service updating data is obtained by processing target service supplementary data, the target service supplementary data is determined according to difference comparison between a first service data set and a second service data set, the first service data set is obtained from a database node on the second processing link, and the second service data set is obtained from a target search node on the second processing link; the data in the database node can be directly stored in the target searching node through the first processing link without the participation of other links, so that the accuracy of data transmission is improved; through the second processing link, data difference comparison is carried out on the data in the database node and the data in the target searching node, and when the data difference is determined, the service updating data corresponding to the difference data can be stored in the target searching node again, so that the data in the database node and the data in the searching equipment can be well consistent, and the accuracy of subsequent data query is facilitated.

Further, please refer to fig. 10, where fig. 10 is a schematic structural diagram of an intelligent device according to an embodiment of the present application. The intelligent device may be the service transmission node in fig. 1 or fig. 2, and the intelligent device may include: a processor 1001, an input device 1002, an output device 1003 and a memory 1004. The processor 1001, the input device 1002, the output device 1003, and the memory 1004 are connected by a bus 1005. The memory 1004 is used to store a computer program comprising program instructions, and the processor 1001 is used to execute the program instructions stored by the memory 1004.

In the embodiment of the present application, the processor 1001 executes the executable program code in the memory 1004 to perform the following operations:

performing data processing on the data in the first service data set to obtain a second service data set of the target service;

In an embodiment, the plurality of service data stored in the database node are allowed to be recorded in a service log node, and the processor 1001, in obtaining the first service data set from the database node through a service intermediate node, may specifically be configured to:

In an embodiment, the processor 1001, when sending the second service data set to the target search node, may specifically be configured to:

In one embodiment, a service agent node is configured for the target search node; the processor 1001, according to the connection resource for accessing the target search node, establishes a TCP connection, specifically configured to:

the processor 1001, when sending the second service data set to the target search node through the established TCP connection by using a write thread, is specifically configured to:

and writing the second service data set into the service proxy node by using a write thread based on the established TCP connection so as to trigger the service proxy node to process the second service data set, and storing the processed second service data set in the target search node.

In one embodiment, the processor 1001 is further configured to:

the second time interval is greater than the first time interval.

In one embodiment, the processor 1001 is further configured to:

It should be understood that, in the embodiment of the present Application, the Processor 1001 may be a Central Processing Unit (CPU), and the Processor 1001 may also be other general-purpose processors, digital Signal Processors (DSPs), application Specific Integrated Circuits (ASICs), field-Programmable Gate arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components, and the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

The memory 1004 may include a read-only memory and a random access memory, and provides instructions and data to the processor 1001. A portion of the memory 1004 may also include non-volatile random access memory.

The input device 1002 may include a keyboard or the like, and inputs service data to the processor 1001; the output device 1003 may include a display or the like.

In specific implementation, the processor 1001, the input device 1002, the output device 1003, and the memory 1004 described in this embodiment may execute the implementation described in fig. 1 and fig. 2 of the above embodiment, and may also execute the implementation described in the above apparatus, which is not described herein again.

Further, an embodiment of the present application also provides a schematic structural diagram of an intelligent device, and the schematic structural diagram of the intelligent device may be seen in fig. 10. The intelligent device may be the supplementary service transmission node in fig. 1 or fig. 4, and the intelligent device may include: a processor 1001, an input device 1002, an output device 1003 and a memory 1004. The processor 1001, the input device 1002, the output device 1003, and the memory 1004 are connected via a bus 1005. The memory 1004 is used to store a computer program comprising program instructions, and the processor 1001 is used to execute the program instructions stored by the memory 1004.

In this embodiment, the processor 1001 executes the executable program code in the memory 1004 to perform the following operations:

sending the service updating data to a target searching node;

In an embodiment, the plurality of service supplementary data stored in the database supplementary node are allowed to be recorded in a service supplementary log node, and the processor 1001 is specifically configured to, when obtaining target service supplementary data from the database supplementary node through the service supplementary intermediate node:

In an embodiment, when the processor 1001 sends the service update data to the target search node, it is specifically configured to:

In one embodiment, a service supplementary proxy node is configured for the target search node; when the processor 1001 establishes a TCP connection according to a connection resource used for accessing the target search node, the processor is specifically configured to:

when the processor 1001 sends the service update data to the target search node through the established TCP connection by using a write thread, it may specifically be configured to:

In one embodiment, the processor 1001 is further configured to:

the second time interval is greater than the first time interval.

In one embodiment, the processor 1001 is further configured to:

In a specific implementation, the processor 1001, the input device 1002, the output device 1003, and the memory 1004 described in this embodiment of the present application may perform the implementation described in fig. 1 and fig. 4 in the above embodiment, and may also perform the implementation described in the above apparatus, which is not described herein again.

Further, an embodiment of the present application also provides a schematic structural diagram of an intelligent device, and the schematic structural diagram of the intelligent device may be referred to in fig. 10. The intelligent device may be the target search node in fig. 1 or fig. 5, and the intelligent device may include: a processor 1001, an input device 1002, an output device 1003 and a memory 1004. The processor 1001, the input device 1002, the output device 1003, and the memory 1004 are connected via a bus 1005. The memory 1004 is used to store a computer program comprising program instructions, and the processor 1001 is used to execute the program instructions stored by the memory 1004.

a plurality of service supplementary data stored in a database supplementary node are allowed to be recorded into a service supplementary log node, and the target service supplementary data are read from a message queue of a service supplementary message queue node by a supplementary service transmission node by using a reading thread; a plurality of analyzed service supplementary data are stored in a message queue of the service supplementary message queue node, and the plurality of analyzed service supplementary data are obtained by performing data analysis on the plurality of service supplementary data stored in the service supplementary log node by a service supplementary intermediate node; the database supplement node, the service supplement log node, the supplementary service transmission node, the service message queue node and the service supplement intermediate node belong to the second processing link.

In one embodiment, a service supplementary proxy node is configured for the target search node; the TCP connection is established with the service supplement proxy node according to the connection resource used for accessing the target search node; and the service updating data is sent to the service supplementary agent node by the supplementary service transmission node through the TCP connection by using a write thread.

The memory 1004 may include both read-only memory and random access memory and provides instructions and data to the processor 1001. A portion of the memory 1004 may also include non-volatile random access memory.

In specific implementation, the processor 1001, the input device 1002, the output device 1003, and the memory 1004 described in this embodiment may execute the implementation described in fig. 1 and fig. 5 in the above embodiment, and may also execute the implementation described in the above apparatus, which is not described herein again.

In the embodiment of the application, a second service data set obtained by processing a first processing link is received, where the second service data set is data obtained by processing a first service data set, and the first service data set is obtained from a database node on the first processing link through a service intermediate node on the first processing link; and receiving service updating data processed by the second processing link, wherein the service updating data is obtained by processing target service supplementary data, the target service supplementary data is determined according to difference comparison between a first service data set and a second service data set, the first service data set is obtained from a database node on the second processing link, and the second service data set is obtained from a target search node on the second processing link. The data in the database node can be directly stored in the target search node through the first processing link without the participation of other links, so that the accuracy of data transmission is improved; through the second processing link, data difference comparison is carried out on the data in the database node and the data in the target searching node, and when the data difference is determined, the service updating data corresponding to the difference data can be stored in the target searching node again, so that the data in the database node and the data in the searching equipment can be well consistent, and the accuracy of subsequent data query is facilitated.

The embodiment of the present application further provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and a processor runs the computer program, so that the intelligent device executes the method provided in the foregoing embodiment.

Embodiments of the present application also provide a computer program product or computer program comprising computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the intelligence performs the method provided by the foregoing embodiments.

It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. The storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), or the like.

While the invention has been described with reference to a preferred embodiment, it will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the invention.

Claims

1. A method for processing service data is characterized by comprising the following steps:

2. The method of claim 1, wherein the plurality of service data stored in the database node are allowed to be recorded in a service log node, and the obtaining the first set of service data from the database node by the service intermediate node comprises:

3. The method of claim 1, wherein said sending the second set of traffic data to the target search node comprises:

4. The method of any one of claims 1-3, further comprising:

the second time interval is greater than the first time interval.

5. A method for processing service data is characterized by comprising the following steps:

sending the service updating data to a target searching node;

6. The method as claimed in claim 5, wherein the plurality of service supplementary data stored in the database supplementary node are allowed to be recorded in a service supplementary log node, and the obtaining of the target service supplementary data from the database supplementary node by the service supplementary intermediate node comprises:

7. The method of claim 5, wherein said sending said traffic update data to a target search node comprises:

8. The method of any one of claims 5-7, further comprising:

traversing the service updating data in the supplementary service polling queue according to a second time interval, and utilizing a write thread to rewrite the traversed service updating data in the target searching node;

the second time interval is greater than the first time interval.

9. A method for processing service data is characterized by comprising the following steps:

10. The method of claim 9, wherein a plurality of service data stored in the database node are allowed to be recorded in a service log node, and the first service data set is read from a message queue of a service message queue node by a service transmission node using a read thread; a plurality of analyzed service data are stored in a message queue of the service message queue node, and the plurality of analyzed service data are obtained by performing data analysis on the plurality of service data stored in the service log node by the service intermediate node; the service log node, the service transmission node and the service message queue node belong to the first processing link;

11. The method of claim 9, wherein the first processing link comprises a TCP connection; the TCP connection is established according to connection resources of a target search node accessed to the first processing link; the connection resource comprises one of an available port number or an available communication address; the connection resource of the target search node accessed to the first processing link is obtained from a Transmission Control Protocol (TCP) long connection pool; the second set of traffic data is sent by a traffic transport node on the first processing link using a write thread and over the TCP connection.

12. The method of claim 11, wherein a service proxy node is configured for the target search node; the TCP connection is established with the service agent node according to the connection resource for accessing the target search node; the second service data set is sent by the service transmission node to the service proxy node by using a write thread and through the TCP connection.

13. The method of any of claims 9-12, wherein the second processing link comprises a TCP connection; the TCP connection is established according to the connection resource accessed to the target searching node; the connection resource accessed to the target search node is obtained from a TCP long connection pool; the traffic update data is sent by a supplementary traffic transfer node on the second processing link using a write thread and over the TCP connection.

14. A smart device, comprising: a storage device and a processor;

the storage device is used for storing a computer program;

the processor for executing the computer program, for implementing the method of any one of claims 1-4, or for implementing the method of any one of claims 5-8, or for implementing the method of any one of claims 9-13.

15. A computer-readable storage medium, in which a computer program is stored which, when executed, implements the method of any one of claims 1-4, or implements the method of any one of claims 5-8, or implements the method of any one of claims 9-13.