WO2017181890A1 - Data processing system and method based on server cluster - Google Patents

Data processing system and method based on server cluster Download PDF

Info

Publication number
WO2017181890A1
WO2017181890A1 PCT/CN2017/080200 CN2017080200W WO2017181890A1 WO 2017181890 A1 WO2017181890 A1 WO 2017181890A1 CN 2017080200 W CN2017080200 W CN 2017080200W WO 2017181890 A1 WO2017181890 A1 WO 2017181890A1
Authority
WO
WIPO (PCT)
Prior art keywords
data processing
server
response
address translation
request
Prior art date
Application number
PCT/CN2017/080200
Other languages
French (fr)
Chinese (zh)
Inventor
杨森
陈林
徐立伟
张飞
许斌
Original Assignee
中国银联股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 中国银联股份有限公司 filed Critical 中国银联股份有限公司
Publication of WO2017181890A1 publication Critical patent/WO2017181890A1/en

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/40—Support for services or applications
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/104—Peer-to-peer [P2P] networks
    • H04L67/1044—Group management mechanisms 
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/50—Network services
    • H04L67/60—Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources

Definitions

  • the present invention relates to data processing systems and methods, and more particularly to server cluster based data processing systems and methods.
  • load balancing of multiple data processing servers in the server cluster of the back end is usually implemented in the front end by using an application layer-based load dispatcher, that is, the load dispatcher will wait
  • the processed data processing requests are distributed to different data processing servers to maintain load balancing between the data processing servers.
  • the present invention proposes a server cluster-based data processing system and method with high data processing performance (ie, data processing speed and efficiency) and good load balancing capability.
  • a server cluster-based data processing system includes:
  • An address translation server configured to receive a data processing request from at least one client and forward it to a dispatch server, and receive a data processing response from any of the at least two data processing servers, and Send it back to the corresponding client;
  • a scheduling server configured to receive a data processing request forwarded by the address translation server and then distribute it to one of the at least two data processing servers according to a predetermined scheduling algorithm to ensure load balancing ;
  • At least two data processing servers each of the at least two data processing servers configured to process the received data processing request and construct a data processing response based on the processing result, and subsequently the data processing response Transfer to the address translation server.
  • the system further includes a task status storage device configured to store a data processing task status table that records data processing indicated by each data processing request Task and its processing status.
  • each of the at least two data processing servers writes a data processing task indicated by the data processing task to one of the data processing task status tables after receiving the data processing request In the new record, set the processing status of the data processing task to "Pending.”
  • each of the at least two data processing servers accesses the data processing task status table when having idle resources to acquire one or more processing states as "waiting" according to a predetermined rule Processes the data processing task, and then processes the data processing task, sets its processing state to "Processing", and sets its processing state to "Processed” after processing is complete.
  • the task status storage device periodically polls the records in the data processing task status table and sets the task status to "in process” and duration in the state The status of the data processing task that exceeds the predetermined threshold is reset to "Pending.”
  • the address translation server generates a globally unique identifier according to a predetermined rule after receiving the data processing request, and associates the identifier with the data processing request, and then includes A data processing request for the associated identifier is forwarded to the dispatch server and a mapping relationship between the identifier and the source address of the data processing request is saved.
  • the address translation server processes the identifier included in the response according to the data and saves The mapping relationship between the identifier and the source address of the data processing request determines the destination address of the data processing response, and then transmits it back to the client pointed to by the destination address end.
  • a data processing method based on server clusters includes the following steps:
  • the address translation server receives the data processing request from the at least one client and forwards it to the dispatch server;
  • the scheduling server distributes the data processing request forwarded by the address translation server to one of the at least two data processing servers according to a predetermined scheduling algorithm to ensure load balancing;
  • each of the at least two data processing servers processing the received data processing request, and constructing a data processing response based on the processing result, and subsequently transmitting the data processing response to the address translation server ;
  • the address translation server transmits the data processing response back to the corresponding client after receiving the data processing response.
  • the server cluster-based data processing system and method disclosed by the present invention has the following advantages: high data processing performance (ie, high data processing speed and efficiency) and good load balancing capability.
  • FIG. 1 is a schematic structural diagram of a server cluster-based data processing system according to an embodiment of the present invention
  • FIG. 2 is a flow chart of a server cluster based data processing method in accordance with an embodiment of the present invention.
  • the server cluster-based data processing system disclosed by the present invention includes an address translation server 1, a scheduling server 2, and at least two data processing servers 3.
  • the address translation server 1 is configured to receive a data processing request from at least one client and forward it to The server 2 is scheduled, and a data processing response from any of the at least two data processing servers 3 is received and subsequently transmitted back to the corresponding client.
  • the scheduling server 2 is configured to receive a data processing request forwarded by the address translation server 1 and then distribute it to one of the at least two data processing servers 3 according to a predetermined scheduling algorithm to ensure load balancing .
  • Each of the at least two data processing servers 3 is configured to process the received data processing request and construct a data processing response based on the processing result, and subsequently transmit the data processing response to the address translation Server 1.
  • the server cluster-based data processing system disclosed herein further includes a task state storage device 4 configured to store a data processing task status table that records each data processing request indication Data processing tasks and their processing status.
  • each of the at least two data processing servers 3 writes the data processing task indicated by the at least two data processing servers after the data processing request is received.
  • the data is processed in a new record in the task status table, and the processing status of the data processing task is set to "pending".
  • each of the at least two data processing servers 3 accesses the data processing task status table when having idle resources to acquire one according to a predetermined rule. Or multiple data processing tasks whose processing status is "Pending”, and then process the data processing task, set its processing status to "Processing”, and set its processing status to "Processed” after processing is completed. ".
  • the task state storage device 4 periodically polls the records in the data processing task status table and sets the task status to "in process”. And the state of the data processing task in which the duration exceeds the predetermined threshold in this state is reset to "pending". From the above, when a certain or some data processing server fails, it can ensure that the processing of the data processing task is not terminated unexpectedly.
  • the address translation server 1 generates a globally unique identifier according to a predetermined rule after receiving the data processing request, and associates the identifier with the data processing. Requesting, then forwarding a data processing request containing the associated identifier to the dispatch server 2 and saving the identifier with the source address of the data processing request (eg, the IP address of the client sending the data processing request) Mapping relations.
  • the source address of the data processing request eg, the IP address of the client sending the data processing request
  • the address translation server 1 processes the data according to the data.
  • the identifier included in the response and the mapping relationship between the saved identifier and the source address of the data processing request determine the destination address of the data processing response (ie, the source address of the data processing request corresponding to the response), and then Send back to the client pointed to by the destination address.
  • the server cluster-based data processing system disclosed by the present invention has the following advantages: high data processing performance (ie, high data processing speed and efficiency) and good load balancing capability.
  • the server cluster-based data processing method disclosed by the present invention includes the following steps: (A1) an address translation server receives a data processing request from at least one client and forwards it to a scheduling server; (A2) After receiving the data processing request forwarded by the address translation server, the scheduling server distributes it to one of the at least two data processing servers according to a predetermined scheduling algorithm to ensure load balancing; (A3) the at least two data processing Each of the servers processes the received data processing request and constructs a data processing response based on the processing result, and subsequently transmits the data processing response to the address translation server; (A4) the address translation server is After receiving the data processing response, it is transmitted back to the corresponding client.
  • the step (A3) further includes: each of the at least two data processing servers instructing the data processing request after receiving the data processing request The data processing task is written into a new record in the data processing task status table, and the processing status of the data processing task is set to "pending", wherein the data processing task status table records each data processing request The indicated data processing tasks and their processing status.
  • the step (A3) further comprises: each of the at least two data processing servers accessing the data processing task when having idle resources a status table to obtain one or more data processing tasks whose processing status is "pending" according to a predetermined rule, and then process the data processing task, and set its processing status to "in process", and after the processing is completed Its processing status is set to "Processed”.
  • the steps (A3) further comprising: a state of the data processing task in which the record in the data processing task status table is periodically polled and the task status is "in process” and the time duration in the state exceeds a predetermined threshold Reset to "Pending.”
  • the step (A1) further includes: the address translation server generates a globally unique identifier according to a predetermined rule after receiving the data processing request, and Associating the identifier with the data processing request, and subsequently forwarding a data processing request including the associated identifier to the dispatch server, and saving the identifier with a source address of the data processing request (eg, transmitting the data)
  • the mapping relationship of the IP address of the client that handles the request is not limited to the address of the client.
  • the step (A4) further comprises: after receiving a data processing response from any one of the at least two data processing servers, Determining, by the address translation server, the destination address of the data processing response according to the identifier included in the data processing response and the mapping relationship between the saved identifier and the source address of the data processing request (ie, the source of the data processing request corresponding to the response) Address) and then transfer it back to the client pointed to by the destination address.
  • the server cluster-based data processing method disclosed by the present invention has the following advantages: high data processing performance (ie, high data processing speed and efficiency) and good load balancing capability.

Abstract

The present invention provides a data processing method and system based on a server cluster. The method comprises: an address translation server receives a data processing request from at least one client, and forwards the data processing request to a scheduling server; after the scheduling server receives the data processing request forwarded by the address translation server, the scheduling server distributes the data processing request to one of at least two data processing servers according to a preset scheduling algorithm, so as to ensure load balancing; each of the at least two data processing servers processes the received data processing request, construct a data processing response according to the processing result, and consequently transmit the data processing response to the address translation server; and after the address translation server receives the data processing response, the address translation server transmits the data processing response back to the corresponding client. The data processing method and system disclosed in the present invention have high data processing performance and a good load balancing capability.

Description

基于服务器集群的数据处理系统及方法Data processing system and method based on server cluster 技术领域Technical field
本发明涉及数据处理系统及方法，更具体地，涉及基于服务器集群的数据处理系统及方法。The present invention relates to data processing systems and methods, and more particularly to server cluster based data processing systems and methods.
背景技术Background technique
目前，随着计算机和网络应用的日益广泛以及不同领域的业务种类的日益丰富，针对来自用户的海量业务处理请求的数据处理系统变得越来越重要。At present, with the increasing popularity of computers and network applications and the growing variety of services in different fields, data processing systems for massive service processing requests from users are becoming more and more important.
在现有的基于服务器集群的数据处理系统中，通常在前端采用基于应用层的负载调度机的方式实现针对后端的服务器集群中的多个数据处理服务器的负载均衡，即由负载调度机将待处理的数据处理请求分发给不同的数据处理服务器，以保持所述数据处理服务器之间的负载均衡。In the existing data processing system based on the server cluster, load balancing of multiple data processing servers in the server cluster of the back end is usually implemented in the front end by using an application layer-based load dispatcher, that is, the load dispatcher will wait The processed data processing requests are distributed to different data processing servers to maintain load balancing between the data processing servers.
然而，上述现有的技术方案存在如下问题：由于通常请求报文较短而应答报文包含大量的数据，故由于请求报文和应答报文均需要负载调度机进行调度，当待处理的数据处理请求较多时，负载调度机将成为整个系统的性能瓶颈。However, the above existing technical solutions have the following problems: since the request message is short and the response message contains a large amount of data, both the request message and the response message require the load scheduler to perform scheduling, and the data to be processed When there are more processing requests, the load scheduler will become the performance bottleneck of the entire system.
因此，存在如下需求：提供具有高的数据处理性能(即数据处理速度和效率)以及良好的负载均衡能力的基于服务器集群的数据处理系统及方法。Therefore, there is a need to provide a server cluster-based data processing system and method with high data processing performance (ie, data processing speed and efficiency) and good load balancing capabilities.
发明内容Summary of the invention
为了解决上述现有技术方案所存在的问题，本发明提出了具有高的数据处理性能(即数据处理速度和效率)以及良好的负载均衡能力的基于服务器集群的数据处理系统及方法。In order to solve the problems of the above prior art solutions, the present invention proposes a server cluster-based data processing system and method with high data processing performance (ie, data processing speed and efficiency) and good load balancing capability.
本发明的目的是通过以下技术方案实现的：The object of the invention is achieved by the following technical solutions:
一种基于服务器集群的数据处理系统，所述基于服务器集群的数据处理系统包括：A server cluster-based data processing system, the server cluster-based data processing system includes:
地址转换服务器，所述地址转换服务器被配置为接收来自至少一个客户端的数据处理请求，并将其转发至调度服务器，以及接收来自至少两个数据处理服务器中的任一个的数据处理应答，并随之将其传送回对应的客户端； An address translation server configured to receive a data processing request from at least one client and forward it to a dispatch server, and receive a data processing response from any of the at least two data processing servers, and Send it back to the corresponding client;
调度服务器，所述调度服务器被配置为接收经所述地址转换服务器转发的数据处理请求，并随之根据预定的调度算法将其分发至所述至少两个数据处理服务器中的一个以确保负载均衡；a scheduling server configured to receive a data processing request forwarded by the address translation server and then distribute it to one of the at least two data processing servers according to a predetermined scheduling algorithm to ensure load balancing ;
至少两个数据处理服务器，所述至少两个数据处理服务器中的每个被配置为处理接收到的所述数据处理请求，并基于处理结果构建数据处理应答，以及随之将所述数据处理应答传送至所述地址转换服务器。At least two data processing servers, each of the at least two data processing servers configured to process the received data processing request and construct a data processing response based on the processing result, and subsequently the data processing response Transfer to the address translation server.
在上面所公开的方案中，优选地，所述系统进一步包括任务状态存储装置，其被配置为存储数据处理任务状态表，所述数据处理任务状态表记录每个数据处理请求所指示的数据处理任务及其处理状态。In the solution disclosed above, preferably, the system further includes a task status storage device configured to store a data processing task status table that records data processing indicated by each data processing request Task and its processing status.
在上面所公开的方案中，优选地，所述至少两个数据处理服务器中的每一个在接收到数据处理请求后将其所指示的数据处理任务写入所述数据处理任务状态表中的一条新的记录中，并将该数据处理任务的处理状态设置为“待处理”。In the solution disclosed above, preferably, each of the at least two data processing servers writes a data processing task indicated by the data processing task to one of the data processing task status tables after receiving the data processing request In the new record, set the processing status of the data processing task to "Pending."
在上面所公开的方案中，优选地，所述至少两个数据处理服务器中的每一个在具有空闲资源时访问所述数据处理任务状态表以便按预定规则获取一个或多个处理状态为“待处理”的数据处理任务，并随之处理该数据处理任务，并将其处理状态设置为“处理中”，以及在处理完成后将其处理状态设置为“已处理”。In the solution disclosed above, preferably, each of the at least two data processing servers accesses the data processing task status table when having idle resources to acquire one or more processing states as "waiting" according to a predetermined rule Processes the data processing task, and then processes the data processing task, sets its processing state to "Processing", and sets its processing state to "Processed" after processing is complete.
在上面所公开的方案中，优选地，所述任务状态存储装置周期性地轮询所述数据处理任务状态表中的记录，并将任务状态为“处理中”且在该状态下持续的时间超过预定的阈值的数据处理任务的状态重新设置为“待处理”。In the solution disclosed above, preferably, the task status storage device periodically polls the records in the data processing task status table and sets the task status to "in process" and duration in the state The status of the data processing task that exceeds the predetermined threshold is reset to "Pending."
在上面所公开的方案中，优选地，所述地址转换服务器在接收到数据处理请求后按预定规则生成全局唯一的标识符，并将所述标识符关联于该数据处理请求，随之将包含关联的标识符的数据处理请求转发至所述调度服务器，并保存该标识符与该数据处理请求的来源地址的映射关系。In the solution disclosed above, preferably, the address translation server generates a globally unique identifier according to a predetermined rule after receiving the data processing request, and associates the identifier with the data processing request, and then includes A data processing request for the associated identifier is forwarded to the dispatch server and a mapping relationship between the identifier and the source address of the data processing request is saved.
在上面所公开的方案中，优选地，在接收来自所述至少两个数据处理服务器中的任一个的数据处理应答后，所述地址转换服务器根据该数据处理应答中包含的标识符以及所保存标识符与数据处理请求的来源地址之间的映射关系确定该数据处理应答的目的地址，并随之将其传送回该目的地址所指向的客户 端。In the solution disclosed above, preferably, after receiving the data processing response from any one of the at least two data processing servers, the address translation server processes the identifier included in the response according to the data and saves The mapping relationship between the identifier and the source address of the data processing request determines the destination address of the data processing response, and then transmits it back to the client pointed to by the destination address end.
本发明的目的也可以通过以下技术方案实现：The object of the invention can also be achieved by the following technical solutions:
一种基于服务器集群的数据处理方法。所述方法包括下列步骤：A data processing method based on server clusters. The method includes the following steps:
(A1)地址转换服务器接收来自至少一个客户端的数据处理请求，并将其转发至调度服务器；(A1) the address translation server receives the data processing request from the at least one client and forwards it to the dispatch server;
(A2)所述调度服务器在接收到所述地址转换服务器转发的数据处理请求后根据预定的调度算法将其分发至至少两个数据处理服务器中的一个以确保负载均衡；(A2) the scheduling server distributes the data processing request forwarded by the address translation server to one of the at least two data processing servers according to a predetermined scheduling algorithm to ensure load balancing;
(A3)所述至少两个数据处理服务器中的每个处理接收到的所述数据处理请求，并基于处理结果构建数据处理应答，以及随之将所述数据处理应答传送至所述地址转换服务器；(A3) each of the at least two data processing servers processing the received data processing request, and constructing a data processing response based on the processing result, and subsequently transmitting the data processing response to the address translation server ;
(A4)所述地址转换服务器在接收到所述数据处理应答后将其传送回对应的客户端。(A4) The address translation server transmits the data processing response back to the corresponding client after receiving the data processing response.
本发明所公开的基于服务器集群的数据处理系统及方法具有下列优点：具有高的数据处理性能(即较高的数据处理速度和效率)以及良好的负载均衡能力。The server cluster-based data processing system and method disclosed by the present invention has the following advantages: high data processing performance (ie, high data processing speed and efficiency) and good load balancing capability.
附图说明DRAWINGS
结合附图，本发明的技术特征以及优点将会被本领域技术人员更好地理解，其中：The technical features and advantages of the present invention will be better understood by those skilled in the art, in which:
图1是根据本发明的实施例的基于服务器集群的数据处理系统的示意性结构图；1 is a schematic structural diagram of a server cluster-based data processing system according to an embodiment of the present invention;
图2是根据本发明的实施例的基于服务器集群的数据处理方法的流程图。2 is a flow chart of a server cluster based data processing method in accordance with an embodiment of the present invention.
具体实施方式detailed description
图1是根据本发明的实施例的基于服务器集群的数据处理系统的示意性结构图。如图1所示，本发明所公开的基于服务器集群的数据处理系统包括地址转换服务器1、调度服务器2、至少两个数据处理服务器3。其中，所述地址转换服务器1被配置为接收来自至少一个客户端的数据处理请求，并将其转发至 调度服务器2，以及接收来自所述至少两个数据处理服务器3中的任一个的数据处理应答，并随之将其传送回对应的客户端。所述调度服务器2被配置为接收经所述地址转换服务器1转发的数据处理请求，并随之根据预定的调度算法将其分发至所述至少两个数据处理服务器3中的一个以确保负载均衡。所述至少两个数据处理服务器3中的每个被配置为处理接收到的所述数据处理请求，并基于处理结果构建数据处理应答，以及随之将所述数据处理应答传送至所述地址转换服务器1。1 is a schematic structural diagram of a server cluster-based data processing system in accordance with an embodiment of the present invention. As shown in FIG. 1, the server cluster-based data processing system disclosed by the present invention includes an address translation server 1, a scheduling server 2, and at least two data processing servers 3. Wherein the address translation server 1 is configured to receive a data processing request from at least one client and forward it to The server 2 is scheduled, and a data processing response from any of the at least two data processing servers 3 is received and subsequently transmitted back to the corresponding client. The scheduling server 2 is configured to receive a data processing request forwarded by the address translation server 1 and then distribute it to one of the at least two data processing servers 3 according to a predetermined scheduling algorithm to ensure load balancing . Each of the at least two data processing servers 3 is configured to process the received data processing request and construct a data processing response based on the processing result, and subsequently transmit the data processing response to the address translation Server 1.
优选地，本发明所公开的基于服务器集群的数据处理系统进一步包括任务状态存储装置4，其被配置为存储数据处理任务状态表，所述数据处理任务状态表记录每个数据处理请求所指示的数据处理任务及其处理状态。Preferably, the server cluster-based data processing system disclosed herein further includes a task state storage device 4 configured to store a data processing task status table that records each data processing request indication Data processing tasks and their processing status.
优选地，在本发明所公开的基于服务器集群的数据处理系统中，所述至少两个数据处理服务器3中的每一个在接收到数据处理请求后将其所指示的数据处理任务写入所述数据处理任务状态表中的一条新的记录中，并将该数据处理任务的处理状态设置为“待处理”。Preferably, in the server cluster-based data processing system disclosed by the present invention, each of the at least two data processing servers 3 writes the data processing task indicated by the at least two data processing servers after the data processing request is received. The data is processed in a new record in the task status table, and the processing status of the data processing task is set to "pending".
优选地，在本发明所公开的基于服务器集群的数据处理系统中，所述至少两个数据处理服务器3中的每一个在具有空闲资源时访问所述数据处理任务状态表以便按预定规则获取一个或多个处理状态为“待处理”的数据处理任务，并随之处理该数据处理任务，并将其处理状态设置为“处理中”，以及在处理完成后将其处理状态设置为“已处理”。Preferably, in the server cluster-based data processing system disclosed by the present invention, each of the at least two data processing servers 3 accesses the data processing task status table when having idle resources to acquire one according to a predetermined rule. Or multiple data processing tasks whose processing status is "Pending", and then process the data processing task, set its processing status to "Processing", and set its processing status to "Processed" after processing is completed. ".
优选地，在本发明所公开的基于服务器集群的数据处理系统中，所述任务状态存储装置4周期性地轮询所述数据处理任务状态表中的记录，并将任务状态为“处理中”且在该状态下持续的时间超过预定的阈值的数据处理任务的状态重新设置为“待处理”。由上，在某个或某些数据处理服务器发生故障时能够保证数据处理任务的处理不被意外终止。Preferably, in the server cluster-based data processing system disclosed by the present invention, the task state storage device 4 periodically polls the records in the data processing task status table and sets the task status to "in process". And the state of the data processing task in which the duration exceeds the predetermined threshold in this state is reset to "pending". From the above, when a certain or some data processing server fails, it can ensure that the processing of the data processing task is not terminated unexpectedly.
优选地，在本发明所公开的基于服务器集群的数据处理系统中，地址转换服务器1在接收到数据处理请求后按预定规则生成全局唯一的标识符，并将所述标识符关联于该数据处理请求，随之将包含关联的标识符的数据处理请求转发至所述调度服务器2，并保存该标识符与该数据处理请求的来源地址(例如，发送该数据处理请求的客户端的IP地址)的映射关系。 Preferably, in the server cluster-based data processing system disclosed by the present invention, the address translation server 1 generates a globally unique identifier according to a predetermined rule after receiving the data processing request, and associates the identifier with the data processing. Requesting, then forwarding a data processing request containing the associated identifier to the dispatch server 2 and saving the identifier with the source address of the data processing request (eg, the IP address of the client sending the data processing request) Mapping relations.
优选地，在本发明所公开的基于服务器集群的数据处理系统中，在接收来自所述至少两个数据处理服务器3中的任一个的数据处理应答后，所述地址转换服务器1根据该数据处理应答中包含的标识符以及所保存标识符与数据处理请求的来源地址之间的映射关系确定该数据处理应答的目的地址(即该应答对应的数据处理请求的来源地址)，并随之将其传送回该目的地址所指向的客户端。Preferably, in the server cluster-based data processing system disclosed in the present invention, after receiving a data processing response from any of the at least two data processing servers 3, the address translation server 1 processes the data according to the data. The identifier included in the response and the mapping relationship between the saved identifier and the source address of the data processing request determine the destination address of the data processing response (ie, the source address of the data processing request corresponding to the response), and then Send back to the client pointed to by the destination address.
由上可见，本发明所公开的基于服务器集群的数据处理系统具有下列优点：具有高的数据处理性能(即较高的数据处理速度和效率)以及良好的负载均衡能力。It can be seen from the above that the server cluster-based data processing system disclosed by the present invention has the following advantages: high data processing performance (ie, high data processing speed and efficiency) and good load balancing capability.
图2是根据本发明的实施例的基于服务器集群的数据处理方法的流程图。如图2所示，本发明所公开的基于服务器集群的数据处理方法包括下列步骤：(A1)地址转换服务器接收来自至少一个客户端的数据处理请求，并将其转发至调度服务器；(A2)所述调度服务器在接收到所述地址转换服务器转发的数据处理请求后根据预定的调度算法将其分发至至少两个数据处理服务器中的一个以确保负载均衡；(A3)所述至少两个数据处理服务器中的每个处理接收到的所述数据处理请求，并基于处理结果构建数据处理应答，以及随之将所述数据处理应答传送至所述地址转换服务器；(A4)所述地址转换服务器在接收到所述数据处理应答后将其传送回对应的客户端。2 is a flow chart of a server cluster based data processing method in accordance with an embodiment of the present invention. As shown in FIG. 2, the server cluster-based data processing method disclosed by the present invention includes the following steps: (A1) an address translation server receives a data processing request from at least one client and forwards it to a scheduling server; (A2) After receiving the data processing request forwarded by the address translation server, the scheduling server distributes it to one of the at least two data processing servers according to a predetermined scheduling algorithm to ensure load balancing; (A3) the at least two data processing Each of the servers processes the received data processing request and constructs a data processing response based on the processing result, and subsequently transmits the data processing response to the address translation server; (A4) the address translation server is After receiving the data processing response, it is transmitted back to the corresponding client.
优选地，在本发明所公开的基于服务器集群的数据处理方法中，所述步骤(A3)进一步包括：所述至少两个数据处理服务器中的每一个在接收到数据处理请求后将其所指示的数据处理任务写入数据处理任务状态表中的一条新的记录中，并将该数据处理任务的处理状态设置为“待处理”，其中，所述数据处理任务状态表记录每个数据处理请求所指示的数据处理任务及其处理状态。Preferably, in the server cluster-based data processing method disclosed in the present invention, the step (A3) further includes: each of the at least two data processing servers instructing the data processing request after receiving the data processing request The data processing task is written into a new record in the data processing task status table, and the processing status of the data processing task is set to "pending", wherein the data processing task status table records each data processing request The indicated data processing tasks and their processing status.
优选地，在本发明所公开的基于服务器集群的数据处理方法中，所述步骤(A3)进一步包括：所述至少两个数据处理服务器中的每一个在具有空闲资源时访问所述数据处理任务状态表以便按预定规则获取一个或多个处理状态为“待处理”的数据处理任务，并随之处理该数据处理任务，并将其处理状态设置为“处理中”，以及在处理完成后将其处理状态设置为“已处理”。Preferably, in the server cluster-based data processing method disclosed in the present invention, the step (A3) further comprises: each of the at least two data processing servers accessing the data processing task when having idle resources a status table to obtain one or more data processing tasks whose processing status is "pending" according to a predetermined rule, and then process the data processing task, and set its processing status to "in process", and after the processing is completed Its processing status is set to "Processed".
优选地，在本发明所公开的基于服务器集群的数据处理系统中，所述步骤 (A3)进一步包括：所述数据处理任务状态表中的记录被周期性的轮询，并将任务状态为“处理中”且在该状态下持续的时间超过预定的阈值的数据处理任务的状态重新设置为“待处理”。由上，在某个或某些数据处理服务器发生故障时能够保证数据处理任务的处理不被意外终止。Preferably, in the server cluster-based data processing system disclosed in the present invention, the steps (A3) further comprising: a state of the data processing task in which the record in the data processing task status table is periodically polled and the task status is "in process" and the time duration in the state exceeds a predetermined threshold Reset to "Pending." From the above, when a certain or some data processing server fails, it can ensure that the processing of the data processing task is not terminated unexpectedly.
优选地，在本发明所公开的基于服务器集群的数据处理方法中，所述步骤(A1)进一步包括：所述地址转换服务器在接收到数据处理请求后按预定规则生成全局唯一的标识符，并将所述标识符关联于该数据处理请求，随之将包含关联的标识符的数据处理请求转发至所述调度服务器，并保存该标识符与该数据处理请求的来源地址(例如，发送该数据处理请求的客户端的IP地址)的映射关系。Preferably, in the server cluster-based data processing method disclosed in the present invention, the step (A1) further includes: the address translation server generates a globally unique identifier according to a predetermined rule after receiving the data processing request, and Associating the identifier with the data processing request, and subsequently forwarding a data processing request including the associated identifier to the dispatch server, and saving the identifier with a source address of the data processing request (eg, transmitting the data) The mapping relationship of the IP address of the client that handles the request.
优选地，在本发明所公开的基于服务器集群的数据处理方法中，所述步骤(A4)进一步包括：在接收到来自所述至少两个数据处理服务器中的任一个的数据处理应答后，所述地址转换服务器根据该数据处理应答中包含的标识符以及所保存标识符与数据处理请求的来源地址之间的映射关系确定该数据处理应答的目的地址(即该应答对应的数据处理请求的来源地址)，并随之将其传送回该目的地址所指向的客户端。Preferably, in the server cluster-based data processing method disclosed in the present invention, the step (A4) further comprises: after receiving a data processing response from any one of the at least two data processing servers, Determining, by the address translation server, the destination address of the data processing response according to the identifier included in the data processing response and the mapping relationship between the saved identifier and the source address of the data processing request (ie, the source of the data processing request corresponding to the response) Address) and then transfer it back to the client pointed to by the destination address.
由上可见，本发明所公开的基于服务器集群的数据处理方法具有下列优点：具有高的数据处理性能(即较高的数据处理速度和效率)以及良好的负载均衡能力。It can be seen from the above that the server cluster-based data processing method disclosed by the present invention has the following advantages: high data processing performance (ie, high data processing speed and efficiency) and good load balancing capability.
尽管本发明是通过上述的优选实施方式进行描述的，但是其实现形式并不局限于上述的实施方式。应该认识到：在不脱离本发明主旨和范围的情况下，本领域技术人员可以对本发明做出不同的变化和修改。 Although the invention has been described in terms of the preferred embodiments described above, the implementation forms are not limited to the embodiments described above. It will be appreciated that various changes and modifications can be made in the present invention without departing from the spirit and scope of the invention.

Claims (8)

  1. 一种基于服务器集群的数据处理系统，所述基于服务器集群的数据处理系统包括：A server cluster-based data processing system, the server cluster-based data processing system includes:
    地址转换服务器，所述地址转换服务器被配置为接收来自至少一个客户端的数据处理请求，并将其转发至调度服务器，以及接收来自至少两个数据处理服务器中的任一个的数据处理应答，并随之将其传送回对应的客户端；An address translation server configured to receive a data processing request from at least one client and forward it to a dispatch server, and receive a data processing response from any of the at least two data processing servers, and Send it back to the corresponding client;
    调度服务器，所述调度服务器被配置为接收经所述地址转换服务器转发的数据处理请求，并随之根据预定的调度算法将其分发至所述至少两个数据处理服务器中的一个以确保负载均衡；a scheduling server configured to receive a data processing request forwarded by the address translation server and then distribute it to one of the at least two data processing servers according to a predetermined scheduling algorithm to ensure load balancing ;
    至少两个数据处理服务器，所述至少两个数据处理服务器中的每个被配置为处理接收到的所述数据处理请求，并基于处理结果构建数据处理应答，以及随之将所述数据处理应答传送至所述地址转换服务器。At least two data processing servers, each of the at least two data processing servers configured to process the received data processing request and construct a data processing response based on the processing result, and subsequently the data processing response Transfer to the address translation server.
  2. 根据权利要求1所述的基于服务器集群的数据处理系统，其特征在于，所述系统进一步包括任务状态存储装置，其被配置为存储数据处理任务状态表，所述数据处理任务状态表记录每个数据处理请求所指示的数据处理任务及其处理状态。The server cluster-based data processing system of claim 1 wherein said system further comprises task state storage means configured to store a data processing task status table, said data processing task status table recording each The data processing task indicated by the data processing request and its processing status.
  3. 根据权利要求2所述的基于服务器集群的数据处理系统，其特征在于，所述至少两个数据处理服务器中的每一个在接收到数据处理请求后将其所指示的数据处理任务写入所述数据处理任务状态表中的一条新的记录中，并将该数据处理任务的处理状态设置为“待处理”。A server cluster-based data processing system according to claim 2, wherein each of said at least two data processing servers writes said indicated data processing task to said said after receiving said data processing request The data is processed in a new record in the task status table, and the processing status of the data processing task is set to "pending".
  4. 根据权利要求3所述的基于服务器集群的数据处理系统，其特征在于，所述至少两个数据处理服务器中的每一个在具有空闲资源时访问所述数据处理任务状态表以便按预定规则获取一个或多个处理状态为“待处理”的数据处理任务，并随之处理该数据处理任务，并将其处理状态设置为“处理中”，以及在处理完成后将其处理状态设置为“已处理”。The server cluster-based data processing system according to claim 3, wherein each of said at least two data processing servers access said data processing task status table when having idle resources to acquire one according to a predetermined rule Or multiple data processing tasks whose processing status is "Pending", and then process the data processing task, set its processing status to "Processing", and set its processing status to "Processed" after processing is completed. ".
  5. 根据权利要求4所述的基于服务器集群的数据处理系统，其特征在于，所述任务状态存储装置周期性地轮询所述数据处理任务状态表中的记录，并将任务状态为“处理中”且在该状态下持续的时间超过预定的阈值的数据处理任 务的状态重新设置为“待处理”。The server cluster-based data processing system according to claim 4, wherein said task state storage means periodically polls records in said data processing task status table and sets the task status to "in process" And the data processing in which the duration continues for more than a predetermined threshold The status of the service is reset to "Pending."
  6. 根据权利要求5所述的基于服务器集群的数据处理系统，其特征在于，所述地址转换服务器在接收到数据处理请求后按预定规则生成全局唯一的标识符，并将所述标识符关联于该数据处理请求，随之将包含关联的标识符的数据处理请求转发至所述调度服务器，并保存该标识符与该数据处理请求的来源地址的映射关系。The server cluster-based data processing system according to claim 5, wherein the address translation server generates a globally unique identifier according to a predetermined rule after receiving the data processing request, and associates the identifier with the identifier The data processing request, and then forwards the data processing request including the associated identifier to the dispatch server, and saves the mapping relationship between the identifier and the source address of the data processing request.
  7. 根据权利要求6所述的基于服务器集群的数据处理系统，其特征在于，在接收来自所述至少两个数据处理服务器中的任一个的数据处理应答后，所述地址转换服务器根据该数据处理应答中包含的标识符以及所保存标识符与数据处理请求的来源地址之间的映射关系确定该数据处理应答的目的地址，并随之将其传送回该目的地址所指向的客户端。The server cluster-based data processing system according to claim 6, wherein the address translation server processes the response according to the data processing response after receiving the data processing response from any of the at least two data processing servers The identifier contained in the identifier and the mapping relationship between the saved identifier and the source address of the data processing request determine the destination address of the data processing response and then transmit it back to the client pointed to by the destination address.
  8. 一种基于服务器集群的数据处理方法。所述方法包括下列步骤：A data processing method based on server clusters. The method includes the following steps:
    (A1)地址转换服务器接收来自至少一个客户端的数据处理请求，并将其转发至调度服务器；(A1) the address translation server receives the data processing request from the at least one client and forwards it to the dispatch server;
    (A2)所述调度服务器在接收到所述地址转换服务器转发的数据处理请求后根据预定的调度算法将其分发至至少两个数据处理服务器中的一个以确保负载均衡；(A2) the scheduling server distributes the data processing request forwarded by the address translation server to one of the at least two data processing servers according to a predetermined scheduling algorithm to ensure load balancing;
    (A3)所述至少两个数据处理服务器中的每个处理接收到的所述数据处理请求，并基于处理结果构建数据处理应答，以及随之将所述数据处理应答传送至所述地址转换服务器；(A3) each of the at least two data processing servers processing the received data processing request, and constructing a data processing response based on the processing result, and subsequently transmitting the data processing response to the address translation server ;
    (A4)所述地址转换服务器在接收到所述数据处理应答后将其传送回对应的客户端。 (A4) The address translation server transmits the data processing response back to the corresponding client after receiving the data processing response.
PCT/CN2017/080200 2016-04-19 2017-04-12 Data processing system and method based on server cluster WO2017181890A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201610241628.XA CN105959346A (en) 2016-04-19 2016-04-19 Data processing system and data processing method based on server cluster
CN201610241628.X 2016-04-19

Publications (1)

Publication Number Publication Date
WO2017181890A1 true WO2017181890A1 (en) 2017-10-26

Family

ID=56917945

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2017/080200 WO2017181890A1 (en) 2016-04-19 2017-04-12 Data processing system and method based on server cluster

Country Status (2)

Country Link
CN (1) CN105959346A (en)
WO (1) WO2017181890A1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114448960A (en) * 2021-12-27 2022-05-06 天翼云科技有限公司 Scheduling system, method, device, equipment and medium

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105959346A (en) * 2016-04-19 2016-09-21 中国银联股份有限公司 Data processing system and data processing method based on server cluster
CN106790610B (en) * 2016-12-29 2020-01-17 华为技术有限公司 Cloud system message distribution method, device and system
CN107872539B (en) * 2017-12-15 2021-01-15 安徽长泰信息安全服务有限公司 Data processing system and method based on cloud computing platform
CN108282520A (en) * 2017-12-29 2018-07-13 广东电网有限责任公司电力科学研究院 Power plant optimization system and its operation method
CN108874548B (en) * 2018-07-11 2021-04-02 深圳市东微智能科技股份有限公司 Data processing scheduling method and device, computer equipment and data processing system
CN111552748A (en) * 2020-05-08 2020-08-18 深圳知路科技有限公司 Data processing method, device and system

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101815033A (en) * 2010-03-12 2010-08-25 成都市华为赛门铁克科技有限公司 Method, device and system for load balancing
WO2010127017A1 (en) * 2009-04-28 2010-11-04 Cisco Technology, Inc. Methods and apparatus to get feedback information in virtual environment for server load balancing
CN102752188A (en) * 2011-04-21 2012-10-24 北京邮电大学 Transmission control protocol connection migratory method and system
CN103220354A (en) * 2013-04-18 2013-07-24 广东宜通世纪科技股份有限公司 Method for achieving load balancing of server cluster
CN105959346A (en) * 2016-04-19 2016-09-21 中国银联股份有限公司 Data processing system and data processing method based on server cluster

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102158512B (en) * 2010-02-11 2016-03-30 联想(北京)有限公司 A kind of load equilibration scheduling method, Apparatus and system
CN102761608B (en) * 2012-06-28 2016-06-01 深圳市深信服电子科技有限公司 The method of UDP session multiplexing and load balancing equipment
CN103677752B (en) * 2012-09-19 2017-02-08 腾讯科技（深圳）有限公司 Distributed data based concurrent processing method and system
CN104468174B (en) * 2013-09-25 2017-10-13 北京新媒传信科技有限公司 The method and apparatus that a kind of cluster server performs task
CN105389209B (en) * 2015-12-25 2019-04-26 中国建设银行股份有限公司 A kind of asynchronous batch tasks processing method and system

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2010127017A1 (en) * 2009-04-28 2010-11-04 Cisco Technology, Inc. Methods and apparatus to get feedback information in virtual environment for server load balancing
CN101815033A (en) * 2010-03-12 2010-08-25 成都市华为赛门铁克科技有限公司 Method, device and system for load balancing
CN102752188A (en) * 2011-04-21 2012-10-24 北京邮电大学 Transmission control protocol connection migratory method and system
CN103220354A (en) * 2013-04-18 2013-07-24 广东宜通世纪科技股份有限公司 Method for achieving load balancing of server cluster
CN105959346A (en) * 2016-04-19 2016-09-21 中国银联股份有限公司 Data processing system and data processing method based on server cluster

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114448960A (en) * 2021-12-27 2022-05-06 天翼云科技有限公司 Scheduling system, method, device, equipment and medium

Also Published As

Publication number Publication date
CN105959346A (en) 2016-09-21

Similar Documents

Publication Publication Date Title
WO2017181890A1 (en) Data processing system and method based on server cluster
KR102498777B1 (en) Messaging platform for delivering real-time messages
US9043382B2 (en) Stream processing using a client-server architecture
US10367852B2 (en) Multiplexed demand signaled distributed messaging
US9705752B2 (en) Reliably updating a messaging system
JP4144897B2 (en) Optimal server in common work queue environment
US9104488B2 (en) Support server for redirecting task results to a wake-up server
US9160791B2 (en) Managing connection failover in a load balancer
US20070282964A1 (en) Method and apparatus for processing remote shell commands
US20150341470A1 (en) Subscribing to multiple resources through a common connection
KR101564965B1 (en) Method and server for assigning relative order to message using vector clock and delivering the message based on the assigned relative order under distributed environment
US20130066828A1 (en) Scale-out system to acquire event data
JP2006235831A (en) Condition management device, method, and program
US20030149773A1 (en) Network abstraction of input/output devices
KR20190035592A (en) Distributed Broker Coordinator System and Method in a Distributed Cloud Environment
JP2019532399A (en) Data replication in scalable messaging systems
US20090034545A1 (en) Multicasting
US20220345536A1 (en) Publish and subscribe communication system
TWI502922B (en) Method and server for keeping apparatuses in alive state
TWI502941B (en) Cluster image recognition service system
JP2011076603A (en) Apparatus, system, and method for improving performance of real time application in intermittent connection environment
JP2006031592A (en) Message communication method, device and program

Legal Events

Date Code Title Description
NENP Non-entry into the national phase

Ref country code: DE

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17785375

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 17785375

Country of ref document: EP

Kind code of ref document: A1