WO2022120806A1

WO2022120806A1 - Multi-cloud distributed messaging method and system for high performance computing

Info

Publication number: WO2022120806A1
Application number: PCT/CN2020/135768
Authority: WO
Inventors: 林帅康; 马健; 刘阳; 朱和胜; 温书豪; 赖力鹏
Original assignee: 深圳晶泰科技有限公司
Priority date: 2020-12-11
Filing date: 2020-12-11
Publication date: 2022-06-16

Abstract

A multi-cloud distributed messaging method and system for high performance computing. The method comprises: creating a message model according to the usage scenario, acquiring information to be transmitted, and filling in information in a required format of a message template according to the information to be transmitted and the message model, so as to generate a message instance, wherein the message template comprises a message header, a message carrier, and a message trailer; a message producer creating a network connection to a message service by means of an SDK provided by a multi-cloud distributed message queuing service, and pushing, by means of the connection, the generated message instance into a specified message queue in the multi-cloud distributed message queuing service; and the multi-cloud distributed message queuing service interpreting the pushed message, performing message routing according to message header information, and pushing related messages to message consumers on different message models. By means of the method and the system, the message queue is oriented towards the definition of message transmission formats for multi-cloud high performance computing and the support of multi-message models, such that high performance computing development and integration on a cloud are more efficient and convenient.

Description

Distributed message delivery method and system for high performance computing and multi-cloud

technical field

The invention relates to the field of communications, in particular to a distributed message delivery method and system for high-performance computing and multi-cloud.

Background technique

High Performance Computing (HPC) refers to the use of aggregated computing power to handle data-intensive computing tasks that cannot be accomplished by standard workstations, including modeling, simulation, and rendering. It plays an increasingly important role in modern scientific research and industrial production.

With the popularity of cloud computing, many traditional services have also moved to the cloud, including HPC. Compared with traditional supercomputing, high-performance computing on the cloud is endowed with more flexibility, such as computing clusters used on demand, computing resources that can be quickly and elastically scaled, and large clusters built across multiple regions or multiple cloud vendors. , more diverse heterogeneous computing resources, etc.

The multi-cloud high-performance computing platform provides a powerful computing cluster that can rapidly expand and contract. The multi-cloud computing cluster resources break through geographical restrictions, enabling computing resources to be distributed globally across regions. The task also has the distributed characteristics. This feature is very different from traditional high-performance computing clusters built in the same region and network. For this reason, in order to improve the programming and development efficiency of enterprise developers and reduce the difficulty of multi-cloud and high-performance programming for enterprise developers, rich cloud middleware services are essential. The multi-cloud distributed messaging method is the information delivery bridge between tasks and tasks and between tasks and services in the cloud computing environment.

SUMMARY OF THE INVENTION

Based on this, it is necessary to provide a distributed messaging method for high-performance computing and multi-cloud that can improve the convenience of communication.

At the same time, a distributed message delivery system for high-performance computing and multi-cloud is provided, which can improve the convenience of communication.

A distributed messaging approach for HPC multi-cloud, including:

Production message: According to the usage scenario, create a message model, obtain the information to be delivered, fill in the corresponding information according to the information to be delivered and the message model according to the requirements of the message template format, and generate a message instance. The message template includes: message header, message carrier, and message trailer ;

Message entering the queue: The message producer creates a network connection to the message service through the SDK provided by the multi-cloud distributed message queue service, and pushes the generated message instance to the specified message queue in the multi-cloud distributed message queue service through the connection;

Message interpretation routing: The multi-cloud distributed message queue service interprets the pushed messages, performs message routing according to the relevant field information in the message header, and pushes the relevant messages to message consumers on different message models.

In a preferred embodiment, the message header includes: message queue name, message authentication information, whether the message is persistent, message sender, message receiver, message type, creation time, and message version number; different information is distinguished according to the message header information. message model;

The message interpretation route includes: the multi-cloud distributed message queue service disassembles the pushed message, obtains the message header information of the message, searches the currently created message queue according to the message queue name in the message header, and finds the current queue, Compare whether the consumption type of the message header is consistent with the attributes of the message queue where it is located, confirm whether the current message producer or message consumer has the right to operate the message queue according to the message authentication information in the message header, and determine one-to-one or one-to-one according to the message model To-many correlation, establish a one-to-one or one-to-many message consumption model.

In a preferred embodiment, the message entry queue further includes message confirmation: if a message confirmation instruction or an enabled message confirmation mechanism is received, the multi-cloud distributed message service receives the message fed back by the message consumer and the message is normally consumed. The message service feeds back the received message and feeds back to the message producer that the message has been received normally.

In a preferred embodiment, the message entry queue further includes message persistence: if message persistence is enabled or a message persistence instruction is received when the message queue is created, the multi-cloud distributed message queue service will send each message on the queue to the service. The received information is persisted.

In a preferred embodiment, the message model includes: one or more of a point-to-point model, a publish-subscribe model, a remote call model, a broadcast model, and a work queue model;

The step of entering the message into the queue further includes: operating according to different message models;

If the message model is the remote call model, the multi-cloud distributed message queue service will generate a unique ID for the message producer. When multiple associated message producers exchange messages with message consumers at the same time, the message will be consumed according to the associated ID. The producer replies to the message and sends it to the correct message producer;

If the message model is a publish-subscribe model, receive subscription instructions, perform multi-queue delivery according to the number of subscribed message consumers, subscribe topic messages according to different message queue names, and control the sending of messages of the corresponding subscribed topics to message consumers according to the message queue name. ;

If the consumption model is the work queue model, different messages are taken out from the consumption queue one by one and assigned to the message consumers, and the same message will not be consumed by different consumers twice.

In a preferred embodiment, it also includes interconnection: setting up message service intermediaries in different areas, deploying message service intermediaries in each area, interconnecting message service intermediaries, and message producers by directly accessing message service intermediaries in their regions Communicate with consumers in other regions.

In a preferred embodiment, it also includes nearby access: deploying a message queue cluster in each public cloud, the message queue service is accessed nearby, and deploying a message queue cluster in different clouds uses the same API interface for access;

The message tail includes: message digest algorithm, digest value, check value;

Message queue attributes include: one or more of queue name and queue persistence.

A distributed messaging system for high-performance computing multi-cloud, including:

Node server: apply for multiple node servers in each cloud, deploy clusters, and create access portals for cluster services respectively;

Message queue center controller: deploy the message queue center controller, and configure the access entry of each cluster service to the message queue center controller;

The message queue center controller includes:

Production message module: According to the usage scenario, create a message model, obtain the information to be delivered, fill in the corresponding information according to the information to be delivered and the message model according to the requirements of the message template format, and generate a message instance. The message template includes: message header, message carrier, message tail;

Message entry queue module: The message producer creates a network connection to the message service through the SDK provided by the multi-cloud distributed message queue service, and pushes the generated message instance to the specified message queue in the multi-cloud distributed message queue service through the connection;

Message interpretation and routing module: The multi-cloud distributed message queue service interprets the pushed messages, performs message routing according to the relevant field information in the message header, and pushes the relevant messages to message consumers on different message models.

The message interpretation and routing module also includes: the multi-cloud distributed message queue service disassembles the pushed message, obtains the message header information of the message, searches the currently created message queue according to the message queue name in the message header, and finds the current message queue. Queue, compare whether the consumption type of the message header is consistent with the attribute of the message queue, confirm whether the current message producer or message consumer has permission to operate the message queue according to the message authentication information in the message header, and determine one-to-one according to the message model Or one-to-many correlation, establish a one-to-one or one-to-many message consumption model;

The message entry queue module further includes a message confirmation unit: if a message confirmation instruction or an enabled message confirmation mechanism is received, the multi-cloud distributed message service receives an instruction that the message fed back by the message consumer is normally consumed, and the multi-cloud distributed message service responds to the received message. The received message is fed back, and it is fed back to the message producer that the message has been received normally;

and message persistence unit: If message persistence is enabled when the message queue is created or a message persistence command is received, the multi-cloud distributed message queue service performs data persistence for each received message on the queue.

In a preferred embodiment, it also includes: an interconnection module: setting up message service intermediate stations in different areas, deploying message service intermediate stations in each area, and interconnecting message service intermediate stations, and message producers directly access the message service in the area where they are located. Intermediate stations communicate with consumers in other areas;

And the nearest access module: Deploy message queue clusters in each public cloud, the message queue service is accessed nearby, and the same API interface is used to deploy message queue clusters in different clouds.

The above-mentioned distributed message delivery method and system for high-performance computing and multi-cloud, the message queue definition of message transmission format for multi-cloud high-performance computing, and the support of multi-message models make the development and integration of high-performance computing on the cloud more efficient and convenient. Distributed messaging services on multiple clouds, running on computing resources and executing programs with high performance can achieve cross-regional message delivery without caring about the actual network location where the program is currently located. The distributed message service shields the complexity of cross-regional and heterogeneous networks. Developers only need to access the interface through a unified distributed message service and focus on the realization of computing workflow and upper-layer business.

The present invention realizes the nearby access of message queue services by deploying message queue clusters in major public cloud vendors. The message queue cluster access deployed in different cloud vendors uses the same API interface, which reduces the difficulty of service integration, and developers do not need to integrate according to different cloud vendors. The multi-cloud distributed message queue service is naturally distributed globally and has strong service high availability.

Description of drawings

FIG. 1 is a partial flowchart of a distributed message delivery method for high-performance computing and multi-cloud according to an embodiment of the present invention;

FIG. 2 is a schematic diagram of a message model according to an embodiment of the present invention.

Detailed ways

The present invention can realize cross-regional message transmission through the distributed message service built on the multi-cloud, running on the computing resources and executing the program with high performance without caring about the current real network location of the program. The distributed message service shields the complexity of cross-regional and heterogeneous networks. Developers only need to access the interface through a unified distributed message service and focus on the realization of computing workflow and upper-layer business.

The design of a distributed messaging approach in a multi-cloud environment needs to consider the following factors:

1. Transmission delay and stability in a real network environment across regions;

2. The protocol of distributed message transmission and its format definition. The definition of the format should be concise and meet the needs of computing tasks.

3. Support of distributed message multi-model, the support of message model depends on the usage requirements in the current multi-cloud computing environment. Usually, a message queue is a container for storing messages. When a message is needed, only the message is taken from the message queue for use. The message queue is an important component in the distributed system. The definition of the message transmission format for multi-cloud high-performance computing and the support of the multi-message model make the development and integration of high-performance computing on the cloud more efficient and convenient.

The present invention provides a distributed multi-message transfer method suitable for multi-cloud high-performance computing scenarios, establishes an information and data transfer mechanism between tasks and service systems in a multi-cloud high-performance computing platform, shields the complexity of cross-regional network communication, and is oriented towards It provides a reference implementation for building a communication model on a multi-cloud high-performance computing environment.

As shown in FIG. 1 , a distributed message delivery method for high-performance computing and multiple clouds according to an embodiment of the present invention includes:

Step S101, producing a message: according to the usage scenario, create a message model, obtain the information to be delivered, fill in the corresponding information according to the information to be delivered and the message model according to the requirements of the message template format, and generate a message instance, the message template includes: a message header, a message carrier , message tail;

Step S103, the message enters the queue: the message producer creates a network connection to the message service through the SDK provided by the multi-cloud distributed message queue service, and pushes the generated message instance to the designated message queue in the multi-cloud distributed message queue service through the connection ;

Step S105, message interpretation and routing: the multi-cloud distributed message queue service interprets the pushed message, performs message routing according to the relevant field information in the message header, and pushes the relevant message to message consumers on different message models.

A multi-cloud distributed message queue service which accepts and forwards messages. Think of it as a post office: when you put mail to be published in a mailbox, you can ensure that the mail is eventually delivered to the recipient. By analogy, the Distributed Message Queuing Service is a PO Box, a Post Office, and a Postman. The main difference between the Distributed Message Queuing Service and the Post Office is that it does not handle paper, but instead receives, stores and forwards binary data messages.

As shown in FIG. 2 , the message model of this embodiment includes one or more of a point-to-point model, a publish-subscribe model, a remote calling model, a broadcast model, and a work queue model.

The message producer can be a producer, a sender, a client, etc. according to different message models, as long as the function of the message producer of the present invention is implemented.

The message consumers can be consumers, receivers, servers, etc. according to different message models or different usage scenarios, as long as the functions of the message consumers of the present invention are implemented.

If the message model is a point-to-point model: the producer produces the message and sends it to the message queue, and the consumer consumes the message from the message queue.

If the message model is a publish-subscribe model, receive subscription instructions, perform multi-queue delivery according to the number of subscribed message consumers, subscribe topic messages according to different message queue names, and control the sending of messages of the corresponding subscribed topics to message consumers according to the message queue name. .

In the publish-subscribe model, consumer A and consumer B subscribe to messages of different topic types through message queues, such as messages with a topic of log type and messages with a topic of monitoring type. Through the publish-subscribe model, message consumers only subscribe to the message topics they care about, and messages on other topics are ignored.

If the message model is the remote call model, the multi-cloud distributed message queue service will generate a unique ID for the message producer. When multiple associated message producers exchange messages with message consumers at the same time, the message will be consumed according to the associated ID. The producer replies to the message and sends it to the correct message producer.

The difference between the remote call model and other message models is that it is a response model. The remote call model is suitable for scenarios where the message loss rate is extremely low and the message delivery reliability is extremely high. Blocking waiting, waiting for the server to return a signal after consuming the message to tell the sender that the message has been delivered safely.

If the message model is the broadcast model, the message queue under the broadcast model supports the same message to be consumed multiple times, the sender sends the message to the broadcast message queue, and both receiver A and receiver B can receive the message, so as to simulate Mechanisms for consuming broadcasts.

The implementation mechanism of the work queue model is similar to that of the broadcast model, the difference is that the same message cannot be received by different consumers, and different messagers will fetch different messages from the message queue one by one.

The network connection created by the message producer to the message service through the SDK (Software Development Kit) provided by the multi-cloud distributed message service can be performed in or before the message production step, or in the message entry queue step, as long as the generated message is processed. The message instance can be pushed to the specified message queue in the multi-cloud distributed message queue service through this connection.

Further, to produce a message, the message producer creates a network connection to the message service through the SDK (Software Development Kit) provided by the multi-cloud distributed message queue service, fills in the corresponding fields according to the requirements of the message template format, and generates a message instance.

Further, after the message enters the queue, the message producer creates a network connection to the message service through the SDK provided by the multi-cloud distributed message queue service, and pushes the generated message instance to the specified message queue in the message queue service through the connection. message model to operate.

Further, for message interpretation and routing, the multi-cloud distributed message service interprets the pushed messages, performs message routing according to the relevant field information in the message header, and pushes the relevant messages to message consumers on different message models.

Since the multi-cloud distributed message queue service acts as an intermediate depositor of messages, it is necessary to receive and forward messages to consumers for message confirmation to ensure message delivery.

The message entering the queue also includes message confirmation: if the message confirmation command or the enabled message confirmation mechanism is received, the multi-cloud distributed message service receives the message fed back by the message consumer and is instructed to be consumed normally, and the multi-cloud distributed message service processes the received message. Feedback, feedback to the message producer that the message has been received normally.

The message entry queue also includes message persistence: if message persistence is enabled when the message queue is created or a message persistence command is received, the multi-cloud distributed message queue service performs data persistence for each received message on the queue.

In a distributed environment, due to the characteristics of cross-regional networks, network interruptions, service failures, etc., messages may be lost during transmission, so persistence is necessary for services that require high reliability. , but message persistence involves the I/O (Input/Output) operations of third-party storage or local hard disk storage, so the transmission efficiency will inevitably decrease, so the message persistence feature is an option in the message model. When the persistence option is enabled when the message queue is created, the multi-cloud distributed message service refers to data persistence for each received message on the queue to prevent the loss of messages in the memory of the storage computer due to service interruption. Through the persistence mechanism, the multi-cloud distributed message service can restore the original message data in the message queue from the persistent data service during the interruption and restart process.

In the process of cross-border message delivery, the public network transmission is used by default. For example, if an enterprise has built a cross-border dedicated line channel, the distributed message queue service can also transmit messages through the dedicated line. The transmission stability and privacy of private line messages will be better, but it is also necessary to pay attention to whether the bandwidth of the private line is sufficient.

The message header, the message model is distinguished by the message header. The fields contained in the message header are: message queue name, message authentication information, whether the message is persistent, message sender, message receiver, message type, creation time, and message version number.

The message carrier, which stores the actual data content of the message.

The message tail is mainly to ensure the integrity of the data during the message transmission. It contains fields: message digest algorithm, digest value, and check value.

The message interpretation route includes: the multi-cloud distributed message queue service disassembles the pushed message, obtains the message header information of the message, searches the currently created message queue according to the message queue name in the message header, finds the current queue, and compares Whether the consumption type of the message header is consistent with the attributes of the message queue where it is located, confirm whether the current message producer or message consumer has permission to operate the message queue according to the message authentication information in the message header, and determine one-to-one or one-to-many according to the message model The correlation, establish a one-to-one or one-to-many message consumption model.

Further, the distributed message delivery method for high-performance computing and multi-cloud in this embodiment further includes interconnection: setting up message service intermediate stations in different regions, deploying message service intermediate stations in each region, interconnecting message service intermediate stations, and producing messages. Consumers communicate with consumers in other regions by directly accessing the message service intermediate station in their region.

Further, the multi-cloud-oriented distributed message delivery method for high-performance computing in this embodiment further includes nearby access: deploying a message queue cluster in each public cloud, the message queue service is accessed nearby, and deploying a message queue cluster in different clouds uses the same access method. API interface.

A distributed message delivery system for high-performance computing and multiple clouds of the present invention includes:

Message Queue Center Controller: Deploy the message queue center controller, and configure the access entry of each cluster service to the message queue center controller.

The message queue center controller of the present invention includes:

High Performance Computing (HPC): refers to the use of aggregated computing power to handle data-intensive computing tasks that cannot be accomplished by standard workstations, including modeling, simulation, and rendering.

High-performance computing on the cloud: It refers to moving traditional high-performance computing clusters to the cloud, and endows them with the characteristics of on-demand expansion charges, diverse heterogeneous computing resources, and the ability to build clusters across the world.

Multi-cloud: refers to the hybrid integration of multiple public clouds to expand the scale of computing resources.

Distributed system: A distributed system is a loosely coupled system composed of multiple processors interconnected through communication lines. From a processor in the system, other processors and corresponding resources are remote, and only its own resources are local.

Message Queuing: A "message" is a unit of data sent between two computers. Messages can be very simple, such as containing only text strings, or more complex, possibly containing embedded objects. The message is sent to the queue. A "message queue" is a container that holds messages during their transmission. A message queue manager acts as a middleman when relaying messages from its source to its destination. The main purpose of queues is to provide routing and guarantee delivery of messages; if a receiver is unavailable when a message is sent, message queues hold the message until it can be successfully delivered.

Leader Node: The message queue deployed by a single node has the risk of a single point of failure, that is to say, if the message queue service deployed on a single node fails, the entire message queue service will be unavailable, directly affecting the message producer. Messaging with the messager. The message queue deployed by multiple nodes solves the problem of single-point failure in a large program. In order to maintain the message delivery management under multiple nodes, it is necessary to elect a leader node, and the producer and the messager communicate with the leader node. .

Kubernetes: K8s for short, is an abbreviation formed by replacing the 8-character "ubernete" with 8. It is an open source, used to manage containerized applications on multiple hosts in the cloud platform. The goal of Kubernetes is to make the deployment of containerized applications simple and efficient (powerful). Kubernetes provides application deployment, planning, updating, and maintenance. a mechanism.

Rabbitmq: RabbitMQ is an open source message broker software (also known as message-oriented middleware) that implements the Advanced Message Queuing Protocol (AMQP).

The system of the invention includes: distributed deployment design in multi-cloud environment, message model design, message template format design, multi-cloud distributed message transmission process, and message persistence.

The invention is oriented to the field of high-performance computing multi-cloud distributed message queue service. The design goal of the distributed multi-message queue service is to meet the data interaction requirements of tasks and services in the high-performance computing environment built by multi-cloud, and to solve the complex cross-regional network arising from the multi-cloud environment. Improve the reliability of message transmission and reduce the complexity of cross-regional communication programming.

Further, the message header in this embodiment includes: message queue name, message authentication information, whether the message is persistent, message sender, message receiver, message type, creation time, and message version number. Different message models are distinguished according to different message header information.

Further, the message interpretation routing module of the present embodiment also includes: the multi-cloud distributed message queue service disassembles the pushed message, obtains the message header information of the message, and searches the currently created message according to the message queue name in the message header Queue, find the current queue, compare whether the consumption type of the message header is consistent with the attributes of the message queue, and confirm whether the current message producer or message consumer has permission to operate the message queue according to the message authentication information in the message header. According to the message model Determine one-to-one or one-to-many dependencies, and establish a one-to-one or one-to-many message consumption model.

Further, the message entry queue module of this embodiment further includes a message confirmation unit: if a message confirmation instruction or an enabled message confirmation mechanism is received, the multi-cloud distributed message service receives an instruction that the message fed back by the message consumer is normally consumed, and the multi-cloud distributed message service The message service feeds back the received message and feeds back to the message producer that the message has been received normally.

Further, the message entry queue module of this embodiment further includes a message persistence unit: if message persistence is enabled or a message persistence instruction is received when the message queue is created, the multi-cloud distributed message queue service will send each message on the queue. The received information is persisted.

Persistence writes the data in the message service memory (which is easy to lose after power failure) to reliable storage that is not lost after power failure, such as disks and databases.

Multi-cloud distributed messaging services may be cross-regional or cross-border, and private lines generally refer to private networks built by enterprises and operators. The biggest difference between them and the public network is better security and stability. There is not much difference in the deployment of message service, except that the message producer and messager use the private network address instead of the public network address when establishing a connection with the message service. The private network addresses will be forwarded by the internal gateway route of the enterprise on the local area network or private line network.

Further, the high-performance computing multi-cloud-oriented distributed messaging system in this embodiment further includes: an interconnection module: setting up message service intermediate stations in different regions, deploying message service intermediate stations in each region, and interconnecting message service intermediate stations, Message producers communicate with consumers in other regions by directly accessing the message service intermediate station in their region.

Further, the multi-cloud-oriented distributed messaging system for high-performance computing in this embodiment further includes: a nearby access module: deploying a message queue cluster in each public cloud, the message queue service is accessed nearby, and the message queue cluster is deployed in different clouds to access and use The same API interface.

In the publish-subscribe model, consumer A and consumer B subscribe to messages of different topic types through message queues, such as messages with a topic of log type and messages with a topic of monitoring type. Through the publish-subscribe model, message consumers only subscribe to the message topics they care about, and messages on other topics are ignored. In the publish-subscribe model, message consumers subscribe to topic messages according to different message queue names. For example, if the queue name is: /log, all messages prefixed with /log are received by consumers. If the queue name is: /log/warning, then Only warning level logs are received by consumers.

In this embodiment, a message service intermediate station is deployed nearby in each area, and the message service intermediate stations are interconnected using a public network or a dedicated line. For message producers and consumers, it looks like a direct connection to the same message service. The reliability of transmission mainly depends on distributed message service deployment, and each message service intermediate station implements message flow control, retransmission and other mechanisms.

Further, specifically, in this embodiment, three node servers are applied for in each cloud, and Kubernetes can be selected as the cluster management software. Deploy the RabbitMQ cluster through Kubernetes, and create access entries for the services respectively.

Further, in this embodiment, the message queue center controller: the message queue center controller is deployed in the AWS Cloud center, and the access portals of each RabbitMQ cluster service are configured in the message queue center controller.

Message queues with different message models:

The formal representation of the multi-cloud distributed message queue for high-performance computing is Queue, and the message model is Model:

Queue=Create of {Name, Model, Durable}

where Name is the unique name of the message queue.

Model types include: point-to-point model, publish-subscribe model, remote call model, broadcast model, and work queue model.

Durable specifies whether the message queue starts the persistence mechanism.

Message template format:

The formal representation of multi-cloud distributed messages for high-performance computing is as follows: Let Message be a message, then

Message=consist of{Message-Header,Message-Body,Message-Tail}

header

message carrier

message trailer

The message header consists of the following:

Message-Header=consist of{Name,Auth,Durable,Sender,Receiver,Time,Version}

Among them, Name is the name of the message queue, Auth is the message authentication information, Durable is the message persistence, Sender is the sender ID, Receiver is the receiver ID, Time is the message creation time, and Version is the message version number.

Multi-cloud distributed messaging

Multi-cloud distributed messaging boils down to the following basic operations. Each operation takes a message as a parameter.

CreateQueue(Name,Model,Durable); message queue creation operation.

CreateMessage(Message-Header,Message-Body,Message-Tail); message creation operation.

SendMessage(Message); the message is sent to the queue operation.

ReceiveMessage(Queue-Name); Get the message operation from the message queue.

SendBatchMessages([]Message); batch send message operation.

ReceiveBatchMessages(Queue-Name); Get message operations in batches from the message queue.

The multi-cloud distributed messaging approach includes the following steps:

According to different usage scenarios, call CreateQueue through the Mode parameter to create different message queue models (Model types include: point-to-point model, publish-subscribe model, remote call model, broadcast model, work queue model).

Prepare Message-Header, Message-Body, Message-Tail, and call CreateMessage to create a message instance.

The message producer calls the SendMessage or SendBatchMessages operation to add the message instance to the queue in the distributed message server.

Message consumers call ReceiveMessage or ReceiveBatchMessages to get message instances from the specified message queue.

Describe in detail below in conjunction with a specific embodiment:

The following uses the Golang programming language as an example to illustrate the message transfer process between the message sender and the receiver. The current basic component of the distributed message queue takes the RabbitMQ open source message middleware as an example, but the implementation of the SDK and the message queue center controller in the present invention does not depend on this system.

Definition of message headers

字段名称Field Name	类型type	长度(Byte)length (Byte)	字段说明Field Description
消息队列各称message queue	StringString	<＝128B<=128B	消息队列的唯一标识The unique identifier of the message queue
消息认证信息message authentication information	StringString	<＝512B<=512B	Authorization:BasicAuthorization:Basic
持久化Persistence	BoolBool	4B4B	是否消息持久标识Whether the message is persistently identified
消息发送方message sender	StringString	32B32B	IP+端口号IP+Port number

消息接收方message recipient	StringString	32B32B	IP+端口号IP+Port number
消息类型message type	StringString	8B8B
创建时间creation time	StringString	128B128B	格式:YYYY/MM/DD-hhmmssFormat: YYYY/MM/DD-hhmmss
消息版本号message version number	StringString	8B8B

Definition of message body

message class definition

type CloudMessage struct{

messageHeader MessageHeader

messageBody MessageBody

messageTail MessageTail

}

func(msg*CloudMessage)CreateQueue(Name,Model,Durable){}

func(msg*CloudMessage)CreateMessage(Message-Header,Message-Body,Message-Tail){}

func(msg*CloudMessage)SendMessage(Message){}

func(msg*CloudMessage)ReceiveMessage(Queue-Name){}

func(msg*CloudMessage)SendBatchMessages([]Message){}

func(msg*CloudMessage)ReceiveBatchMessages(Queue-Name){}

Messaging steps:

Create a message queue with a specified message model, taking the point-to-point model as an example

Queue=CloudMessage.CreateQueue("hello", "point-to-point", true)

Generate message instances from message template definitions

Msg=CloudMessage.CreateMessage(Message-Header,Message-Body,Message-Tail)()

Send a message

CloudMessage.SendMessage(Msg)

receive message

CloudMessage.ReceiveMessage(Queue.Name)

In the present invention, a message service intermediate station is deployed nearby in each area, and the message service intermediate stations are interconnected by public network or special line. For message producers and consumers, it looks like a direct connection to the same message service. The reliability of transmission mainly depends on the deployment of distributed message services, and each message service intermediate station implements mechanisms such as message flow control and retransmission. By deploying a message service intermediate station nearby in each area, the intermediate stations are interconnected by public network or dedicated line. For a message producer to directly access the message service in its region, it can communicate with consumers in other regions.

The present invention solves that the message service provided by the existing public cloud cannot satisfy the multiple message models simultaneously integrated by the present invention, and the API interfaces used by various clouds are inconsistent, which is not friendly to multi-cloud integration development. And the message service provided by a single public cloud is more inclined to serve services in a single cloud, and the reliability of cross-cloud and cross-region communication delays can be guaranteed.

Taking the above ideal embodiments according to the present application as inspiration, and through the above descriptions, relevant personnel can make various changes and modifications without departing from the technical idea of the present application. The technical scope of the present application is not limited to the content in the description, and the technical scope must be determined according to the scope of the claims.

As will be appreciated by those skilled in the art, the embodiments of the present application may be provided as a method, a system, or a computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It will be understood that each process and/or block in the flowchart illustrations and/or block diagrams, and combinations of processes and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing device to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing device produce Means for implementing the functions specified in a flow or flow of a flowchart and/or a block or blocks of a block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory result in an article of manufacture comprising instruction means, the instructions The apparatus implements the functions specified in the flow or flow of the flowcharts and/or the block or blocks of the block diagrams.

These computer program instructions can also be loaded on a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process such that The instructions provide steps for implementing the functions specified in the flow or blocks of the flowcharts and/or the block or blocks of the block diagrams.

Claims

A distributed message delivery method for high-performance computing and multi-cloud, characterized by comprising:

Production message: According to the usage scenario, create a message model, obtain the information to be delivered, fill in the corresponding information according to the information to be delivered and the message model according to the requirements of the message template format, and generate a message instance. The message template includes: message header, message carrier, and message trailer ;

Message entering the queue: The message producer creates a network connection to the message service through the SDK provided by the multi-cloud distributed message queue service, and pushes the generated message instance to the specified message queue in the multi-cloud distributed message queue service through the connection;

Message interpretation routing: The multi-cloud distributed message queue service interprets the pushed messages, performs message routing according to the relevant field information in the message header, and pushes the relevant messages to message consumers on different message models.
The multi-cloud-oriented distributed message delivery method according to claim 1, wherein the message header includes: message queue name, message authentication information, whether the message is persistent, message sender, message receiver, Message type, creation time, message version number; distinguish different message models according to different message header information;

The message interpretation route includes: the multi-cloud distributed message queue service disassembles the pushed message, obtains the message header information of the message, searches the currently created message queue according to the message queue name in the message header, and finds the current queue, Compare whether the consumption type of the message header is consistent with the attributes of the message queue where it is located, confirm whether the current message producer or message consumer has the right to operate the message queue according to the message authentication information in the message header, and determine one-to-one or one-to-one according to the message model To-many correlation, establish a one-to-one or one-to-many message consumption model.
The distributed message delivery method for high-performance computing and multi-cloud according to claim 1, wherein the message entering the queue further comprises message confirmation: if a message confirmation instruction or an enabled message confirmation mechanism is received, the multi-cloud distributed message The service receives the message fed back by the message consumer and is instructed to consume normally, and the multi-cloud distributed message service feeds back the received message and feeds back to the message producer that the message has been received normally.
The high-performance computing multi-cloud-oriented distributed message delivery method according to claim 1, wherein the message entering the queue further includes message persistence: if message persistence is enabled when the message queue is created, or if the message persistence is received command, the multi-cloud distributed message queue service performs data persistence for each piece of information received on the queue.
The multi-cloud-oriented distributed message delivery method according to any one of claims 1 to 4, wherein the message model includes: a point-to-point model, a publish-subscribe model, a remote call model, a broadcast model, and a work queue one or more models;

The step of entering the message into the queue further includes: operating according to different message models;

If the message model is the remote call model, the multi-cloud distributed message queue service will generate a unique ID for the message producer. When multiple associated message producers exchange messages with message consumers at the same time, the message will be consumed according to the associated ID. The producer replies to the message and sends it to the correct message producer;

If the message model is a publish-subscribe model, receive subscription instructions, perform multi-queue delivery according to the number of subscribed message consumers, subscribe topic messages according to different message queue names, and control the message of the corresponding subscription topic to be sent to message consumers according to the message queue name. ;

If the consumption model is the work queue model, different messages are taken out one by one in the consumption queue and assigned to the message consumers, and the same message will not be consumed by different consumers twice.
The high-performance computing multi-cloud-oriented distributed message delivery method according to any one of claims 1 to 4, further comprising interconnecting: setting up message service intermediate stations in different regions, and deploying message service intermediate stations in each region, The message service intermediate stations are interconnected, and the message producer communicates with consumers in other regions by directly accessing the message service intermediate station in the region.
The high-performance computing multi-cloud-oriented distributed message delivery method according to claim 2, further comprising nearby access: deploying a message queue cluster in each public cloud, the message queue service nearby access, and deploying messages in different clouds Queue cluster access uses the same API interface;

The message tail includes: message digest algorithm, digest value, check value;

Message queue attributes include: one or more of queue name and queue persistence.
A distributed messaging system for high-performance computing and multi-cloud, characterized by comprising:

Node server: apply for multiple node servers in each cloud, deploy clusters, and create access portals for cluster services respectively;

Message queue center controller: deploy the message queue center controller, and configure the access entry of each cluster service to the message queue center controller;

The message queue center controller includes:

Production message module: According to the usage scenario, create a message model, obtain the information to be delivered, fill in the corresponding information according to the information to be delivered and the message model according to the requirements of the message template format, and generate a message instance. The message template includes: message header, message carrier, message tail;

Message entry queue module: The message producer creates a network connection to the message service through the SDK provided by the multi-cloud distributed message queue service, and pushes the generated message instance to the specified message queue in the multi-cloud distributed message queue service through the connection;

Message interpretation and routing module: The multi-cloud distributed message queue service interprets the pushed messages, performs message routing according to the relevant field information in the message header, and pushes the relevant messages to message consumers on different message models.
The high-performance computing multi-cloud-oriented distributed message delivery system according to claim 8, wherein the message header includes: message queue name, message authentication information, whether the message is persistent, message sender, message receiver, Message type, creation time, message version number; distinguish different message models according to different message header information;

The message interpretation and routing module also includes: the multi-cloud distributed message queue service disassembles the pushed message, obtains the message header information of the message, searches the currently created message queue according to the message queue name in the message header, and finds the current message queue. Queue, compare whether the consumption type of the message header is consistent with the attributes of the message queue, confirm whether the current message producer or message consumer has permission to operate the message queue according to the message authentication information in the message header, and determine one-to-one according to the message model Or one-to-many correlation, establish a one-to-one or one-to-many message consumption model;

The message entry queue module further includes a message confirmation unit: if a message confirmation instruction or an enabled message confirmation mechanism is received, the multi-cloud distributed message service receives an instruction that the message fed back by the message consumer is normally consumed, and the multi-cloud distributed message service responds to the received message. The received message is fed back, and it is fed back to the message producer that the message has been received normally;

And message persistence unit: If message persistence is enabled when the message queue is created or a message persistence command is received, the multi-cloud distributed message queue service performs data persistence for each received message on the queue.
The high-performance computing multi-cloud-oriented distributed message delivery system according to claim 8 or 9, further comprising: an interconnection module: setting up a message service intermediate station in different areas, deploying a message service intermediate station in each area, The service intermediate stations are interconnected, and the message producer communicates with consumers in other areas by directly accessing the message service intermediate station in the area;

And the nearest access module: deploy message queue clusters in each public cloud, the message queue service is accessed nearby, and the same API interface is used to deploy message queue clusters in different clouds.