CN112527523A - Distributed message transmission method and system for high-performance computing multiple clouds - Google Patents

Abstract

A distributed message transmission method and system facing high-performance computing multi-cloud comprises the following steps: creating a message model according to a use scene, acquiring information to be transmitted, filling information according to the information to be transmitted and the message model according to the requirement of a message template format, and generating a message example, wherein the message template comprises a message header, a message carrier and a message tail; a message producer establishes network connection to a message service through an SDK provided by the multi-cloud distributed message queue service, and pushes a generated message instance to a specified message queue in the multi-cloud distributed message queue service through the connection; the multi-cloud distributed message queue service explains the pushed messages, performs message routing according to message header information, and pushes related messages to message consumers on different message models; according to the method and the system, the message queue faces to the multi-cloud high-performance computing message transmission format definition and the support of a multi-message model, so that the high-performance computing development and integration on the cloud are more efficient and convenient.

Description

Distributed message transmission method and system for high-performance computing multiple clouds

Technical Field

The invention relates to the field of communication, in particular to a distributed message transmission method and system for high-performance computing multiple clouds.

Background

High Performance Computing (HPC) refers to processing data-intensive Computing tasks that cannot be completed by standard workstations, including modeling, simulation, rendering, etc., using aggregated Computing power. It plays an increasingly important role in modern scientific research and industrial production.

With the popularity of cloud computing, many traditional services have moved to the cloud, including HPC. Compared with traditional supercomputing, high-performance computing on the cloud is endowed with more flexibility, such as computing clusters used on demand, computing resources with rapid elastic scaling capacity, super-large clusters built across multiple regions or multiple cloud vendors, heterogeneous computing resources with more diversity and the like.

The multi-cloud computing cluster resource breaks through the limitation of regions, so that the computing resources can be distributed across the regions in the world, and the computing tasks running on the computing resources have the distributed characteristic. This characteristic is very different from that of the traditional high-performance computing cluster which is constructed in the same region and the same network. Therefore, in order to improve the programming development efficiency of the enterprise developer and reduce the high-performance multi-cloud programming difficulty of the enterprise developer, abundant cloud middleware services are essential. The multi-cloud distributed message transmission method is an information transmission bridge between tasks and services in the cloud computing environment.

Disclosure of Invention

Based on this, there is a need for a distributed messaging method oriented to high-performance computing clouds, which can improve communication convenience.

Meanwhile, the distributed message transmission system facing high-performance computing cloud is capable of improving communication convenience.

A distributed message delivery method facing high-performance computing multi-cloud comprises the following steps:

producing messages: according to the using scene, creating a message model, acquiring information to be transmitted, filling corresponding information according to the information to be transmitted and the message model according to the requirement of a message template format, and generating a message example, wherein the message template comprises: a message header, a message carrier and a message tail;

the message enters the queue: a message producer establishes network connection to a message service through an SDK provided by the multi-cloud distributed message queue service, and pushes a generated message instance to a specified message queue in the multi-cloud distributed message queue service through the connection; message interpretation routing: the multi-cloud distributed message queue service explains the pushed messages, performs message routing according to related field information in a message header, and pushes the related messages to message consumers on different message models.

In a preferred embodiment, the message header comprises: the message queue name, the message authentication information, whether the message is persistent, the message sender, the message receiver, the message type, the creation time and the message version number; distinguishing different message models according to different message header information;

the message interpretation routing comprises the steps that the multi-cloud distributed message queue service disassembles the pushed message to obtain message header information of the message, searches a message queue created currently according to the message queue name in the message header, finds out the current queue, compares whether the consumption type of the message header is consistent with the attribute of the message queue, confirms whether a current message producer or a message consumer has authority to operate the message queue according to the message authentication information of the message header, determines one-to-one or one-to-many correlation according to a message model, and establishes a one-to-one or one-to-many message consumption model.

In a preferred embodiment, the message entry queue further comprises a message acknowledgement: if a message confirmation instruction or an enabled message confirmation mechanism is received, the multi-cloud distributed message service receives a message normally consumed instruction fed back by a message consumer, and feeds back the received message to a message producer, wherein the message is normally received.

In a preferred embodiment, the message entry queue further comprises a message persistence: if the message persistence is enabled or a message persistence instruction is received when the message queue is created, the multi-cloud distributed message queue service performs data persistence on each piece of received information on the queue.

In a preferred embodiment, the message model comprises: one or more of a point-to-point model, a publish-subscribe model, a remote invocation model, a broadcast model, and a work queue model;

the message queuing step further comprises: operating according to different message models;

if the message model is a remote calling model, the multi-cloud distributed message queue service can generate a unique ID for a message producer, and when a plurality of associated message producers exchange messages with the message consumers at the same time, the message consumer reply messages are sent to the correct message producer according to the associated IDs;

if the message model is a publish-subscribe model, receiving a subscription instruction, performing multi-queue delivery according to the number of subscribed message consumers, subscribing topic messages according to different message queue names, and controlling to send messages of corresponding subscription topics to the message consumers according to the message queue names;

if the consumption model is a working queue model, different messages are taken out one by one in the consumption queue and distributed to message consumers, and the same message is not consumed by different consumers for the second time.

In a preferred embodiment, further comprising an interconnect: and setting message service intermediate stations in different areas, deploying the message service intermediate stations in each area, interconnecting the message service intermediate stations, and communicating a message producer with consumers in other areas through the message service intermediate station directly accessing the area.

In a preferred embodiment, further comprising a nearby access: deploying message queue clusters in each public cloud, accessing the message queue service nearby, and deploying the message queue clusters in different clouds to access by using the same API (application programming interface);

the message tail includes: a message digest algorithm, a digest value and a check value;

the message queue attributes include: queue name, queue persistence.

A high performance computing multi-cloud oriented distributed messaging system, comprising:

a node server: applying for a plurality of node servers in each cloud, deploying clusters, and respectively creating access entries of cluster services; message queue central controller: deploying a message queue central controller, and configuring an access entrance of each cluster service to the message queue central controller;

the message queue center controller comprises:

a production message module: according to the using scene, creating a message model, acquiring information to be transmitted, filling corresponding information according to the information to be transmitted and the message model according to the requirement of a message template format, and generating a message example, wherein the message template comprises: a message header, a message carrier and a message tail;

the message entry queue module: a message producer establishes network connection to a message service through an SDK provided by the multi-cloud distributed message queue service, and pushes a generated message instance to a specified message queue in the multi-cloud distributed message queue service through the connection;

message interpretation routing module: the multi-cloud distributed message queue service explains the pushed messages, performs message routing according to related field information in a message header, and pushes the related messages to message consumers on different message models.

In a preferred embodiment, the message header comprises: the message queue name, the message authentication information, whether the message is persistent, the message sender, the message receiver, the message type, the creation time and the message version number; distinguishing different message models according to different message header information;

the message interpretation routing module also comprises a multi-cloud distributed message queue service, a message header information acquisition module, a message queue management module and a message generation module, wherein the multi-cloud distributed message queue service disassembles the pushed message to obtain the message header information of the message, searches the currently created message queue according to the message queue name in the message header, finds the current queue, compares whether the consumption type of the message header is consistent with the attribute of the message queue, confirms whether a current message producer or a message consumer has the authority to operate the message queue according to the message authentication information of the message header, determines the one-to-one or one-to-many correlation according to the message model, and establishes a one-to-one or one-to-many message consumption model;

the message entering queue module further comprises a message confirmation unit: if a message confirmation instruction or a started message confirmation mechanism is received, the multi-cloud distributed message service receives a message normally consumed instruction fed back by a message consumer, and feeds back the received message to a message producer, wherein the message is normally received;

and a message persistence unit: if the message persistence is enabled or a message persistence instruction is received when the message queue is created, the multi-cloud distributed message queue service performs data persistence on each piece of received information on the queue.

In a preferred embodiment, further comprising: an interconnection module: setting message service intermediate stations in different areas, deploying the message service intermediate stations in each area, interconnecting the message service intermediate stations, and communicating a message producer with consumers in other areas through the message service intermediate station directly accessing the area;

and a nearby access module: message queue clusters are deployed in the public clouds, message queue services are accessed nearby, and the same API (application programming interface) interfaces are used for deploying the message queue clusters in different clouds.

According to the distributed message transmission method and system for the high-performance computing multi-cloud, the message queue faces to the multi-cloud high-performance computing message transmission format definition and the multi-message model support, so that the high-performance computing development and integration on the cloud are more efficient and convenient, the distributed message service on the multi-cloud is constructed, and the distributed message service can be operated on computing resources, can realize cross-region message transmission on a high-performance execution program and does not need to care about the real network position where the program is currently located. The complexity of cross-region and heterogeneous networks is shielded by the distributed message service, and developers only need to focus on the realization of the calculation workflow and the upper-layer business through a uniform distributed message service access interface.

The invention realizes the nearby access of the message queue service by deploying the message queue cluster in each large public cloud manufacturer. The same API interface is used for accessing the message queue clusters deployed in different cloud manufacturers, the difficulty of service integration is reduced, and developers do not need to integrate according to different cloud manufacturers. The natural global distributed deployment of the multi-cloud distributed message queue service has extremely high service availability.

Drawings

Fig. 1 is a partial flowchart of a distributed messaging method for high performance computing multi-cloud oriented according to an embodiment of the present invention; fig. 2 is a schematic diagram of a message model according to an embodiment of the present invention.

Detailed Description

By the distributed message service constructed on multiple clouds, the invention can realize cross-region message transmission by executing the program on the computing resources at high performance without concerning the real network position of the program. The complexity of cross-region and heterogeneous networks is shielded by the distributed message service, and developers only need to focus on the realization of the calculation workflow and the upper-layer business through a uniform distributed message service access interface.

The design of the distributed message transmission method in the cloud environment needs to consider the following factors:

1. transmission delay and stability under a real network environment across regions;

2. the protocol of distributed message transmission and the format definition thereof should be concise and meet the requirement of running a computing task.

3. And supporting a distributed message multi-model, wherein the supporting of the message model depends on the use requirement under the current multi-cloud computing environment. Usually, a message queue is a container for storing messages, and only messages are taken out from the message queue for use when the messages are needed. The message queue is an important component in a distributed system, and the high-performance computing development and integration on the cloud is more efficient and convenient due to the message transmission format definition oriented to the multi-cloud high-performance computing and the support of a multi-message model.

The invention provides a distributed multi-message transfer method suitable for a multi-cloud high-performance computing scene, which is used for establishing an information data transfer mechanism between tasks and service systems in a multi-cloud high-performance computing platform, shielding the complexity of cross-region network communication and providing reference realization for establishing a communication model on a multi-cloud-oriented high-performance computing environment.

As shown in fig. 1, a distributed message delivery method for high performance computing multiple clouds according to an embodiment of the present invention includes: step S101, producing a message: according to the using scene, creating a message model, acquiring information to be transmitted, filling corresponding information according to the information to be transmitted and the message model according to the requirement of a message template format, and generating a message example, wherein the message template comprises: a message header, a message carrier and a message tail;

step S103, the message enters a queue: a message producer establishes network connection to a message service through an SDK provided by the multi-cloud distributed message queue service, and pushes a generated message instance to a specified message queue in the multi-cloud distributed message queue service through the connection;

step S105, message interpretation routing: the multi-cloud distributed message queue service explains the pushed messages, performs message routing according to related field information in a message header, and pushes the related messages to message consumers on different message models.

The multi-cloud distributed message queue service accepts and forwards messages. It can be considered as a post office: when the mail to be issued is placed in the mailbox, it can be ensured that the mail is finally delivered to the recipient. By analogy, the distributed message queue service is a mailbox, a post office and a postman. The main difference between the distributed message queue service and the post office is that it does not process paper, but receives, stores and forwards binary data messages.

As shown in fig. 2, the message model of the present embodiment includes: one or more of a point-to-point model, a publish-subscribe model, a remote invocation model, a broadcast model, a work queue model.

The message producer can be a producer, a sender, a Client, etc. according to different message models, as long as the functions of the message producer of the present invention are realized.

The message consumer can be a consumer, a receiver, a Server and the like according to different message models or different using scenes, as long as the functions of the message consumer are realized.

If the message model is a point-to-point model: the producer produces and sends messages to the message queue, while the consumer consumes messages from the message queue.

And if the message model is a publish-subscribe model, receiving a subscription instruction, performing multi-queue delivery according to the number of subscribed message consumers, subscribing the subject message according to different message queue names, and controlling to send the message of the corresponding subscription subject to the message consumers according to the message queue names.

In the publish-subscribe model, the consumer a and the consumer B subscribe to messages of different topic types through message queues respectively, such as messages with a topic of a log type and messages with a topic of a monitoring type. Through the publish-subscribe model, message consumers only subscribe to message topics of interest to them, and messages for other topics are ignored.

If the message model is a remote calling model, the multi-cloud distributed message queue service can generate a unique ID for the message producer, and when a plurality of associated message producers exchange messages with the message consumers at the same time, the message consumer reply messages are sent to the correct message producer according to the associated IDs.

The remote calling model is different from other message models in that the remote calling model is a response model, and the remote calling model is suitable for scenes with extremely low tolerance of message loss rate and extremely high message transmission reliability, so that a message sending end needs to wait for blocking after sending the message, and a signal is returned by a waiting server end after consuming the message to tell the sending end that the message is safely transmitted.

If the message model is a broadcast model, the message queue under the broadcast model supports the same message to be consumed for multiple times, the sender sends the message to the broadcast message queue, and both the receiver A and the receiver B can receive the message, so that the mechanism of consumption broadcast is simulated.

If the consumption model is a working queue model, different messages are taken out one by one in the consumption queue and distributed to message consumers, and the same message is not consumed by different consumers for the second time.

The work queue model is similar to the broadcast model implementation mechanism, except that the same message cannot be received by different consumers, who will take different messages from the message queue one by one.

The message producer can establish a network connection to the message service through an SDK (software Development kit) provided by the multi-cloud distributed message service, wherein the network connection can be established in or before the message production step, and can also be established in the message queue entering step, as long as the generated message instance is pushed to a specified message queue in the multi-cloud distributed message queue service through the connection.

Further, a message is produced, a message producer creates a network connection to the message service through an SDK (software Development kit) provided by the multi-cloud distributed message queue service, and fills corresponding fields according to the requirements of the message template format to generate a message instance.

Further, the messages enter a queue, after a message producer establishes network connection to the message service through an SDK provided by the multi-cloud distributed message queue service, the generated message instance is pushed to a designated message queue in the message queue service through the connection, and operation is performed according to different message models.

Further, the message is interpreted and routed, the multi-cloud distributed message service interprets the pushed message, and performs message routing according to related field information in a message header, and pushes the related message to message consumers on different message models.

The message queue attributes include: queue name, queue persistence.

Since the multi-cloud distributed message queue service serves as an intermediate depositor of messages, message confirmation is required for push receiving and forwarding of the messages to consumers so as to ensure the communication of the messages.

The message entry queue further comprises a message acknowledgement: if a message confirmation instruction or an enabled message confirmation mechanism is received, the multi-cloud distributed message service receives a message normally consumed instruction fed back by a message consumer, and feeds back the received message to a message producer, wherein the message is normally received.

The message entry queue also includes message persistence: if the message persistence is enabled or a message persistence instruction is received when the message queue is created, the multi-cloud distributed message queue service performs data persistence on each piece of received information on the queue.

In a distributed environment, due to the characteristics of a cross-regional network, network interruption, service failure and the like, messages may be lost in a transmission process, so that persistence is necessary for services with high reliability requirements, but due to the fact that I/O (Input/Output) operations of third-party storage or local hard disk storage are involved in message persistence, certain transfer efficiency is necessarily reduced, and therefore the message persistence characteristics are an option in a message model. When a persistence option is enabled during message queue creation, the multi-cloud distributed message service may refer to data persistence of each received message on the queue to prevent loss of messages stored in a computer memory due to service interruption. Through the persistence mechanism, the multi-cloud distributed message service can recover original message data in the message queue from the persistent data service in the interrupt restart process.

The default use in the transnational message transmission process is public network transmission, and if enterprises have already constructed transnational private line channels, the distributed message queue service can also carry out message transmission through private lines. The delivery stability and privacy of private line messages may be better, but also care needs to be taken whether the bandwidth of the private line is sufficient.

And the message header and the message model are distinguished by using the message header. The message header contains the following fields: message queue name, message authentication information, whether a message is persistent, message sender, message receiver, message type, creation time, message version number.

A message carrier storing the actual data content of the message.

The message tail, which mainly ensures the integrity of data in the message transmission process, comprises the following fields: message digest algorithm, digest value, check value.

The message interpretation routing comprises the steps that the multi-cloud distributed message queue service disassembles the pushed message to obtain message header information of the message, searches a message queue created currently according to the message queue name in the message header, finds out the current queue, compares whether the consumption type of the message header is consistent with the attribute of the message queue, confirms whether a current message producer or a message consumer has authority to operate the message queue according to the message authentication information of the message header, determines one-to-one or one-to-many correlation according to a message model, and establishes a one-to-one or one-to-many message consumption model.

Further, the distributed message delivery method for high-performance computing multiple clouds of the embodiment further includes interconnection: and setting message service intermediate stations in different areas, deploying the message service intermediate stations in each area, interconnecting the message service intermediate stations, and communicating a message producer with consumers in other areas through the message service intermediate station directly accessing the area.

Further, the distributed message delivery method facing high performance computing multiple clouds of the embodiment further includes access nearby: message queue clusters are deployed in the public clouds, message queue services are accessed nearby, and the same API (application programming interface) interfaces are used for deploying the message queue clusters in different clouds.

The invention realizes the nearby access of the message queue service by deploying the message queue cluster in each large public cloud manufacturer. The same API interface is used for accessing the message queue clusters deployed in different cloud manufacturers, the difficulty of service integration is reduced, and developers do not need to integrate according to different cloud manufacturers. The natural global distributed deployment of the multi-cloud distributed message queue service has extremely high service availability.

The invention relates to a distributed message transmission system facing high-performance computing multi-cloud, which comprises:

a node server: applying for a plurality of node servers in each cloud, deploying clusters, and respectively creating access entries of cluster services; message queue central controller: and deploying a message queue central controller, and configuring an access entry of each cluster service to the message queue central controller.

The message queue center controller of the present invention comprises:

a production message module: according to the using scene, creating a message model, acquiring information to be transmitted, filling corresponding information according to the information to be transmitted and the message model according to the requirement of a message template format, and generating a message example, wherein the message template comprises: a message header, a message carrier and a message tail;

the message entry queue module: a message producer establishes network connection to a message service through an SDK provided by the multi-cloud distributed message queue service, and pushes a generated message instance to a specified message queue in the multi-cloud distributed message queue service through the connection;

message interpretation routing module: the multi-cloud distributed message queue service explains the pushed messages, performs message routing according to related field information in a message header, and pushes the related messages to message consumers on different message models.

High Performance Computing (HPC): refers to the utilization of aggregated computing power to process data intensive computing tasks that standard workstations cannot accomplish, including modeling, simulation, rendering, and the like.

High performance computing on the cloud: the method is characterized in that a traditional high-performance computing cluster is moved to the cloud, charging is expanded on the cloud according to needs, heterogeneous computing resources are diversified, and the cluster can be constructed across the globe.

Cloud: the method refers to hybrid integration of a plurality of public clouds to enlarge the scale of computing resources.

Distributed system: a distributed system is a loosely coupled system of multiple processors interconnected by communication lines. From the perspective of a processor in the system, the remaining processors and corresponding resources are remote and local to itself.

Message queue: a "message" is a unit of data that is transferred between two computers. The message may be very simple, e.g. containing only text strings; or may be more complex, possibly containing embedded objects. The message is sent to a queue. A "message queue" is a container that holds messages during their transmission. The message queue manager acts as a man-in-the-middle in relaying a message from its source to its destination. The main purpose of the queues is to provide routing and guarantee delivery of messages; if the recipient is not available when the message is sent, the message queue will hold the message until it can be successfully delivered.

Leader Node (Leader Node): the message queue deployed by a single node has the risk of single point failure, that is, if the message queue service deployed on the single node fails, the whole message queue service cannot be used, and the message sending and receiving of a message producer and a message receiver are directly influenced. The message queue deployed by a plurality of nodes solves the problem of single-point failure on a large program, a leader node needs to be elected for maintaining message transmission management under the plurality of nodes, and a producer and a messenger communicate with the leader node.

Kubernetes, K8s for short, is an abbreviation for 8 characters "ubernet" instead. The Kubernetes is an open source and used for managing containerized applications on a plurality of hosts in a cloud platform, aims to make the application of the containerization simple and efficient to deploy (powerfull), and provides a mechanism for deploying, planning, updating and maintaining the applications.

Rabbitmq is an open source message broker software (also known as message-oriented middleware) that implements the Advanced Message Queuing Protocol (AMQP).

The system of the present invention comprises: the method comprises the steps of distributed deployment design in a multi-cloud environment, design of a message model, design of a message template format, a multi-cloud distributed message transmission process and persistence of messages.

The invention relates to the field of high-performance computing multi-cloud distributed message queue service. The design goal of the distributed multi-message queue service is to meet the data interaction requirements of tasks and services in a high-performance computing environment constructed by multiple clouds and solve the problem of cross-region complex networks caused by the multi-cloud environment. The reliability of message transmission delay is improved, and the complexity of cross-region communication programming is reduced.

Further, the message header of the present embodiment includes: message queue name, message authentication information, whether a message is persistent, message sender, message receiver, message type, creation time, message version number. And distinguishing different message models according to different message header information.

The message interpretation routing module further comprises a multi-cloud distributed message queue service which disassembles the pushed message to obtain message header information of the message, searches a message queue created currently according to the message queue name in the message header, finds the current queue, compares whether the consumption type of the message header is consistent with the attribute of the message queue, confirms whether a current message producer or a message consumer has the authority to operate the message queue according to the message authentication information of the message header, determines one-to-one or one-to-many correlation according to the message model, and establishes one-to-one or one-to-many message consumption model.

Further, the message entry queue module of this embodiment further includes a message confirmation unit: if a message confirmation instruction or an enabled message confirmation mechanism is received, the multi-cloud distributed message service receives a message normally consumed instruction fed back by a message consumer, and feeds back the received message to a message producer, wherein the message is normally received.

Further, the message entry queue module of this embodiment further includes a message persistence unit: if the message persistence is enabled or a message persistence instruction is received when the message queue is created, the multi-cloud distributed message queue service performs data persistence on each piece of received information on the queue.

The persistence writes the data in the message service memory (easy to lose when power is off) into a reliable storage that is not lost when power is off, such as a definite disk, a database, etc.

The cloud distributed message service has the possibility of crossing regions or countries, and the private line generally refers to a private network constructed by an enterprise and an operator, and the greatest difference from a public network is better security and stability. The distinction is not great for messaging service deployments, except that the producer and the messenger of a message use a private network address rather than a public network address when establishing a connection with the messaging service. The private network address is forwarded by the gateway route in the enterprise to the local area network or private network.

Further, the distributed messaging system for high performance computing multiple clouds of the embodiment further includes: an interconnection module: and setting message service intermediate stations in different areas, deploying the message service intermediate stations in each area, interconnecting the message service intermediate stations, and communicating a message producer with consumers in other areas through the message service intermediate station directly accessing the area.

Further, the distributed messaging system for high performance computing multiple clouds of the embodiment further includes: a nearby access module: message queue clusters are deployed in the public clouds, message queue services are accessed nearby, and the same API (application programming interface) interfaces are used for deploying the message queue clusters in different clouds.

As shown in fig. 2, the message model of the present embodiment includes: one or more of a point-to-point model, a publish-subscribe model, a remote invocation model, a broadcast model, a work queue model.

The message producer can be a producer, a sender, a Client, etc. according to different message models, as long as the functions of the message producer of the present invention are realized.

The message consumer can be a consumer, a receiver, a Server and the like according to different message models or different using scenes, as long as the functions of the message consumer are realized.

If the message model is a point-to-point model: the producer produces and sends messages to the message queue, while the consumer consumes messages from the message queue.

And if the message model is a publish-subscribe model, receiving a subscription instruction, performing multi-queue delivery according to the number of subscribed message consumers, subscribing the subject message according to different message queue names, and controlling to send the message of the corresponding subscription subject to the message consumers according to the message queue names.

In the publish-subscribe model, the consumer a and the consumer B subscribe to messages of different topic types through message queues respectively, such as messages with a topic of a log type and messages with a topic of a monitoring type. Through the publish-subscribe model, message consumers only subscribe to the message topics they are interested in, and messages are ignored for other topics. In the publish-subscribe model, a message consumer subscribes to subject messages according to different message queue names, wherein if the queue name is:/log, all messages with/log prefix are received by the consumer, and if the queue name is:/log/warning, only the log with warning level is received by the consumer.

If the message model is a remote calling model, the multi-cloud distributed message queue service can generate a unique ID for the message producer, and when a plurality of associated message producers exchange messages with the message consumers at the same time, the message consumer reply messages are sent to the correct message producer according to the associated IDs.

The remote calling model is different from other message models in that the remote calling model is a response model, and the remote calling model is suitable for scenes with extremely low tolerance of message loss rate and extremely high message transmission reliability, so that a message sending end needs to wait for blocking after sending the message, and a signal is returned by a waiting server end after consuming the message to tell the sending end that the message is safely transmitted.

If the message model is a broadcast model, the message queue under the broadcast model supports the same message to be consumed for multiple times, the sender sends the message to the broadcast message queue, and both the receiver A and the receiver B can receive the message, so that the mechanism of consumption broadcast is simulated.

If the consumption model is a working queue model, different messages are taken out one by one in the consumption queue and distributed to message consumers, and the same message is not consumed by different consumers for the second time.

The work queue model is similar to the broadcast model implementation mechanism, except that the same message cannot be received by different consumers, who will take different messages from the message queue one by one.

In the embodiment, a message service intermediate station is deployed nearby in each area, and the message service intermediate stations are interconnected by using a public network or a private line. It appears to the message producer and consumer that the same message service is directly connected. The reliability of transmission mainly depends on distributed message service deployment, and simultaneously, each message service intermediate station realizes the mechanisms of message flow control, retransmission and the like.

Further, specifically, in this embodiment, 3 node servers are applied in each cloud, and the cluster management software may be Kubernetes. Deploying the RabbitMQ cluster through Kubernets, and respectively creating an access portal of the service.

Further, in this embodiment, the message queue center controller: and deploying a message queue center controller in the AWS Cloud center, and configuring an access inlet of each RabbitMQ cluster service into the message queue center controller.

Message queues for different message models:

the formalization of the multi-cloud distributed message Queue oriented to high-performance computing is represented as Queue, and if the message Model is Model: queue of Name, Model, Durable

Where the Name is the unique Name of the message queue.

Model types include: a point-to-point model, a publish-subscribe model, a remote invocation model, a broadcast model, a work queue model.

The Durable specifies whether the message queue initiates a persistence mechanism.

Message template format:

the high-performance computing-oriented multi-cloud distributed message formalization is represented as follows: if the Message is a Message, then

Message＝consist of{Message-Header,Message-Body,Message-Tail}

Message header

Message carrier

Message tail

The message header comprises the following components:

Message-Header＝consist of{Name,Auth,Durable,Sender,Receiver,Time,Version}

the Name is a message queue Name, Auth is message authentication information, the Durable identifies message persistence, Sender is a Sender identifier, Receiver is a Receiver identifier, Time is message creation Time, and Version is a message Version number.

Multi-cloud distributed messaging

The multi-cloud distributed messaging is summarized as the following several basic operations. Each operation takes a message as a parameter.

CreateQueue (Name, Model, Durable); a message queue creation operation.

Createssage (Message-Header, Message-Body, Message-Tail); a message creation operation.

Sendmessage (message); the message is sent to a queue operation.

ReceiveMessage (Queue-Name); and acquiring the message operation from the message queue.

SendBatchMessages ([ ] Message); and (5) sending messages in batches.

receiveBatchMessages (Queue-Name); and obtaining message operations in batches from the message queue.

The multi-cloud distributed message transmission method comprises the following steps:

according to different use scenarios, creating different message queue Model types by calling createQueue through a Mode parameter comprises the following steps: point-to-point model, publish-subscribe model, remote-call model, broadcast model, work queue model).

Prepare Message-Header, Message-Body, Message-Tail, call createmeessage create Message instance.

The message producer invokes a SendMessage or SendBatchMessages operation to add a message instance to a queue in the distributed message server.

The message consumer calls a ReceiveMessage or receiveBatchMessages to obtain a message instance from the specified message queue.

The following is a detailed description of a specific embodiment:

the message passing process between the sender and the receiver of the message is described below by taking Golang programming language as an example. The current distributed message queue base component takes RabbitMQ open source message middleware as an example, but the SDK and the message queue central controller in the invention are realized without depending on the system.

Definition of message header

Definition of message body

Message class definition

type CloudMessage struct{

messageHeader MessageHeader

messageBody MessageBody

messageTail MessageTail

}

func(msg*CloudMessage)CreateQueue(Name,Model,Durable){}

func(msg*CloudMessage)CreateMessage(Message-Header,Message-Body,Message-Tail){}

func(msg*CloudMessage)SendMessage(Message){}

func(msg*CloudMessage)ReceiveMessage(Queue-Name){}

func(msg*CloudMessage)SendBatchMessages([]Message){}

func(msg*CloudMessage)ReceiveBatchMessages(Queue-Name){}

A message transmission step:

creating message queues specifying message models, e.g. point-to-point model

createQueue ("hello", "point-to-point", true)

Generating message instances from message template definitions

Msg＝CloudMessage.CreateMessage(Message-Header,Message-Body,Message-Tail)()

Sending messages

CloudMessage.SendMessage(Msg)

Receiving messages

CloudMessage.ReceiveMessage(Queue.Name)

The invention arranges a message service intermediate station nearby in each area, and the message service intermediate stations are interconnected by using a public network or a private line. It appears to the message producer and consumer that the same message service is directly connected. The reliability of transmission mainly depends on distributed message service deployment, and simultaneously, each message service intermediate station realizes the mechanisms of message flow control, retransmission and the like. By deploying a message service intermediate station nearby in each area, the intermediate stations are interconnected by using a public network or a private line. It is possible for a message producer to communicate with consumers in other areas by directly accessing the messaging service in the area in which it is located.

The invention realizes the nearby access of the message queue service by deploying the message queue cluster in each large public cloud manufacturer. The same API interface is used for accessing the message queue clusters deployed in different cloud manufacturers, the difficulty of service integration is reduced, and developers do not need to integrate according to different cloud manufacturers. The natural global distributed deployment of the multi-cloud distributed message queue service has extremely high service availability.

The invention solves the problems that the message service provided by the existing public cloud can not simultaneously meet various message models integrated by the invention, and API interfaces used by various clouds are inconsistent, thus being unfriendly to multi-cloud integration development. And the message service provided by the single public cloud is more prone to serve the service in the single cloud, and the reliability of the cross-cloud and cross-region communication delay can be guaranteed.

In light of the foregoing description of the preferred embodiments according to the present application, it is to be understood that various changes and modifications may be made without departing from the spirit and scope of the invention. The technical scope of the present application is not limited to the contents of the specification, and must be determined according to the scope of the claims.

As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

Claims (10)

1. A distributed message transmission method facing high-performance computing multi-cloud is characterized by comprising the following steps:

producing messages: according to the using scene, creating a message model, acquiring information to be transmitted, filling corresponding information according to the information to be transmitted and the message model according to the requirement of a message template format, and generating a message example, wherein the message template comprises: a message header, a message carrier and a message tail;

the message enters the queue: a message producer establishes network connection to a message service through an SDK provided by the multi-cloud distributed message queue service, and pushes a generated message instance to a specified message queue in the multi-cloud distributed message queue service through the connection;

message interpretation routing: the multi-cloud distributed message queue service explains the pushed messages, performs message routing according to related field information in a message header, and pushes the related messages to message consumers on different message models.

2. The high-performance computing multi-cloud oriented distributed messaging method of claim 1, wherein the message header comprises: the message queue name, the message authentication information, whether the message is persistent, the message sender, the message receiver, the message type, the creation time and the message version number; distinguishing different message models according to different message header information;

the message interpretation routing comprises the steps that the multi-cloud distributed message queue service disassembles the pushed message to obtain message header information of the message, searches a message queue created currently according to the message queue name in the message header, finds out the current queue, compares whether the consumption type of the message header is consistent with the attribute of the message queue, confirms whether a current message producer or a message consumer has authority to operate the message queue according to the message authentication information of the message header, determines one-to-one or one-to-many correlation according to a message model, and establishes a one-to-one or one-to-many message consumption model.

3. The high-performance computing multi-cloud oriented distributed messaging method of claim 1, wherein the message entry queue further comprises a message acknowledgement: if a message confirmation instruction or an enabled message confirmation mechanism is received, the multi-cloud distributed message service receives a message normally consumed instruction fed back by a message consumer, and feeds back the received message to a message producer, wherein the message is normally received.

4. The high-performance computing multi-cloud oriented distributed messaging method of claim 1, wherein the message entry queue further comprises message persistence: if the message persistence is enabled or a message persistence instruction is received when the message queue is created, the multi-cloud distributed message queue service performs data persistence on each piece of received information on the queue.

5. The high-performance computing multi-cloud oriented distributed messaging method according to any one of claims 1 to 4, wherein the message model comprises: one or more of a point-to-point model, a publish-subscribe model, a remote invocation model, a broadcast model, and a work queue model;

the message queuing step further comprises: operating according to different message models;

if the message model is a remote calling model, the multi-cloud distributed message queue service can generate a unique ID for a message producer, and when a plurality of associated message producers exchange messages with the message consumers at the same time, the message consumer reply messages are sent to the correct message producer according to the associated IDs;

if the message model is a publish-subscribe model, receiving a subscription instruction, performing multi-queue delivery according to the number of subscribed message consumers, subscribing topic messages according to different message queue names, and controlling to send messages of corresponding subscription topics to the message consumers according to the message queue names;

if the consumption model is a working queue model, different messages are taken out one by one in the consumption queue and distributed to message consumers, and the same message is not consumed by different consumers for the second time.

6. The distributed messaging method for high performance computing multi-cloud oriented according to any one of claims 1 to 4, further comprising interconnecting: and setting message service intermediate stations in different areas, deploying the message service intermediate stations in each area, interconnecting the message service intermediate stations, and communicating a message producer with consumers in other areas through the message service intermediate station directly accessing the area.

7. The high performance computing multi-cloud oriented distributed messaging method of claim 2, further comprising nearby access: deploying message queue clusters in each public cloud, accessing the message queue service nearby, and deploying the message queue clusters in different clouds to access by using the same API (application programming interface);

the message tail includes: a message digest algorithm, a digest value and a check value;

the message queue attributes include: queue name, queue persistence.

8. A distributed messaging system oriented to high performance computing clouds, comprising:

a node server: applying for a plurality of node servers in each cloud, deploying clusters, and respectively creating access entries of cluster services;

message queue central controller: deploying a message queue central controller, and configuring an access entrance of each cluster service to the message queue central controller;

the message queue center controller comprises:

a production message module: according to the using scene, creating a message model, acquiring information to be transmitted, filling corresponding information according to the information to be transmitted and the message model according to the requirement of a message template format, and generating a message example, wherein the message template comprises: a message header, a message carrier and a message tail;

the message entry queue module: a message producer establishes network connection to a message service through an SDK provided by the multi-cloud distributed message queue service, and pushes a generated message instance to a specified message queue in the multi-cloud distributed message queue service through the connection;

message interpretation routing module: the multi-cloud distributed message queue service explains the pushed messages, performs message routing according to related field information in a message header, and pushes the related messages to message consumers on different message models.

9. The high performance computing multi-cloud oriented distributed messaging system of claim 8, wherein the message header comprises: the message queue name, the message authentication information, whether the message is persistent, the message sender, the message receiver, the message type, the creation time and the message version number; distinguishing different message models according to different message header information;

the message interpretation routing module also comprises a multi-cloud distributed message queue service, a message header information acquisition module, a message queue management module and a message generation module, wherein the multi-cloud distributed message queue service disassembles the pushed message to obtain the message header information of the message, searches the currently created message queue according to the message queue name in the message header, finds the current queue, compares whether the consumption type of the message header is consistent with the attribute of the message queue, confirms whether a current message producer or a message consumer has the authority to operate the message queue according to the message authentication information of the message header, determines the one-to-one or one-to-many correlation according to the message model, and establishes a one-to-one or one-to-many message consumption model;

the message entering queue module further comprises a message confirmation unit: if a message confirmation instruction or a started message confirmation mechanism is received, the multi-cloud distributed message service receives a message normally consumed instruction fed back by a message consumer, and feeds back the received message to a message producer, wherein the message is normally received;

and a message persistence unit: if the message persistence is enabled or a message persistence instruction is received when the message queue is created, the multi-cloud distributed message queue service performs data persistence on each piece of received information on the queue.

10. The high performance computing multi-cloud oriented distributed messaging system according to claim 8 or 9, further comprising: an interconnection module: setting message service intermediate stations in different areas, deploying the message service intermediate stations in each area, interconnecting the message service intermediate stations, and communicating a message producer with consumers in other areas through the message service intermediate station directly accessing the area;

and a nearby access module: message queue clusters are deployed in the public clouds, message queue services are accessed nearby, and the same API (application programming interface) interfaces are used for deploying the message queue clusters in different clouds.

Images