CN116938850A

CN116938850A - SDTP message analysis method and device and electronic equipment

Info

Publication number: CN116938850A
Application number: CN202210331651.3A
Authority: CN
Inventors: 高东生
Original assignee: China Mobile Communications Group Co Ltd; China Mobile Group Liaoning Co Ltd
Current assignee: China Mobile Communications Group Co Ltd; China Mobile Group Liaoning Co Ltd
Priority date: 2022-03-31
Filing date: 2022-03-31
Publication date: 2023-10-24

Abstract

The application discloses a shared data transmission protocol SDTP message analysis method, a device and electronic equipment. The SDTP message analysis method comprises the following steps: receiving a plurality of SDTP messages sent by a message sending terminal based on a Shared Data Transmission Protocol (SDTP); caching a plurality of SDTP messages by a data buffering component; acquiring a cached first SDTP message through a first processing node; registering the association relationship between the first SDTP message and the first processing node in a memory pool through the first processing node under the condition that the first SDTP message is fragment data and is not a data packet header; determining, by the first processing node, a second SDTP message required for the first SDTP message, in the case that the first SDTP message is fragmented data and is a data packet header; searching a second processing node associated with a second SDTP message in the registered association relation in the memory pool; and acquiring a second SDTP message from the second processing node and splicing. According to the embodiment of the application, the SDTP fragment data can be quickly searched and synthesized.

Description

SDTP message analysis method and device and electronic equipment

Technical Field

The present application belongs to the field of communication technologies, and in particular, to a method, an apparatus, and an electronic device for resolving an SDTP message.

Background

At present, a mobile communication system is constructed based on a function layering idea, all interface data of a user plane and a signaling plane are collected from core network equipment at a bottom layer by a unified deep packet inspection (Deep Packet Inspection, DPI) system, then data verification, backfilling and synthesis are carried out by a synthesis layer, and finally the fragmented data is sent to an application system or a receiving application system in real time by adopting a shared data transmission protocol (Shared Data Transfer Protocol, SDTP) protocol.

After receiving the fragmented data transmitted by adopting the SDTP protocol, the application system or the receiving application system needs to splice according to the content of each SDTP message, and commonly adopts an analysis method to write a multithreaded processing program for Java language or adopts a large data acquisition component such as a Flume to develop, however, the methods have certain limitations. The utilization rate of the program hardware resources developed by adopting Java language is not high, and if the integrated structure design of the receiving and distributing model is adopted, the original data is lost when the pressure of the processing module is too high. The jump is an acquisition tool of a big data platform, adopts a distributed architecture, but aiming at complex data warehouse technology (ETL) processing, complex configuration is needed, a user cannot dynamically control scheduling of bottom hardware resources, the utilization rate of the hardware resources is low, and when encountering sudden increase of data volume or failure of analysis nodes, the data is extremely easy to permanently lose.

Disclosure of Invention

The embodiment of the application provides a method and a device for resolving SDTP messages and electronic equipment, which can quickly find and synthesize SDTP fragmented data.

In a first aspect, an embodiment of the present application provides a method for resolving an SDTP message of a shared data transmission protocol, where the method includes:

receiving a plurality of SDTP messages sent by a message sending terminal based on a Shared Data Transmission Protocol (SDTP);

caching a plurality of SDTP messages by a data buffering component;

acquiring a cached first SDTP message through a first processing node;

registering the association relationship between the first SDTP message and the first processing node in a memory pool through the first processing node under the condition that the first SDTP message is fragment data and is not a data packet header;

determining, by the first processing node, a second SDTP message required for the first SDTP message, in the case that the first SDTP message is fragmented data and is a data packet header; searching a second processing node associated with a second SDTP message in the registered association relation in the memory pool; and acquiring a second SDTP message from the second processing node and splicing.

Optionally, in a case where the first SDTP message is a fragment data and is a packet header, and in a case where the second SDTP message required for the first SDTP message is a plurality of the first SDTP messages, the method further includes:

Acquiring corresponding second SDTP messages from second processing nodes corresponding to each second SDTP message;

and splicing the first SDTP message and a plurality of second SDTP messages required by the first SDTP message to obtain a complete data packet.

Optionally, registering, by the first processing node, the association relationship between the first SDTP message and the first processing node in the memory pool includes:

establishing a first association relationship between the number of the receiving relationship of the first SDTP message and the number of the first processing node; the receiving relation number is used as a key word used by a third processing node for searching a first processing node associated with the first SDTP message based on a third SDTP message, wherein the third SDTP message is a data packet header corresponding to the first SDTP message;

and registering the first association relation in the memory pool.

Optionally, the SDTP message is fragmented data of a ticket; the receiving relation number comprises a receiving server number and a receiving ticket serial number; the third processing node searches for the first processing node associated with the first SDTP message, and comprises:

adding a preset offset to the received ticket serial number of the third SDTP message to obtain the received ticket serial number of the first SDTP message;

determining that the number of the receiving server of the first SDTP message is the number of the receiving server of the third SDTP message;

And searching the corresponding number of the first processing node in the registered association relation in the memory pool according to the number of the receiving server of the first SDTP message and the number of the receiving ticket serial number.

Optionally, before the cached first SDTP message is obtained by the first processing node, the method further comprises:

establishing a second association relationship between the buffer memory position of the first SDTP message and the receiving relationship number of the first SDTP message;

registering the second association relationship in the memory pool;

after obtaining the cached first SDTP message by the first processing node, further comprising:

and searching the receiving relation number of the first SDTP message in the registered association relation in the memory pool according to the cache position of the first SDTP message.

In a second aspect, an embodiment of the present application provides a shared data transmission protocol SDTP message parsing apparatus, where the apparatus includes:

a receiving unit, configured to receive a plurality of SDTP messages sent by a message sending end based on a shared data transmission protocol SDTP;

a buffer unit for buffering a plurality of SDTP messages through a data buffer component;

an obtaining unit, configured to obtain, by using a first processing node, a cached first SDTP message;

a registration unit, configured to register, by the first processing node, an association relationship between the first SDTP message and the first processing node in the memory pool, where the first SDTP message is fragment data and is not a packet header;

An execution unit, configured to determine, by the first processing node, a second SDTP message required for the first SDTP message, where the first SDTP message is fragment data and is a packet header; searching a second processing node associated with a second SDTP message in the registered association relation in the memory pool; and acquiring a second SDTP message from the second processing node and splicing.

Optionally, in the case that the first SDTP message is fragment data and is a packet header, and in the case that the second SDTP message required by the first SDTP message is a plurality of second SDTP messages, the obtaining unit is further configured to obtain a corresponding second SDTP message from a second processing node corresponding to each second SDTP message;

the device also comprises a splicing unit for splicing the first SDTP message and a plurality of second SDTP messages required by the first SDTP message to obtain a complete data packet.

Optionally, the registration unit includes:

the establishing subunit is used for establishing a first association relation between the number of the receiving relation of the first SDTP message and the number of the first processing node; the receiving relation number is used as a key word used by a third processing node for searching a first processing node associated with the first SDTP message based on a third SDTP message, wherein the third SDTP message is a data packet header corresponding to the first SDTP message;

And the registration subunit is used for registering the first association relation in the memory pool.

Optionally, the SDTP message is fragmented data of a ticket; the receiving relation number comprises a receiving server number and a receiving ticket serial number; the apparatus further comprises:

the calculating unit is used for increasing the received ticket serial number of the third SDTP message by a preset offset to obtain the received ticket serial number of the first SDTP message;

a determining unit, configured to determine that the number of the receiving server of the first SDTP message is the number of the receiving server of the third SDTP message;

the first searching unit is used for searching the corresponding number of the first processing node according to the number of the receiving server and the number of the receiving ticket serial number of the first SDTP message in the registered association relation in the memory pool.

Optionally, the apparatus further comprises:

the establishing unit is used for establishing a second association relationship between the buffer memory position of the first SDTP message and the receiving relationship number of the first SDTP message before the first buffered SDTP message is acquired through the first processing node;

the registration unit is also used for registering the second association relation in the memory pool;

and the second searching unit is used for searching the receiving relation number of the first SDTP message in the registered association relation in the memory pool according to the cache position of the first SDTP message after the first cached SDTP message is acquired through the first processing node.

In a third aspect, an embodiment of the present application provides an electronic device, including: a processor and a memory storing program instructions; the processor executes the program instructions to implement the shared data transfer protocol SDTP message parsing method according to the first aspect.

In a fourth aspect, an embodiment of the present application provides a readable storage medium, where program instructions are stored, where the program instructions, when executed by a processor, implement the method for parsing a SDTP message in a shared data transfer protocol according to the first aspect.

In a fifth aspect, an embodiment of the present application provides a program product, where instructions in the program product, when executed by a processor of an electronic device, enable the electronic device to perform the shared data transfer protocol SDTP message parsing method according to the first aspect.

According to the shared data transmission protocol SDTP message analysis method, device, electronic equipment, readable storage medium and program product, a message sending end receives a plurality of SDTP messages sent by a shared data transmission protocol SDTP, a data buffer component caches the plurality of SDTP messages, a first processing node obtains the cached first SDTP message, the first SDTP message is divided into pieces of data and is not a data packet header, the first processing node registers the association relation between the first SDTP message and the first processing node in a memory pool, and the first processing node determines a second SDTP message required by the first SDTP message when the first SDTP message is divided into pieces of data and is the data packet header; searching a second processing node associated with a second SDTP message in the registered association relation in the memory pool; and acquiring the second SDTP message from the second processing node and splicing the second SDTP message, so that the received SDTP message realizes the registration and the search of the association relationship in a shared memory pool mode, the efficiency of searching and synthesizing SDTP sliced data is improved, different sliced data can be processed through different processing nodes, and the utilization rate of hardware resources is improved.

Drawings

In order to more clearly illustrate the technical solution of the embodiments of the present application, the drawings that are needed to be used in the embodiments of the present application will be briefly described, and it is possible for a person skilled in the art to obtain other drawings according to these drawings without inventive effort.

Fig. 1 is a flow chart of a method for resolving SDTP messages of a shared data transmission protocol according to an embodiment of the present application;

fig. 2 is a system schematic diagram of a method for resolving SDTP messages in a shared data transmission protocol according to an embodiment of the present application;

fig. 3 is a flow chart of a method for resolving SDTP messages in a shared data transmission protocol according to another embodiment of the present application;

fig. 4 is a schematic diagram of a shared data transmission protocol SDTP message parsing method according to an embodiment of the present application;

fig. 5 is a schematic structural diagram of an SDTP message parsing device according to another embodiment of the present application;

fig. 6 is a schematic structural diagram of an electronic device according to another embodiment of the present application.

Detailed Description

Features and exemplary embodiments of various aspects of the present application will be described in detail below, and in order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be described in further detail below with reference to the accompanying drawings and the detailed embodiments. It should be understood that the particular embodiments described herein are meant to be illustrative of the application only and not limiting. It will be apparent to one skilled in the art that the present application may be practiced without some of these specific details. The following description of the embodiments is merely intended to provide a better understanding of the application by showing examples of the application.

It is noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises the element.

In order to solve the problems in the prior art, the embodiment of the application provides a shared data transmission protocol SDTP message parsing method, a device, equipment and a readable storage medium. The method for resolving the SDTP message provided by the embodiment of the present application is first described below.

Fig. 1 is a flow chart illustrating a method for resolving SDTP messages in a shared data transmission protocol according to an embodiment of the present application. As shown in fig. 1, the method comprises the steps of:

in step 401, the receiving message transmitting end transmits a plurality of SDTP messages based on the shared data transmission protocol SDTP.

The method for resolving an SDTP message provided in the embodiment of the present application may be performed by an SDTP receiving and resolving end, as shown in fig. 2, which is a system schematic diagram of the method for resolving an SDTP message provided in the embodiment of the present application, where the system mainly includes an SDTP sending end 101 and an SDTP receiving and resolving end 110, the SDTP sending end includes a network deep packet inspection (Deep Packet Inspection, DPI) segmentation and encapsulation processing module 1011 and a data distribution module 1012, and the SDTP receiving and resolving end 110 includes a unified configuration management and monitoring module 102, an original code stream receiving module 103, a data buffering component 104, a data resolving module 105 and a data distribution module 106.

The unified configuration management and monitoring module 102 is a core management module of the SDTP receiving and analyzing end 110, and can perform unified management on modules such as management, scheduling and monitoring of the whole system, and is an execution center of the whole system of the SDTP receiving and analyzing end 110, which is not only used for controlling operation parameters and start and stop of each component, but also is responsible for functions such as node capacity expansion and online monitoring, and when abnormal conditions occur in part of components, the abnormal conditions can be alerted to operation and maintenance personnel in a short message mode.

The SDTP message may be sent by the SDTP sender 101, and after the SDTP receiver and analyzer receives the SDTP message at 110, the SDTP message may be analyzed by executing the SDTP analysis method provided by the embodiment of the present application, so as to obtain complete data.

Alternatively, each SDTP message sent by the SDTP sender 101 may include a complete data packet, or may be fragmented for a complete data packet to obtain fragmented data, where a complete data packet is sent through multiple SDTP messages. The embodiment of the present application is not limited thereto.

In one example, one complete packet may be a complete ticket and the SDTP sender 101 may be a network DPI system. The network DPI system encapsulates various network element tickets that need to be transmitted in real time to the outside through a DPI fragmentation and encapsulation processing module 1011, and then sends binary code stream to the SDTP receiving and analyzing end 110 through a data distributing module 1012 by adopting a Socket communication mode.

Referring to fig. 2, the SDTP receiving and parsing end 110 may receive the binary code stream sent by the SDTP sending end 101 through the original code stream receiving module 103. That is, step 401 may be performed by the original code stream receiving module 103, wherein the received SDTP message is received in the form of a binary code stream.

Specifically, the original code stream receiving module 103 may be configured to connect with the SDTP sender 101. The unified configuration management and monitoring module 102 of the SDTP receiving and analyzing end 110 may configure servers of the original code stream receiving module 103, including the number of servers, the IP of the designated receiving server, and the number of channels started by each server, and may establish a corresponding relationship between each channel and the SDTP transmitting end 101.

Step 402, buffering a plurality of SDTP messages by a data buffering component.

After receiving the plurality of SDTP messages, the received messages may be buffered in real time in a message queue through a data buffering component. When receiving a plurality of SDTP messages, the SDTP messages can be received in the form of binary code streams, and accordingly, the binary code streams can be cached in a message queue by the data buffering component.

Referring to the system shown in fig. 2, the original code stream receiving module 103 may use a Socket interface to push data received in real time to the data buffering component 104. The data buffering component 104 may serve as a data buffering layer, so as to implement decoupling between the receiving logic and the processing logic, and specifically, the data buffering component 104 may adopt a message subscription mode to push the original code stream data to be processed to the message queue. Illustratively, the data buffering component 104 may be a Kafka system, and accordingly, raw code stream data is to be pushed to a Kafka message queue.

Step 403, obtaining, by the first processing node, the cached first SDTP message.

The processing node may be an electronic device for parsing and processing data. A system may include multiple processing nodes that perform tasks of data parsing in parallel. Referring to fig. 2, each processing node may include the data parsing module 105 shown in fig. 2, and the binary data code stream is fetched from the message queue by the data parsing module 105, so as to implement transcoding of the data code stream, and SDTP message splicing. Taking the first processing node as an example of extracting the cached first SDTP message from the cached message queue, the step of parsing the SDTP message is described, and the parsing and splicing of the SDTP message may be implemented in steps 404 and 405.

In step 404, in the case that the first SDTP message is a fragment data and is not a packet header, the association between the first SDTP message and the first processing node is registered in the memory pool by the first processing node.

If the first SDTP message is fragmented data and not a header, then the presence of a header requires the first SDTP message. In order to facilitate the processing node for processing the data packet header to find the required SDTP message, in the embodiment of the application, the association relation between the SDTP message of the non-data packet header and the corresponding processing node is registered (stored) in the memory pool, and the processing node for processing the data packet header conveniently and rapidly finds the processing node for the required SDTP message in a mode that all the processing nodes share the memory pool, so that the efficiency of obtaining the fragmented data and splicing is improved.

The association between the registered first SDTP message and the first processing node may be represented by a mapping between the unique identifier of the first SDTP message and the unique identifier of the first processing node. For example, registering, by the first processing node, the association of the first SDTP message with the first processing node in the memory pool may include the steps of:

step 4041, a first association between the number of the reception relationship of the first SDTP message and the number of the first processing node is established.

The receive relationship number is the number of the relationship that received the first SDTP message. For example, for ticket data, the receive relationship number may include a receive server number and a receive ticket serial number to uniquely identify the SDTP message.

The receiving relationship number may be used as a key used by the third processing node to find the first processing node associated with the first SDTP message based on a third SDTP message, where the third SDTP message is a packet header corresponding to the first SDTP message. That is, when the data packet header searches for the SDTP message required by the data packet header, the accept relationship number may be used as a key (key) during searching, and a value corresponding to the key is an identifier of the corresponding processing node.

In step 4042, the first association is registered in the memory pool.

All processing nodes may share a memory pool, and therefore, all associations stored in the memory pool may be queried by all processing nodes.

After registering the association, the first processing node may cache the first SDTP message locally, e.g., locally, at the first processing node, waiting for its corresponding packet header to be extracted.

Step 405, in the case that the first SDTP message is a fragment data and is a packet header, determining, by the first processing node, a second SDTP message required for the first SDTP message; searching a second processing node associated with a second SDTP message in the registered association relation in the memory pool; and acquiring a second SDTP message from the second processing node and splicing.

And in the case that the first SDTP message is fragment data and is a data packet header, the first SDTP message needs other fragment data, the unique identifier of other SDTP messages needed by the first SDTP message can be determined based on the format of the SDTP protocol, and the corresponding processing node is determined in the memory pool through the identifier of the SDTP message, so that the stored SDTP message is obtained in the corresponding processing node.

Optionally, in the case that the first SDTP message is fragment data and is a packet header, the second SDTP message required by the first SDTP message may be multiple, in this case, the corresponding second SDTP message may be obtained from the second processing node corresponding to each second SDTP message, and the first SDTP message and the multiple second SDTP messages required by the first SDTP message may be spliced to obtain a complete packet. That is, when the packet header needs multiple pieces of fragmented data, after all the required SDTP messages are acquired, the complete packet is spliced.

Referring to fig. 2, after the data parsing module 105 concatenates the generated complete data, the complete data may be pushed to the data buffering component 104 to be buffered, so that the complete data is in a ready state, and waiting for the data distribution module 106 to push the complete data to other storage media, such as hfs, kafka, relational databases, and the like. The data distribution module 106 is used as a final data output module and is responsible for pushing the complete data to other data sources according to the configured pushing strategy. In an alternative application scenario, the complete data may be a complete DPI ticket.

According to the shared data transmission protocol SDTP message analysis device, a message sending end is used for receiving a plurality of SDTP messages sent by a shared data transmission protocol SDTP, the plurality of SDTP messages are cached through a data buffer component, a first cached SDTP message is obtained through a first processing node, the association relation between the first SDTP message and the first processing node is registered in a memory pool through the first processing node when the first SDTP message is fragmented data and is not a data packet header, and a second SDTP message required by the first SDTP message is determined through the first processing node when the first SDTP message is fragmented data and is a data packet header; searching a second processing node associated with a second SDTP message in the registered association relation in the memory pool; and acquiring the second SDTP message from the second processing node and splicing the second SDTP message, so that the received SDTP message realizes the registration and the search of the association relationship in a shared memory pool mode, the efficiency of searching and synthesizing SDTP sliced data is improved, different sliced data can be processed through different processing nodes, and the utilization rate of hardware resources is improved.

In an alternative embodiment, the SDTP message is a fragmented data of a ticket, and accordingly, the receive relationship number may include a receive server number and a receive ticket serial number.

In this case, taking the third SDTP message currently processed by the third processing node as an example of a packet header, in the case that the third SDTP message needs the first SDTP message, the method for searching the first processing node associated with the first SDTP message by the third processing node may specifically include the following steps:

step 4051, adding the preset offset to the received ticket serial number of the third SDTP message to obtain the received ticket serial number of the first SDTP message. The preset offset may be 1, for example.

Step 4052, determining that the receiving server number of the first SDTP message is the receiving server number of the third SDTP message.

Step 4053, searching the corresponding number of the first processing node according to the number of the receiving server and the number of the receiving ticket serial number of the first SDTP message in the registered association relationship in the memory pool.

Fig. 3 is an alternative implementation of the SDTP message parsing method provided in fig. 1 according to an embodiment of the present application, in which the procedure is performed after step 402. In the application scenario, the content of the SDTP message is ticket data.

Referring to fig. 3, after the received data (binary stream of SDTP messages) is buffered in the message queue, the processing node may retrieve the data from the data buffering component 104 and the message queue shown in fig. 2, step 201 is performed. Alternatively, the data buffering component 104 may be a Kafka system, and when acquiring data, continuous transcoded call ticket data, that is, SDTP messages obtained after transcoding a binary code stream, may be extracted from a certain Broker (which may include one or more servers) of the Kafka system. Here, extracting continuous data may improve the efficiency of parsing.

After executing step 201, step 202 is executed to determine whether the acquired SDTP message is a wharf.

Specifically, the received data packet may include a packet header and an SDTP message, where the packet header includes: SDTP generic header, ethernet header, IP header and TCP header. The SDTP message includes: the SDTP message body comprises a packet head of one of at least two fragment packets and a packet body of each fragment packet.

After the original binary code stream is obtained, whether the data packet is a number header can be judged according to the information of the SDTP message header.

If not, the processing node may establish an association in the shared memory pool, step 203 is performed. Because the distribution of the data in the message queue and the distribution of the processing nodes are random, in order to improve the efficiency of the splicing link, the accurate association relation of the sliced data needs to be quickly positioned, the sliced data is pushed to the processing node where the data packet header is located, and then the processing node where the data packet header is located completes the splicing work.

Specifically, the association relationship between the processing node and the SDTP message may be stored in the form of a Key-Value Key Value pair, where the Key may include a receiving server number of the SDTP message+a receiving ticket serial number, and the Value may include a number of the processing node, optionally, an instance number of the processing node, so as to establish a correspondence between receiving and processing situations of the incomplete ticket.

In step 204, after the processing node registers the association relationship of the data packet of the non-number wharf in the memory pool, the data packet may be cached locally, and waiting for loading of the data packet header node file.

If it is determined in step 202 that the data packet is a header, step 205 may be further performed to determine whether fragmentation data is required.

If no additional fragmented data is needed, indicating that the SDTP message is a complete call ticket data packet, step 206 may be performed to encapsulate the data packet, and then push the encapsulated data to a data cache queue (step 207), specifically, to the message queue by the data cache component 104.

If other sliced data is needed, it is indicated that the SDTP message is not a complete call ticket packet, step 208 may be executed to extract the sliced data to the processing node where the sliced data is located according to the association relationship corresponding to the other sliced data stored in the memory pool.

Specifically, the current processing data packet serial number "+1" (preset offset is increased) can be used for obtaining the receiving ticket serial number of the next piece of data, and the Key of the required piece of data is constructed according to the receiving server serial number+the receiving ticket serial number, so that the processing host (processing node) in the memory is queried, and then the piece of data is obtained from the corresponding processing node through a Socket interface; and then repeatedly executing the flow of increasing the offset by the serial number, searching and acquiring until a complete ticket is formed.

Optionally, before the first processing node obtains the cached first SDTP message, an association relationship between a cache location of the first SDTP message and a receiving relationship number of the first SDTP message may be registered in the memory pool, so as to facilitate searching for the receiving relationship number of the first SDTP message when the receiving relationship number of the first SDTP message and the number of the processing node are established.

Specifically, the method for resolving the SDTP message provided by the embodiment of the present application may further include the following steps:

step 501, before the first processing node obtains the cached first SDTP message, a second association relationship between the cache location of the first SDTP message and the reception relationship number of the first SDTP message is established.

Step 502, registering the second association relationship in the memory pool.

In step 503, after the first processing node obtains the cached first SDTP message, the receiving relationship number of the first SDTP message may be searched for in the registered association relationship in the memory pool according to the cache location of the first SDTP message.

Thus, the receiving relation number of the first SDTP message can be rapidly determined according to the buffer memory position of the first SDTP message.

Optionally, the second association relationship may be regarded as a receiving cache relationship of the first SDTP message, and may specifically be in a Key-Value form. In one example, the Key may be a specific location of the Kafka message queue written by the first SDTP message, may be encoded by using a partition+offset (partition+offset) encoding manner, and Value may be set to be a receiving server encoding+a receiving ticket serial number.

Referring to fig. 4, which is a schematic diagram of an example of ticket stitching, the process may include the following steps:

step 1: processing node 1 (numbered DataNode 01) and processing node 2 (numbered DataNode 02) extract the SDTP message from the message queue for processing.

The information obtained by the processing node 2 comprises a data packet head, a p05 partition cached in a message queue, a serial number (offset) of 30000001, and a second association relationship established in a memory pool of [ p05-30000001, machinery 01-20000 ]).

The processing node 1 is the sliced data which does not comprise the data packet head, and is cached in the p02 partition, the serial number is 10000001, and the second association relation established in the memory pool is [ p02-10000001, machinery 01-20001 ]).

Step 2: after determining that the message is the fragment data, the processing node 1 registers the first association relationship in the database of the shared memory according to the information extracted from the message.

Specifically, step 2 may include the following two steps:

step 1) executing query operation, and using the cache address of the Kafka message queue as a Key, for example: p02-10000001, wherein p02 is a partition, 10000001 is a serial number, and related information of the piece of data is obtained in the memory pool, and the value is obtained as follows: machine01-20001. Wherein Machine01 is the number of the receiving server and 20001 is the number of the receiving ticket stream.

Step 2) establishing a Key-Value Key Value pair of a first association relation in a memory pool, wherein the Key is Machine01-20001 (receiving server number-receiving ticket serial number), and the Value is DataNode01 (processing node number).

Step 3: when the processing node 2 determines that the message is a data packet header and the fragmentation information is needed, the following two steps of operations are executed to obtain the fragmentation data:

Step 1) using p05-30000001 as a query condition in a memory pool, and searching to obtain a value of Machine01-20000.

Step 2) adding 1 to the received ticket serial number 20000 (i.e. adding a preset offset) to obtain 20001, and constructing a new Key: and inquiring the Machine01-20001 to obtain the corresponding processing node number as DataNode01.

Step 4: and acquiring SDTP information from the memory cache of the DataNode01 processing node by adopting a socket mode to the DataNode01 processing node, so as to realize the splicing work of the complete ticket.

The embodiment of the application can realize quick search and synthesis of the SDTP sliced data crossing the processing nodes by constructing the index based on the memory pool, is suitable for the condition of ultra-large scale data, can fully exert the resource efficiency of the hardware server, and saves a large number of hardware servers and matched electricity consumption and maintenance cost. And the hierarchical structure design of the data caching mechanism is applied to decouple the receiving process and the processing process of large data volume data, so that the fault tolerance and the expandability are improved.

Fig. 5 is a schematic structural diagram of an SDTP message parsing apparatus according to an embodiment of the present application. The shared data transmission protocol SDTP message analysis device provided by the embodiment of the application can be used for executing the shared data transmission protocol SDTP message analysis method provided by the embodiment of the application. In the part not described in detail in the embodiment of the SDTP message parsing device of the shared data transmission protocol provided in the embodiment of the present application, reference may be made to the description in the embodiment of the SDTP message parsing method of the shared data transmission protocol provided in the embodiment of the present application.

As shown in fig. 5, the shared data transmission protocol SDTP message parsing apparatus provided by the embodiment of the present application includes a receiving unit 11, a buffering unit 12, an obtaining unit 13, a registering unit 14 and an executing unit 15.

The receiving unit 11 is configured to receive a plurality of SDTP messages sent by the message sending end based on a shared data transmission protocol SDTP;

the caching unit 12 is configured to cache a plurality of SDTP messages through the data buffering component;

the acquiring unit 13 is configured to acquire, by using the first processing node, the cached first SDTP message;

the registration unit 14 is configured to register, in the memory pool, an association relationship between the first SDTP message and the first processing node, where the first SDTP message is fragment data and is not a packet header;

the execution unit 15 is configured to determine, by the first processing node, a second SDTP message required for the first SDTP message, in a case where the first SDTP message is fragment data and is a packet header; searching a second processing node associated with a second SDTP message in the registered association relation in the memory pool; and acquiring a second SDTP message from the second processing node and splicing.

Alternatively, in the case that the first SDTP message is a fragment data and is a packet header, and in the case that the second SDTP message required for the first SDTP message is a plurality of second SDTP messages, the obtaining unit 13 may be further configured to obtain a corresponding second SDTP message from a second processing node corresponding to each second SDTP message;

The apparatus may further include a concatenation unit configured to concatenate the first SDTP message and the plurality of second SDTP messages required by the first SDTP message to obtain a complete data packet.

Alternatively, the registration unit 14 may include:

Alternatively, the SDTP message may be fragmented data of a ticket; the receiving relationship number may include a receiving server number and a receiving ticket serial number; the apparatus may further include:

Optionally, the apparatus may further include:

the registration unit may be further configured to register the second association relationship in the memory pool;

Fig. 6 shows a schematic hardware structure of an electronic device according to an embodiment of the present application.

The electronic device may comprise a processor 301 and a memory 302 in which program instructions are stored.

In particular, the processor 301 may include a Central Processing Unit (CPU), or an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), or may be configured as one or more integrated circuits that implement embodiments of the present application.

Memory 302 may include mass storage for data or instructions. By way of example, and not limitation, memory 302 may comprise a Hard Disk Drive (HDD), floppy Disk Drive, flash memory, optical Disk, magneto-optical Disk, magnetic tape, or universal serial bus (Universal Serial Bus, USB) Drive, or a combination of two or more of the foregoing. Memory 302 may include removable or non-removable (or fixed) media, where appropriate. Memory 302 may be internal or external to the integrated gateway disaster recovery device, where appropriate. In a particular embodiment, the memory 302 is a non-volatile solid-state memory.

In particular embodiments, memory 302 includes Read Only Memory (ROM). The ROM may be mask programmed ROM, programmable ROM (PROM), erasable PROM (EPROM), electrically Erasable PROM (EEPROM), electrically rewritable ROM (EAROM), or flash memory, or a combination of two or more of these, where appropriate.

The memory may include Read Only Memory (ROM), random Access Memory (RAM), magnetic disk storage media devices, optical storage media devices, flash memory devices, electrical, optical, or other physical/tangible memory storage devices. Thus, in general, the memory comprises one or more tangible (non-transitory) readable storage media (e.g., memory devices) encoded with software comprising computer-executable instructions and when the software is executed (e.g., by one or more processors) it is operable to perform the operations described with reference to a method according to an aspect of the application.

The processor 301 implements the shared data transfer protocol SDTP message parsing method of any of the above embodiments by reading and executing program instructions stored in the memory 302.

In one example, the electronic device may also include a communication interface 303 and a bus 310. As shown in fig. 6, the processor 301, the memory 302, and the communication interface 303 are connected to each other by a bus 310 and perform communication with each other.

The communication interface 303 is mainly used to implement communication between each module, device, unit and/or apparatus in the embodiment of the present application.

Bus 310 includes hardware, software, or both, that couple components of the electronic device to one another. By way of example, and not limitation, the buses may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an infiniband interconnect, a Low Pin Count (LPC) bus, a memory bus, a micro channel architecture (MCa) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a video electronics standards association local (VLB) bus, or other suitable bus, or a combination of two or more of the above. Bus 310 may include one or more buses, where appropriate. Although embodiments of the application have been described and illustrated with respect to a particular bus, the application contemplates any suitable bus or interconnect.

In combination with the method for resolving the SDTP message of the shared data transmission protocol in the above embodiment, the embodiment of the application may be implemented by providing a readable storage medium. The readable storage medium has program instructions stored thereon; the program instructions, when executed by the processor, implement any of the shared data transfer protocol SDTP message parsing methods of the embodiments described above.

It should be understood that the application is not limited to the particular arrangements and instrumentality described above and shown in the drawings. For the sake of brevity, a detailed description of known methods is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method processes of the present application are not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between steps, after appreciating the spirit of the present application.

The functional blocks shown in the above-described structural block diagrams may be implemented in hardware, software, firmware, or a combination thereof. When implemented in hardware, it may be, for example, an electronic circuit, an Application Specific Integrated Circuit (ASIC), suitable firmware, a plug-in, a function card, or the like. When implemented in software, the elements of the application are the programs or code segments used to perform the required tasks. The program or code segments may be stored in a machine readable medium or transmitted over transmission media or communication links by a data signal carried in a carrier wave. A "machine-readable medium" may include any medium that can store or transfer information. Examples of machine-readable media include electronic circuitry, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio Frequency (RF) links, and the like. The code segments may be downloaded via computer networks such as the internet, intranets, etc.

It should also be noted that the exemplary embodiments mentioned in this disclosure describe some methods or systems based on a series of steps or devices. However, the present application is not limited to the order of the above-described steps, that is, the steps may be performed in the order mentioned in the embodiments, or may be performed in a different order from the order in the embodiments, or several steps may be performed simultaneously.

Aspects of the present application are described above with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by program instructions. These program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions/acts specified in the flowchart and/or block diagram block or blocks. Such a processor may be, but is not limited to being, a general purpose processor, a special purpose processor, an application specific processor, or a field programmable logic circuit. It will also be understood that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware which performs the specified functions or acts, or combinations of special purpose hardware and computer instructions.

In the foregoing, only the specific embodiments of the present application are described, and it will be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes of the systems, modules and units described above may refer to the corresponding processes in the foregoing method embodiments, which are not repeated herein. It should be understood that the scope of the present application is not limited thereto, and any equivalent modifications or substitutions can be easily made by those skilled in the art within the technical scope of the present application, and they should be included in the scope of the present application.

Claims

1. A method for resolving a shared data transmission protocol SDTP message, comprising:

caching the plurality of SDTP messages by a data buffering component;

acquiring a cached first SDTP message through a first processing node;

registering, by the first processing node, an association of the first SDTP message with the first processing node in a memory pool, where the first SDTP message is fragment data and is not a packet header;

determining, by the first processing node, a second SDTP message required for the first SDTP message in the case that the first SDTP message is fragmented data and is a data packet header; searching a second processing node associated with the second SDTP message in the registered association relation in the memory pool; and acquiring the second SDTP message from the second processing node and splicing.

2. The method of claim 1, wherein if the first SDTP message is fragmented data and is a header of a packet, and if there are multiple second SDTP messages required for the first SDTP message, the method further comprises:

3. The method of claim 1, wherein the registering, by the first processing node, the association of the first SDTP message with the first processing node in a memory pool comprises:

establishing a first association relationship between the receiving relationship number of the first SDTP message and the number of the first processing node; the receiving relation number is used as a key word used by a third processing node when searching a first processing node associated with the first SDTP message based on a third SDTP message, wherein the third SDTP message is a data packet header corresponding to the first SDTP message;

registering the first association relation in the memory pool.

4. The method of claim 3, wherein the SDTP message is a ticket's fragmented data; the receiving relation number comprises a receiving server number and a receiving ticket serial number; the third processing node searches the first processing node associated with the first SDTP message, and comprises:

determining that the receiving server number of the first SDTP message is the receiving server number of the third SDTP message;

and searching the number of the corresponding first processing node in the registered association relation in the memory pool according to the number of the receiving server of the first SDTP message and the number of the receiving ticket serial number.

5. The method of claim 3, further comprising, prior to the retrieval of the cached first SDTP message by the first processing node:

registering the second association relationship in the memory pool;

6. A shared data transfer protocol, SDTP, message parsing apparatus comprising:

a buffer unit, configured to buffer the plurality of SDTP messages through a data buffering component;

a registration unit, configured to register, by the first processing node, an association relationship between the first SDTP message and the first processing node in a memory pool, where the first SDTP message is fragment data and is not a packet header;

an execution unit, configured to determine, by the first processing node, a second SDTP message required for the first SDTP message, where the first SDTP message is fragment data and is a packet header; searching a second processing node associated with the second SDTP message in the registered association relation in the memory pool; and acquiring the second SDTP message from the second processing node and splicing.

7. The apparatus according to claim 6, wherein, in a case where the first SDTP message is fragmented data and is a packet header, and in a case where there are a plurality of second SDTP messages required for the first SDTP message, the acquiring unit is further configured to acquire a corresponding second SDTP message from a second processing node corresponding to each of the second SDTP messages;

8. An electronic device, the electronic device comprising: a processor and a memory storing program instructions;

the processor, when executing the program instructions, implements a shared data transfer protocol SDTP message parsing method as defined in any one of claims 1-5.

9. A readable storage medium, wherein program instructions are stored on the readable storage medium, and when executed by a processor, implement the shared data transfer protocol SDTP message parsing method according to any one of claims 1-5.

10. A program product, characterized in that instructions in the program product, when executed by a processor of an electronic device, cause the electronic device to perform the shared data transfer protocol, SDTP, message parsing method according to any of claims 1-5.