CN106897316B - Method and device for processing signaling data - Google Patents
Method and device for processing signaling data Download PDFInfo
- Publication number
- CN106897316B CN106897316B CN201510964842.3A CN201510964842A CN106897316B CN 106897316 B CN106897316 B CN 106897316B CN 201510964842 A CN201510964842 A CN 201510964842A CN 106897316 B CN106897316 B CN 106897316B
- Authority
- CN
- China
- Prior art keywords
- signaling data
- buffer queue
- memory buffer
- module
- signaling
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2255—Hash tables
Abstract
The embodiment of the invention discloses a method for processing signaling data, which comprises the following steps: the embodiment of the invention also discloses a device for processing the signaling data.
Description
Technical Field
The present invention relates to a signaling data processing technology in the field of mobile communications, and in particular, to a method and an apparatus for processing signaling data.
Background
In the field of mobile communication, signaling data includes time, position, calling and called numbers, failure reasons, mobile phone types, user startup, shutdown, position updating and other information of each call of a user, so that real-time analysis and marketing of mobile communication users can be performed through the signaling data, and in order to support precise marketing and real-time marketing scenes, the processing of the signaling data must have high real-time characteristics.
In the prior art, Data such as signaling is processed, collected signaling Data needs to be transmitted to a signaling processing system through a Transmission Control Protocol/Internet Protocol (TCP/IP) or a User Datagram Protocol (UDP), the signaling processing system puts all Data into a database, and the signaling processing system searches a Data storage medium in response to a Data Manipulation Language (DML) statement submitted by a User and returns a query result; the signaling data is often a disk or a tape as a medium, and a large amount of Input/Output (I/O) exchanges are required to perform the query operation, so that the signaling processing system is inefficient and cannot meet the real-time requirement.
Therefore, the technical problem that the processing signaling data is lack of real-time performance exists in the prior art.
Disclosure of Invention
In view of this, embodiments of the present invention are expected to provide a method and an apparatus for processing signaling data, so as to meet the real-time requirement of signaling data processing.
In order to achieve the purpose, the technical scheme of the invention is realized as follows:
in a first aspect, an embodiment of the present invention provides a method for processing signaling data, where the method includes: the memory buffer queue control module obtains signaling data from at least one network element, stores the signaling data in a memory buffer queue, extracts the signaling data from the memory buffer queue through the ETL task management module for conversion and loading, and transmits the signaling data to the distributed memory database module, and the distributed memory database module stores the signaling data.
Further, the obtaining, by the memory buffer queue control module, signaling data from at least one network element includes: the signaling transmission module collects the signaling data from the at least one network element, the signaling transmission module sends the signaling data to the memory buffer queue control module in a multithreading transmission mode based on reliable User Datagram Protocol (UDP) connection, and the memory buffer queue control module receives the signaling data.
Further, after the signaling transmission module sends the signaling data to the memory buffer queue control module in a multithread transmission mode based on a reliable user datagram protocol UDP connection, the method further includes: when the memory buffer queue control module does not receive the signaling data within the preset time, the signaling data is retransmitted by the signaling acquisition module, and when the memory buffer queue control module receives the signaling data and is abnormal, the memory buffer queue control module returns an abnormal prompt message to the signaling transmission module.
Further, the storing the signaling data in a memory buffer queue by the memory buffer queue control module includes: the memory buffer queue control module adjusts the length of the memory buffer queue according to the flow of the signaling data, and stores the signaling data in the adjusted memory buffer queue.
Further, the memory buffer queue control module adjusts the length of the memory buffer queue according to the flow of the signaling data, including: and when the flow of the signaling data meets a preset condition, adjusting the length of the memory buffer queue to enable the length of the memory buffer queue to be matched with the flow of the signaling data.
Further, the distributed memory database module stores the signaling data, including: and the distributed memory database module determines a target node of the signaling data according to the number of memory database nodes contained in the distributed memory database module, and stores the signaling data to the target node.
In a second aspect, an embodiment of the present invention provides an apparatus for processing signaling data, where the apparatus includes: the system comprises a memory buffer queue control module, an ETL task management module for extracting the signaling data from the memory buffer queue and transmitting the signaling data to a distributed memory database module, and the distributed memory database module is used for storing the signaling data.
Further, the apparatus further comprises: and the signaling transmission module is used for acquiring the signaling data from the at least one network element and sending the signaling data to the memory buffer queue control module in a multithreading transmission mode based on reliable User Datagram Protocol (UDP) connection.
Further, the signaling collection module is further configured to: when the memory buffer queue control module does not receive the signaling data within a preset time, resending the signaling data, wherein the memory buffer queue control module is further configured to: and when the memory buffer queue control module receives the signaling data and the signaling data is abnormal, returning an abnormal prompt message to the signaling transmission module.
Further, the memory buffer queue control module includes: the adjusting module is used for adjusting the length of the memory buffer queue according to the flow of the signaling data, and the storing module is used for storing the signaling data in the adjusted memory buffer queue.
Further, the adjustment module includes: and when the flow of the signaling data meets a preset condition, adjusting the length of the memory buffer queue to enable the length of the memory buffer queue to be matched with the flow of the signaling data.
Further, the distributed memory database module is configured to determine a target node of the signaling data according to the number of memory database nodes included in the distributed memory database module, and store the signaling data to the target node.
The method and device for processing signaling data provided by the embodiment of the invention comprises the steps that firstly, a memory buffer queue control module obtains signaling data from at least one network element and stores the signaling data in a memory buffer queue, then an ETL task management module extracts the signaling data from the memory buffer queue and transmits the signaling data to a distributed memory database module, and finally the distributed memory database module stores the signaling data, so that the memory buffer queue control module on a memory stores the signaling data in the memory buffer queue after obtaining the signaling data from at least one network element, the ETL task management module on the memory extracts the signaling data from the memory buffer queue and stores the signaling data in the distributed memory database, and the signaling data is processed on the memory to enable the processing of the signaling data to be faster, thus, the method effectively solves the technical problem that the processing of the signaling data in the prior art lacks real-time performance, and further realizes the requirement on the real-time performance in the signaling data processing.
Drawings
Fig. 1 is a schematic flow chart of a method for processing signaling data according to an embodiment of the present invention;
fig. 2 is a schematic flow chart of a preferred embodiment of a method for processing signaling data according to an embodiment of the present invention;
fig. 3 is a schematic flow chart of another preferred embodiment of a signaling data processing method according to an embodiment of the present invention;
FIG. 4 is a schematic structural diagram of a preferred embodiment of a distributed memory database module according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of a signaling data processing apparatus according to an embodiment of the present invention.
Detailed Description
The technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.
An embodiment of the present invention provides a method for processing signaling data, and fig. 1 is a schematic flow chart of the method for processing signaling data in the embodiment of the present invention, where the method includes:
s101: the memory buffer queue control module obtains the signaling data from at least one network element and stores the signaling data in the memory buffer queue;
in the S101, the memory buffer queue control module obtains signaling data from at least one network element, where common signaling data includes an a-port signaling, a Gn-port signaling, a Gb signaling, and S1-U and S1-MME signaling of the fourth Generation mobile communication technology (4G), but is not limited thereto; after the memory buffer queue module obtains the signaling data, the signaling data is stored in the memory buffer queue, so that preparation is made for the ETL task management module to extract the signaling data.
In another embodiment, fig. 2 is a schematic flowchart of a preferred embodiment of a method for processing signaling data in the embodiment of the present invention, and as shown in fig. 2, S101 may include:
s11: the signaling transmission module collects signaling data from at least one network element;
s12: the signaling transmission module sends signaling data to the memory buffer queue control module in a multithreading transmission mode based on reliable User Datagram Protocol (UDP) connection;
s13: the memory buffer queue control module receives signaling data.
In the above S11 to S13, first, after the signaling transmission module collects the signaling data of each network element, the signaling transmission module transmits the signaling data based on a multi-thread transmission mode of reliable UDP connection, specifically, first, the signaling transmission module allocates the signaling data collected from each network element on a plurality of threads according to pre-configured configuration information, and then transmits the signaling data to the memory buffer queue control module through a parallel pipeline, so that the transmission efficiency in signaling transmission is increased; in addition, the embodiment of the invention greatly improves the transmission capability of the signaling data by a multi-thread connection transmission technology.
In another embodiment, after S12, the method further comprises:
when the memory buffer queue control module does not receive the signaling data within the preset time, the signaling acquisition module retransmits the signaling data; when the memory buffer queue control module receives the signaling data and is abnormal, the memory buffer queue control module returns an abnormal prompt message to the signaling transmission module.
In the method, during the process of distributing the collected signaling data of each network element to a plurality of transmission threads based on UDP for processing transmission, when the memory buffer queue control module does not receive the signaling data within the preset time, the signaling collection module resends the signaling data, and the preset time can be set by a retransmission timer; for example: when a retransmission timer is used for monitoring, after each message is sent, the retransmission timer is set, a response is sent to a sending end after a receiving end receives the message, the sending end cancels the timer after receiving the response, and the retransmission timer automatically retransmits if the response message is not received;
when the memory buffer queue control module receives the signaling data and the signaling data is abnormal, the memory buffer queue control module returns the abnormal prompt information to the signaling transmission module, and the method can realize the transmission of a plurality of transmission threads based on reliable UDP through a retransmission counter, a message responder, a keep-alive checker and a flow controller; the method comprises the following specific steps:
when a retransmission counter is used for monitoring, a signaling transmission module records the retransmission times of the message, and error processing is carried out until the retransmission times is greater than a preset threshold value;
when the message responder is used for monitoring, the message responder is responsible for sending messages to the receiving end, when the message receiving end judges that the received messages are not in sequence according to the message serial number, the receiving end feeds back message responses to the sending end to inform the sending end that the sending end is possibly damaged or lost in the information transmission process, and after the sending end receives the feedback message responses, the sending end calculates the messages which are possibly lost according to the original message serial number and the disordered message serial number in the responses and then retransmits the messages;
when monitoring is carried out by using the keep-alive checker, the activity conditions of the sending end and the receiving end are checked so as to find out in time when one party is abnormal and automatically reestablish connection;
when the flow controller is used for monitoring, based on the flow control of the sliding window, when the sequence number of the maximum message which can be received by the receiving end is coincident with the received maximum message sequence number, the receiving end window is closed, and meanwhile, the sending end window is also closed, so that the retransmission, window detection and link keep-alive actions of the message can be carried out under the condition.
By the method, the reliability and stability of UDP connection are greatly improved by the technology based on reliable UDP connection under the condition of ensuring the high-speed characteristic of the original UDP network connection, and a plurality of problems caused by the adoption of TCP/IP connection and UDP connection in the traditional scheme are solved; finally, the signaling data can be efficiently and stably transmitted from the transmitting end to the receiving end, and the characteristics of real-time, continuity and unboundness of the signaling data are adapted.
It should be noted that the receiving end corresponds to the memory buffer queue control module in the present invention, and the transmitting end corresponds to the signaling transmission module in the present invention.
In another embodiment, fig. 3 is a schematic flowchart of another preferred embodiment of a method for processing signaling data in the embodiment of the present invention, and as shown in fig. 3, in the above S101, further includes:
s14: the memory buffer queue control module adjusts the length of the memory buffer queue according to the flow of the signaling data;
s15: the memory buffer queue control module stores the signaling data in the adjusted memory buffer queue.
In the above S14 to S15, the memory buffer queue control module may adjust the length of the memory buffer by using a memory buffer queue control algorithm, and more flexibly adjust the length of the memory buffer queue to a variable fixed length, so as to effectively solve the risk of data loss caused by the incompatibility between the signaling transmission capability and the signaling processing capability in the conventional scheme;
the length change of the memory buffer queue can be adjusted through a memory buffer queue control algorithm, so that a memory buffer queue with a variable fixed length is generated, and then the memory buffer queue control module stores the signaling data in the adjusted memory buffer queue; the size of the memory buffer queue is managed by the method, the signaling data is temporarily stored in the memory buffer queue for subsequent processing and calling, the capacity of the memory buffer area is properly increased and decreased under the condition that the memory buffer queue is full, and the memory buffer queue with variable fixed length is realized by a memory buffer queue control algorithm, so that the length of the memory buffer queue is controlled.
In a specific implementation, S14 includes: and when the flow of the signaling data meets the preset condition, adjusting the length of the memory buffer queue to enable the length of the memory buffer queue to be matched with the flow of the signaling data.
Specifically, the length change of the memory buffer queue can be adjusted through a memory buffer queue control algorithm, so that the memory buffer queue with variable fixed length is generated. For example, assume the maximum length L of the memory buffer queuemax10GB, minimum length L of memory buffer queuemin3GB, the actual memory buffer queue size is Lt，LtThe conditions are satisfied: 3G<Lt<10G, setting the current L of the systemt5GB, the actual signalling data size V after a time period t has elapsedtWill increase to 7GB, at which time Vt>LtThen, the memory buffer queue control module adjusts the memory buffer queue LtEach increment is Δ xn, where Δ ═ L (L)max-Lmin) The value of the parameter can be used for controlling the adjusting speed, and the adjustment of the memory buffer queue is carried out once every time period t until L is mett≥VtOr Lt＝LmaxUntil now. In addition, when Vt<LtIn time, every time period t, the length of the memory buffer queue of delta multiplied by N is reduced until the length is adjusted to meet Lt-Vt<Δ × N.
Then, through the above-mentioned method, write the signalling data into the memory buffering queue, still manage the size of the memory buffering queue at the same time, store the signalling data in the memory buffering queue temporarily, for the subsequent processing to call, there is the situation that the queue is full in the memory buffering queue, increase and reduce the capacity of the buffer area of the memory properly, have realized the memory buffering queue of variable fixed length through the control algorithm of the memory buffering queue, thus control the length of the memory buffering queue, this method presumes the initial buffering queue length of the system as a certain value, according to the magnitude trend of the processing capacity of the signalling data, increase or reduce the size of the memory buffering queue correspondingly; therefore, the deterioration of the reading swallowing capacity and the corresponding time of the whole system caused by the fact that the burst flow exceeds the processing capacity of the system is avoided, the risk of data loss is possibly generated in the more serious situation, and the influence on the system is reduced to the maximum extent.
S102: the ETL task management module extracts the signaling data from the memory buffer queue and transmits the signaling data to the distributed memory database module;
specifically, in the above S102, the ETL task management module extracts the signaling data of the memory buffer queue in parallel through a parallel pipeline technique with multiple threads, and then the ETL task management module transmits the signaling data to the distributed memory database module.
Here, the ETL task management module is mainly responsible for: ETL task monitoring, ETL task engine, ETL task management, and metadata management, etc., the present invention is not limited in particular.
S103: the distributed memory database module stores signaling data.
Specifically, the distributed memory database module stores signaling data, including: and the distributed memory database module determines a target node of the signaling data according to the number of memory database nodes contained in the distributed memory database module, and stores the signaling data to the target node.
The distributed memory database module stores the signaling data after receiving the signaling data, provides external services and facilitates access and query of users, is a memory database, stores all data in a memory and can exert the super-high speed advantage of memory access, ensures data reliability through full data files and database logs, and is distributed and deployed on a plurality of nodes of a network to provide a uniform access interface for the outside.
In a specific implementation process, S103 further includes: and the distributed memory database module determines a target node of the signaling data according to the number of memory database nodes contained in the distributed memory database module, and stores the signaling data to the target node.
Specifically, the distributed memory database module calculates a hash value, i.e., hash (KEY), of each signaling data record according to a distribution KEY value (KEY) specified in the signaling data, and then determines a target node h of the signaling data according to the following formula (1).
h＝Hash(KEY)％N (1)
The KEY is a distribution KEY value appointed in the signaling data, and the N is the number of the memory database nodes contained by the distributed memory database module according to the distributed memory database module.
In the embodiment of the present invention, since the memory database nodes are numbered according to 0 to N-1, the distributed memory database module may calculate the number h of the target node of the signaling data according to the formula (1), and then store the signaling data to the target node h.
Here, the access interface of the distributed memory database module is identified by an IP address and a port number;
the distributed memory database module provides an IP address and a port number to identify a distributed memory, so that each application connected with the distributed memory database module can identify and establish connection only through the IP address and the port number; the method can realize the linear expansion of the performance of the memory database, effectively solve the problems of the capacity and the expansibility of the system, greatly improve the concurrency performance and be very suitable for processing the signaling data.
Fig. 4 is a schematic structural diagram of a preferred embodiment of the distributed memory database module in the embodiment of the present invention, and as shown in fig. 4, the distributed memory database module includes: a unified service interface library 41, a management node 42, an SQL parser 43, a routing rule 44 and a memory database cluster 45;
wherein, the unified interface service interface library 41: the method comprises the steps that the method is deployed together with an application program, the application program accesses a distributed memory database module by calling a unified service interface library 41, a proxy mode of a traditional database is abandoned by adopting a framework of a fat client, cross-host access is effectively reduced, and after the unified service interface library 41 acquires routing rules and node information from a management node 42, operations such as routing analysis, memory library access, result combination and the like are directly carried out on an SQL request;
the management node 42 is mainly used for managing routing rules, maintaining memory database nodes, and monitoring external connections;
the SQL parser 43 is used to receive and verify the syntax semantics of SQL statements, analyze the environment and optimize the methods that meet SQL, so that the distributed memory database can be accessed as if a single database is accessed, the system automatically determines the route according to the SQL statements, supports distributed things, two-phase commit, supports rollback on the transaction that was successfully pre-committed, and executes in a non-blocking mode.
In the routing rule 44, the nodes and the routing information are stored in the master library and the slave library of each node in the form of a system table, and data consistency is ensured through distributed transactions, it should be noted that the routing rule 44 may support the following routing types: 0 single bank, 1 full bank and 2 partitions;
0 single library: all data of the table are stored in a certain designated node, and the operation on the table is performed on the node, such as a shared table;
1, whole library: the table has a complete data on each node, and the data consistency of each node is ensured through distributed transactions, such as a configuration class table;
2, partitioning: the data of the table are respectively stored on different nodes according to the calculated value of the designated field, and the operation on the table record needs to be routed to a corresponding node, such as an account list table, according to the value of the designated field.
A memory database cluster 45, which is mainly used for managing the distributed memory database examples, storing data, copying the main nodes and the standby nodes, and checking consistency; wherein the primary and standby node replication comprises HA bidirectional replication.
The method for processing signaling data provided by the embodiment of the invention comprises the steps that firstly, a memory buffer queue control module obtains signaling data from at least one network element and stores the signaling data in a memory buffer queue, then an ETL task management module extracts the signaling data from the memory buffer queue and transmits the signaling data to a distributed memory database module, and finally the distributed memory database module stores the signaling data, so that the memory buffer queue control module on a memory stores the signaling data in the memory buffer queue after obtaining the signaling data from at least one network element, the ETL task management module on the memory extracts the signaling data from the memory buffer queue and stores the signaling data in the distributed memory database module, the signaling data is processed on the memory to enable the processing of the signaling data to be faster, and therefore, the method effectively solves the technical problem that the processing of the signaling data in the prior art lacks real-time performance, and further realizes the requirement on the real-time performance in the signaling data processing.
Based on the same inventive concept, an embodiment of the present invention provides a device for processing signaling data, and fig. 5 is a schematic structural diagram of the device for processing signaling data in the embodiment of the present invention, as shown in fig. 5, the device includes: a memory buffer queue control module 51, an ETL task management module 52 and a distributed memory database module 53;
the memory buffer queue control module 51 is configured to obtain signaling data from at least one network element, and store the signaling data in a memory buffer queue;
in the memory buffer queue control module 51, the memory buffer queue control module 51 obtains signaling data from at least one network element, where common signaling data includes an a-port signaling, a Gn-port signaling, a Gb signaling, and 4G S1-U and S1-MME signaling, but is not limited thereto; after obtaining the signaling data, the memory buffer queue module 31 stores the signaling data in the memory buffer queue, so as to prepare for the ETL task management module 52 to extract the signaling data.
The ETL task management module 52 is configured to extract signaling data from the memory buffer queue, and transmit the signaling data to the distributed memory database module 53;
in another embodiment, the above apparatus further comprises: and a signaling transmission module, configured to collect signaling data from at least one network element, and send the signaling data to the memory buffer queue control module 51 in a multithread transmission manner based on a reliable user datagram protocol UDP connection.
In the signaling transmission module, firstly, after the signaling transmission module collects the signaling data of each network element, the signaling transmission module transmits the signaling data based on a multithread transmission mode of reliable UDP connection, specifically, firstly, the signaling transmission module distributes the signaling data collected from each network element on a plurality of threads according to the pre-configured configuration information, and then transmits the signaling data to the memory buffer queue control module through a parallel pipeline, thereby increasing the transmission efficiency in signaling transmission; in addition, the embodiment of the invention greatly improves the transmission capability of the signaling data by a multi-thread connection transmission technology.
In another embodiment, the signaling collection module is further configured to: when the memory buffer queue control module 51 does not receive the signaling data within the preset time, the memory buffer queue control module 51 is further configured to: when the memory buffer queue control module 51 receives the signaling data, it returns an exception prompt message to the signaling transmission module.
In the signaling collection module and the memory buffer queue control module 51, during the process of distributing the collected signaling data of each network element to a plurality of transmission threads based on UDP for processing and transmission, when the memory buffer queue control module does not receive the signaling data within a preset time, the signaling collection module resends the signaling data, and the preset time can be set by a retransmission timer; for example: when a retransmission timer is used for monitoring, after each message is sent, the retransmission timer is set, a response is sent to a sending end after a receiving end receives the message, the sending end cancels the timer after receiving the response, and the retransmission timer automatically retransmits if the response message is not received;
when the memory buffer queue control module 51 receives the signaling data and the signaling data is abnormal, the memory buffer queue control module 51 returns the abnormal prompt information to the signaling transmission module, and the device can realize the transmission of a plurality of transmission threads based on reliable UDP by a retransmission counter, a message responder, a keep-alive checker and a flow controller; the method comprises the following specific steps:
when a retransmission counter is used for monitoring, a signaling transmission module records the retransmission times of the message, and error processing is carried out until the retransmission times is greater than a preset threshold value;
when the message responder is used for monitoring, the message responder is responsible for sending messages to the receiving end, when the message receiving end judges that the received messages are not in sequence according to the message serial number, the receiving end feeds back message responses to the sending end to inform the sending end that the sending end is possibly damaged or lost in the information transmission process, and after the sending end receives the feedback message responses, the sending end calculates the messages which are possibly lost according to the original message serial number and the disordered message serial number in the responses and then retransmits the messages;
when monitoring is carried out by using the keep-alive checker, the activity conditions of the sending end and the receiving end are checked so as to find out in time when one party is abnormal and automatically reestablish connection;
when the flow controller is used for monitoring, based on the flow control of the sliding window, when the sequence number of the maximum message which can be received by the receiving end is coincident with the received maximum message sequence number, the receiving end window is closed, and meanwhile, the sending end window is also closed, so that the retransmission, window detection and link keep-alive actions of the message can be carried out under the condition.
The device realizes that the technology based on reliable UDP connection greatly improves the reliability and stability of UDP connection under the condition of ensuring the high-speed characteristic of the original UDP network connection, solves a plurality of problems caused by adopting TCP/IP connection and UDP connection in the traditional scheme, and also greatly improves the transmission capability of signaling by the multithread transmission technology realized by the invention, and can ensure the stability in the whole signaling data transmission process; finally, the signaling data can be efficiently and stably transmitted from the transmitting end to the receiving end, and the characteristics of real-time, continuity and unboundness of the signaling data are adapted.
It should be noted that the receiving end corresponds to the memory buffer queue control module 51 in the present invention, and the transmitting end corresponds to the signaling transmission module in the present invention.
In another embodiment, the memory buffer queue control module 51 includes: an adjusting module and a storing module;
the adjusting module is used for adjusting the length of the memory buffer queue according to the flow of the signaling data;
and the storage module is used for storing the signaling data in the adjusted memory buffer queue.
In the adjusting module and the storage module, the adjusting module can adjust the length of the memory buffer through a memory buffer queue control algorithm, the length of the memory buffer queue is more flexibly changed into a variable fixed length by adjusting the length of the memory buffer queue, and the risk of data loss caused by the incongruity of signaling transmission capacity and signaling processing capacity in the traditional device can be effectively solved;
the length change of the memory buffer queue can be adjusted through a memory buffer queue control algorithm, so that a memory buffer queue with a variable fixed length is generated, and then the memory buffer queue control module 51 stores the signaling data in the adjusted memory buffer queue; the size of the memory buffer queue is managed through the device, the signaling data are temporarily stored in the memory buffer queue for subsequent processing and calling, the capacity of the memory buffer area is properly increased and reduced under the condition that the memory buffer queue is full, the memory buffer queue with variable fixed length is realized through a memory buffer queue control algorithm, and the length of the memory buffer queue is controlled.
In a specific implementation process, the adjusting module includes: and when the flow of the signaling data meets the preset condition, adjusting the length of the memory buffer queue to enable the length of the memory buffer queue to be matched with the flow of the signaling data.
Specifically, the length change of the memory buffer queue can be adjusted through a memory buffer queue control algorithm, so that the memory buffer queue with variable fixed length is generated. For example, assume the maximum length L of the memory buffer queuemax10GB, minimum length L of memory buffer queuemin3GB, the actual memory buffer queue size is Lt，LtThe conditions are satisfied: 3G<Lt<10G, setting the current L of the systemt5GB, the actual signalling data size V after a time period t has elapsedtWill increase to 7GB, at which time Vt>LtThen, the memory buffer queue control module adjusts the memory buffer queue LtEach increment is Δ xn, where Δ ═ L (L)max-Lmin) The value of the parameter can be used for controlling the adjusting speed, and the adjustment of the memory buffer queue is carried out once every time period t until L is mett≥VtOr Lt＝LmaxUntil now. In addition, when Vt<LtIn time, every time period t, the length of the memory buffer queue of delta multiplied by N is reduced until the length is adjusted to meet Lt-Vt<Δ × N.
Then, the signaling data is written into the memory buffer queue through the device, the size of the memory buffer queue is managed at the same time, the signaling data is temporarily stored in the memory buffer queue for subsequent processing and calling, the capacity of the memory buffer area is properly increased and decreased under the condition that the memory buffer queue is full, the memory buffer queue with variable fixed length is realized through a memory buffer queue control algorithm, so that the length of the memory buffer queue is controlled, the device sets the length of the initial buffer queue of the system to be a certain value, and the size of the memory buffer queue is correspondingly increased or decreased according to the size trend of the processing amount of the signaling data; therefore, the deterioration of the reading swallowing capacity and the corresponding time of the whole system caused by the fact that the burst flow exceeds the processing capacity of the system is avoided, the risk of data loss is possibly generated in the more serious situation, and the influence on the system is reduced to the maximum extent.
The ETL task management module 52 is configured to extract signaling data from the memory buffer queue, and transmit the signaling data to the distributed memory database module 53;
in the ETL task management module 52, the ETL task management module 52 extracts the signaling data of the memory buffer queue in parallel through a parallel pipeline technique with multiple threads, and then the ETL task management module transmits the signaling data to the distributed memory database module 53.
Here, the ETL task management module 52 is mainly responsible for: ETL task monitoring, ETL task engine, ETL task management, metadata management, and other services, which are not specifically limited in the present invention.
A distributed memory database module 53 for storing signaling data.
In another embodiment, the distributed memory database module 53 is configured to: and determining a target node of the signaling data according to the number of the memory database nodes contained in the target node, and storing the signaling data to the target node.
In the distributed memory database module 53, the distributed memory database module 53 stores the signaling data after receiving the signaling data, and provides an external service, which is convenient for a user to access and query, in addition, the distributed memory database module 53 is a memory database, all data are stored in a memory, and can exert the super-high speed advantage of memory access, the distributed memory database module 53 ensures the reliability of the data through a full-scale data file and a database log, and in addition, the distributed memory database module 53 is distributed, is deployed on a plurality of nodes of a network, and provides a uniform access interface to the outside.
In a specific implementation process, the distributed memory database module 53 is configured to: and determining a target node of the signaling data according to the number of the memory database nodes contained in the target node, and storing the signaling data to the target node.
Specifically, the distributed memory database module 53 calculates a hash value, i.e., hash (KEY), of each signaling data record according to a distribution KEY value (KEY) specified in the signaling data, and then determines a target node h of the signaling data according to the following formula (1).
h＝Hash(KEY)％N (1)
Wherein, KEY is a distribution KEY value designated in the signaling data, and N is the number of memory database nodes included by the distributed memory database module 53 according to itself.
In the embodiment of the present invention, since the memory database nodes are numbered according to 0 to N-1, the distributed memory database module 53 may calculate the number h of the target node of the signaling data according to the formula (1), and then store the signaling data to the target node h.
Here, the access interface of the distributed memory database module 53 is identified by an IP address and port number;
the distributed memory database module 53 provides an IP address and a port number to identify a distributed memory, so that each application connected to the distributed memory database module can identify and establish connection only through the IP address and the port number, and in addition, the number of internal nodes, the IP address of each node, and the port number of the distributed memory database module 53 are transparent to the application; the method can realize the linear expansion of the performance of the memory database, effectively solve the problems of the capacity and the expansibility of the system, greatly improve the concurrency performance and be very suitable for processing the signaling data.
For example, as shown in fig. 4, the distributed memory database module 53 includes: a unified service interface library 41, a management node 42, an SQL parser 43, a routing rule 44 and a memory database cluster 45;
wherein, the unified interface service interface library 41: as shown in fig. 4, the application accesses the distributed memory database by calling the unified service interface library 41, and adopts the architecture of the fat client, so as to abandon the proxy mode of the conventional database, thereby effectively reducing the cross-host access, and after the unified service interface library 41 acquires the routing rules and the node information from the management node 42, the unified service interface library 41 directly performs operations such as routing analysis, memory library access, result merging and the like on the SQL request;
the management node 42 is mainly used for managing routing rules, maintaining memory database nodes, and monitoring external connections;
the SQL parser 43 is used to receive and verify the syntax semantics of SQL statements, analyze the environment and optimize the methods that meet SQL, so that the distributed memory database can be accessed as if a single database is accessed, the system automatically determines the route according to the SQL statements, supports distributed things, two-phase commit, supports rollback on the transaction that was successfully pre-committed, and executes in a non-blocking mode.
In the routing rule 44, the nodes and the routing information are stored in the master library and the slave library of each node in the form of a system table, and data consistency is ensured through distributed transactions, it should be noted that the routing rule 44 may support the following routing types: 0 single bank, 1 full bank and 2 partitions;
0 single library: all data of the table are stored in a certain designated node, and the operation on the table is performed on the node, such as a shared table;
1, whole library: the table has a complete data on each node, and the data consistency of each node is ensured through distributed transactions, such as a configuration class table;
2, partitioning: the data of the table are respectively stored on different nodes according to the calculated value of the designated field, and the operation on the table record needs to be routed to a corresponding node, such as an account list table, according to the value of the designated field.
A memory database cluster 45, which is mainly used for managing the distributed memory database examples, storing data, copying the main nodes and the standby nodes, and checking consistency; wherein the primary and standby node replication comprises HA bidirectional replication.
The signaling data processing device provided in the embodiment of the present invention first obtains signaling data from at least one network element through the memory buffer queue control module 51, stores the signaling data in the memory buffer queue, then extracts the signaling data from the memory buffer queue through the ETL task management module 52, transmits the signaling data to the distributed memory database module 53, and finally stores the signaling data through the distributed memory database module 53, so that the memory buffer queue control module 31 on the memory obtains the signaling data from at least one network element and then stores the signaling data in the memory buffer queue, the ETL task management module 52 on the memory extracts the signaling data from the memory buffer queue and stores the signaling data in the distributed memory database module 53, and the processing of the signaling data on the memory makes the processing of the signaling data faster, therefore, the technical problem that the processing of the signaling data lacks real-time performance in the prior art is effectively solved, and the requirement on the real-time performance in the signaling data processing is further realized.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The above description is only a preferred embodiment of the present invention, and is not intended to limit the scope of the present invention.
Claims (6)
1. A method for processing signaling data, the method comprising:
the signaling transmission module collects the signaling data from at least one network element;
the signaling transmission module sends the signaling data to a memory buffer queue control module in a multithreading transmission mode based on reliable User Datagram Protocol (UDP) connection;
when the memory buffer queue control module does not receive the signaling data within the preset time, the signaling acquisition module retransmits the signaling data;
when the memory buffer queue control module receives the signaling data and is abnormal, the memory buffer queue control module returns an abnormal prompt message to the signaling transmission module; the signaling transmission module retransmits the signaling data;
the memory buffer queue control module receives the signaling data and stores the signaling data in a memory buffer queue;
the ETL task management module extracts the signaling data from the memory buffer queue and transmits the signaling data to the distributed memory database module;
the distributed memory database module stores the signaling data; the number of internal nodes of the distributed memory database module, the IP address and the port number of each node are transparent to the application.
2. The method of claim 1, wherein the memory buffer queue control module stores the signaling data in a memory buffer queue, comprising:
the memory buffer queue control module adjusts the length of the memory buffer queue according to the flow of the signaling data;
and the memory buffer queue control module stores the signaling data in the adjusted memory buffer queue.
3. The method of claim 2, wherein the adjusting the length of the memory buffer queue by the memory buffer queue control module according to the traffic of the signaling data comprises:
and when the flow of the signaling data meets a preset condition, adjusting the length of the memory buffer queue to enable the length of the memory buffer queue to be matched with the flow of the signaling data.
4. An apparatus for processing signaling data, the apparatus comprising:
the system comprises a signaling transmission module, a memory buffer queue control module and a data transmission module, wherein the signaling transmission module is used for acquiring signaling data from at least one network element and transmitting the signaling data to the memory buffer queue control module in a multithreading transmission mode based on reliable User Datagram Protocol (UDP) connection;
the memory buffer queue control module is used for receiving the signaling data; when the signaling data is not received within the preset time, receiving the retransmitted signaling data; when the signaling data is received to be abnormal, returning abnormal prompt information to the signaling transmission module, receiving the retransmitted signaling data, and storing the signaling data in a memory buffer queue;
the ETL task management module is used for extracting the signaling data from the memory buffer queue and transmitting the signaling data to the distributed memory database module;
the distributed memory database module is used for storing the signaling data; the number of internal nodes of the distributed memory database module, the IP address and the port number of each node are transparent to the application.
5. The apparatus of claim 4, wherein the memory buffer queue control module comprises:
the adjusting module is used for adjusting the length of the memory buffer queue according to the flow of the signaling data;
and the storage module is used for storing the signaling data in the adjusted memory buffer queue.
6. The apparatus of claim 5, wherein the adjustment module comprises:
and when the flow of the signaling data meets a preset condition, adjusting the length of the memory buffer queue to enable the length of the memory buffer queue to be matched with the flow of the signaling data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510964842.3A CN106897316B (en) | 2015-12-21 | 2015-12-21 | Method and device for processing signaling data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510964842.3A CN106897316B (en) | 2015-12-21 | 2015-12-21 | Method and device for processing signaling data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106897316A CN106897316A (en) | 2017-06-27 |
| CN106897316B true CN106897316B (en) | 2021-08-10 |
Family
ID=59191004
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510964842.3A Active CN106897316B (en) | 2015-12-21 | 2015-12-21 | Method and device for processing signaling data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106897316B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107835327B (en) * | 2017-10-30 | 2020-01-07 | 北京佳讯飞鸿电气股份有限公司 | Method, device and system for maintaining protocol stack queue of signaling gateway |
| CN109561389A (en) * | 2018-12-28 | 2019-04-02 | 海南中智信信息技术有限公司 | A method of based on flow of the people real-time in mobile phone signaling zoning |
| CN111770030B (en) * | 2019-05-17 | 2023-08-08 | 北京京东尚科信息技术有限公司 | Message persistence processing method, device and storage medium |
| CN110929102B (en) * | 2019-11-13 | 2023-01-17 | 讯飞幻境(北京)科技有限公司 | Data processing method and device and electronic equipment |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6105034A (en) * | 1997-12-12 | 2000-08-15 | Nortel Networks Corporation | Non-volatile mission-ready database for signaling transfer point |
| CN104468736A (en) * | 2014-11-19 | 2015-03-25 | 天津中兴智联科技有限公司 | System and method for carrying out signaling tracking on RFID reader |
-
2015
- 2015-12-21 CN CN201510964842.3A patent/CN106897316B/en active Active
Non-Patent Citations (3)
| Title |
|---|
| 列车固体录音装置的音频系统设计与实现;张俊涛 等;《计算机测量与控制》;20110430;第19卷(第4期);第938-940页 * |
| 基于网络信令快速处理技术的实时营销;敖锦蓉 等;《移动通信》;20131231(第13期);第76-78页 * |
| 自动路测系统中智能数据采集终端设计;刘瑞兰 等;《计算机技术与发展》;20130731;第23卷(第7期);第226-229页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106897316A (en) | 2017-06-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10567506B2 (en) | Data storage method, SDN controller, and distributed network storage system | |
| US20200358848A1 (en) | Methods, systems, and media for providing distributed database access during a network split | |
| CN106897316B (en) | Method and device for processing signaling data | |
| US11068499B2 (en) | Method, device, and system for peer-to-peer data replication and method, device, and system for master node switching | |
| JP2017531250A (en) | Granular / semi-synchronous architecture | |
| US9367261B2 (en) | Computer system, data management method and data management program | |
| CN105493474B (en) | System and method for supporting partition level logging for synchronizing data in a distributed data grid | |
| JPWO2013018808A1 (en) | Distributed storage system and method | |
| US20230273866A1 (en) | Methods, systems, and media for controlling append-only file rewrites | |
| JP2018503159A (en) | Method, apparatus, and system for executing distributed transaction resources | |
| CN105138679A (en) | Data processing system and method based on distributed caching | |
| CN113703917B (en) | Multi-cluster resource data processing system and method and non-transient storage medium | |
| CN112069152B (en) | Database cluster upgrading method, device, equipment and storage medium | |
| US9767023B2 (en) | Method of controlling data writing to persistent storage device | |
| US9158817B2 (en) | Enhancing online querying service using client memorization | |
| EP3709173B1 (en) | Distributed information memory system, method, and program | |
| Afonso | Mechanisms for providing causal consistency on edge computing | |
| WO2017094194A1 (en) | Computer system and device management method | |
| EP4066117B1 (en) | Managing provenance information for data processing pipelines | |
| KR101973537B1 (en) | Apparatus and method for managing of heartbeat exchange period in the hadoop distributed file system | |
| US10412198B1 (en) | Methods for improved transmission control protocol (TCP) performance visibility and devices thereof | |
| CN106227791B (en) | Data storage architecture | |
| US20120303582A1 (en) | Local differential compression | |
| WO2022085013A1 (en) | System and method to resync flow rules | |
| JP2013182430A (en) | Replication system, information processing apparatus, replication method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |