CN116776275A - Multithreading-based multilink data receiving fusion method - Google Patents
Multithreading-based multilink data receiving fusion method Download PDFInfo
- Publication number
- CN116776275A CN116776275A CN202310467072.6A CN202310467072A CN116776275A CN 116776275 A CN116776275 A CN 116776275A CN 202310467072 A CN202310467072 A CN 202310467072A CN 116776275 A CN116776275 A CN 116776275A
- Authority
- CN
- China
- Prior art keywords
- data
- node
- nodes
- protocol
- database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000007500 overflow downdraw method Methods 0.000 title claims abstract description 18
- 238000000034 method Methods 0.000 claims abstract description 69
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 41
- 238000001914 filtration Methods 0.000 claims abstract description 34
- 230000008569 process Effects 0.000 claims abstract description 24
- 230000004927 fusion Effects 0.000 claims abstract description 23
- 238000004458 analytical method Methods 0.000 claims abstract description 17
- 238000010276 construction Methods 0.000 claims abstract description 8
- 238000005516 engineering process Methods 0.000 claims description 37
- 238000012545 processing Methods 0.000 claims description 20
- 230000000903 blocking effect Effects 0.000 claims description 11
- 230000007246 mechanism Effects 0.000 claims description 8
- 230000002159 abnormal effect Effects 0.000 claims description 4
- 238000013507 mapping Methods 0.000 claims description 4
- 230000001360 synchronised effect Effects 0.000 claims description 4
- 230000001550 time effect Effects 0.000 claims description 4
- 238000004364 calculation method Methods 0.000 claims description 3
- 238000012544 monitoring process Methods 0.000 claims description 3
- 238000007493 shaping process Methods 0.000 claims description 3
- 238000012795 verification Methods 0.000 claims description 2
- 230000005540 biological transmission Effects 0.000 abstract description 16
- 238000012423 maintenance Methods 0.000 abstract description 5
- 239000000725 suspension Substances 0.000 abstract description 2
- 230000008034 disappearance Effects 0.000 description 5
- 238000011161 development Methods 0.000 description 4
- 230000018109 developmental process Effects 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 238000011084 recovery Methods 0.000 description 3
- 238000005096 rolling process Methods 0.000 description 3
- 230000032683 aging Effects 0.000 description 2
- 239000000203 mixture Substances 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000005856 abnormality Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000012827 research and development Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S19/00—Satellite radio beacon positioning systems; Determining position, velocity or attitude using signals transmitted by such systems
- G01S19/38—Determining a navigation solution using signals transmitted by a satellite radio beacon positioning system
- G01S19/39—Determining a navigation solution using signals transmitted by a satellite radio beacon positioning system the satellite radio beacon positioning system transmitting time-stamped messages, e.g. GPS [Global Positioning System], GLONASS [Global Orbiting Navigation Satellite System] or GALILEO
- G01S19/42—Determining position
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/547—Messaging middleware
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Radar, Positioning & Navigation (AREA)
- Remote Sensing (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Artificial Intelligence (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Computer Networks & Wireless Communication (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to a multi-link data receiving and fusion method based on multithreading, and belongs to the field of high-availability data fusion. The method is applied to the remote sending and storage of the data with high reliability, the construction of the node service cluster, the high reliability, the application environment such as GPS or Beidou positioning object and object tracking display service, and the like, so that a data provider does not need to worry about the shutdown and outage of a server or the data transmission suspension problem caused by the stop of the node application process and the like. When one of the service clusters of the multiple machine rooms has a power failure phenomenon, a user does not need to worry about data interrupt transmission, so that the stability of data transmission is improved, the operation and maintenance cost of a supervisor is reduced, and the user only worries about data display service. Meanwhile, the data source connection is configurable, the data protocol analysis is configurable, the fusion filtering algorithm is configurable, the user can customize the data source connection, and the data protocol and the data filtering method are customized and used for analyzing, filtering and forwarding data.
Description
Technical Field
The invention belongs to the field of high-availability data fusion, and particularly relates to a multi-link data receiving and fusion method based on multithreading.
Background
Along with the development of Beidou positioning technology, the mobile phone terminal, the microcomputer terminal and the webpage terminal are used more and more widely, and provide convenient and quick user positioning experience service for users, so that the Beidou technology is used as a conventional technology for serving a plurality of fields such as politics, military, civilian use and the like. The Beidou data protocol has more various content compositions, and has the digital forms of longitude, latitude, time and date, and the character string forms of daily texts and the like. However, the conventional positioning technology cannot solve the problems of positioning accuracy of data, long data time transmission period and lack of data sustainable transmission, and thus is no longer suitable for positioning services which are used more and more frequently and require high reliability.
In order to solve the problems that the Beidou positioning technology has server shutdown, server application process disappearance, data loss, data time sequence confusion, beidou positioning data offset and the like in the application process, and to enable data transmission to be more stable and efficient in the process of combining the Beidou positioning technology with the application, the invention aims to apply a multithreading technology and a fusion filtering algorithm technology to data source connection, data protocol analysis and error data filtering of Beidou positioning so as to solve various problems of data in the receiving and forwarding process.
The invention is a data transmission service applied to Beidou/GPS positioning data, and mainly solves the problems of multi-link data filtering fusion, multi-protocol analysis generalization and multi-connection generalized sustainable transmission of basic data. The problems to be solved specifically include:
the problem that data such as shutdown, disappearance of application processes and the like are not lost when the shutdown occurs is solved, and the node cluster building mode is needed, so that the stability is improved, and the operation and maintenance cost is reduced.
The method solves the problems of excessive types of processing protocols, processing of new protocols in uncertain formats and the like, and needs to design a set of general configuration files for analyzing various data protocols, and the user can customize the data protocol formats by configuring the specification through the configuration files, so that the operation steps of the user are reduced, and the research and development cost is reduced.
The method solves the problems of data redundancy and error data transmission, and according to the characteristics of user-defined data, the user can self-define a data filtering algorithm to filter the error data.
The problem of complex and various data sources is solved.
The method solves the problem of state synchronization of node cluster construction.
The node resumes the started state reply problem.
The method solves the problems that single or multiple servers are stopped, processes are lost and the like in the state synchronization process of all nodes, and the transmission of requests fails.
Disclosure of Invention
First, the technical problem to be solved
The technical problem to be solved by the invention is how to provide a multithreading-based multilink data receiving fusion method so as to solve the problems of server shutdown, disappearance of server application processes, data loss, disordered data time sequence, beidou positioning data offset and the like in the application process of the Beidou positioning technology.
(II) technical scheme
In order to solve the technical problems, the invention provides a multi-thread-based multi-link data receiving fusion method, which is based on a fusion system, wherein the fusion system consists of a plurality of nodes to form a cluster, each node is connected with three targets, the first target is a data source, the data source is used for receiving data, the second target is a database, the database is a relational database and is used for recording the cluster state, and the third target is a cache middleware Redis and is used for executing a data fusion script protocol; the method comprises the following steps:
defining a receiver form, a data protocol form and a data filtering algorithm in each node through a custom configuration file, establishing a plurality of threads through a thread pool technology, establishing a receiver for each thread, simultaneously operating data receiving and data processing services, enabling the data processing services of each receiver not to be influenced, and summarizing multiple data source data after analysis, so as to perform data de-duplication and data filtering operation;
each node is an independently operated node, and each node is connected with a plurality of data sources through a multithreading technology and operates in parallel, each node removes data received by the data sources, then calculates the distance between the data at the two last points and calculates the receiving time difference, judges whether the data is error data, if so, filters the data, finally sends the processed data to a cache middleware Redis, performs data de-duplication and data sending by means of a script technology and a SET SET of the cache middleware Redis, and timely destroys the cached outdated data;
the cache middleware Redis is arranged outside the application cluster, each node of the cluster is connected with the cache middleware Redis through a TCP protocol, data are sent to a SET SET of the cache middleware Redis, when the SET SET does not exist, the data are sent to a blocking queue, the cache middleware Redis supports multiple language connection, and the blocking queue data can be consumed;
each node is deployed on different servers, each node is a process in the servers, the clusters are built on a plurality of servers, the servers can be distributed in a plurality of machine rooms, if one server is down or the servers are restarted, the processes disappear, the clusters lose one node, other nodes still operate, and data cannot be lost; because the data of all nodes receives the same data, the transmitted data also has the same property, even if a server of one node fails, the other nodes are not influenced to receive the transmitted data, and the data of all nodes are transmitted to the SET SET to judge whether the data exist or not due to the characteristics that the cache middleware Redis script technology and the SET SET are not repeated, if the data exist, the data are already received, if the data do not exist, the data are not yet received, so when one node is hung up, the data can be restored to be transmitted within 1 second, and the data have the uniqueness;
the data received by each node has data time, firstly, each node filters out the data with disordered time, however, the time that each node sends to the cache middleware Redis is different, so the customization of the script protocol of the cache middleware Redis has two solutions, the first solution is that because the data of each node has order, only the SET is SET in the cache middleware dis, the data of all nodes are sent to the SET, judging whether the data is received or not by utilizing the non-repeatability of the SET SET, if the SET SET has no data, caching the data to the SET and sending the data to a blocking queue, wherein the queue has a first-in first-out characteristic, so that the data is finally consumed in order; setting a right Key in a cache middleware Redis, wherein the right Key has 5-second time effect, the 5-second time effect can be customized to be any positive shaping numerical value, the value of the Key is the IP address of a node, the right allocation rule is that the node which firstly transmits data has the right, that is, the value of the Key is the IP address of the node which firstly transmits data, the node has the right of transmitting data to a SET SET in 5 seconds, judging whether the SET receives the same data or not, if not, caching the data to the SET SET, and transmitting the data to a blocking queue; after 5 seconds, an empty Key is created again, or any node sends data first, then the IP address of the node is assigned to the Key, and the data of each node has time sequence, so that the data finally collected to the queue has time sequence;
the user sends a request to the node 1, the node 1 obtains cluster configuration through a configuration file, asynchronously forwards the request to other nodes, and the synchronous state is achieved, so that all the nodes of the cluster have the same receiving and forwarding state at the same time, and when the synchronization is successful, the current state information is stored in a database; when one node is down or the request is not passed, the node can be removed, and the request operation of other nodes is not influenced; if the request is not down or the request is not passed, when one node service request fails, the last-step state of the database is read by using the last-step state of the node stored in the database so as to make the states of all the nodes consistent, and the last-step state of the node is recovered; the state of the nodes, connection and forwarding information are stored by using a database, and when the nodes are hung up, the nodes are restarted, so that the connection state and other states can be recovered.
(III) beneficial effects
The invention provides a multithreading-based multilink data receiving fusion method, which discloses a multithreading-based multilink data fusion filtering method, and has the main advantages that:
1. the multi-link data fusion solution is designed, even if the problems of server shutdown, disappearance of application processes and the like occur, the data can be restored to be sent within one second, so that a client does not need to worry about the problems of power failure and the like, and the client can operate the application by starting a server without excessive operations due to a starting script.
2. The solution of cluster construction is designed, a node is requested to send by utilizing a multithreading technology, and the node can send the same request to all other nodes through multithreading asynchronously, so that the problems of uploading configuration files, synchronizing sending states and the like of the cluster can be solved. However, when a server is down, the application node on the server disappears, and other nodes remove the node, so that the normal operation of other nodes is not affected. However, when a request problem occurs to one node, the operation of rolling back the states of all nodes can be set, so that the states of all nodes are consistent, and the operation and maintenance operations are reduced.
3. A method for supporting a custom data filtering algorithm is designed. The user can customize what kind of data is obtained according to own data requirements, and customize own filtering method by uploading binary algorithm files and configuration files, so that the method is simple and effective.
4. A method for supporting simple construction service for connecting multiple data sources is designed, a node can be connected with multiple data sources by utilizing a multithreading technology, basic information of the multiple data sources, forwarding destinations, routing tables and the like are configured, configuration files are uploaded, application nodes required by users can be customized and stored in a database, and the node is restarted and still uses the last configuration connection information without uploading the configuration files again.
5. The method has the advantages that a generalized solution for data protocol analysis is designed, the function of processing the data protocol by the application node is universal through customizing the configuration file protocol, development of a developer for a new protocol is reduced, and the environment of a custom data protocol analysis algorithm can be built only by uploading the configuration file once, so that operation of a user-defined data protocol is reduced.
6. An overall architecture is designed, which comprises a cache middleware Redis, a database and node application, and supports clients written in multiple languages to access and acquire data.
Drawings
FIG. 1 is a system architecture diagram of the present invention;
FIG. 2 is a business flow diagram;
FIG. 3 is a diagram of a multi-link deployment;
FIG. 4 is a synchronization diagram of node cluster states;
FIG. 5 is a difference data processing flow;
fig. 6 is a flow chart of a difference algorithm.
Detailed Description
To make the objects, contents and advantages of the present invention more apparent, the following detailed description of the present invention will be given with reference to the accompanying drawings and examples.
The invention belongs to the field of high-availability data fusion, and relates to a multi-link data receiving fusion method based on multithreading, in particular to a multithreading technology, a filtering algorithm technology and a data protocol analysis technology.
The invention provides a solution for data filtering fusion, node state synchronization and node state recovery, and combines a Beidou technology with a multithreading technology.
The method of the invention is applied to the remote sending and storage of the data with high reliability, the node service cluster is built, and the method can be applied to the application environments such as GPS or Beidou positioning objects, object tracking display services and the like, so that a data provider does not need to worry about the problems of shutdown and outage of a server, or data transmission suspension caused by the stop of the operation of a node application process and the like. When one of the service clusters of the multiple machine rooms has a power failure phenomenon, a user does not need to worry about data interrupt transmission, so that the stability of data transmission is improved, the operation and maintenance cost of a supervisor is reduced, and the user only worries about data display service. Meanwhile, the application has the configuration functions of configurable data source connection, configurable data protocol analysis, configurable fusion filtering algorithm and the like, and a user can customize the data source connection, customize the data protocol and the data filtering method and is used for analyzing, filtering and forwarding data.
The invention discloses a multi-link data receiving fusion method based on multithreading, which can start a plurality of nodes as a cluster, each node has the functions of data receiving, transmitting and data fusion, so that the situation of repeated data transmission or data loss cannot occur, and the specific architecture is shown in figure 1.
The method of the invention is based on a fusion system which consists of a plurality of nodes, each node is connected with three targets, the first target is a data source, a plurality of data sources can be used for receiving data, the second target is a database, the database is a relational database, such as Oracle, mySQL or domestic database dream and miracle, and the like, is used for recording the cluster state, the third target is a cache middleware Redis, is used for executing a data fusion script protocol, and the scheme comprises the following several problem solutions.
1.1 multithreading Multi-data Source data
The method comprises the steps of defining a receiver form, a data protocol form and a data filtering algorithm in each node through a custom configuration file, establishing a plurality of threads through a thread pool technology, establishing a receiver for each thread, simultaneously operating data receiving and data processing services, enabling the data processing services of each receiver not to be influenced, and summarizing multiple data source data after analysis, and performing data de-duplication and data filtering operation.
1.2 multilink data fusion
As shown in FIG. 1, the cluster is composed of a plurality of nodes, each node is an independently operated node, and each node is connected with a plurality of data sources through a multithreading technology and operates in parallel, however, in order that the data received by each node is not redundant and will not generate errors, firstly, each node de-weights the data received by the data sources, then calculates the distance between the data at the latest two points and calculates the receiving time difference, judges whether the data is error data, if so, filters the data, finally sends the processed data to a cache middleware Redis, performs data de-duplication and data transmission by means of the script technology and the SET SET of the cache middleware Redis, destroys the outdated data at regular time, and can play a role in data recovery transmission within 1 second by means of the high efficiency of the script technology, and the specific script protocol is shown in FIG. 2.
(1) The invention provides data outwards, can support different programming languages,
the cache middleware Redis is arranged outside the application cluster, so long as the network is communicated, the cache middleware Redis can be arranged in different machine rooms or even different areas. The cache middleware Redis is a cluster building, each node of the cluster is connected with the cache middleware Redis through a TCP protocol, data is written into a SET SET and a blocking queue of the cache middleware Redis, and the cache middleware Redis supports multiple language connection, so that system consumption data developed by different development languages can be provided.
(2) The invention has high availability, and the fused data has uniqueness
Each node is deployed on different servers, each node is a process in the servers, the cluster is built on a plurality of servers, the servers can be distributed in a plurality of machine rooms, if one server is down or the server is restarted, the process disappears, the cluster can lose one node, other nodes still operate, and data cannot be lost, as shown in fig. 3. The data of all nodes are received by the same data, so that the transmitted data also has the same property, even if a server of one node fails, the other nodes are not influenced to receive the transmitted data, and the data of all nodes are transmitted to the SET SET to judge whether the data exist or not due to the characteristics that the Redis script technology of the caching middleware and the SET SET are not repeated, if the data exist, the data are already received, if the data do not exist, the data are not received, therefore, when one node is hung, the data can be restored to be transmitted within 1 second, the data have uniqueness, and then the data are received by a system needing to display the motion state of an object, and the running condition of the object on a map is displayed.
(3) The fused data has order
The data received by each node has data time, as shown in fig. 2, firstly, each node filters out the data with disordered time, however, the time when each node sends the data to the cache middleware Redis is different, so that the customization of the script protocol of the cache middleware Redis has two solutions, the first solution is that the data of each node has order, only a SET is needed to be SET in the cache middleware Redis, the data is sent to the SET by using the non-repeatability of the SET, whether the data is received or not is judged, if the SET has no data, the data is cached to the SET, and then the data is sent to a blocking queue, and the queue has the first-in first-out characteristic, so that the data has order finally consumed. The second scheme is that a permission Key is SET in a cache middleware Redis, and has an aging of 5 seconds, the 5-second aging can be defined as any positive shaping numerical value, the value of the Key is the IP address of a node, the permission is assigned by a permission rule that the node which sends data first has the permission, that is, the value of the Key is the IP address of the node which sends data first, the node has the permission to send data to a SET SET in 5 seconds, whether the SET receives the same data is judged, if not, the data is cached to the SET SET, and then the data is sent to a blocking queue. After 5 seconds, an empty Key is created again, or any node sends data first, then the IP address of the node is assigned to the Key, and the data of each node has time sequence, so that the data finally collected to the queue has time sequence.
1.3 Cluster setup
As shown in FIG. 3, a cluster is composed of one or more nodes, each node has the same data receiving and data sending services, each node is connected to the same data source, and there may be multiple data sources, so multiple data sources send the same data to each node at the same time, so long as the IP addresses and ports of all nodes are configured in an application configuration file, node cluster configurations are written using server socket writing rules and comma connections, for example 127.0.0.1:8080,127.0.0.1:8081,127.0.0.1:8082
1.4 data protocol commonality resolution scheme
The method comprises the steps of configuring a protocol name, a protocol ID and a byte composition of a protocol through a configuration file data protocol format, for example, a first byte is a word connector, a second byte is a data type, a third byte is a data quantity, a fourth byte to a seventh byte are longitude values of data and the like, configuring each byte or each byte array analysis method name of the protocol data in the configuration file through a programming language reflection technology, mapping the name into an executable method of the byte data through a reflection mechanism and a mapping mechanism of a programming language, and analyzing the data through the methods. The user also uploads the own method file to the database, loads the method file into the memory, acquires an executable method through a reflection technology, finally caches the method names and the methods into the application through the mapping relation, and when the data needs to be analyzed, the application can analyze the data through the methods only by acquiring an analysis method by analyzing the data method names through a configuration file protocol.
1.5 node connection State setup
The databases are relational databases, such as Oracle, mySQL or domestic databases, dream and general, and the databases can be connected by adding a database driver and a configuration file connection URL if the databases are needed, a connection target and a forwarding target are set in the databases, and the node can automatically read the databases to establish receiving and forwarding connection after starting.
1.6 node State setup
(1) State synchronization mechanism
The user sends a request to the node 1, the node 1 obtains cluster configuration through a configuration file, asynchronously forwards the request to the nodes such as the node 2, the node 3, the node 4 and the node … …, and the like, and the synchronous state enables all the nodes of the cluster to have the same receiving and forwarding state at the same time, and when the synchronous is successful, the current state information is stored in a database.
(2) Node removal mechanism
When one node is down or the request is not passed, the node can be removed, and the request operation of other nodes is not influenced.
(2) State rollback mechanism
And storing the last-step state of the nodes by using the database, and when one node service request fails, reading the last-step state of the database and rolling back to recover the last-step state.
(3) State recovery mechanism
The state of the nodes, connection and forwarding information are stored by using a database, and when the nodes are hung up, the nodes are restarted, so that the connection state and other states can be recovered.
1.7 customizable data Filter protocol
(1) Custom data filtering algorithm
The user can customize the filtering data algorithm by uploading the binary file of the data filtering algorithm. The application loads the binary file into the memory, and then obtains the filtering algorithm through the reflection technology of the programming language, so that the application can filter the data which the user wants to filter through the algorithm.
1.8 custom data connection scheme
(1) Custom data source connection
The invention supports various data sources, and configures the data source connection protocol TCP/UDP through the configuration file, the data connection mode, the data source is configured for a client or a server and the like
(2) Custom TCP data sticky half-packet data processing algorithm
TCP data sticky packet half-packet data processing algorithm configured through configuration file
(3) Custom data protocol parsing rules
And configuring data protocol construction and analysis rules through the configuration file, and analyzing data.
1.9 startup script
An application starting script is established, the script is placed in a folder in which a server system is started and operated, when the server is started, the application is automatically operated, and applications such as a cache middleware Redis, a database, a message queue middleware and the like are cached and have an execution sequence.
2.0 Supervisory script
And writing a monitoring script, monitoring the names of application processes, caching middleware Redis and database processes, and the method is used for starting the applications such as a database, caching middleware Redis, message queue middleware and the like, and has an execution sequence.
Example 1:
the invention is an application developed based on Java open source technology framework, which supports various databases, various data sources, various data protocols and various stream data filtering algorithms. The process of setting up is that firstly, a database and a cache middleware Redis are set up, then a data source is started, node application is started on a plurality of servers, configuration files are uploaded, and finally, the data source is started to send data, so that data fusion filtering processing can be carried out on the data.
(1) Connector configuration: by establishing multiple data source connections through multiple threads, the connector may configure multiple connection protocols, TCP or UDP connections, with UDP supporting multicast and unicast modes, and TCP connections supporting handling data sticky and half-packet problems. The invention provides two data sticking packet half-packet processing algorithms, wherein one of the two data sticking packet half-packet processing algorithms is that each frame in data is provided with a data head and a data length field, whether the current data is complete one-frame data is judged according to the data head and the data length, if not, the current data is directly discarded, and the second data sticking packet half-packet processing algorithm is that the current data is supported to be provided with a frame head, a frame tail and a verification byte for detecting whether the data is complete one-frame data or not, if the current data is the half-packet, the current data is added into a buffer memory, and the next frame data is to be processed.
(2) Protocol configuration: configuring a protocol name, a protocol ID and a protocol sticky packet half-packet analysis algorithm, and configuring a data type and an analysis algorithm represented by each byte of protocol data;
(3) Filter configuration: the invention provides two algorithms by configuring a custom algorithm. An algorithm for calculating the speed of a Beidou positioning object according to time sequence data, judging whether the current data or the object is abnormal according to the speed, and performing abnormality processing, wherein the calculation formula is as follows:
distence＝R*arccos(sin(latitude1)*sin(latitude 2)+cos(latitude1)*cos(latitude 2)*cos(longitude1-longitude 2))
v＝distence/(time2–time1)
where distance is distance, R is radius, latitude1 is the first point latitude, latitude2 is the second point latitude, longitude1 is the first point longitude, longitude2 is the second point longitude, v is speed, time2 is the second point time, and time1 is the first point time.
The algorithm can also be customized, if the abnormal data is Gaussian distribution, a Gaussian filtering algorithm can be used for establishing a confidence interval, the confidence interval data is reserved, and the error data is filtered.
(4) Configuring a difference algorithm: if the GPS data or the Beidou data is adopted, configuration options can be added. Due to the adoption of the multithreading technology, the difference value calculation of a plurality of objects can be supported.
(5) The method in the field configuration is configured to analyze the protocol data, and the node can also be used to apply the self protocol data analysis method.
The invention discloses a data fusion filtering method of multithreading and multilink data, which has the main advantages that:
1. the multi-link data fusion solution is designed, even if the problems of server shutdown, disappearance of application processes and the like occur, the data can be restored to be sent within one second, so that a client does not need to worry about the problems of power failure and the like, and the client can operate the application by starting a server without excessive operations due to a starting script.
2. The solution of cluster construction is designed, a node is requested to be sent by utilizing a multithreading technology, and the same request is sent to other nodes by multithreading asynchronization, so that the problems of uploading configuration files, synchronizing sending states and the like of the cluster can be solved. However, when a server is down, the application node on the server disappears, and other nodes remove the node, so that the normal operation of other nodes is not affected. However, when a request problem occurs to one node, the operation of rolling back the states of all nodes can be set, so that the states of all nodes are consistent, and the operation and maintenance operations are reduced.
3. A method for supporting a custom data filtering algorithm is designed. The user can customize what kind of data is obtained according to own data requirements, and customize own filtering method by uploading binary files and configuration files, so that the method is simple and effective.
4. A method for supporting simple construction service for connecting multiple data sources is designed, a node can be connected with multiple data sources by utilizing a multithreading technology, basic information of the multiple data sources, forwarding destinations, routing tables and the like are configured, configuration files are uploaded, application nodes required by users can be customized and stored in a database, and the node is restarted and still uses the last configuration connection information without uploading the configuration files again.
5. The method has the advantages that a generalized solution for data protocol analysis is designed, the function of processing the data protocol by the application node is universal through customizing the configuration file protocol, development of a developer for a new protocol is reduced, and the environment of a custom data protocol analysis algorithm can be built only by uploading the configuration file once, so that operation of a user-defined data protocol is reduced.
6. An overall architecture is designed, which comprises a cache middleware Redis, a database and node application, and supports clients written in multiple languages to access and acquire data.
The foregoing is merely a preferred embodiment of the present invention, and it should be noted that modifications and variations could be made by those skilled in the art without departing from the technical principles of the present invention, and such modifications and variations should also be regarded as being within the scope of the invention.
Claims (10)
1. The method is characterized in that the method is based on a fusion system, the fusion system consists of a plurality of nodes to form a cluster, each node is connected with three targets, the first target is a data source, the data source is used for receiving data, the second target is a database, the database is a relational database and is used for recording cluster states, and the third target is a cache middleware Redis and is used for executing a data fusion script protocol; the method comprises the following steps:
defining a receiver form, a data protocol form and a data filtering algorithm in each node through a custom configuration file, establishing a plurality of threads through a thread pool technology, establishing a receiver for each thread, simultaneously operating data receiving and data processing services, enabling the data processing services of each receiver not to be influenced, and summarizing multiple data source data after analysis, so as to perform data de-duplication and data filtering operation;
each node is an independently operated node, and each node is connected with a plurality of data sources through a multithreading technology and operates in parallel, each node removes data received by the data sources, then calculates the distance between the data at the two last points and calculates the receiving time difference, judges whether the data is error data, if so, filters the data, finally sends the processed data to a cache middleware Redis, and executes data de-duplication and data sending by means of the script technology and the SET SET of the cache middleware Redis, and timely destroys the cached outdated data;
the method comprises the steps that a cache middleware Redis is arranged outside an application cluster, each node of the cluster is connected with the cache middleware Redis through a TCP protocol, data are sent to a SET SET of the cache middleware Redis, when the SET SET does not exist, the data are sent to a blocking queue, the cache middleware Redis supports multiple language connection, and blocking queue data can be consumed;
each node is deployed on different servers, each node is a process in the servers, the clusters are built on a plurality of servers, the servers are distributed in a plurality of machine rooms, if one server is down or the servers are restarted, the processes disappear, the clusters lose one node, however, other nodes still operate, and data cannot be lost; the data of all nodes are received by the same data, so that the transmitted data have the same property, even if a server of one node fails, the other nodes are not influenced to receive the transmitted data, and the data of all nodes are transmitted to the SET SET to judge whether the data exist or not due to the characteristics that the Redis script technology of the caching middleware and the SET SET are not repeated, if the data exist, the data are received, if the data do not exist, the data are not received, so that when one node is hung, the data can be restored to be transmitted within 1 second, and the data have uniqueness;
the data received by each node has data time, firstly, each node filters out the data with disordered time, however, the time that each node sends to the cache middleware Redis is different, so the customization of the script protocol of the cache middleware Redis has two solutions, the first solution is that because the data of each node has order, only a SET is SET in a cache middleware Redis, data of all nodes are sent to the SET, whether the data are received or not is judged by utilizing the unrepeatability of the SET, if the SET SET has no data, the data is cached in the SET SET and sent to a blocking queue, and the queue has the characteristic of first-in first-out, so that the data is finally consumed in order; setting a right Key in a cache middleware Redis, wherein the right Key has 5 seconds of time effect, the 5 seconds of time effect can be defined as any positive shaping numerical value, the value of the Key is the IP address of a node, the right allocation rule is that the node which firstly transmits data has the right, that is, the value of the Key is the IP address of the node which firstly transmits data, the node has the right of transmitting data to a SET SET in 5 seconds, judging whether the SET receives the same data or not, if not, caching the data to the SET SET, and then transmitting the data to a blocking queue; after 5 seconds, an empty Key is created again, or any node sends data first, then the IP address of the node is assigned to the Key, and the data of each node has time sequence, so that the data finally collected to the queue has time sequence;
the user sends a request to the node 1, the node 1 obtains cluster configuration through a configuration file, asynchronously forwards the request to other nodes, and the synchronous state is achieved, so that all the nodes of the cluster have the same receiving and forwarding state at the same time, and when the synchronization is successful, the current state information is stored in a database; when one node is down or the request is not passed, removing the node, and not affecting the request operation of other nodes; if the request is not down or the request is not passed, when one node service request fails, the last-step state of the database is read by using the last-step state of the node stored in the database so as to make the states of all the nodes consistent, and the last-step state of the node is recovered; the state of the nodes, connection and forwarding information are stored by using a database, and when the nodes are hung up, the nodes are restarted, and the connection state and other states are restored.
2. The multi-thread based multi-link data reception fusion method of claim 1, wherein the relational database is Oracle, mySQL, dream or miracle.
3. The multithreading-based multilink data reception fusion method of claim 1, wherein multiple data sources simultaneously transmit the same data to each node, and node cluster configuration is written using server socket writing rules and comma connections as long as IP addresses and ports of all nodes are configured in an application configuration file.
4. The multithreading-based multilink data reception fusion method of claim 1, wherein the protocol name, protocol ID, byte array of the protocol are configured by a configuration file data protocol format configuration, and the method name is resolved by configuring each byte or each byte array of the protocol data in the configuration file using a programming language reflection technique, and the names are mapped into executable methods of byte data by which the data is resolved using a reflection mechanism and a mapping mechanism of the programming language.
5. The multi-thread based multi-link data receiving fusion method according to claim 1, wherein the user loads the binary file into the memory by uploading the binary file of the data filtering algorithm, and then obtains the filtering algorithm by the reflection technology of the programming language, so that the application can filter the data which the user wants to filter by the algorithm.
6. The multi-thread based multi-link data receiving fusion method according to claim 1, wherein the user customizes the data source connection, configures the data source connection protocol TCP/UDP through the configuration file, configures the data connection mode, and the data source for the client or the server, configures the TCP data sticky packet half packet data processing algorithm through the configuration file, configures the data protocol construction and parsing rules through the configuration file, and parses the data.
7. The multithreading-based multilink data receiving fusion method of claim 1, wherein an application start script is established, the script is placed in a folder in which a server system is started and is operated, and when the server is started, an application is automatically operated, a middleware dis is cached, a database and a message queue middleware application are executed, and an execution sequence is provided.
8. The multithreading-based multilink data receiving fusion method of claim 1, wherein a monitoring script is written, application process names are monitored, middleware dis and database processes are cached, and the method is used for starting a database, middleware dis, message queue middleware application and has an execution sequence.
9. The multithreading-based multilink data reception fusion method according to any one of claims 1 to 8, wherein a plurality of data source connections are established through multithreading, a connector configures a plurality of connection protocols, TCP or UDP connections, wherein multicast and unicast modes are supported using UDP, TCP connections are used, and data sticky and half-packet problems are supported; the data sticking half-packet processing algorithm comprises two types, wherein one type is that each frame in the data is provided with a data head and a data length field, whether the current data is complete one-frame data is judged according to the data head and the data length, if not, the current data is directly discarded, and the second type is that the data is supported to be provided with a frame head, a frame tail and a verification byte and is used for detecting whether the data is complete one-frame data or not, if the data is sticking, the data is unpacked, and if the data is half-packet, a buffer is added, and the arrival of the next frame data is processed.
10. The multithreading-based multilink data reception fusion method of claim 9, wherein the filter configuration comprises: configuring a custom algorithm, namely calculating the speed of a Beidou positioning object according to data with time sequence, judging whether the current data or the object is abnormal according to the speed, and performing exception handling, wherein the calculation formula is as follows:
distence＝R*arccos(sin(latitude1)*sin(latitude2)+cos(latitude1)*cos(latitude2)*cos(longitude1-longitude2))
v＝distence/(time2–time1)
where distance is distance, R is radius, latitude1 is the first point latitude, latitude2 is the second point latitude, longitude1 is the first point longitude, longitude2 is the second point longitude, v is speed, time2 is the second point time, and time1 is the first point time;
the second is a custom algorithm, if the abnormal data is gaussian distribution, a confidence interval is established by using a gaussian filtering algorithm, confidence interval data is reserved, and error data is filtered.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310467072.6A CN116776275A (en) | 2023-04-26 | 2023-04-26 | Multithreading-based multilink data receiving fusion method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310467072.6A CN116776275A (en) | 2023-04-26 | 2023-04-26 | Multithreading-based multilink data receiving fusion method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116776275A true CN116776275A (en) | 2023-09-19 |
Family
ID=87986820
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310467072.6A Pending CN116776275A (en) | 2023-04-26 | 2023-04-26 | Multithreading-based multilink data receiving fusion method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116776275A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117540151A (en) * | 2023-12-08 | 2024-02-09 | 深圳市亲邻科技有限公司 | Data preprocessing method of data pushing system |
-
2023
- 2023-04-26 CN CN202310467072.6A patent/CN116776275A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117540151A (en) * | 2023-12-08 | 2024-02-09 | 深圳市亲邻科技有限公司 | Data preprocessing method of data pushing system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2021121370A1 (en) | Message loss detection method and apparatus for message queue | |
| CN111787066B (en) | Internet of things data platform based on big data and AI | |
| US8195757B2 (en) | Method, apparatus and computer program for controlling retention of publications | |
| CN105959151B (en) | A kind of Stream Processing system and method for High Availabitity | |
| US7353259B1 (en) | Method and apparatus for exchanging configuration information between nodes operating in a master-slave configuration | |
| CN107391276B (en) | Distributed monitor method, interception control device and system | |
| CN112084098A (en) | Resource monitoring system and working method | |
| US20070094659A1 (en) | System and method for recovering from a failure of a virtual machine | |
| CN112118174B (en) | Software defined data gateway | |
| CN112506702B (en) | Disaster recovery method, device, equipment and storage medium for data center | |
| CN107682169B (en) | Method and device for sending message by Kafka cluster | |
| CN110209507A (en) | Data processing method, device, system and storage medium based on message queue | |
| CN110708352B (en) | Data backup method for Internet of things equipment and Internet of things equipment | |
| CN116776275A (en) | Multithreading-based multilink data receiving fusion method | |
| WO2003003656A1 (en) | Method and system for auto discovery of ip-based network elements | |
| CN111464612A (en) | Method for providing stable computing service in severe environment | |
| CN113472469B (en) | Data synchronization method, device, equipment and storage medium | |
| CN113032477B (en) | Long-distance data synchronization method and device based on GTID and computing equipment | |
| CN115617536A (en) | Data synchronization method, device, network equipment and cluster | |
| CN113190546A (en) | Eureka service control method, system and readable storage medium | |
| CN112732482A (en) | Cifs service backup switching method, device, equipment and readable storage medium | |
| CN115022392B (en) | IOT-oriented distributed publishing and subscribing service method and system | |
| KR100377864B1 (en) | System and method of communication for multiple server system | |
| CN114826892B (en) | Cluster node control method, device, equipment and medium | |
| CN114661538B (en) | Distributed database monitoring method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |