CN1964346A - A system and method to process mass GPS data based on distributed system structure - Google Patents

Abstract

The large-volume GPS data process system based on distributed architecture comprises: the server module, which includes a global data share module, a receiving sub-module to build communication with data source and receive real-time GPS data, and a distribution sub-module to build communication with client and send extracted data from share module and wait for back information from client; and a client module with every client connected to one server to receive GPS data package, extract package, process data, extract information, and return process result. This invention improves data process sped and efficiency.

Description

Mass GPS data processing system and method based on distributed architecture

Technical Field

The invention relates to a system and a method for processing mass GPS data.

Background

The gps (global Positioning system) is a navigation system developed by the U.S. military in the 70 th generation on the basis of the "satellite navigation and Positioning with meridian instrument" technology. In the aspect of positioning, the GPS has the advantages of all weather, globality, high precision, high speed, real-time three-dimensional positioning, no error accumulation along with positioning time and the like. The application of the method causes revolutionary changes in the fields of navigation positioning, public safety, deformation monitoring, precision farming, meteorological service and the like. With the maturity of technology and the popularization of application, how to quickly and effectively process massive GPS data generated in real time in various industries and extract required information from the massive GPS data becomes a hot topic to be urgently solved in the current society. The system provides a simple and general massive GPS data processing method based on a distributed architecture just around the content.

The term distributed architecture is used to refer to a dual layer architecture, client/server, in the traditional sense. In this architecture, the application is divided into two parts: one part is that a plurality of users share required functions and information, namely a server terminal part; the other part is the functionality and information required by each user, i.e. the client side part. The client end sends request to the server end through the special interface, after receiving the request, the server end activates corresponding service and processes correspondingly, and then returns the processing information to the client end.

Disclosure of Invention

The invention aims to provide a massive GPS data processing system and method based on a distributed architecture.

The invention adopts the following technical scheme:

a massive GPS data processing system based on a distributed architecture comprises a server module and a client module, wherein the server module comprises a global packet data sharing pool, a receiving submodule and a distributing submodule, and the receiving submodule is used for establishing communication with a data source, receiving real-time GPS data, packaging and placing the real-time GPS data into the global packet data sharing pool; the distribution submodule is used for establishing communication with the client end, taking out the packet data from the global packet data sharing pool and sending the packet data, and then waiting for feedback information sent back by the client end;

each client end of the client module is connected to the same server end, receives the GPS data packet, unpacks the GPS data packet, performs corresponding processing on the GPS data packet, extracts required information, and feeds back a processing result to the server end.

A massive GPS data processing method based on a distributed system structure adopts the system, and is characterized by comprising the following steps:

1) the server module sets each operation parameter in the module configuration file according to the actual GPS data flow; initializing the module according to the configuration file and starting the module; the module starts two management threads, one is responsible for initializing a GPS data receiving submodule instance for each newly connected data source, and the other is responsible for initializing a packet data sending submodule instance for each newly connected client; adjusting system operation parameters according to packet processing feedback information and real-time GPS data receiving conditions returned by each client end in real time;

2) the client side sets each operation parameter in the module configuration file according to the actual application environment, initializes the module according to the module configuration file and starts the module; and receiving the GPS data packet, performing corresponding processing on the data after unpacking, extracting required information, and feeding back a processing result to the server side.

The invention adopts a distributed system structure to deploy system load (GPS data processing task) to a plurality of processing nodes, thereby realizing the rapid and effective processing of massive GPS data; introducing a real-time feedback mechanism in the data processing process to dynamically adjust the system operation parameters; a rectification mechanism is introduced to shape the random concurrent GPS data flow into regular data packet flow with ordered intervals, and each processing node is provided with a uniform data format, so that the overall data processing speed of the system is improved.

The invention is further described with reference to the following figures and examples.

Drawings

FIG. 1 is a system architecture diagram according to an embodiment of the present invention;

FIG. 2 is a diagram of a server module;

FIG. 3 is a flow diagram of a server module technique;

FIG. 4 is a flow chart of a receive submodule technique;

FIG. 5 is a flow diagram of a distribution submodule technique;

FIG. 6 is a client module technology flow diagram;

FIG. 7 is a schematic diagram of a GPS data processing module of a bottom vehicle of a Shanghai city traffic information service grid system.

Detailed Description

A massive GPS data processing system based on a distributed architecture is disclosed, wherein a system framework is shown in figure 1 and comprises a server module for receiving multisource GPS data, packaging and distributing the multisource GPS data, and a client module for unpacking and processing the GPS data. Wherein,

a server module: two sub-modules are included: a receiving submodule and a distributing submodule, as shown in fig. 2. The receiving submodule is mainly responsible for establishing communication with a data source, receiving real-time GPS data, packaging and placing the real-time GPS data into a global packet data sharing pool. The distribution submodule is mainly responsible for establishing communication with the client, taking out the packet data from the global packet data sharing pool and sending the packet data, and then waiting for feedback information sent back by the client. The server side is functionally equivalent to a rectifier, and processes original rough GPS data stream received from data source into stable and easily-processed data stream (the rough data means that the time intervals of each piece of data received by the server side are uneven due to different acquisition rates of original GPS data of each data source, and the stable data means that the original GPS data are packaged into data stream with the same time interval after being processed and then are sequentially distributed to each client side.

A client module: each client end is connected to the same server end, receives the GPS data packet with the uniform format, unpacks the GPS data packet, correspondingly processes the data, extracts required information, and feeds back the processing condition to the server end to provide decision basis for adjusting system operation parameters.

A massive GPS data processing method based on a distributed system structure adopts the system. Because the system operates in a distributed multi-computer environment, and the operation of each node (including a server and a plurality of client ends) has spatial independence and temporal parallelism, the overall technical process of the system can be divided into the following two parts and processing steps according to the functions of the node:

the server side is as shown in fig. 2:

1. setting various operation parameters (the number of nodes of a client, the size of a global packet data sharing pool and the sending time interval of a GPS data packet) in a module configuration file according to the actual GPS data flow;

2. initializing the module according to the configuration file and starting the module;

3. the module will start two management threads, and one will initialize a GPS data receiving submodule instance for each newly connected data source according to the determination result of whether there is a new data source connection, as shown in fig. 4, which specifically includes the steps of: 1) establishing connection, 2) obtaining original GPS data, and 3) packaging; 4) and (5) putting the data into a global packet data sharing pool, and turning to the step 2). And another, according to the judgment result of whether there is a new client connection, initializing a packet data transmission submodule instance for each newly connected client side, as shown in fig. 5. The method specifically comprises the following steps: 1) establishing connection; 2) acquiring a data packet from the global packet data sharing pool; 3) sending a bag; 4) and (5) waiting for feedback information, and turning to the step 2).

4. According to the packet processing feedback information and the real-time GPS data receiving condition returned by each client terminal in real time, the system operation parameters are adjusted, and the system operation efficiency is improved;

a client side:

1. setting various operation parameters in the module configuration file according to the actual application environment, such as a server-side IP address and a connection port;

2. initializing and starting a module according to the module configuration file;

3. the specific technical flow for running the client module is shown in fig. 6. The method comprises the following steps:

1) firstly, establishing connection with a server;

2) receiving a data packet;

3) processing data in the data packet;

4) feeding the processed information back to the server, and turning to the step 2);

the system and the method are applied to the process of processing the massive GPS data of the bottom layer vehicles of the Shanghai traffic information service grid system. The system mainly comprises a receiving and processing deviation rectifying module. The data receiving is responsible for the server module, and the data rectifying is responsible for the client module. The specific composition of the module is shown in fig. 7. Wherein:

[ distributed cluster environment ]

Server (1 station):

P42.8G, memory 512M, hard disk 80G, gigabit network card, Linux FC3, j2sdk1.4.2

Client (4 pieces):

P42.8G, memory 512M, hard disk 80G, gigabit network card, Linux FC3, j2sdk1.4.2

[ GPS data amount ]

6000 taxis in Shanghai city generate 1 piece of GPS data every 30 seconds;

150 buses in Shanghai city generate 1 piece of GPS data every 15 seconds. I.e., 75 million 6 pieces of GPS data per hour, 1814 million 4 pieces of GPS data per day, and 5 hundred million 4432 pieces of GPS data per month.

[ System operation flow ]

1. Deploying a server module and each client module in the cluster;

2. initializing a system example according to a configuration file, and starting a server side (a vehicle GPS data receiver) and a plurality of client sides (GPS data deviation correction nodes);

3. when each client end is connected to the server end, the server end starts an example of a distribution submodule for the client end (see fig. 3);

4. waiting for the data source (Shanghai city information center) to connect to the server side, and starting a receiving submodule example for the server side (see figure 3) once the data source is connected;

5. the server side packages and distributes the received GPS data packets to each client side;

6. the client receives the real-time vehicle GPS data packet from the server, corrects the error, stores the error into the database, and feeds back the relevant processing information to the server;

7. the server dynamically adjusts the operation parameters of the system instance (increase or decrease the number of nodes of the client end, increase or decrease,

Changing the global packet data sharing pool size and adjusting the GPS packet transmission time interval).

The embodiment of the invention adopts a distributed system structure, and the system load (GPS data processing task) is deployed to a plurality of processing nodes, thereby realizing the rapid and effective processing of mass GPS data; introducing a real-time feedback mechanism in the data processing process to dynamically adjust the system operation parameters; a rectification mechanism is introduced to shape the random concurrent GPS data flow into regular data packet flow with ordered intervals, and each processing node is provided with a uniform data format, so that the overall data processing speed of the system is improved.

Claims (4)

1. A mass GPS data processing system based on a distributed architecture comprises a server module and a client module, and is characterized in that: the server module comprises a global packet data sharing pool, a receiving submodule and a distributing submodule, wherein the receiving submodule is used for establishing communication with a data source, receiving real-time GPS data, packaging and putting the real-time GPS data into the global packet data sharing pool; the distribution module is used for establishing communication with the client end, taking out the packet data from the global packet data sharing pool and sending the packet data, and then waiting for feedback information sent back by the client end;

each client end of the client module is connected to the same server end, receives the GPS data packet, unpacks the GPS data packet, performs corresponding processing on the GPS data packet, extracts required information, and feeds back a processing result to the server end.

2. The mass GPS data processing system based on distributed architecture according to claim 1, wherein: the GPS data packet is in a uniform format.

3. The mass GPS data processing system based on distributed architecture according to claim 2, wherein: the GPS data packets are data streams of the same time interval.

4. A massive GPS data processing method based on a distributed architecture, which adopts the system of any one of claims 1 to 3, and is characterized by comprising the following steps:

1) the server module sets each operation parameter in the module configuration file according to the actual GPS data flow; initializing the module according to the configuration file and starting the module; the module starts two management threads, one is responsible for initializing a GPS data receiving submodule instance for each newly connected data source, and the other is responsible for initializing a packet data sending submodule instance for each newly connected client; adjusting system operation parameters according to packet processing feedback information and real-time GPS data receiving conditions returned by each client end in real time;

2) the client side sets each operation parameter in the module configuration file according to the actual application environment, initializes the module according to the module configuration file and starts the module; and receiving the GPS data packet, performing corresponding processing on the data after unpacking, extracting required information, and feeding back a processing result to the server side.

Images