CN104158757B - A kind of gps data enters library processing method and system - Google Patents
A kind of gps data enters library processing method and system Download PDFInfo
- Publication number
- CN104158757B CN104158757B CN201410414439.9A CN201410414439A CN104158757B CN 104158757 B CN104158757 B CN 104158757B CN 201410414439 A CN201410414439 A CN 201410414439A CN 104158757 B CN104158757 B CN 104158757B
- Authority
- CN
- China
- Prior art keywords
- gps
- message
- messages
- gps messages
- task pool
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Landscapes
- Position Fixing By Use Of Radio Waves (AREA)
Abstract
The present invention provides a kind of gps data and enters library processing method, realize transmitting GPS messages between the Communication Gateway of bottom and TCP service ends by I/O completing port technologies, process hierarchical cache after message, batch write into Databasce using message task pool and worker thread pond and improve error handle.It is an advantage of the current invention that not only increasing systematic function and volume of transmitted data, and the data blocking of transmitting procedure is avoided, while improving the integrality to mass GPS data processing in actual applications.
Description
Technical field
The present invention relates to a kind of data loading processing method, more specifically to a kind of place of mass GPS data loading
Reason method.
Background technology
In current industry, gps data treatment storage platform accesses the comparing that quantity reaches mass GPS message amount level
It is few, usually as a part for system.For receiving message, processing message and be shown to client, now not by this
Used as single one service, an important module being divided into whole large scale system is optimized link, so will deposit
In some defects.Prior art exist shortcoming have it is following some：
1st, after Communication Gateway receives the data of GPS terminal, directly process message and be put in storage, the ring for the treatment of message and storage
Festival-gathering takes most resource, and the easily therefore queue of barrage reception, causes queue full, message dropping to reduce communication network
Close the load capacity of receiving terminal message；
2nd, Communication Gateway is directly put in storage, and intermediate link needs caching to receive the queue of message, the queue for the treatment of message, treatment
The queue of the data of complete wait storage, Memory Load easily reaches the limit, influences the handling capacity of processing data；
3rd, the connection number of database is limited, when connection number is excessive, there may be resource contention and competition during operating database
Probability it is bigger, the storage service that place can be accessed is limited, and the number of Communication Gateway is restricted, it is impossible to born
Carry balanced.
The content of the invention
One of the technical problem to be solved in the present invention, is to provide a kind of gps data to enter library processing method, complete by I/O
Realize transmitting GPS messages between the Communication Gateway of bottom and TCP service ends into port technique, and use message task pool and work
Batch write into Databasce and error handle is improved after thread pool treatment message, not only increases systematic function and volume of transmitted data,
And the data blocking of transmitting procedure is avoided, while improving in actual applications to mass GPS message storage error handle
Integrality.
A kind of one of the technical problem to be solved in the present invention, gps data enters what library processing method was realized in：The side
Method includes that process, message task pool and the worker thread pond of I/O completing port technical transmission GPS messages process the mistake of GPS messages
Process and improve the process that GPS messages are put in storage error handle that journey, GPS messages are put in storage：
The process of the I/O completing ports technical transmission GPS messages is：Communication Gateway seals GPS messages as client
In plurality of data transmission socket, gps data is put in storage processing system as TCP service ends, and client is by the plural number
Individual data transfer socket is sent to the reception that TCP service ends complete GPS messages by the port of computer, and specific steps are such as
Under：
After step 11, initialization, one is created in the service end and intercepts socket, and by the socket that intercepts with clothes
The a port binding at business end, to realize the one-to-many transmission of GPS messages；
Step 12, one connecting object of SocketAsyncEventArgs classes of establishment in TCP service ends, by the company
Connect object and intercept socket bindings with described, whether the connecting object starts there are data in the port that asynchronous detection is bound with it
Transmission socket is accessed：When there is a data transfer socket requests connection, the connecting object just accesses the data
Transmission socket, and the socket that intercepts then continues the port that detection is bound with it；
Step 13, one reception object of SocketAsyncEventArgs classes of establishment, the reception in TCP service ends
Object is bound with the data transfer socket for accessing, and starts to receive GPS messages：If the data of the GPS messages for receiving
Length is equal to zero, then close the data transfer socket of correspondence access and discharge the reception object bound with it；If receiving
The data length of GPS messages is more than zero, then carry out packet splicing treatment：If the packet of the GPS messages for receiving is complete, directly
Connect and be put into message buffering queue, if the packet of the GPS messages for receiving is incomplete, be first put into reception buffering area etc.
Wait to continue to until packet completely places into message buffering queue；
After step 14, TCP service ends receive GPS messages, the sequencing according to received GPS messages makes packet receiving successively
Normal response returns to client；
The process of the message task pool and worker thread pond treatment GPS messages includes：
Step 21, create a plurality of packet parsing threads in TCP service ends the GPS messages in message buffering queue are entered
Row parsing, according to the different type that parsing is obtained, the GPS messages of variant type are put into message by the packet parsing thread
In a plurality of message queues of task pool, a plurality of message queues are corresponded with the GPS messages of variant type；
Step 22, each message queue of the message task pool realize read and write abruption using double buffering concept：Each message
Queue corresponds to two task pool buffer areas, is respectively task pool buffer area 1 and task pool buffer area 2, first by each message queue
In GPS messages write corresponding task pool buffer area 1, when the storage treatment of GPS messages is slow or TCP service ends connect
Connect number it is very many in the case of, the GPS messages in task pool buffer area 1 can slowly be piled up, and reach maximum, at this moment each message team
Remaining GPS messages can be written to corresponding task pool buffer area 2 in row, finally from the corresponding task pool of each message queue
GPS messages are read out in buffer area 1 and task pool buffer area 2 in batches, correspondence is written in the cache file of each message queue；
Step 23, the worker thread pond are according to different types of message queue corresponding configuration at least in message task pool
Individual work disposal thread, work disposal thread is responsible for the GPS reports in the corresponding cache file of each message queue described in step 22
Text reads out and is processed accordingly；
The process of GPS messages storage is：TCP service ends call data access layer, are handled well described in write-in in batches
GPS messages are to database；
It is described to improve the process that GPS messages are put in storage error handle：
When GPS messages are put in storage, if SQL statement occur performs error, the GPS messages for performing mistake are written to message and are appointed
Wrong buffer queue in business pond is processed：If what is run into is network problem, repeats and perform the SQL statement two
It is secondary, still in the case of mistake, wrong cache file is created in systems, during GPS messages are write into the wrong cache file,
If the SQL statement is problematic in itself, then directly by GPS message write error cache files, in case reclaiming or using
Investigate error reason in family.
Further, each message queue of the message task pool processes GPS message amounts more than corresponding two
During the capacity of task pool buffer area, waited in each message queue storage GPS messages be first written to each message queue caching text
In part, the occupancy situation of system CPU is read by creating monitoring thread timing:When system CPU occupancy rate is less than 20 percent
When, notify that work disposal thread reclaims the corresponding cache file of each message queue using semaphore.
Further, also including alarming short message service in methods described, the work disposal thread is by the report in GPS messages
After warning message is parsed, the alert if set according to user judges whether to need alarm, if meeting alert if, sends out
Alarming short message is sent to corresponding contact person.
Further, also checked including daily record in methods described, during whole GPS messages storage, user can
Daily record is checked, the daily record is divided into master log, alarming short message daily record, cache garbage collection daily record and error handle daily record, they are pressed
Date point file is preserved, and is so retained separately, is separately checked, has not only been reduced the time of user search daily record but also has conveniently been checked daily record
Details.
The second technical problem to be solved by the present invention, is to provide a kind of gps data storage processing system, complete by I/O
Realize transmitting GPS messages between the Communication Gateway of bottom and TCP service ends into port technique, and use message task pool and work
Batch write into Databasce and error handle is improved after thread pool treatment message, not only increases systematic function and volume of transmitted data,
And the data blocking of transmitting procedure is avoided, while improving in actual applications to mass GPS message storage error handle
Integrality.
The second technical problem to be solved by the present invention, what a kind of gps data storage processing system was realized in：The system
System includes TCP message receiver module, message processing module (MPM), data loading module and error handling module：
The TCP message receiver module：By I/O completing port technical transmission GPS messages：Communication Gateway is used as client
End, GPS messages are encapsulated in plurality of data transmission socket, and gps data is put in storage processing system as TCP service ends, visitor
The plurality of data is transmitted into socket connecing for TCP service ends completion GPS messages is sent to by the port of computer in family end
Receive, comprise the following steps that：
After step 11, initialization, one is created in the service end and intercepts socket, and by the socket that intercepts with clothes
The a port binding at business end, to realize the one-to-many transmission of GPS messages；
Step 12, one connecting object of SocketAsyncEventArgs classes of establishment in TCP service ends, by the company
Connect object and intercept socket bindings with described, whether the connecting object starts there are data in the port that asynchronous detection is bound with it
Transmission socket is accessed：When there is a data transfer socket requests connection, the connecting object just accesses the data
Transmission socket, and the socket that intercepts then continues the port that detection is bound with it；
Step 13, one reception object of SocketAsyncEventArgs classes of establishment, the reception in TCP service ends
Object is bound with the data transfer socket for accessing, and starts to receive GPS messages：If the data of the GPS messages for receiving
Length is equal to zero, then close the data transfer socket of correspondence access and discharge the reception object bound with it；If receiving
The data length of GPS messages is more than zero, then carry out packet splicing treatment：If the packet of the GPS messages for receiving is complete, directly
Connect and be put into message buffering queue, if the packet of the GPS messages for receiving is incomplete, be first put into reception buffering area etc.
Wait to continue to until packet completely places into message buffering queue；
After step 14, TCP service ends receive GPS messages, the sequencing according to received GPS messages makes packet receiving successively
Normal response returns to client；
The message processing module (MPM)：GPS messages are processed by message task pool and worker thread pond, is specifically included：
Step 21, create a plurality of packet parsing threads in TCP service ends the GPS messages in message buffering queue are entered
Row parsing, according to the different type that parsing is obtained, the GPS messages of variant type are put into message by the packet parsing thread
In a plurality of message queues of task pool, a plurality of message queues are corresponded with the GPS messages of variant type；
Step 22, each message queue of the message task pool realize read and write abruption using double buffering concept：Each message
Queue corresponds to two task pool buffer areas, is respectively task pool buffer area 1 and task pool buffer area 2, first by each message queue
In GPS messages write corresponding task pool buffer area 1, when the storage treatment of GPS messages is slow or TCP service ends connect
Connect number it is very many in the case of, the GPS messages in task pool buffer area 1 can slowly be piled up, and reach maximum, at this moment each message team
Remaining GPS messages can be written to corresponding task pool buffer area 2 in row, finally from the corresponding task pool of each message queue
GPS messages are read out in buffer area 1 and task pool buffer area 2 in batches, correspondence is written to the corresponding cache file of each message queue
In；
Step 23, the worker thread pond are according to different types of message queue corresponding configuration at least in message task pool
Individual work disposal thread, work disposal thread is responsible for reading the GPS messages in the cache file of each message queue described in step 22
Take out and processed accordingly；
The data loading module：TCP service ends call data access layer, batch write-in GPS messages to database；
The error handling module：When GPS messages are put in storage, if SQL statement occur performs error, mistake will be performed
The wrong buffer queue that GPS messages are written in message task pool is processed：If what is run into is network problem, repeat
Perform the SQL statement twice, still in the case of mistake, wrong cache file is created in systems, by the write-in of GPS messages
In the wrong cache file, if the SQL statement is problematic in itself, then directly GPS messages write error is cached
File, in case reclaiming or user's investigation error reason.
Further, each message queue of the message task pool processes GPS message amounts more than corresponding two
During the capacity of task pool buffer area, waited in each message queue storage GPS messages be first written to each message queue caching text
In part, the occupancy situation of system CPU is read by creating monitoring thread timing:When system CPU occupancy rate is less than 20 percent
When, notify that work disposal thread reclaims the corresponding cache file of each message queue using semaphore.
Further, alarming short message service module is also included in the system：The work disposal thread is by GPS messages
Warning message parse after, the alert if set according to user judges whether to need alarm, if meeting alert if,
Alarming short message is then sent to corresponding contact person.
Further, log pattern is also included in the system, during whole GPS messages storage, user can
Daily record is checked, the daily record is divided into master log, alarming short message daily record, cache garbage collection daily record and error handle daily record, they are pressed
Date point file is preserved, and is so retained separately, is separately checked, has not only been reduced the time of user search daily record but also has conveniently been checked daily record
Details.
The invention has the advantages that：
1st, realize transmitting GPS messages between Communication Gateway and TCP service ends by I/O completing port technologies：Improve and be
The handling capacity of system performance and transmission data, can support one-to-many framework, be easy to underlying platform to extend, and realize load balancing；
2nd, message task pool and worker thread pond are used：The data for avoiding the transmission link that treatment message link causes are blocked up
Plug, improves throughput of system；Number of threads can be configured as needed, farthest using server resource；Can shorten
The response time of task requests；
3rd, hierarchical cache, batch writing mechanism：The access frequency to database is reduced, the efficiency of storage is also improved；
4th, mistake caching mechanism：It is easy to failure reason analysis afterwards and investigation, improves in actual applications to mass GPS
The integrality of data processing.
Brief description of the drawings
The present invention is further illustrated in conjunction with the embodiments with reference to the accompanying drawings.
Fig. 1 is the application schematic diagram of present system.
Fig. 2 receives flow chart for the GPS messages of the inventive method.
Fig. 3 is the embodiment packet splicing algorithm schematic diagram of the inventive method one.
Data processings and storage schematic diagram of the Fig. 4 for the inventive method.
Fig. 5 is that the embodiment of the inventive method one writes GPS message schematic diagrames.
Fig. 6 is that the embodiment of the inventive method one reads GPS message schematic diagrames.
Specific embodiment
The system application schematic diagram shown in Fig. 1 is refer to, a kind of gps data storage processing system of the invention can be received
GPS messages from Communication Gateway simultaneously place into database after being processed, and the system is widely used in various onboard systems.
A kind of gps data of the present invention enters library processing method and specifically includes：The mistake of I/O completing port technical transmission GPS messages
Process and improve GPS messages storage that the process of journey, message task pool and worker thread pond treatment GPS messages, GPS messages are put in storage
The process of error handle：
The process of the I/O completing ports technical transmission GPS messages is：Communication Gateway seals GPS messages as client
In plurality of data transmission socket, gps data is put in storage processing system as TCP service ends, and client is by the plural number
Individual data transfer socket is sent to the reception that TCP service ends complete GPS messages by the port of computer, and specific steps are as schemed
Shown in 2：
After step 11, initialization, one is created in the service end and intercepts socket, and by the socket that intercepts with clothes
The a port binding at business end, to realize the one-to-many transmission of GPS messages；
Step 12, in TCP service ends create a SocketAsyncEventArgs class it is (described
SocketAsyncEventArgs classes are the asynchronous socket classes encapsulated in .NET) connecting object (connecting object can be with
It is acceptEventArgs objects), the connecting object is intercepted into socket bindings with described, the connecting object starts different
Whether there is data transfer socket to access in the port that step detection is bound with it：Asked when there is a data transfer socket
During connection, the connecting object just accesses data transfer socket, and it is described intercept socket then continue detection bound with it
Port；
Step 13, one reception object (reception of SocketAsyncEventArgs classes of establishment in TCP service ends
Object can be ReceiveEventArgs objects), the object that receives is bound with the data transfer socket of access, and
Start to receive GPS messages：If the data length of the GPS messages for receiving is equal to zero, the data transfer that correspondence is accessed is closed
Socket simultaneously discharges the reception object bound with it；If the data length of the GPS messages for receiving is more than zero, carry out at packet splicing
Reason：If the packet of the GPS messages for receiving is complete, message buffering queue is directly placed into, if the GPS reports for receiving
The packet of text is incomplete, then be first put into reception buffering area wait and continue to until packet completely places into message and delays
Rush queue；
In the present embodiment, the packet splicing treatment uses Packetization, and main process is as shown in Figure 3：
The buffering area initial value for receiving object is set to 4M, including receives buffering area and unused area, whole buffering
The size in area can be adjusted according to operating system and Installed System Memory.To receive the data that receive of order caching area of object by
Data are taken out according to starting position and side-play amount, finds packet header mark and bag length, subpackage is carried out according to bag length, by complete data
Bag puts into packet buffering queue, if also remaining data, circulation according to the step subpackage, until reaching described slow
Rush the position of area's side-play amount：If just data are a plurality of complete bags, the reception of the reception buffering area for receiving object is set
Original position is 0, will otherwise not receive complete bag (residual bag) and copies the original position for receiving buffering area to, sets original position
It is the length of residual bag, setting reception buffer size is：Buffering area actual size subtracts residual packet length；Then proceed to receive, after
It is continuous to carry out above-mentioned subpackage step.Because logical when some data that sender sends to recipient are received during TCP transmission
A bag can be often sticked into, so needing to carry out packet splicing treatment；
After step 14, TCP service ends receive GPS messages, the sequencing according to received GPS messages makes packet receiving successively
Normal response returns to client；
Fig. 4 is refer to, the process that the message task pool and worker thread pond process GPS messages is as follows：
Step 21, create a plurality of packet parsing threads in TCP service ends the GPS messages in message buffering queue are entered
Row parsing, according to the different type that parsing is obtained, the GPS messages of variant type are put into message by the packet parsing thread
In a plurality of message queues of task pool, a plurality of message queues are corresponded with the GPS messages of variant type；
As shown in figure 4, as a example by creating four packet parsing threads, four packet parsing lines are created in TCP service ends
Journey is parsed to the GPS messages for receiving, and according to the different type that obtains of parsing, the packet parsing thread is by variant class
The GPS messages of type are put into the corresponding message queue of message task pool, including position message queue, warning message queue, answer
Answer message queue, peripheral hardware message queue and self-inspection message queue etc.；
Step 22, each message queue of the message task pool realize read and write abruption using double buffering concept：Each message
Queue corresponds to two task pool buffer areas, is respectively task pool buffer area 1 and task pool buffer area 2, first by each message queue
In GPS messages write corresponding task pool buffering area 1, when the storage treatment of GPS messages is slow or TCP service ends connect
Connect number it is very many in the case of, the GPS messages in task pool buffer area 1 can slowly be piled up, and reach maximum, at this moment each message team
Remaining GPS messages can be written to corresponding task pool buffer area 2 in row, finally from the corresponding task pool of each message queue
GPS messages are read out in buffer area 1 and task pool buffer area 2 in batches, correspondence is written in the cache file of each message queue；
A message queue is now illustrated to realize write-in GPS messages by two task pool buffer areas and read GPS messages
Process：
The process of said write GPS messages is as shown in Figure 5：
Step 221, first setting task pool buffer area sequence number buffID；
If step 222, buffID=1, the GPS messages in message queue are added to task pool buffer area 1, work as task pool
When GPS messages in buffer area 1 reach maximum, buffID=2 is set, now remaining GPS messages can be write in message queue
Enter in task pool buffer area 2；If buffID=2, the GPS messages in message queue are added to task pool buffer area 2, when appoint
When GPS messages in business pond buffer area 2 reach maximum, buffID=1 is set, now remaining GPS messages in message queue
Can be written in task pool buffer area 1；
Step 223, write write buffer thread signal amount has been set after GPS messages；
The reading GPS message processes are as shown in Figure 6：
Step 224, reading write buffer thread signal amount；
If step 225, buffID=2, the GPS messages in task pool buffer area 1 are read, if buffID=1, read task
GPS messages in pond buffer area 2；
Step 226, the cache file that the GPS messages taken out from two task pool buffer areas are written to message queue
In；
Step 227, the cache file title for obtaining the message queue；
Step 228, according to message queue cache file title judge the CACHE DIRECTORY where disk space capacity, if
The capacity is less than or equal to 100M, then send alarm email and abandon the GPS messages in the cache file, if the capacity is more than
, then still be written to GPS messages in the cache file of message queue by 100M；
But, when each message queue treatment GPS message amounts of message task pool exceed two corresponding task pools
During the capacity of buffer area, wait the GPS messages of storage to be first written in the cache file of each message queue in each message queue, lead to
Cross and create the occupancy situation that system CPU is read in monitoring thread timing:When system CPU occupancy rate is less than 20 percent, utilize
Semaphore notifies that work disposal thread reclaims the corresponding cache file of each message queue；
Step 23, the worker thread pond are according to different types of message queue corresponding configuration at least in message task pool
Individual work disposal thread, for example, position message queue and configure eight positioning Message processing threads, and warning message queue configures four
(referring to Fig. 4) such as warning message treatment threads；Work disposal thread is responsible for each message queue described in step 22 is corresponding slow
The GPS messages deposited in file read out and are processed accordingly, and such as warning message, worker thread parses warning message
The alert if set according to user after out judges whether that needs issue alarm, if it is desired, then send alarming short message to right
The contact person for answering；
The process of GPS messages storage is：TCP service ends call data access layer, are handled well described in write-in in batches
GPS messages are to database；
It is described to improve the process that GPS messages are put in storage error handle：
When GPS messages are put in storage, if SQL statement occur performs error, the GPS messages for performing mistake are written to message and are appointed
Wrong buffer queue in business pond is processed：If what is run into is network problem, repeats and perform the SQL statement two
It is secondary, still in the case of mistake, wrong cache file is created in systems, during GPS messages are write into the wrong cache file,
If the SQL statement is problematic in itself, then directly by GPS message write error cache files, in case reclaiming or using
Investigate error reason in family.
During whole GPS messages storage, user can check daily record, and the daily record is divided into master log, alarms short
Letter daily record, cache garbage collection daily record and error handle daily record, by date divide them file to preserve, and are so retained separately, separate
Check, the time for not only having reduced user search daily record but also the details for conveniently checking daily record.
Based on above method, at a kind of gps data storage processing system, including TCP message receiver module, message of the invention
Reason module, data loading module and error handling module：
The TCP message receiver module：By I/O completing port technical transmission GPS messages：Communication Gateway is used as client
End, GPS messages are encapsulated in plurality of data transmission socket, and gps data is put in storage processing system as TCP service ends, visitor
The plurality of data is transmitted into socket connecing for TCP service ends completion GPS messages is sent to by the port of computer in family end
Receive, comprise the following steps that：
After step 11, initialization, one is created in the service end and intercepts socket, and by the socket that intercepts with clothes
The a port binding at business end, to realize the one-to-many transmission of GPS messages；
Step 12, one connecting object of SocketAsyncEventArgs classes of establishment in TCP service ends, by the company
Connect object and intercept socket bindings with described, whether the connecting object starts there are data in the port that asynchronous detection is bound with it
Transmission socket is accessed：When there is a data transfer socket requests connection, the connecting object just accesses the data
Transmission socket, and the socket that intercepts then continues the port that detection is bound with it；
Step 13, one reception object of SocketAsyncEventArgs classes of establishment, the reception in TCP service ends
Object is bound with the data transfer socket for accessing, and starts to receive GPS messages：If the data of the GPS messages for receiving
Length is equal to zero, then close the data transfer socket of correspondence access and discharge the reception object bound with it；If receiving
The data length of GPS messages is more than zero, then carry out packet splicing treatment：If the packet of the GPS messages for receiving is complete, directly
Connect and be put into message buffering queue, if the packet of the GPS messages for receiving is incomplete, be first put into reception buffering area etc.
Wait to continue to until packet completely places into message buffering queue；
After step 14, TCP service ends receive GPS messages, the sequencing according to received GPS messages makes packet receiving successively
Normal response returns to client；
The message processing module (MPM)：GPS messages are processed by message task pool and worker thread pond, is specifically included：
Step 21, create a plurality of packet parsing threads in TCP service ends the GPS messages in message buffering queue are entered
Row parsing, according to the different type that parsing is obtained, the GPS messages of variant type are put into message by the packet parsing thread
In a plurality of message queues of task pool, a plurality of message queues are corresponded with the GPS messages of variant type；
Step 22, each message queue of the message task pool realize read and write abruption using double buffering concept：Each message
Queue corresponds to two task pool buffer areas, is respectively task pool buffer area 1 and task pool buffer area 2, first by each message queue
In GPS messages write corresponding task pool buffer area 1, when the storage treatment of GPS messages is slow or TCP service ends connect
Connect number it is very many in the case of, the GPS messages in task pool buffer area 1 can slowly be piled up, and reach maximum, at this moment each message team
Remaining GPS messages can be written to corresponding task pool buffer area 2 in row, finally from the corresponding task pool of each message queue
GPS messages are read out in buffer area 1 and task pool buffer area 2 in batches, correspondence is written in the cache file of each message queue；
But, when each message queue treatment GPS message amounts of message task pool exceed two corresponding task pools
During the capacity of buffer area, wait the GPS messages of storage to be first written in the cache file of each message queue in each message queue, lead to
Cross and create the occupancy situation that system CPU is read in monitoring thread timing:When system CPU occupancy rate is less than 20 percent, utilize
Semaphore notifies that work disposal thread reclaims the corresponding cache file of each message queue；
Step 23, the worker thread pond are according to different types of message queue corresponding configuration at least in message task pool
Individual work disposal thread, work disposal thread is responsible for the GPS reports in the corresponding cache file of each message queue described in step 22
Text reads out and is processed accordingly；
The data loading module：TCP service ends call data access layer, batch write-in GPS messages to database；
The error handling module：When GPS messages are put in storage, if SQL statement occur performs error, mistake will be performed
The wrong buffer queue that GPS messages are written in message task pool is processed：If what is run into is network problem, repeat
Perform the SQL statement twice, still in the case of mistake, wrong cache file is created in systems, by the write-in of GPS messages
In the wrong cache file, if the SQL statement is problematic in itself, then directly GPS messages write error is cached
File, in case reclaiming or user's investigation error reason.
Alarming short message service module：After the work disposal thread parses the warning message in GPS messages, according to
The alert if of user's setting judges whether to need alarm, if meeting alert if, sends alarming short message to corresponding
It is people.
Log pattern：During whole GPS messages storage, user can check daily record, and the daily record is divided into Lord's Day
Will, alarming short message daily record, cache garbage collection daily record and error handle daily record, by date divide them file to preserve, and so separate
Preserve, separately check, the time for not only having reduced user search daily record but also the details for conveniently checking daily record.
Although the foregoing describing specific embodiment of the invention, those familiar with the art should manage
Solution, the specific embodiment described by us is merely exemplary, and rather than for the restriction to the scope of the present invention, is familiar with this
The technical staff in field should be covered of the invention in the equivalent modification and change made according to spirit of the invention
In scope of the claimed protection.
Claims (8)
1. a kind of gps data enters library processing method, it is characterised in that methods described is reported including I/O completing port technical transmissions GPS
Process and improve GPS reports that the process of text, the process of message task pool and worker thread pond treatment GPS messages, GPS messages are put in storage
The process of text storage error handle：
The process of the I/O completing ports technical transmission GPS messages is：Be encapsulated in for GPS messages as client by Communication Gateway
In plurality of data transmission socket, gps data is put in storage processing system as TCP service ends, and client is by the plural number
The reception that TCP service ends complete GPS messages is sent to by the port of computer according to transmission socket, is comprised the following steps that：
After step 11, initialization, socket is intercepted in establishment one in the service end, and intercepts socket and service end by described
A port binding, to realize the one-to-many transmission of GPS messages；
Step 12, one connecting object of SocketAsyncEventArgs classes of establishment in TCP service ends, the connection is right
As intercepting socket bindings with described, whether the connecting object starts have data transfer in the port that asynchronous detection is bound with it
Socket is accessed：When there is a data transfer socket requests connection, the connecting object just accesses the data transfer
Socket, and the socket that intercepts then continues the port that detection is bound with it；
Step 13, one reception object of SocketAsyncEventArgs classes of establishment, the reception object in TCP service ends
Bound with the data transfer socket for accessing, and start to receive GPS messages：If the data length of the GPS messages for receiving
Equal to zero, then close the data transfer socket of correspondence access and discharge the reception object bound with it；If the GPS reports for receiving
The data length of text is more than zero, then carry out packet splicing treatment：If the packet of the GPS messages for receiving is complete, directly put
Enter message buffering queue, if the packet of the GPS messages for receiving is incomplete, be first put into reception buffering area wait after
Continued access is received until packet completely places into message buffering queue；
After step 14, TCP service ends receive GPS messages, it is normal that the sequencing according to received GPS messages makes packet receiving successively
Response return to client；
The process of the message task pool and worker thread pond treatment GPS messages includes：
Step 21, create a plurality of packet parsing threads in TCP service ends the GPS messages in message buffering queue are solved
Analysis, according to the different type that parsing is obtained, the GPS messages of variant type are put into message task by the packet parsing thread
In a plurality of message queues in pond, a plurality of message queues are corresponded with the GPS messages of variant type；
Step 22, each message queue of the message task pool realize read and write abruption using double buffering concept：Each message queue
Two task pool buffer areas are corresponded to, is respectively task pool buffer area 1 and task pool buffer area 2, first by each message queue
GPS messages write corresponding task pool buffer area 1, when the storage treatment of GPS messages is slow or TCP service ends connection number
In the case of more than very, the GPS messages in task pool buffer area 1 can slowly be piled up, and reach maximum, at this moment in each message queue
Remaining GPS messages can be written to corresponding task pool buffer area 2, finally from the corresponding task pool caching of each message queue
GPS messages are read out in area 1 and task pool buffer area 2 in batches, correspondence is written in the cache file of each message queue；
Step 23, the worker thread pond are according to the different types of work of message queue corresponding configuration at least one in message task pool
Thread is dealt with, work disposal thread is responsible for reading the GPS messages in the corresponding cache file of each message queue described in step 22
Take out and processed accordingly；
The process of GPS messages storage is：TCP service ends call data access layer, the GPS reports handled well described in write-in in batches
Text arrives database；
It is described to improve the process that GPS messages are put in storage error handle：
When GPS messages are put in storage, if SQL statement occur performs error, the GPS messages for performing mistake are written to message task pool
In wrong buffer queue processed：If what is run into is network problem, repeats and perform the SQL statement twice,
Still in the case of mistake, wrong cache file is created in systems, during GPS messages are write into the wrong cache file, such as
Fruit is that the SQL statement is problematic in itself, then directly by GPS message write error cache files, in case reclaiming or user
Investigation error reason.
2. a kind of gps data according to claim 1 enters library processing method, it is characterised in that：The step 22 also includes,
When each message queue treatment GPS message amounts of message task pool exceed two corresponding capacity of task pool buffer area
When, wait the GPS messages of storage to be first written in the cache file of each message queue in each message queue, by creating monitoring line
The occupancy situation of system CPU is read in journey timing:When system CPU occupancy rate is less than 20 percent, notified using semaphore
Work disposal thread reclaims the corresponding cache file of each message queue.
3. a kind of gps data according to claim 1 enters library processing method, it is characterised in that：The step 23 includes report
Alert short message service：After the work disposal thread parses the warning message in GPS messages, according to the alarm that user sets
Whether condition judgment needs alarm, if meeting alert if, sends alarming short message to corresponding contact person.
4. a kind of gps data according to claim 1 enters library processing method, it is characterised in that：In whole GPS messages storage
During, user can check daily record, and the daily record is divided at master log, alarming short message daily record, cache garbage collection daily record and mistake
Reason daily record, they are divided by date file preserve, be so retained separately, separately check, both reduced user search daily record when
Between conveniently check the details of daily record again.
5. a kind of gps data is put in storage processing system, it is characterised in that：The system includes TCP message receiver module, Message processing
Module, data loading module and error handling module：
The TCP message receiver module：By I/O completing port technical transmission GPS messages：Communication Gateway, will used as client
GPS messages are encapsulated in plurality of data transmission socket, and gps data is put in storage processing system as TCP service ends, and client will
The plurality of data transmission socket is sent to the reception that TCP service ends complete GPS messages by the port of computer, specifically
Step is as follows：
After step 11, initialization, socket is intercepted in establishment one in the service end, and intercepts socket and service end by described
A port binding, to realize the one-to-many transmission of GPS messages；
Step 12, one connecting object of SocketAsyncEventArgs classes of establishment in TCP service ends, the connection is right
As intercepting socket bindings with described, whether the connecting object starts have data transfer in the port that asynchronous detection is bound with it
Socket is accessed：When there is a data transfer socket requests connection, the connecting object just accesses the data transfer
Socket, and the socket that intercepts then continues the port that detection is bound with it；
Step 13, one reception object of SocketAsyncEventArgs classes of establishment, the reception object in TCP service ends
Bound with the data transfer socket for accessing, and start to receive GPS messages：If the data length of the GPS messages for receiving
Equal to zero, then close the data transfer socket of correspondence access and discharge the reception object bound with it；If the GPS reports for receiving
The data length of text is more than zero, then carry out packet splicing treatment：If the packet of the GPS messages for receiving is complete, directly put
Enter message buffering queue, if the packet of the GPS messages for receiving is incomplete, be first put into reception buffering area wait after
Continued access is received until packet completely places into message buffering queue；
After step 14, TCP service ends receive GPS messages, it is normal that the sequencing according to received GPS messages makes packet receiving successively
Response return to client；
The message processing module (MPM)：GPS messages are processed by message task pool and worker thread pond, is specifically included：
Step 21, create a plurality of packet parsing threads in TCP service ends the GPS messages in message buffering queue are solved
Analysis, according to the different type that parsing is obtained, the GPS messages of variant type are put into message task by the packet parsing thread
In a plurality of message queues in pond, a plurality of message queues are corresponded with the GPS messages of variant type；
Step 22, each message queue of the message task pool realize read and write abruption using double buffering concept：Each message queue
Two task pool buffer areas are corresponded to, is respectively task pool buffer area 1 and task pool buffer area 2, first by each message queue
GPS messages write corresponding task pool buffer area 1, when the storage treatment of GPS messages is slow or TCP service ends connection number
In the case of more than very, the GPS messages in task pool buffer area 1 can slowly be piled up, and reach maximum, at this moment in each message queue
Remaining GPS messages can be written to corresponding task pool buffer area 2, finally from the corresponding task pool caching of each message queue
GPS messages are read out in area 1 and task pool buffer area 2 in batches, correspondence is written in the cache file of each message queue；
Step 23, the worker thread pond are according to the different types of work of message queue corresponding configuration at least one in message task pool
Thread is dealt with, work disposal thread is responsible for reading the GPS messages in the corresponding cache file of each message queue described in step 22
Take out and processed accordingly；
The data loading module：TCP service ends call data access layer, batch write-in GPS messages to database；
The error handling module：When GPS messages are put in storage, if SQL statement occur performs error, the GPS reports of mistake will be performed
The wrong buffer queue that text is written in message task pool is processed：If what is run into is network problem, execution is repeated
The SQL statement twice, still in the case of mistake, creates wrong cache file in systems, and the write-in of GPS messages is described
In mistake cache file, if the SQL statement is problematic in itself, then directly by GPS message write error cache files,
In case reclaiming or user's investigation error reason.
6. a kind of gps data according to claim 5 is put in storage processing system, it is characterised in that：The message task pool
When each message queue treatment GPS message amounts exceed the capacity of two corresponding task pool buffer areas, in each message queue
Wait the GPS messages of storage to be first written in the cache file of each message queue, system is read by creating monitoring thread timing
The occupancy situation of CPU:When system CPU occupancy rate is less than 20 percent, notify that work disposal thread is returned using semaphore
Receive the corresponding cache file of each message queue.
7. a kind of gps data according to claim 5 is put in storage processing system, it is characterised in that：Also taken including alarming short message
Business module：After the work disposal thread parses the warning message in GPS messages, according to the alert if that user sets
Judge whether to need alarm, if meeting alert if, send alarming short message to corresponding contact person.
8. a kind of gps data according to claim 5 is put in storage processing system, it is characterised in that：Also include log pattern,
During whole GPS messages storage, user can check daily record, and the daily record is divided into master log, alarming short message daily record, caches back
Daily record and error handle daily record are received, they is divided by date file preserve, be so retained separately, separately check, both reduce use
The time of family retrieval daily record conveniently checks the details of daily record again.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410414439.9A CN104158757B (en) | 2014-08-21 | 2014-08-21 | A kind of gps data enters library processing method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410414439.9A CN104158757B (en) | 2014-08-21 | 2014-08-21 | A kind of gps data enters library processing method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104158757A CN104158757A (en) | 2014-11-19 |
| CN104158757B true CN104158757B (en) | 2017-07-07 |
Family
ID=51884165
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410414439.9A Active CN104158757B (en) | 2014-08-21 | 2014-08-21 | A kind of gps data enters library processing method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104158757B (en) |
Families Citing this family (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104281668A (en) * | 2014-09-28 | 2015-01-14 | 墨仕（厦门）电子商务有限公司 | Data processing method |
| CN104598574A (en) * | 2015-01-13 | 2015-05-06 | 北京中交兴路车联网科技有限公司 | Method and device for storing massive GPS (global positioning system) data |
| CN105260378A (en) * | 2015-09-08 | 2016-01-20 | 上海上讯信息技术股份有限公司 | Database audit method and device |
| CN105760452B (en) * | 2016-02-04 | 2020-03-10 | 深圳市嘉力达实业有限公司 | Method and system for acquiring, processing and storing high-concurrency mass data |
| CN106227782A (en) * | 2016-07-15 | 2016-12-14 | 广东亿迅科技有限公司 | A kind of method being inserted into data base based on multi-data source |
| CN106656863A (en) * | 2016-12-31 | 2017-05-10 | 广东欧珀移动通信有限公司 | Business monitoring method and apparatus, and computer device |
| CN107562790B (en) * | 2017-07-31 | 2020-05-01 | 北京北信源软件股份有限公司 | Method and system for realizing batch warehousing of data processing |
| CN109525632A (en) * | 2018-09-30 | 2019-03-26 | 珠海派诺科技股份有限公司 | Gateway data uploads database connection digital control method, device, equipment and medium |
| CN110908798B (en) * | 2019-11-08 | 2022-11-25 | 丁剑明 | Multi-process cooperative network traffic analysis method and device |
| CN111898116B (en) * | 2019-12-26 | 2021-09-24 | 长扬科技（北京）有限公司 | Cache-based industrial white list learning method and system |
| CN112748707A (en) * | 2020-12-29 | 2021-05-04 | 飞马智科信息技术股份有限公司 | System and method for improving steelmaking component stokehole acquisition efficiency |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1964346A (en) * | 2005-11-09 | 2007-05-16 | 同济大学 | A system and method to process mass GPS data based on distributed system structure |
| CN102542026A (en) * | 2011-12-22 | 2012-07-04 | 深圳市赛格导航科技股份有限公司 | Mass global position system (GPS) data access method and system |
| CN103064890A (en) * | 2012-12-11 | 2013-04-24 | 泉州豪杰信息科技发展有限公司 | Global position system (GPS) mass data processing method |
| KR101318903B1 (en) * | 2011-11-15 | 2013-10-17 | 동아대학교 산학협력단 | Location Data Server and Method for Managing of Location Data using the same |
-
2014
- 2014-08-21 CN CN201410414439.9A patent/CN104158757B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1964346A (en) * | 2005-11-09 | 2007-05-16 | 同济大学 | A system and method to process mass GPS data based on distributed system structure |
| KR101318903B1 (en) * | 2011-11-15 | 2013-10-17 | 동아대학교 산학협력단 | Location Data Server and Method for Managing of Location Data using the same |
| CN102542026A (en) * | 2011-12-22 | 2012-07-04 | 深圳市赛格导航科技股份有限公司 | Mass global position system (GPS) data access method and system |
| CN103064890A (en) * | 2012-12-11 | 2013-04-24 | 泉州豪杰信息科技发展有限公司 | Global position system (GPS) mass data processing method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104158757A (en) | 2014-11-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104158757B (en) | A kind of gps data enters library processing method and system | |
| US11449519B2 (en) | Method, system, and apparatus for enterprise wide storage and retrieval of large amounts of data | |
| CN110795257B (en) | Method, device, equipment and storage medium for processing multi-cluster job record | |
| CN100590620C (en) | System and method for moving records between partitions | |
| CN111770002B (en) | Test data forwarding control method and device, readable storage medium and electronic equipment | |
| CN110188103A (en) | Data account checking method, device, equipment and storage medium | |
| CN107613529A (en) | Message treatment method and base station | |
| CN106095678A (en) | Automatization's result inspection method of data bank service operation under windows platform | |
| US20070245171A1 (en) | Methods and apparatus to perform distributed memory checking | |
| CN104618304A (en) | Data processing method and data processing system | |
| CN106503008A (en) | File memory method and device and file polling method and apparatus | |
| CN111740868A (en) | Alarm data processing method and device and storage medium | |
| CN109842621A (en) | A kind of method and terminal reducing token storage quantity | |
| CN109918221A (en) | A kind of hard disk reports an error analytic method, system, terminal and storage medium | |
| CN105939364A (en) | Intelligent message push method and system | |
| CN105930504B (en) | A kind of network management timing file cocurrent processing system and concurrent processing method | |
| CN204425400U (en) | Application server system | |
| CN100435514C (en) | Filtering method and system for Ethernet driving bottom layer | |
| US9703739B2 (en) | Return available PPI credits command | |
| CN105162837A (en) | Method and system for improving I/O throughput rate in massive data storage environment | |
| KR102421150B1 (en) | Apparatus and method for distributed processing of identical packet in high-speed network security equipment | |
| US20080034054A1 (en) | System and method for reservation flow control | |
| CN113760799B (en) | Scalable communication method, device, computer equipment and storage medium of UPP interface | |
| CN114092211A (en) | Processing method of accounting data, electronic device and readable storage medium | |
| CN107102901A (en) | A kind of task processing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |