CN109089301A - network data processing system and method - Google Patents
network data processing system and method Download PDFInfo
- Publication number
- CN109089301A CN109089301A CN201810798348.8A CN201810798348A CN109089301A CN 109089301 A CN109089301 A CN 109089301A CN 201810798348 A CN201810798348 A CN 201810798348A CN 109089301 A CN109089301 A CN 109089301A
- Authority
- CN
- China
- Prior art keywords
- probe
- data
- probe data
- module
- distribution module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04W—WIRELESS COMMUNICATION NETWORKS
- H04W48/00—Access restriction; Network selection; Access point selection
- H04W48/16—Discovering, processing access restriction or access information
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/10—Active monitoring, e.g. heartbeat, ping or trace-route
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04W—WIRELESS COMMUNICATION NETWORKS
- H04W28/00—Network traffic management; Network resource management
- H04W28/02—Traffic management, e.g. flow control or congestion control
- H04W28/06—Optimizing the usage of the radio link, e.g. header compression, information sizing, discarding information
- H04W28/065—Optimizing the usage of the radio link, e.g. header compression, information sizing, discarding information using assembly or disassembly of packets
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04W—WIRELESS COMMUNICATION NETWORKS
- H04W28/00—Network traffic management; Network resource management
- H04W28/02—Traffic management, e.g. flow control or congestion control
- H04W28/10—Flow control between communication endpoints
- H04W28/14—Flow control between communication endpoints using intermediate storage
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Health & Medical Sciences (AREA)
- Cardiology (AREA)
- General Health & Medical Sciences (AREA)
- Computer Security & Cryptography (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Mobile Radio Communication Systems (AREA)
Abstract
The invention discloses a kind of network data processing system and methods, system includes probe device and cloud probe server, probe device includes probe data collection module, probe data sorting module and probe client modules, probe data collection module obtains probe data by analysis air message to collect the relevant information of surrounding wireless device, and probe data is stored in memory；Probe data distribution module is connect with probe data collection module；Probe client modules are connect with probe data distribution module, obtain probe data by registration probe data distribution module, according to reporting agreement to assemble data message, and the data message after assembling are reported to cloud probe server.The present invention can guarantee the quality of equipment, the later maintenance for reducing equipment, meet different user the different scenes the needs of, guarantee that equipment meets requirement of the public security to network security, can save the server resource that data upload the broadband resource needed and cloud storage.
Description
Technical field
The present invention relates to field of wireless data communication, in particular to a kind of network data processing system and method.
Background technique
WiFi is the industrial standard (IEEE802.11) for the wireless communication that IEEE is defined.First version of WiFi
It is published in 1997, there is defined medium access access control layer (MAC layer) and physical layers.Physical layer defines work and exists
Two kinds of wireless frequency modulation modes and a kind of mode of Infrared Transmission, overall data transmission rate in the ISM band of 2.4GHz are designed as
2Mbits.Communication between two equipment can the mode of freely directly (ad hoc) carry out, can also be in base station (Base
Station, BS) or the coordination of access point (Access Point, AP) under carry out.
When an equipment sends information by Radio Transmission Technology to another equipment, other same categories of device of surrounding
These information can be all received, wireless probe monotechnics is namely based on this principle.Specifically, as long as one wireless
Equipment is listened in range wireless probe, when this wireless device (either terminal, router or other wireless devices)
When sending an any frame (Frame), no matter whom is intended for, probe can be intercepted and captured, and analyze the one of this frame MAC layer and physical layer
A little information, for example send and receive the MAC Address of equipment, frame type, signal strength etc..For the wireless device of surrounding,
Probe is transparent.Probe does not need have any interaction with the equipment of surrounding, itself does not need to issue any wireless signal.
Active scan refers in the wireless client course of work, can periodically search for the wireless network of surrounding, that is, main
Wireless network around dynamic scanning.Whether SSID is carried according to Probe Request frame (probe request), will can actively swept
It retouches and is divided into two kinds:
Client sends Probe Request frame, and (SSID is sky, that is, 0) length of SSID IE is: client can determine
Phase its support channel list in, transmission detect claim frame (Probe Request) scanning wireless network.When AP receives spy
After looking into claim frame, probe response frame (Probe Response) can be responded, the radio network information that can be provided is provided.Wireless visitor
Family end can actively know workable wireless service, wireless client can according to need selection later by active scan
Wireless network access appropriate.
Client sends Probe Request (Probe Request carries specified SSID): when wireless client configures
Wish the wireless network of connection or be successfully connected in the case of a wireless network, client also can periodically send and detect
Claim frame (Probe Request) (message carries the SSID of wireless network for being configured or having connected), when can
It provides and replys probe response after specifying the AP of SSID wireless service to receive probe requests thereby.By this method, wireless client
The wireless network that can be specified with active scan.
Drive sweep refers to client by listening to the wireless network around the Beacon frame discovery that AP is periodically sent.It provides
The AP equipment of any wireless network services all can periodicity sending Beacon frame, so wireless client can be periodically in the channel of support
The radio network information that beacon frame obtains surrounding is monitored in list.When user needs to save electricity, drive sweep can be used.
Probe device can scan the wireless signal of surrounding, and record the record that each is parsed, and upload at one
In period, specified server is transferred data to, this operating method redundant data is more, leads to bandwidth resources in transmission process
Waste and cloud initial data storage resource waste.
Summary of the invention
The technical problem to be solved in the present invention is that in view of the above drawbacks of the prior art, providing one kind can guarantee equipment
Quality, reduce equipment later maintenance, meet different user the different scenes the needs of, guarantee that equipment meets public security to network
The requirement of safety can save the network data processing system that data upload the server resource of the broadband resource needed and cloud storage
System and method.
The technical solution adopted by the present invention to solve the technical problems is: a kind of network data processing system is constructed, including
Probe device and cloud probe server, the probe device include probe data collection module, probe data distribution module and
Probe client modules, the probe data collection module work are collected in wireless driver module, by analysis air message
The relevant information of surrounding wireless device obtains probe data, and the probe data is stored in memory；The probe data
Distribution module is connect with the probe data collection module, is worked in User space, for distributing probe data；The probe client
End module is connect with the probe data distribution module, is obtained probe data by registering the probe data distribution module, is pressed
According to reporting agreement to assemble data message, and the data message after assembling is reported to the cloud probe server.
In network data processing system of the present invention, the probe data collection module passes through enabled switch control
System is persistently collected work, and carry out duplicate removal processing using terminal MAC and Radio Frequency ID as unique identification when function enables；
When function is closed, all cache informations are emptied, are not regathered.
In network data processing system of the present invention, the probe data distribution module by IOCTL mechanism from
The probe data collection module obtains probe data, obtains once every the first setting time, after acquisition, the probe data
The data cached are carried out null clear operation by collection module；The probe data distribution module provides login mechanism, supports multiple
Business module is registered, and filtration types and time interval, when there is business module to be registered, the probe are provided when registration
Data distribution module enables the probe data collection module, if all business all register completion, the probe data distributes mould
Block closes the probe data collection module；The probe data distribution module provides probe data according to registration content, timing,
The specified time interval in business module registration, and the probe data is carried out at duplicate removal according to the time interval
Reason.
The invention further relates to a kind of network data processing method, it is applied to above-mentioned network data processing system, including initial
Change process, main thread process, caching thread process, send thread flow and timer process, the initialization process includes such as
Lower step:
A the relevant configuration information of probe device) is obtained by configuration file；The relevant configuration information includes reporting week
Phase, average value time interval, the relevant information of terminal type and cloud probe server；
B probe data distribution module) is registered according to the average value time interval；
C) initialization timer module, for sending heartbeat message；
D the initialization of data structure related content) is carried out；
E the directory index for having saved data) is obtained；
F after) initializing, caching thread is respectively created and sends thread；
G after the completion of) creating, the probe data sended over from probe data distribution module is handled using main thread.
In network data processing system of the present invention, the main thread process includes the following steps:
A1 the probe data that the probe data distribution module sends over) is received, and the information in HASH chain is carried out
Update operation；
B1) judge whether current time interval reaches and report interval, if so, executing step C1)；Otherwise, return step
A1)；
C1 the HASH chain) is traversed, according to reporting agreement to assemble message, executes step D1)；
D1) by the packet storage being completed into single-track link table.
In network data processing system of the present invention, the caching thread process includes the following steps:
A2) judge whether system time synchronizes success, if so, executing step B2)；Otherwise, after waiting the second setting time
Continue the judgement in this step；
B2 the data packet being completed) is obtained from the single-track link table；
C2) judge whether there are data in the single-track link table, if so, the timestamp in the data packet that modification is got, holds
Row step D2)；Otherwise, return step B2 after waiting third setting time)；
D2) judge that system with the presence or absence of SD card, is selected as SD card if so, then saving contents, execute step E2)；Otherwise, then
It saves contents and is selected as memory headroom, execute step E2)；
E2 specified file) is written into the data packet in binary form；
F2 compression processing) is carried out to the specified file, and is saved in specified directory.
In network data processing system of the present invention, the transmission thread flow includes the following steps:
A3 file) is obtained；
B3) judge with the presence or absence of file under the memory headroom, if so, executing step D3)；Otherwise, step C3 is executed)；
C3) judge with the presence or absence of file in the SD card, if so, executing step D3)；Otherwise, the 4th setting time is waited
Return step A3 afterwards)；
D3 it) reads file and stores the content in the file into caching；
E3 after the completion of) reading, then respective file is deleted, if deleting corresponding catalogue without any file under corresponding catalogue；
F3) whether judgement is normal with the connection of the cloud probe server, if so, executing step G3)；Otherwise, postpone
It sends and waits the judgement for continuing this step after the 5th setting time；
G3 it) assembles data message and is sent to the cloud probe server；
H3) judge whether to send successfully, if so, return step A3)；Otherwise, return step after the 6th setting time of waiting
F3)。
In network data processing system of the present invention, the timer process includes: the timer module touching
When hair, heartbeat message is sent to specified cloud probe server and is shown and the cloud if being normally received back message
The connection of probe server is normal, if continuous n times do not receive response, and n is greater than max-timeout number, then it is assumed that with the cloud
Probe server disconnects.
Implement network data processing system and method for the invention, has the advantages that due to being equipped with probe device
With cloud probe server, probe device includes probe data collection module, probe data distribution module and probe client mould
Block, probe data collection module obtain probe data by analysis air message to collect the relevant information of surrounding wireless device,
And probe data is stored in memory；Probe data distribution module works in User space, for distributing probe data；Probe visitor
Family end module obtains probe data by registration probe data distribution module, according to reporting agreement to assemble data message, and by group
Data message after dress is reported to cloud probe server, the present invention can guarantee the quality of equipment, the later maintenance for reducing equipment,
Meet different user the different scenes the needs of, guarantee that equipment meets requirement of the public security to network security, can save data and upload
The server resource of the broadband resource needed and cloud storage.
Detailed description of the invention
In order to more clearly explain the embodiment of the invention or the technical proposal in the existing technology, to embodiment or will show below
There is attached drawing needed in technical description to be briefly described, it should be apparent that, the accompanying drawings in the following description is only this
Some embodiments of invention for those of ordinary skill in the art without creative efforts, can be with
It obtains other drawings based on these drawings.
Fig. 1 is the structural schematic diagram of system in inventive network data processing system and method one embodiment；
Fig. 2 is the schematic diagram of multiple threads mode in the embodiment；
Fig. 3 is the flow chart of initialization process in the embodiment；
Fig. 4 is the flow chart of main thread process in the embodiment；
Fig. 5 is the flow chart of caching thread process in the embodiment；
Fig. 6 is the flow chart that thread flow is sent in the embodiment.
Specific embodiment
Following will be combined with the drawings in the embodiments of the present invention, and technical solution in the embodiment of the present invention carries out clear, complete
Site preparation description, it is clear that described embodiments are only a part of the embodiments of the present invention, instead of all the embodiments.It is based on
Embodiment in the present invention, it is obtained by those of ordinary skill in the art without making creative efforts every other
Embodiment shall fall within the protection scope of the present invention.
In inventive network data processing system and embodiment of the method, the structural schematic diagram of network data processing system
As shown in Figure 1.In Fig. 1, which includes probe device and cloud probe server Cloudserver, is visited
Needle equipment includes probe data collection module MACINFO, probe data distribution module WP_SERVER and probe client modules
Cloudclient, probe data collection module MACINFO work are collected in wireless driver module, by analysis air message
The relevant information of surrounding wireless device obtains probe data, and probe data is stored in memory.Probe data collection module
MACINFO is by enabling switch control, when function enables, is persistently collected work, and using terminal MAC and Radio Frequency ID as
Unique identification carries out duplicate removal processing；When function is closed, all cache informations are emptied, are not regathered.It is updated for convenience of the time,
What the acquisition time of terminal was filled in is system operation time, i.e., equipment from start to collect information altogether run how long.
Probe data distribution module WP_SERVER is connect with probe data collection module MACINFO, is worked in User space,
For distributing probe data.The business of probe data distribution module WP_SERVER is divided into three parts, specifically,
Probe data distribution module WP_SERVER is obtained from probe data collection module MACINFO by IOCTL mechanism and is visited
Needle data obtain once every the first setting time, and after acquisition, probe data collection module MACINFO can be the number cached
According to progress null clear operation.In the present embodiment, the first setting time is one second, certainly, in practical applications, the first setting time
Size can adjust accordingly as the case may be.
Probe data distribution module WP_SERVER provides login mechanism, supports that other multiple business modules are registered, infuses
Filtration types and time interval need to be provided when volume, when there is other business modules to be registered, probe data distribution module WP_
SERVER enables probe data collection module MACINFO, if all business all register completion, probe data distribution module WP_
SERVER closes probe data collection module MACINFO.
Probe data distribution module WP_SERVER is according to registration content, and timing provides probe data, in other business modules
When registration, specified time interval is needed, and duplicate removal processing is carried out to probe data according to the time interval.
Probe client modules Cloudclient is connect with probe data distribution module WP_SERVER, probe client mould
Block Cloudclient obtains probe data by registration probe data distribution module WP_SERVER, and according to reporting agreement to assemble
Data message after assembling is reported to cloud probe server Cloudserver by data message, it is final it is complete after report cloud to visit
The work of needle server Cloudserver.Network data processing system of the invention can guarantee the quality of equipment, reduce equipment
Later maintenance, meet different user the different scenes the needs of, guarantee that equipment meets requirement of the public security to network security, can save
Data upload the server resource of the broadband resource needed and cloud storage.
The probe uses the processing mode of multithreading, and the schematic diagram of the multiple threads mode is as shown in Figure 2.
The present embodiment further relates to a kind of network data processing method, and network data processing method application is in this present embodiment
Above-mentioned network data processing system, which includes initialization process, main thread process, caching thread stream
Journey sends thread flow and timer process.The flow chart of the initialization process is as shown in figure 3, in Fig. 3, the initialization process
Include the following steps:
Step S01 obtains the relevant configuration information of probe device by configuration file: in this step, being obtained by configuration file
Take the relevant configuration information of probe device, the relevant configuration information include report cycle, average value time interval, terminal type with
And the relevant information of cloud probe server.
Step S02 registers probe data distribution module according to average value time interval: in this step, according to the average value time
Interval registration probe data distribution module WP_SERVER.
Step S03 initialization timer module, for sending heartbeat message: in this step, initialization timer module is used
In transmission heartbeat message.
The initialization of step S04 progress data structure related content: in this step, the first of data structure related content is carried out
Beginningization.
Step S05 obtains the directory index for having saved data: in this step, obtaining the directory index for having saved data.
After step S06 is initialized, caching thread is respectively created and sends thread: in this step, after initialization,
Caching thread is respectively created and sends thread.Caching thread sends thread to send data to save data to SD card
Cloud probe server Cloudserver.
After the completion of step S07 creation, the probe data sended over from probe data distribution module is handled using main thread:
In this step, after the completion of creation, the number of probes sended over from probe data distribution module WP_SERVER is handled using main thread
According to.
Fig. 4 is the flow chart of main thread process in the present embodiment, and main thread is mainly responsible for the assembly working of probe data.Group
Dress is according to reporting protocol format to be handled.The main thread is related to two data structures: one of data structure is HASH
Table, for the detecting probe information in average storage time interval, the HASH table using the MAC of terminal as unique identification at
Reason, the index of HASH table are up to 128.Another data structure is single-track link table, is assisted for storing according to probe data
Discuss the datagram of conversion.It is configured according to report cycle, every primary, storage one, maximum capacity 8000.
In Fig. 4, which includes the following steps:
Step S11 receives the probe data that probe data distribution module sends over, and carries out to the information in HASH chain
It updates operation: in this step, receiving the probe data that probe data distribution module WP_SERVER is sended over, and to HASH chain
In information be updated operation.Wherein RSSI then carries out covering treatment using additional processing, other information.
Step S12 judges whether current time interval reaches and reports interval: in this step, whether judging current time interval
Reach and report interval, if it is determined that result be it is yes, then follow the steps S13；Otherwise, return step S11 continues to number of probes
According to.
Step S13 traverse HASH chain, according to report agreement assemble message: if the judging result of above-mentioned steps S12 be it is yes,
Then execute this step.In this step, HASH chain is traversed, according to reporting agreement to assemble message, wherein the timestamp in message is to work as
The runing time (system from start to the time run at this time) of preceding system.This step has been executed, step S14 is executed.
Step S14 is by the packet storage being completed into single-track link table: in this step, the packet storage that will be completed
Into single-track link table.When storage, if the free memory of current system is less than 15M, discard processing is carried out to the information.Separately
Outside, same to carry out discard processing if the number of stored messages reaches 8000 in single-track link table, in order to anti-
It is excessive only to occupy Installed System Memory.Main thread process can be realized by step S11 to step S14.
Fig. 5 be the present embodiment in caching thread process flow chart, caching thread be mainly responsible for message to be sent with
File mode is cached in SD card or memory.In Fig. 5, which includes the following steps:
Step S201 judges whether system time synchronizes success: in this step, starting caching thread, when first determining whether system
Between whether synchronize success, if it is determined that result be it is yes, then follow the steps S203；Otherwise, step S202 is executed.
Step S202 wait the second setting time: if the judging result of above-mentioned steps S201 be it is no, execute this step.
In this step, return step S201 after the second setting time is waited to continue to judge.In the present embodiment, the second setting time is
10 seconds, that is, it is primary every detection in 10 seconds, until testing result is to synchronize success.It is noted that in practical applications,
The size of second setting time can adjust accordingly as the case may be.
Step S203 obtains the data packet being completed from single-track link table: if the judging result of above-mentioned steps S201 is
It is no, then execute this step.In this step, the data packet being completed is obtained from single-track link table.This step has been executed, step is executed
Rapid S204.
Step S204 judges whether there are data in single-track link table: in this step, judging whether there are data in single-track link table, such as
The result that fruit judges be it is yes, then follow the steps S206；Otherwise, step S205 is executed.
Step S205 wait third setting time: if the judging result of above-mentioned steps S204 be it is no, i.e., single-track link table is
Sky then executes this step.In this step, return step S203 after third setting time is waited to continue to obtain, third setting time
It is short time, such as: 5 seconds.Certainly, in practical applications, the size of third setting time can carry out as the case may be
Corresponding adjustment.
Timestamp in the data packet that gets of step S206 modification: if the judging result of above-mentioned steps S204 be it is yes,
Data packet is got, then executes this step.In this step, the timestamp in the data packet got is modified, modification mode is such as
Under: timestamp=present system time-(sweep time in current system runing time-probe data).This step is executed, is executed
Step S207.
Step S207 judges system with the presence or absence of SD card: in this step, judge that system whether there is SD card, if it is determined that
As a result be it is yes, then follow the steps S209；Otherwise, step S208 is executed.
Step S208, which saves contents, is selected as memory headroom: if the judging result of above-mentioned steps S207 is not present to be no
SD card then executes this step.In this step, saves contents and be selected as memory headroom, i.e. ,/tmp catalogue, here no longer in residue
Deposit and make a decision, because data packet store single-track link table before it is determined that mistake.This step has been executed, step S210 is executed.
Step S209, which saves contents, is selected as SD card: if the judging result of above-mentioned steps S207 be it is yes, that is, there is SD card,
Then execute this step.It in this step, saves contents and is selected as SD card, and check whether the residual capacity of SD card is greater than 15M, if
Space is too small, then abandons saving, and abandons the data packet.
Determine that the filename for needing to save, format are as follows simultaneously: cy_MAC_TIME_index；Wherein, MAC table shows equipment
Rear three bytes of MAC Address, TIME indicate that present system time, index indicate file serial number, open, be no more than from 0
8000.This step has been executed, step S210 is executed.
Specified file is written in data packet by step S210 in binary form: in this step, by data packet with binary system shape
Specified file is written in formula.It (is 1500 calculating, the data portion of udp protocol according to interface mtu more than 1463 if data are excessive
It is up to 1472, then subtracts datagram protocol HEAD byte 9), then storage is split to data.Purpose is to guarantee to send
UDP message will not be fragmented, and reduce influence of the UDP packet loss to business.For dividing storage mode, all segments are stored in
In same file.It is identified by specific character (LCYL), each fragment byte number is no more than 1463, and guarantees each point
The probe data of piece is complete.This step has been executed, step S211 is executed.
Step S211 carries out compression processing to specified file, and is saved in specified directory: in this step, specified file is saved
Afterwards, compression processing is carried out to specified file, and is saved in specified directory.Purpose is the consumption reduced as far as possible to space.It saves
Bibliographic structure it is as follows:
Wherein, the file number under each subdirectory is conditional.For SD card, each catalogue limits maximum file
Number is 10000；For/tmp catalogue, each catalogue limitation file number is 100.Mainly two aspect reasons, first is that anti-
Only file is excessive, causes inquiry file stuck；In addition it is that system has restriction to the file number under catalogue, prevents from excessively causing most
It can not save eventually.S201 to step S211 can realize caching thread process through the above steps.
Fig. 6 is the flow chart that thread flow is sent in the present embodiment, and transmission thread, which is mainly responsible for, sends the data to cloud
Probe server Cloudserver.In Fig. 6, which includes the following steps:
Step S301 obtains file: in this step, obtaining file.
Step S302 judges under memory headroom with the presence or absence of file: in this step, judging under memory headroom with the presence or absence of text
Under part, i.e. judgement/tmp catalogue whether there is file, if it is determined that result be it is yes, then follow the steps S305；Otherwise, step is executed
Rapid S303.
Step S303 judges in SD card with the presence or absence of file: if the judging result of above-mentioned steps S302 is no, i.e. memory
File is not present under space, then executes this step.In this step, judge in SD card with the presence or absence of file, if it is determined that result
Be it is yes, then follow the steps S305；Otherwise, step S304 is executed.
Step S304 wait the 4th setting time: if the judging result of above-mentioned steps S303 be it is no, i.e. in two catalogues
File is all not present, then executes this step.In this step, return step S301 after the 4th setting time is waited.4th setting
Time is a bit of time, such as: 3 seconds.Certainly, in practical applications, the size of the 4th setting time can be according to specific feelings
Condition adjusts accordingly.
Step S305 reads file and stores the content in file into caching: if the judgement knot of above-mentioned steps S302
Fruit be yes or the judging result of above-mentioned steps S303 be it is yes, then execute this step.In this step, file is read and by file
In content store into caching.After having executed this step, step S306 is executed.
After the completion of step S306 is read, then respective file is deleted, if deleting correspondence without any file under corresponding catalogue
Catalogue: in this step, after the completion of reading, then deleting respective file, if deleting corresponding mesh without any file under corresponding catalogue
Record.This step has been executed, step S307 is executed.
Whether step S307 judgement is normal with the connection of cloud probe server: in this step, judgement takes with cloud probe
Be engaged in device connection it is whether normal, connection status is judged by heartbeat keepalive mechanism, if it is determined that result be it is yes, then follow the steps
S309；Otherwise, step S308 is executed.
Step S308 postpone send simultaneously wait the 5th setting time: if the judging result of above-mentioned steps S307 be it is no, i.e.,
It is in an off state, then executes this step.In this step, postpone sending and waiting the 5th setting time return step S307,
It is exactly to postpone sending, after waiting connection normal, retransmits.The size of 5th setting time can carry out accordingly as the case may be
Adjustment.
Step S309 assembling data message is simultaneously sent to cloud probe server: if the judging result of above-mentioned steps S307
Be it is yes, then execute this step.In this step, assembles data message and be sent to cloud probe server Cloudserver.Assembling
When, judge in data message with the presence or absence of segmental identification.If it is present a point multiple messages are sent.This step has been executed, has been executed
Step S310.
Step S310 judges whether to send successfully: in this step, judge whether to send successfully, if it is determined that result be
It is, then return step S301；Otherwise, step S311 is executed.
Step S311 waits the 6th setting time: if the judging result of above-mentioned steps S310 is no, i.e. transmission failure, then
Execute this step.In this step, return step S307 after the 6th setting time is waited.In the present embodiment, the 6th setting time is 1
Second, that is, wait 1 second, continue to send.Certainly, in practical applications, the size of the 6th setting time can be according to tool
Body situation adjusts accordingly.S301 to step S311 may be implemented to send thread flow through the above steps.
For timer process, when timer process includes: timer module triggering, taken to specified cloud probe
Business device Cloudserver (i.e. heartbeat cloud probe server) sends heartbeat message and shows if being normally received back message
Probe device is connect normally with cloud probe server Cloudserver, if continuous n times do not receive response, and n is greater than maximum
Expired times, then it is assumed that probe device is disconnected with cloud probe server Cloudserver.
In short, equipment of the present invention different from other same industries, first, using the other chip of carrier-class, guarantee equipment
Quality, reduce the later maintenance of equipment.Second, the presence of plurality of devices form meets different user in the need of different scenes
It asks.Third integrates a variety of peripheral hardwares, makes intelligent node, services big data.4th, guarantee that equipment meets public security to network
The requirement of safety provides safeguard for the following orderly development of project.
Data acquisition is done in radio protocol layer and is arranged, and equipment uplink is effectively treated using custom protocol and compression transmission
Problems of liquid flow.Reach the magnitude for simplifying and uploading data, saves the server that data upload the broadband resource needed and cloud storage
Resource.Data are finished with the processing of whole property and accuracy, such as the processing of puppet MAC, optimization of collection algorithm etc. guarantees to using industry
The support of business avoids traditional data blindly upload, simple average and abnormal data.
The foregoing is merely illustrative of the preferred embodiments of the present invention, is not intended to limit the invention, all in essence of the invention
Within mind and principle, any modification, equivalent replacement, improvement and so on be should all be included in the protection scope of the present invention.
Claims (8)
1. a kind of network data processing system, which is characterized in that including probe device and cloud probe server, the probe is set
Standby includes probe data collection module, probe data distribution module and probe client modules, the probe data collection module
It works and obtains probe data in wireless driver module, by analysis air message come the relevant information of wireless device around collecting,
And the probe data is stored in memory；The probe data distribution module is connect with the probe data collection module,
Work is in User space, for distributing probe data；The probe client modules are connect with the probe data distribution module, are led to
It crosses and registers the probe data distribution module and obtain probe data, according to reporting agreement to assemble data message, and will be after assembling
Data message is reported to the cloud probe server.
2. network data processing system according to claim 1, which is characterized in that the probe data collection module passes through
Enabled switch control is persistently collected work when function enables, and using terminal MAC and Radio Frequency ID as unique identification into
Row duplicate removal processing；When function is closed, all cache informations are emptied, are not regathered.
3. network data processing system according to claim 2, which is characterized in that the probe data distribution module passes through
IOCTL mechanism obtains probe data from the probe data collection module, obtains once every the first setting time, after acquisition,
The data cached are carried out null clear operation by the probe data collection module；The probe data distribution module provides register machine
System, supports that multiple business modules are registered, and filtration types and time interval are provided when registration, when there is business module to be registered
When, the probe data distribution module enables the probe data collection module, if all business all register completion, the probe
Data distribution module closes the probe data collection module；The probe data distribution module is periodically mentioned according to registration content
For probe data, the specified time interval in business module registration, and according to the time interval to the probe data
Carry out duplicate removal processing.
4. a kind of network data processing method, which is characterized in that be applied to network data processing system as described in claim 1
System, including initialization process, main thread process, caching thread process, transmission thread flow and timer process, the initialization
Process includes the following steps:
A the relevant configuration information of probe device) is obtained by configuration file；The relevant configuration information includes report cycle, puts down
Mean time interval, terminal type and cloud probe server relevant information；
B probe data distribution module) is registered according to the average value time interval；
C) initialization timer module, for sending heartbeat message；
D the initialization of data structure related content) is carried out；
E the directory index for having saved data) is obtained；
F after) initializing, caching thread is respectively created and sends thread；
G after the completion of) creating, the probe data sended over from probe data distribution module is handled using main thread.
5. network data processing system according to claim 4, which is characterized in that the main thread process includes following step
It is rapid:
A1 the probe data that the probe data distribution module sends over) is received, and the information in HASH chain is updated
Operation；
B1) judge whether current time interval reaches and report interval, if so, executing step C1)；Otherwise, return step A1)；
C1 the HASH chain) is traversed, according to reporting agreement to assemble message, executes step D1)；
D1) by the packet storage being completed into single-track link table.
6. network data processing system according to claim 5, which is characterized in that the caching thread process includes as follows
Step:
A2) judge whether system time synchronizes success, if so, executing step B2)；Otherwise, continue after waiting the second setting time
Carry out the judgement in this step；
B2 the data packet being completed) is obtained from the single-track link table；
C2) judge whether there are data in the single-track link table, if so, the timestamp in the data packet that modification is got, executes step
Rapid D2)；Otherwise, return step B2 after waiting third setting time)；
D2) judge that system with the presence or absence of SD card, is selected as SD card if so, then saving contents, execute step E2)；Otherwise, then it saves
Catalogue is selected as memory headroom, executes step E2)；
E2 specified file) is written into the data packet in binary form；
F2 compression processing) is carried out to the specified file, and is saved in specified directory.
7. network data processing system according to claim 6, which is characterized in that the transmission thread flow includes as follows
Step:
A3 file) is obtained；
B3) judge with the presence or absence of file under the memory headroom, if so, executing step D3)；Otherwise, step C3 is executed)；
C3) judge with the presence or absence of file in the SD card, if so, executing step D3)；Otherwise, it is returned after waiting the 4th setting time
Return step A3)；
D3 it) reads file and stores the content in the file into caching；
E3 after the completion of) reading, then respective file is deleted, if deleting corresponding catalogue without any file under corresponding catalogue；
F3) whether judgement is normal with the connection of the cloud probe server, if so, executing step G3)；Otherwise, postpone sending
And continue the judgement of this step after the 5th setting time of waiting；
G3 it) assembles data message and is sent to the cloud probe server；
H3) judge whether to send successfully, if so, return step A3)；Otherwise, return step F3 after the 6th setting time of waiting).
8. network data processing system according to claim 4, which is characterized in that the timer process includes: described
When timer module triggers, heartbeat message, if being normally received back message, table are sent to specified cloud probe server
Bright to connect normally with the cloud probe server, if continuous n times do not receive response, and n is greater than max-timeout number, then recognizes
To be disconnected with the cloud probe server.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810798348.8A CN109089301B (en) | 2018-07-19 | 2018-07-19 | Network data processing system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810798348.8A CN109089301B (en) | 2018-07-19 | 2018-07-19 | Network data processing system and method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109089301A true CN109089301A (en) | 2018-12-25 |
| CN109089301B CN109089301B (en) | 2021-01-15 |
Family
ID=64837990
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810798348.8A Active CN109089301B (en) | 2018-07-19 | 2018-07-19 | Network data processing system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109089301B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110809026A (en) * | 2019-09-29 | 2020-02-18 | 视联动力信息技术股份有限公司 | File processing method and device, electronic equipment and storage medium |
| CN114900341A (en) * | 2022-04-24 | 2022-08-12 | 京东科技信息技术有限公司 | Scanning detection method, device, system, equipment and medium in mixed cloud environment |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102426536A (en) * | 2011-10-26 | 2012-04-25 | 深圳市亚特尔科技有限公司 | Multitask data communication implementation method and system |
| WO2014068757A1 (en) * | 2012-11-01 | 2014-05-08 | 株式会社日立製作所 | Information processing system, data administration method, and program |
| CN106254246A (en) * | 2016-09-20 | 2016-12-21 | 上海斐讯数据通信技术有限公司 | A kind of method of sniff wireless terminal MAC information and router |
| CN106413004A (en) * | 2016-09-05 | 2017-02-15 | 上海斐讯数据通信技术有限公司 | Wireless MAC sniffing scanning method and apparatus |
| CN106790491A (en) * | 2016-12-14 | 2017-05-31 | 日照职业技术学院 | The implementation method of the Internet of Things movable termination intelligent platform of data-oriented |
| CN106851719A (en) * | 2017-04-18 | 2017-06-13 | 云熠信息科技有限公司 | A kind of Wifi probes, data balancing acquisition system and method |
| CN108243067A (en) * | 2016-12-23 | 2018-07-03 | 上海辰智商务信息咨询有限公司 | A kind of user terminal data acquisition system based on wifi detections |
-
2018
- 2018-07-19 CN CN201810798348.8A patent/CN109089301B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102426536A (en) * | 2011-10-26 | 2012-04-25 | 深圳市亚特尔科技有限公司 | Multitask data communication implementation method and system |
| WO2014068757A1 (en) * | 2012-11-01 | 2014-05-08 | 株式会社日立製作所 | Information processing system, data administration method, and program |
| CN106413004A (en) * | 2016-09-05 | 2017-02-15 | 上海斐讯数据通信技术有限公司 | Wireless MAC sniffing scanning method and apparatus |
| CN106254246A (en) * | 2016-09-20 | 2016-12-21 | 上海斐讯数据通信技术有限公司 | A kind of method of sniff wireless terminal MAC information and router |
| CN106790491A (en) * | 2016-12-14 | 2017-05-31 | 日照职业技术学院 | The implementation method of the Internet of Things movable termination intelligent platform of data-oriented |
| CN108243067A (en) * | 2016-12-23 | 2018-07-03 | 上海辰智商务信息咨询有限公司 | A kind of user terminal data acquisition system based on wifi detections |
| CN106851719A (en) * | 2017-04-18 | 2017-06-13 | 云熠信息科技有限公司 | A kind of Wifi probes, data balancing acquisition system and method |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110809026A (en) * | 2019-09-29 | 2020-02-18 | 视联动力信息技术股份有限公司 | File processing method and device, electronic equipment and storage medium |
| CN110809026B (en) * | 2019-09-29 | 2022-11-01 | 视联动力信息技术股份有限公司 | File processing method and device, electronic equipment and storage medium |
| CN114900341A (en) * | 2022-04-24 | 2022-08-12 | 京东科技信息技术有限公司 | Scanning detection method, device, system, equipment and medium in mixed cloud environment |
| CN114900341B (en) * | 2022-04-24 | 2023-11-03 | 京东科技信息技术有限公司 | Scanning detection method, device, system, equipment and medium in hybrid cloud environment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109089301B (en) | 2021-01-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP5727106B2 (en) | Radio resource optimization method, radio resource optimization apparatus, and radio resource optimization system | |
| US9474101B2 (en) | Apparatus and system for managing a sensor network | |
| US20230247388A1 (en) | Communication system, method and device for miniature intelligent sensor | |
| CN109155933A (en) | Reflective service quality control and management | |
| CN103441939B (en) | Multi-functional border router based on IPv6 wireless sensor network | |
| CN110740490A (en) | Terminal network access method, gateway equipment, system, storage medium and device | |
| CN111263338B (en) | Network distribution method of Bluetooth Mesh network, related network distribution equipment and system | |
| CN110752943B (en) | Distributed fault diagnosis system and method for power transmission line | |
| CN103051672B (en) | End message acquisition methods in a kind of heterogeneous terminals environment and device | |
| CN107623752B (en) | Network management method and device based on link layer | |
| CN105868040A (en) | Log collection method and collection terminal | |
| CN103501356A (en) | Network address configuration method and system of network camera | |
| CN109089301A (en) | network data processing system and method | |
| US8189459B2 (en) | Method and system for guiding packet data protocol activation | |
| CN109246762A (en) | A kind of local service shunt method and device | |
| CN103052094B (en) | Method for acquiring service set identifier of wireless mobile terminal | |
| US20110131255A1 (en) | Wireless base station device and operation information collection method for the same | |
| WO2015070763A1 (en) | Self-establishing method and apparatus for x2 interface | |
| WO2015077979A1 (en) | Maintenance management method and device, and device maintenance system | |
| WO2023125056A1 (en) | Network data control method and apparatus, and storage medium and electronic device | |
| CN106170133A (en) | Multi-mode communication method in a kind of sensing network and device | |
| CN110505648A (en) | A kind of anti-lost line method of wireless device | |
| KR101170141B1 (en) | Mobile terminal, the system and method to trace network charcteristic from mobile terminals | |
| CN113300869B (en) | Communication method with in-band network remote sensing function, network device and storage medium | |
| KR101673755B1 (en) | System for interoperation network of IoT and network of Zigbee based DDS and Method thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |