CN109089301B - Network data processing system and method - Google Patents

Network data processing system and method Download PDF

Info

Publication number
CN109089301B
CN109089301B CN201810798348.8A CN201810798348A CN109089301B CN 109089301 B CN109089301 B CN 109089301B CN 201810798348 A CN201810798348 A CN 201810798348A CN 109089301 B CN109089301 B CN 109089301B
Authority
CN
China
Prior art keywords
probe
data
probe data
module
cloud
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201810798348.8A
Other languages
Chinese (zh)
Other versions
CN109089301A (en
Inventor
刘耀明
王全
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shenzhen Cloudwinner Network Technology Co ltd
Original Assignee
Shenzhen Cloudwinner Network Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shenzhen Cloudwinner Network Technology Co ltd filed Critical Shenzhen Cloudwinner Network Technology Co ltd
Priority to CN201810798348.8A priority Critical patent/CN109089301B/en
Publication of CN109089301A publication Critical patent/CN109089301A/en
Application granted granted Critical
Publication of CN109089301B publication Critical patent/CN109089301B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04W—WIRELESS COMMUNICATION NETWORKS
    • H04W48/00—Access restriction; Network selection; Access point selection
    • H04W48/16—Discovering, processing access restriction or access information
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00—Arrangements for monitoring or testing data switching networks
    • H04L43/10—Active monitoring, e.g. heartbeat, ping or trace-route
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04W—WIRELESS COMMUNICATION NETWORKS
    • H04W28/00—Network traffic management; Network resource management
    • H04W28/02—Traffic management, e.g. flow control or congestion control
    • H04W28/06—Optimizing the usage of the radio link, e.g. header compression, information sizing, discarding information
    • H04W28/065—Optimizing the usage of the radio link, e.g. header compression, information sizing, discarding information using assembly or disassembly of packets
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04W—WIRELESS COMMUNICATION NETWORKS
    • H04W28/00—Network traffic management; Network resource management
    • H04W28/02—Traffic management, e.g. flow control or congestion control
    • H04W28/10—Flow control between communication endpoints
    • H04W28/14—Flow control between communication endpoints using intermediate storage

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Health & Medical Sciences (AREA)
  • Cardiology (AREA)
  • General Health & Medical Sciences (AREA)
  • Computer Security & Cryptography (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Mobile Radio Communication Systems (AREA)

Abstract

The invention discloses a network data processing system and a method, wherein the system comprises probe equipment and a cloud probe server, the probe equipment comprises a probe data collection module, a probe data sorting module and a probe client module, the probe data collection module collects relevant information of surrounding wireless equipment by analyzing wireless messages to obtain probe data, and the probe data is stored in a memory; the probe data distribution module is connected with the probe data collection module; the probe client module is connected with the probe data distribution module, acquires probe data through the registered probe data distribution module, assembles data messages according to a reporting protocol, and reports the assembled data messages to the cloud probe server. The invention can ensure the quality of the equipment, reduce the later maintenance of the equipment, meet the requirements of different users in different scenes, ensure that the equipment meets the requirement of public security on network safety, and save broadband resources required by data uploading and server resources stored in a cloud.

Description

Network data processing system and method
Technical Field
The present invention relates to the field of wireless data communication, and in particular, to a network data processing system and method.
Background
WiFi is an industry standard for wireless network communications defined by IEEE (IEEE 802.11). The first release of WiFi was published in 1997, where the medium access control layer (MAC layer) and the physical layer were defined. The physical layer defines two wireless frequency modulation modes and an infrared transmission mode which work on an ISM frequency band of 2.4GHz, and the total data transmission rate is designed to be 2 Mbits. Communication between two devices can be performed in a free and direct (ad hoc) manner, and can also be performed under the coordination of a Base Station (BS) or an Access Point (AP).
When a device sends information to another device through wireless transmission technology, other surrounding devices of the same type can receive the information, and the wireless probe unit technology is based on the principle. Specifically, when a wireless device (whether a terminal, a router or other wireless device) sends any Frame (Frame) as long as the wireless device is in the listening range of the wireless probe, the probe can intercept and analyze some information of the MAC layer and the physical layer of the Frame, such as the MAC address, the Frame type, the signal strength and the like of the sending and receiving device, no matter to whom the Frame is sent. The probe is transparent to the surrounding wireless devices. The probe need not have any interaction with surrounding equipment and need not itself emit any wireless signals.
The active scanning means that the wireless client periodically searches for the surrounding wireless networks during the operation process, that is, actively scans the surrounding wireless networks. Active scanning can be divided into two types according to whether a Probe Request frame (Probe Request frame) carries an SSID:
the client sends a Probe Request frame (SSID is null, i.e. length of SSID IE is 0): the client periodically sends a Probe Request frame (Probe Request) in its supported channel list to scan for wireless networks. When the AP receives the Probe request frame, it responds with a Probe Response frame (Probe Response) to announce the available wireless network information. The wireless client can actively acquire available wireless services through active scanning, and then the wireless client can select a proper wireless network to access according to the requirement.
The client sends a Probe Request (the Probe Request carries a specified SSID): when a wireless client configures a wireless network that the wireless client desires to connect to or has successfully connected to a wireless network, the client also periodically sends a Probe Request frame (Probe Request) that carries the SSID of the configured or connected wireless network, and replies a Probe response when an AP capable of providing the specified SSID wireless service receives the Probe Request. In this way, the wireless client may actively scan for the designated wireless network.
Passive scanning refers to that a client discovers a surrounding wireless network by listening to Beacon frames periodically sent by an AP. The AP device providing the wireless network service may periodically send Beacon frames, so the wireless client may periodically monitor the Beacon frames in the supported channel list to obtain the surrounding wireless network information. Passive scanning may be used when the user needs to save power.
The probe equipment can scan surrounding wireless signals, record each analyzed record and transmit data to a designated server in an uploading period, and the operation method has more redundant data, so that bandwidth resources and cloud original data storage resources are wasted in the transmission process.
Disclosure of Invention
The technical problem to be solved by the present invention is to provide a network data processing system and method capable of ensuring the quality of the device, reducing the post-maintenance of the device, meeting the requirements of different users in different scenes, ensuring that the device meets the requirements of public security on network security, and saving broadband resources required for data uploading and server resources stored in a cloud.
The technical scheme adopted by the invention for solving the technical problems is as follows: the network data processing system comprises probe equipment and a cloud probe server, wherein the probe equipment comprises a probe data collection module, a probe data distribution module and a probe client module, the probe data collection module works in a wireless driving module, collects relevant information of surrounding wireless equipment by analyzing wireless messages to obtain probe data, and stores the probe data in an internal memory; the probe data distribution module is connected with the probe data collection module, works in a user mode and is used for distributing probe data; the probe client module is connected with the probe data distribution module, acquires probe data by registering the probe data distribution module, assembles data messages according to a reporting protocol, and reports the assembled data messages to the cloud probe server.
In the network data processing system, the probe data collection module is controlled by an enabling switch, when the function is enabled, collection work is continuously carried out, and terminal MAC and radio frequency ID are used as unique identifiers for carrying out duplication elimination processing; when the function is turned off, all cached information is emptied and no longer collected.
In the network data processing system of the present invention, the probe data distribution module acquires probe data from the probe data collection module through an IOCTL mechanism, the acquisition is performed once every first set time, and after the acquisition, the probe data collection module performs an emptying operation on the cached data; the probe data distribution module provides a registration mechanism to support a plurality of service modules to register, a filtering type and a time interval are provided during registration, when a service module registers, the probe data distribution module enables the probe data collection module, and if all services are registered, the probe data distribution module closes the probe data collection module; the probe data distribution module provides probe data at regular time according to registration content, appoints time interval when the business module registers, and performs deduplication processing on the probe data according to the time interval.
The invention also relates to a network data processing method which is applied to the network data processing system and comprises an initialization process, a main thread process, a cache thread process, a sending thread process and a timer process, wherein the initialization process comprises the following steps:
A) acquiring related configuration information of the probe equipment through the configuration file; the relevant configuration information comprises a reporting period, an average value time interval, a terminal type and relevant information of the cloud probe server;
B) registering a probe data distribution module according to the average value time interval;
C) the initialization timer module is used for sending heartbeat messages;
D) initializing related content of the data structure;
E) acquiring a directory index of stored data;
F) after initialization is finished, respectively creating a cache thread and a sending thread;
G) and after the creation is completed, processing the probe data sent from the probe data distribution module by adopting the main thread.
In the network data processing system of the present invention, the main thread flow includes the following steps:
A1) receiving the probe data sent by the probe data distribution module, and updating the information in the HASH chain;
B1) judging whether the current time interval reaches the reporting interval, if so, executing the step C1); otherwise, return to step a 1);
C1) traversing the HASH chain, assembling a message according to a report protocol, and executing the step D1);
D1) and storing the assembled message into a single-direction linked list.
In the network data processing system of the present invention, the cache thread process includes the following steps:
A2) judging whether the system time is successfully synchronized, if so, executing the step B2); otherwise, continuing to judge in the step after waiting for a second set time;
B2) acquiring the assembled data packet from the one-way linked list;
C2) judging whether the one-way linked list has data, if so, modifying the timestamp in the acquired data packet, and executing the step D2); otherwise, returning to the step B2 after waiting for a third set time);
D2) judging whether the system has the SD card, if so, selecting the storage catalog as the SD card, and executing the step E2); otherwise, saving the directory selection as the memory space, and executing step E2);
E2) writing the data packet into a specified file in a binary form;
F2) and compressing the specified file and storing the compressed specified file in a specified directory.
In the network data processing system of the present invention, the sending thread flow includes the following steps:
A3) acquiring a file;
B3) judging whether a file exists in the memory space, if so, executing step D3); otherwise, performing step C3);
C3) judging whether a file exists in the SD card, if so, executing step D3); otherwise, returning to the step A3 after waiting for a fourth set time);
D3) reading a file and storing the content in the file into a cache;
E3) after the reading is finished, deleting the corresponding file, and if no file exists in the corresponding directory, deleting the corresponding directory;
F3) judging whether the connection with the cloud probe server is normal or not, if so, executing step G3); otherwise, suspending sending and continuing to judge the step after waiting for a fifth set time;
G3) assembling a data message and sending the data message to the cloud probe server;
H3) judging whether the transmission is successful, if so, returning to the step A3); otherwise, return to step F3 after waiting for the sixth set time).
In the network data processing system of the present invention, the timer procedure includes: when the timer module is triggered, a heartbeat message is sent to a designated cloud probe server, if a response message is normally received, the connection with the cloud probe server is indicated to be normal, if no response is received for n times continuously, and n is greater than the maximum timeout time, the connection with the cloud probe server is considered to be disconnected.
The network data processing system and the method have the following beneficial effects: the system comprises a probe device and a cloud probe server, wherein the probe device comprises a probe data collection module, a probe data distribution module and a probe client module, the probe data collection module collects relevant information of surrounding wireless devices by analyzing wireless messages to obtain probe data, and stores the probe data in a memory; the probe data distribution module works in a user mode and is used for distributing probe data; the invention can ensure the quality of equipment, reduce the later maintenance of the equipment, meet the requirements of different users in different scenes, ensure that the equipment meets the requirement of public security on network safety, and save broadband resources required by data uploading and server resources stored in a cloud.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a block diagram of a system in accordance with an embodiment of the present invention;
FIG. 2 is a diagram showing a multithread processing method according to the embodiment;
FIG. 3 is a flowchart of an initialization procedure in the embodiment;
FIG. 4 is a flowchart of the main thread flow in the embodiment;
FIG. 5 is a flow chart of the cache thread flow in the embodiment;
FIG. 6 is a flow chart of the thread sending flow in the embodiment.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In the embodiment of the network data processing system and method of the present invention, a schematic structural diagram of the network data processing system is shown in fig. 1. In fig. 1, the network data processing system includes a probe device and a cloud probe SERVER cloud SERVER, the probe device includes a probe data collection module MACINFO, a probe data distribution module WP _ SERVER, and a probe client module client, the probe data collection module MACINFO operates in a wireless driving module, collects relevant information of surrounding wireless devices by analyzing wireless messages to obtain probe data, and stores the probe data in a memory. The probe data collection module MACINFO is controlled by an enabling switch, when the function is enabled, collection work is continuously carried out, and terminal MAC and radio frequency ID are used as unique identifiers for carrying out duplication elimination processing; when the function is turned off, all cached information is emptied and no longer collected. In order to facilitate time updating, the acquisition time of the terminal is filled in the system operation time, namely how long the equipment runs from starting to acquiring information.
The probe data distribution module WP _ SERVER is connected with the probe data collection module MACINFO, works in a user state and is used for distributing probe data. The service of the probe data distribution module WP _ SERVER is divided into three parts, specifically,
the probe data distribution module WP _ SERVER acquires the probe data from the probe data collection module MACINFO through an IOCTL mechanism, the acquisition is carried out once every first set time, and after the acquisition, the probe data collection module MACINFO can clear the cached data. In this embodiment, the first setting time is one second, and certainly, in practical applications, the size of the first setting time may be adjusted accordingly according to specific situations.
The probe data distribution module WP _ SERVER provides a registration mechanism, supports a plurality of other service modules to register, and needs to provide a filtering type and a time interval during registration.
The probe data distribution module WP _ SERVER provides probe data regularly according to the registration content, and when other service modules register, a time interval needs to be appointed, and the probe data is subjected to deduplication processing according to the time interval.
The method comprises the steps that a probe client module cloud is connected with a probe data distribution module WP _ SERVER, the probe client module cloud acquires probe data through a registration probe data distribution module WP _ SERVER, data messages are assembled according to a reporting protocol, the assembled data messages are reported to a cloud probe SERVER cloud, and finally the work of the cloud probe SERVER cloud is reported. The network data processing system can ensure the quality of the equipment, reduce the later maintenance of the equipment, meet the requirements of different users in different scenes, ensure that the equipment meets the requirement of public security on network safety, and save broadband resources required by data uploading and server resources stored in a cloud.
The probe adopts a multi-thread processing mode, and a schematic diagram of the multi-thread processing mode is shown in FIG. 2.
The present embodiment also relates to a network data processing method, which is applied to the network data processing system in the present embodiment, and includes an initialization process, a main thread process, a cache thread process, a sending thread process, and a timer process. Fig. 3 shows a flowchart of the initialization process, and in fig. 3, the initialization process includes the following steps:
step S01 obtains the relevant configuration information of the probe device through the configuration file: in this step, the relevant configuration information of the probe device is obtained through the configuration file, and the relevant configuration information includes a reporting period, an average value time interval, a terminal type and relevant information of the cloud probe server.
Step S02 registers the probe data distribution module according to the average value time interval: in this step, the probe data distribution module WP _ SERVER is registered according to the average value time interval.
Step S03 initializes the timer module, which is used to send the heartbeat message: in this step, a timer module is initialized, which is used to send heartbeat messages.
Step S04 initializes the content related to the data structure: in this step, the initialization of the content related to the data structure is performed.
Step S05 acquires the directory index of the saved data: in this step, the directory index of the stored data is obtained.
After the initialization of step S06 is completed, a cache thread and a sending thread are created respectively: in this step, after the initialization is completed, a cache thread and a sending thread are respectively created. The cache thread is used for storing data to the SD card, and the sending thread is used for sending the data to the cloud probe server Cloudserver.
After the step S07 is completed, the main thread is used to process the probe data sent from the probe data distribution module: in this step, after the creation is completed, the main thread is used to process the probe data sent from the probe data distribution module WP _ SERVER.
Fig. 4 is a flowchart of a main thread flow in the present embodiment, where the main thread is mainly responsible for assembling probe data. And the assembly is processed according to a reporting protocol format. The main thread involves two data structures: one of the data structures is a HASH table for storing probe information over a mean time interval, the HASH table being processed with the MAC of the terminal as a unique identifier, the HASH table being indexed up to 128. Another data structure is a singly linked list for storing datagrams for protocol conversion according to probe data. And storing one at every other time according to the configuration of the reporting period, wherein the maximum capacity is 8000.
In fig. 4, the main thread flow includes the following steps:
step S11 is to receive the probe data sent by the probe data distribution module, and update the information in the HASH chain: in this step, the probe data sent by the probe data distribution module WP _ SERVER is received, and the information in the HASH chain is updated. Wherein, the RSSI adopts additional processing, and other information is covered.
Step S12, determining whether the current time interval reaches the reporting interval: in this step, it is determined whether the current time interval reaches the reporting interval, and if the determination result is yes, step S13 is executed; otherwise, return to step S11 to continue receiving probe data.
Step S13 traverses the HASH chain, assembles the packet according to the report protocol: if the judgment result of the above step S12 is yes, the present step is executed. In this step, the HASH chain is traversed, and a message is assembled according to the reporting protocol, where a timestamp in the message is the running time of the current system (the time from the start of the system to the running of the system). After the present step is executed, step S14 is executed.
Step S14 stores the assembled packet into the one-way linked list: in this step, the assembled packet is stored in the single linked list. And during storage, if the residual memory of the current system is less than 15M, discarding the information. In addition, if the number of the stored messages in the single-direction linked list reaches 8000, the messages are discarded in the same way, so as to prevent the occupation of too large system memory. The main thread flow can be realized through steps S11 to S14.
Fig. 5 is a flowchart of a cache thread flow in this embodiment, where the cache thread is mainly responsible for caching a message to be sent in a SD card or a memory in a file manner. In fig. 5, the cache thread flow includes the following steps:
step S201 determines whether the system time is synchronized successfully: in this step, a cache thread is started, and first, whether the system time is successfully synchronized is judged, and if the judgment result is yes, step S203 is executed; otherwise, step S202 is performed.
Step S202 waits for a second set time: if the judgment result of the step S201 is no, the present step is executed. In this step, after waiting for the second setting time, the process returns to step S201 to continue the determination. In this embodiment, the second setting time is 10 seconds, that is, the detection is performed every 10 seconds until the detection result is that the synchronization is successful. It should be noted that, in practical applications, the second setting time may be adjusted according to specific situations.
Step S203 acquires the assembled data packet from the single linked list: if the judgment result of the step S201 is no, the present step is executed. In this step, the assembled data packet is obtained from the single-direction linked list. After this step is executed, step S204 is executed.
Step S204, judging whether the one-way linked list has data: in this step, it is determined whether there is data in the single-direction linked list, and if yes, step S206 is executed; otherwise, step S205 is executed.
Step S205 waits for a third set time: if the judgment result in the step S204 is no, that is, the one-way linked list is empty, the step is executed. In this step, after waiting for a third set time, returning to step S203 to continue to obtain, where the third set time is a short period of time, such as: for 5 seconds. Of course, in practical applications, the third setting time may be adjusted according to specific situations.
Step S206 modifies the timestamp in the acquired data packet: if the determination result in the step S204 is yes, that is, the data packet is acquired, the step is executed. In this step, the timestamp in the acquired data packet is modified in the following manner: time stamp ═ current system time- (current system run time-scan time in probe data). This step is executed, and step S207 is executed.
Step S207 determines whether the system has an SD card: in this step, it is determined whether the system has an SD card, and if the determination result is yes, step S209 is executed; otherwise, step S208 is performed.
Step S208 saves the directory selection as a memory space: if the judgment result of the step S207 is no, that is, no SD card exists, the present step is executed. In this step, the storage directory is selected as a memory space, i.e., a/tmp directory, where the remaining memory is not judged any more, because the judgment is already made before the data packet is stored in the single linked list. After the present step is executed, step S210 is executed.
Step S209 saves the directory selection as the SD card: if the judgment result of the above step S207 is yes, that is, the SD card exists, this step is executed. In this step, the storage directory is selected as the SD card, and it is checked whether the remaining capacity of the SD card is greater than 15M, if the space is too small, the storage is abandoned, and the packet is discarded.
And simultaneously determining the file name to be saved, wherein the format is as follows: cy _ MAC _ TIME _ index; wherein, MAC represents the last three bytes of the MAC address of the equipment, TIME represents the current system TIME, index represents the file serial number, and is started from 0 and is not more than 8000. After the present step is executed, step S210 is executed.
Step S210 writes the data packet into the designated file in binary form: in this step, the data packet is written in a binary form into a designated file. And if the data is too large and exceeds 1463 (the maximum data part of the UDP protocol is 1472 calculated according to the interface MTU being 1500, and the HEAD byte 9 of the reporting protocol is subtracted), the data is segmented and stored. The method aims to ensure that the sent UDP message cannot be fragmented and reduce the influence of UDP packet loss on the service. For a split storage approach, all the fragments are stored in the same file. The number of bytes per slice is not more than 1463, as identified by a specific character (LCYL), and the probe data of each slice is guaranteed to be complete. After the present step is executed, step S211 is executed.
Step S211 compresses the designated file, and saves it to the designated directory: in this step, after the designated file is saved, the designated file is compressed and saved to the designated directory. The aim is to reduce the consumption of space as much as possible. The directory structure saved is as follows:
Figure BDA0001736496770000101
wherein the number of files under each subdirectory is limited. For the SD card, the maximum file number of each directory is limited to 10000; for/tmp directories, each directory limits the number of files to 100. The method mainly has two reasons, namely, preventing the query file from being blocked due to excessive files; in addition, the system has limitation on the number of files in the directory, so that the problem that the files cannot be stored finally due to excessive files is avoided. The cache thread flow can be realized through the steps S201 to S211.
Fig. 6 is a flowchart of a sending thread flow in this embodiment, where the sending thread is mainly responsible for sending data to the cloud probe server Cloudserver. In fig. 6, the thread sending process includes the following steps:
step S301 acquires a file: in this step, a file is acquired.
Step S302 determines whether a file exists in the memory space: in this step, it is determined whether a file exists in the memory space, that is, it is determined whether a file exists in the tmp directory, and if the determination result is yes, step S305 is executed; otherwise, step S303 is executed.
Step S303 determines whether a file exists in the SD card: if the determination result in the step S302 is negative, that is, no file exists in the memory space, the step is executed. In this step, it is determined whether a file exists in the SD card, and if the determination result is yes, step S305 is executed; otherwise, step S304 is performed.
Step S304 waits for a fourth set time: if the judgment result in the step S303 is no, that is, no file exists in both directories, the present step is executed. In this step, the process returns to step S301 after waiting for the fourth setting time. The fourth setting time is a short time, for example: for 3 seconds. Of course, in practical applications, the size of the fourth setting time may be adjusted according to specific situations.
Step S305 reads the file and stores the content in the file in the cache: if the judgment result of the step S302 is yes or the judgment result of the step S303 is yes, the present step is executed. In this step, the file is read and the content in the file is stored in the cache. After the present step is executed, step S306 is executed.
After the reading in step S306 is completed, the corresponding file is deleted, and if there is no file in the corresponding directory, the corresponding directory is deleted: in this step, after the reading is completed, the corresponding file is deleted, and if no file exists in the corresponding directory, the corresponding directory is deleted. After the present step is executed, step S307 is executed.
Step S307 is to determine whether the connection with the cloud probe server is normal: in this step, whether the connection with the cloud probe server is normal or not is judged, the connection state is judged through a heartbeat keep-alive mechanism, and if the judgment result is yes, step S309 is executed; otherwise, step S308 is executed.
Step S308 suspends transmission and waits for a fifth set time: if the judgment result of the above step S307 is no, that is, in the off state, the present step is executed. In this step, the transmission is suspended and the fifth setting time is waited for, and the process returns to step S307, that is, the transmission is suspended and the transmission is waited for after the connection is normal. The size of the fifth setting time can be adjusted according to specific situations.
Step S309, assembling the data message and sending to the cloud probe server: if the judgment result of the above step S307 is yes, the present step is executed. In this step, the data message is assembled and sent to the cloud probe server Cloudserver. And judging whether the data message has the fragment identifier or not during assembly. If so, sending the message in multiple messages. After the present step is executed, step S310 is executed.
Step S310 determines whether the transmission is successful: in this step, whether the transmission is successful is judged, and if the judgment result is yes, the step returns to the step S301; otherwise, step S311 is performed.
Step S311 waits for the sixth setting time: if the judgment result of the step S310 is no, that is, the transmission fails, the present step is executed. In this step, the process returns to step S307 after waiting for the sixth setting time. In this embodiment, the sixth setting time is 1 second, that is, the transmission is continued after waiting for 1 second. Of course, in practical applications, the magnitude of the sixth setting time can be adjusted accordingly according to specific situations. The sending thread flow can be realized by the above steps S301 to S311.
For the timer procedure, the timer procedure includes: when the timer module is triggered, sending a heartbeat message to a designated cloud probe server cloud (namely, a heartbeat cloud probe server), if a response message is normally received, indicating that the probe device is normally connected with the cloud probe server cloud, and if no response is received for n times continuously and n is greater than the maximum timeout time, considering that the probe device is disconnected with the cloud probe server cloud.
In a word, the invention is different from other equipment in the same industry, firstly, the invention adopts the chips of the operator grade, ensures the quality of the equipment and reduces the later maintenance of the equipment. Secondly, the existence of multiple device forms meets the requirements of different users in different scenes. And thirdly, integrating a plurality of peripherals to form intelligent nodes for serving big data. Fourthly, the equipment is guaranteed to meet the requirement of public security on network safety, and guarantee is provided for future ordered development of projects.
Data acquisition and arrangement are carried out on a wireless protocol layer, and the problem of uplink flow of equipment is effectively solved by adopting a customized protocol and compression transmission. The magnitude of data uploading is simplified, and broadband resources and cloud-stored server resources required by data uploading are saved. The data is processed in an integrity and accuracy mode, for example, pseudo MAC processing, an acquisition optimization algorithm and the like, support of application services is guaranteed, and traditional blind data uploading, simple averaging and abnormal data are avoided.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.

Claims (8)

1. A network data processing system is characterized by comprising probe equipment and a cloud probe server, wherein the probe equipment comprises a probe data collection module, a probe data distribution module and a probe client module, the probe data collection module works in a wireless drive module, collects relevant information of surrounding wireless equipment by analyzing wireless messages to obtain probe data, and stores the probe data in a memory; the probe data distribution module is connected with the probe data collection module, works in a user mode, is used for distributing probe data and providing a registration mechanism, supports a plurality of service modules to register, provides a filtering type and a time interval during registration, provides probe data regularly according to registration content, specifies the time interval during registration of the service modules, and performs deduplication processing on the probe data according to the time interval; the probe client module is connected with the probe data distribution module, acquires probe data by registering the probe data distribution module, assembles data messages according to a reporting protocol, and reports the assembled data messages to the cloud probe server.
2. The network data processing system of claim 1, wherein the probe data collection module is controlled by an enable switch, and when the function is enabled, the collection operation is continuously performed, and the terminal MAC and the radio frequency ID are used as unique identifiers for performing deduplication processing; when the function is turned off, all cached information is emptied and no longer collected.
3. The network data processing system of claim 2, wherein the probe data distribution module obtains probe data from the probe data collection module through an IOCTL mechanism, the obtaining is performed every first set time, and after the obtaining, the probe data collection module performs a flushing operation on the cached data; when a service module is registered, the probe data distribution module enables the probe data collection module, and if all services are registered, the probe data distribution module closes the probe data collection module.
4. A network data processing method applied to the network data processing system according to claim 1, comprising an initialization process, a main thread process, a cache thread process, a sending thread process and a timer process, wherein the initialization process comprises the following steps:
A) acquiring related configuration information of the probe equipment through the configuration file; the relevant configuration information comprises a reporting period, an average value time interval, a terminal type and relevant information of the cloud probe server;
B) registering a probe data distribution module according to the average value time interval;
C) the initialization timer module is used for sending heartbeat messages;
D) initializing related content of the data structure;
E) acquiring a directory index of stored data;
F) after initialization is finished, respectively creating a cache thread and a sending thread;
G) and after the creation is completed, processing the probe data sent from the probe data distribution module by adopting the main thread.
5. The method according to claim 4, wherein the main thread flow comprises the steps of:
A1) receiving the probe data sent by the probe data distribution module, and updating the information in the HASH chain;
B1) judging whether the current time interval reaches the reporting interval, if so, executing the step C1); otherwise, return to step a 1);
C1) traversing the HASH chain, assembling a message according to a report protocol, and executing the step D1);
D1) and storing the assembled message into a single-direction linked list.
6. The method according to claim 5, wherein the cache thread process comprises the steps of:
A2) judging whether the system time is successfully synchronized, if so, executing the step B2); otherwise, continuing to judge in the step after waiting for a second set time;
B2) acquiring the assembled data packet from the one-way linked list;
C2) judging whether the one-way linked list has data, if so, modifying the timestamp in the acquired data packet, and executing the step D2); otherwise, returning to the step B2 after waiting for a third set time);
D2) judging whether the system has the SD card, if so, selecting the storage catalog as the SD card, and executing the step E2); otherwise, saving the directory selection as the memory space, and executing step E2);
E2) writing the data packet into a specified file in a binary form;
F2) and compressing the specified file and storing the compressed specified file in a specified directory.
7. The method of claim 6, wherein the send thread process comprises the steps of:
A3) acquiring a file;
B3) judging whether a file exists in the memory space, if so, executing step D3); otherwise, performing step C3);
C3) judging whether a file exists in the SD card, if so, executing step D3); otherwise, returning to the step A3 after waiting for a fourth set time);
D3) reading a file and storing the content in the file into a cache;
E3) after the reading is finished, deleting the corresponding file, and if no file exists in the corresponding directory, deleting the corresponding directory;
F3) judging whether the connection with the cloud probe server is normal or not, if so, executing step G3); otherwise, suspending sending and continuing to judge the step after waiting for a fifth set time;
G3) assembling a data message and sending the data message to the cloud probe server;
H3) judging whether the transmission is successful, if so, returning to the step A3); otherwise, return to step F3 after waiting for the sixth set time).
8. The network data processing method of claim 4, wherein the timer procedure comprises: when the timer module is triggered, a heartbeat message is sent to a designated cloud probe server, if a response message is normally received, the connection with the cloud probe server is indicated to be normal, if no response is received for n times continuously, and n is greater than the maximum timeout time, the connection with the cloud probe server is considered to be disconnected.
CN201810798348.8A 2018-07-19 2018-07-19 Network data processing system and method Active CN109089301B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201810798348.8A CN109089301B (en) 2018-07-19 2018-07-19 Network data processing system and method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201810798348.8A CN109089301B (en) 2018-07-19 2018-07-19 Network data processing system and method

Publications (2)

Publication Number Publication Date
CN109089301A CN109089301A (en) 2018-12-25
CN109089301B true CN109089301B (en) 2021-01-15

Family

ID=64837990

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201810798348.8A Active CN109089301B (en) 2018-07-19 2018-07-19 Network data processing system and method

Country Status (1)

Country Link
CN (1) CN109089301B (en)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110809026B (en) * 2019-09-29 2022-11-01 视联动力信息技术股份有限公司 File processing method and device, electronic equipment and storage medium
CN114900341B (en) * 2022-04-24 2023-11-03 京东科技信息技术有限公司 Scanning detection method, device, system, equipment and medium in hybrid cloud environment

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102426536A (en) * 2011-10-26 2012-04-25 深圳市亚特尔科技有限公司 Multitask data communication implementation method and system
WO2014068757A1 (en) * 2012-11-01 2014-05-08 株式会社日立製作所 Information processing system, data administration method, and program
CN106254246A (en) * 2016-09-20 2016-12-21 上海斐讯数据通信技术有限公司 A kind of method of sniff wireless terminal MAC information and router
CN106413004A (en) * 2016-09-05 2017-02-15 上海斐讯数据通信技术有限公司 Wireless MAC sniffing scanning method and apparatus
CN106790491A (en) * 2016-12-14 2017-05-31 日照职业技术学院 The implementation method of the Internet of Things movable termination intelligent platform of data-oriented
CN106851719A (en) * 2017-04-18 2017-06-13 云熠信息科技有限公司 A kind of Wifi probes, data balancing acquisition system and method
CN108243067A (en) * 2016-12-23 2018-07-03 上海辰智商务信息咨询有限公司 A kind of user terminal data acquisition system based on wifi detections

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102426536A (en) * 2011-10-26 2012-04-25 深圳市亚特尔科技有限公司 Multitask data communication implementation method and system
WO2014068757A1 (en) * 2012-11-01 2014-05-08 株式会社日立製作所 Information processing system, data administration method, and program
CN106413004A (en) * 2016-09-05 2017-02-15 上海斐讯数据通信技术有限公司 Wireless MAC sniffing scanning method and apparatus
CN106254246A (en) * 2016-09-20 2016-12-21 上海斐讯数据通信技术有限公司 A kind of method of sniff wireless terminal MAC information and router
CN106790491A (en) * 2016-12-14 2017-05-31 日照职业技术学院 The implementation method of the Internet of Things movable termination intelligent platform of data-oriented
CN108243067A (en) * 2016-12-23 2018-07-03 上海辰智商务信息咨询有限公司 A kind of user terminal data acquisition system based on wifi detections
CN106851719A (en) * 2017-04-18 2017-06-13 云熠信息科技有限公司 A kind of Wifi probes, data balancing acquisition system and method

Also Published As

Publication number Publication date
CN109089301A (en) 2018-12-25

Similar Documents

Publication Publication Date Title
JP6138809B2 (en) System and method for fast initial network link setup
JP6073344B2 (en) System and method for fast initial network link setup
KR101494143B1 (en) Radio resource optimization method, device and system
US20230247388A1 (en) Communication system, method and device for miniature intelligent sensor
CN109155933A (en) Reflective service quality control and management
WO2017132991A1 (en) Method and device for communication resource allocation, terminal device, base station, and communication system
JP5976823B2 (en) System and method for fast initial network link setup
US8050218B2 (en) Mobile communications system PDIF and method for peer detection of mobile terminal
JP6129862B2 (en) System and method for fast initial network link setup
CN111586807B (en) Network access method and communication device
CN108282846B (en) Service request processing method and device
CN110740490A (en) Terminal network access method, gateway equipment, system, storage medium and device
CN108141801B (en) Cell reselection method, frequency point information management method and device
CN101361323A (en) Terminal, access point and method for roaming queries prior to association/authentication
CN109168171B (en) Configuration information acquisition method, device, equipment and system
JP5784831B2 (en) Device and method for communication of management information in an ad hoc wireless network
CN109089301B (en) Network data processing system and method
CN107623752B (en) Network management method and device based on link layer
CN109246762B (en) Local service distribution method and device
CN105165068A (en) Entity and a method of operating an entity of a wireless local area network
US9554327B2 (en) System coordinated WLAN scanning
US20110131255A1 (en) Wireless base station device and operation information collection method for the same
WO2017000477A1 (en) Method and device for managing neighbor configuration information
CN112004258B (en) Data acquisition method and communication device
CN111405615B (en) Communication data transmission method, device and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant