CN108243067A - A kind of user terminal data acquisition system based on wifi detections - Google Patents
A kind of user terminal data acquisition system based on wifi detections Download PDFInfo
- Publication number
- CN108243067A CN108243067A CN201611201808.1A CN201611201808A CN108243067A CN 108243067 A CN108243067 A CN 108243067A CN 201611201808 A CN201611201808 A CN 201611201808A CN 108243067 A CN108243067 A CN 108243067A
- Authority
- CN
- China
- Prior art keywords
- data
- server
- wifi
- acquisition system
- wifi probe
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000001514 detection method Methods 0.000 title claims description 9
- 239000000523 sample Substances 0.000 claims abstract description 25
- 230000006835 compression Effects 0.000 claims abstract description 10
- 238000007906 compression Methods 0.000 claims abstract description 10
- 230000004044 response Effects 0.000 claims abstract description 5
- 238000004891 communication Methods 0.000 claims description 11
- 238000012856 packing Methods 0.000 claims description 10
- 230000005540 biological transmission Effects 0.000 claims description 4
- 241001269238 Data Species 0.000 abstract description 3
- 238000004321 preservation Methods 0.000 abstract description 3
- 238000011084 recovery Methods 0.000 abstract description 3
- 238000011897 real-time detection Methods 0.000 abstract description 2
- 238000003860 storage Methods 0.000 description 9
- 238000000034 method Methods 0.000 description 7
- 238000012545 processing Methods 0.000 description 6
- 238000013480 data collection Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 238000013481 data capture Methods 0.000 description 2
- 230000006837 decompression Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000013515 script Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 125000004122 cyclic group Chemical group 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000013213 extrapolation Methods 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 230000014759 maintenance of location Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000004806 packaging method and process Methods 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/10—Active monitoring, e.g. heartbeat, ping or trace-route
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Health & Medical Sciences (AREA)
- Cardiology (AREA)
- General Health & Medical Sciences (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The present invention discloses a kind of user terminal data acquisition system detected based on wifi, including wifi probe devices end and server end.The wifi probe devices end, which acquires mac address dates and mac address dates are carried out compression, is uploaded to server, additionally can send heartbeat bag data to server end by setting time interval.The response for the heartbeat packet that server end reverting equipment end is sent, and it is supplied to equipment end that newer download service is set.The characteristics of wifi probe datas acquisition system proposed by the present invention, has：1) the more flexible data volume that 2) uploads is few, small to bandwidth requirement, while in turn ensures time granularity.3) zmodem, reliability are high, and data are not allowed easy to be lost, have the function of that the network recoveries such as preservation data transmit again again after network interruption.It 4) can be with real-time detection wifi probe device states.5) automatically updating function of equipment setting can be realized.
Description
Technical field
The present invention relates to a kind of acquisition system, specifically a kind of user terminal data acquisition system based on wifi detections.
Background technology
Currently there are some equipment that can be acquired with the mac addresses of user terminal wireless network card.(so-called user is whole
End is mainly the equipment with wifi function of surfing the Net such as mobile phone, tablet computer, notebook).The data of acquisition can carry out flow of the people
Calculating, the calculating of area entry density and the matching that carries out big data etc..These equipment are properly termed as wifi probes and set
It is standby.But existing wifi probe devices gathered data and upload data mode there are it is respective the problem of：Some excessively emphasis
Real-time catches mac address dates and does not just process or done a small amount of processing toward server transmission, but data volume is all very
Greatly, can cause to lose or occupy a large amount of network bandwidths.And the mac address dates acquisition under business scenario has its feature, differs
It is so high to determine requirement of real-time；Also some has done data processing, but is stored in the storage medium of equipment in itself, needs periodically
Slave device copy data is gone to come out, the real-time for leading to data acquisition is not high also inconvenient.
Therefore, it is necessary to improve the method for existing data acquisition, transmitted data amount can be reduced as possible to adapt to certain
A little poor network environments, and the stability of wifi data transmissions can be kept again.In addition, in terms of real-time is ensured, need
Method with the real-time, interactive of data acquisition equipment is provided.
Invention content
The purpose of the present invention is to provide a kind of user terminal datas simple in structure, easy to use based on wifi detections
Acquisition system, to solve the problems mentioned in the above background technology.
To achieve the above object, the present invention provides following technical solution：
A kind of user terminal data acquisition system based on wifi detections, including wifi probe devices end and server end,
The wifi probe devices end, which acquires mac address dates and mac address dates are carried out compression, is uploaded to server, additionally can
By setting time interval heartbeat bag data is sent to server end.The response for the heartbeat packet that server end reverting equipment end is sent,
And it is supplied to equipment end that need to update the download service of configuration.
Preferred embodiment as the present invention：The wifi probe devices end includes data deduplication submodule, data record
Module, data packing block, data uploading module, real-time communication module and server, probe data pass sequentially through duplicate removal submodule
Block, data record submodule, data packing block and data uploading module are transferred to server-side, and real-time communication module also connects clothes
Business device.
Compared with prior art, the beneficial effects of the invention are as follows：Wifi probe datas acquisition system proposed by the present invention
Feature has：1) the more flexible data volume that 2) uploads is few, small to bandwidth requirement, while in turn ensures time granularity.3) zmodem,
Reliability is high, and data are not allowed easy to be lost, have the function of that the network recoveries such as preservation data transmit again again after network interruption.4) may be used
With real-time detection wifi probe device states.5) automatically updating function that can be set with starting device.
Description of the drawings
Fig. 1 is wifi probe data acquisition system schematic diagrames；
Fig. 2 is wifi probe devices end software flow pattern；
Fig. 3 is server software flow chart.
Specific embodiment
Below in conjunction with the attached drawing in the embodiment of the present invention, the technical solution in the embodiment of the present invention is carried out clear, complete
Site preparation describes, it is clear that described embodiment is only part of the embodiment of the present invention, instead of all the embodiments.It is based on
Embodiment in the present invention, those of ordinary skill in the art are obtained every other without making creative work
Embodiment shall fall within the protection scope of the present invention.
- 3 are please referred to Fig.1, a kind of user terminal data acquisition system based on wifi detections, including wifi probe devices end
And server end.The wifi probe devices end, which acquires mac data and mac data are carried out compression, is uploaded to server, in addition
Also heartbeat bag data can be sent to server end by setting time interval.The heartbeat packet that server end reverting equipment end is sent is answered
It answers, and is supplied to equipment end that need to update the download service of configuration.
Wifi probe devices end includes data deduplication submodule, data record submodule, data packing block, data and uploads
Module, real-time communication module and server, probe data pass sequentially through duplicate removal submodule, data record submodule, data and are packaged
Module and data uploading module are transferred to server-side, real-time communication module also Connection Service device.
The present invention operation principle be：Entire data collecting system is divided into wifi probe devices end and server end.Wherein
The software at wifi probe devices end is broadly divided into data capture module, data packing block, data uploading module and real-time communication
Module.The software of server end is broadly divided into ftp data acquisition servo module, data loading servo module and real-time communication servo
Module.
Equipment end implementation method is described as follows：
1st, data capture module is divided into data deduplication submodule and data record sub module.
Data deduplication submodule is can the data that captured in setting time to be carried out duplicate removal processing, to effectively reduce weight
Data are captured again, so as to which the crawl data for making storage largely reduce.By the unduplicated data of duplicate removal processing module just by data
Record sub module processing.
Data record submodule is responsible for the user terminal data of crawl that text file is written, this file is known as data record
File.The data of record include the mac addresses of terminal, crawl timestamp, location information etc., are saved in one by one with text formatting.
The record strip of this file preservation is several to be determined according to data collection strategy.There are mainly two types of data collection strategies：1) individual data
It records file and stores fixed number of data.2) number of data acquired in the individual data record file storage set time.When
When the data of acquisition meet the requirement of data collection strategy, terminate the write-in to this document data record, re-create one newly
Document data record.Notify data packing block that can carry out being packaged the document data record for being over write-in simultaneously.
2nd, data packing block is resident in systems, is responsible for the document data record of production carrying out zip compression packings.It beats
Packet is completed to delete former document data record to save memory space.Then, packaging file is put into the storage catalogue specified.This
Catalogue can be that the flash storages in equipment local can also be the storage medium such as tf cards etc. being inserted into.The compressed package of generation
It must include equipment serial number and timestamp in name, so as to server-side processes.
3rd, data uploading module is responsible for the mesh that will store compression data packet according to the ftp addresses and login method that originally set
The compressed package files of record the inside upload to ftp server.It is given birth to if having multiple files inside this catalogue according to compressed package files
Into time upload successively.It uploads after completing a compressed package, by transmitting file in the file size of the same name on ftp server and this
Size be compared, confirm that upload is completed if consistent, this file can be deleted, otherwise continue to retain this file and taste again
Examination uploads.If network interruption or server, which go wrong, can not complete to upload, compression data packet does not have loss and still deposits
Storage is in this catalogue.
4th, real-time communication module is responsible for reporting the real-time status of server apparatus and inquires the update of server.With setting
Time interval sends designated port of the real-time status to the server of setting.Referred to as send heartbeat packet.Server return receive into
The message of work(.Return message, which is divided into, receives response message and new information.If response message shows correctly to receive what is reported
Heartbeat packet；If new information, then according to the server address of institute's band and port in the new information of return, go to download update
Configuration file.The end of configuration file carries the cyclic redundancy check of 16, if the verification of the CRC16 of this configuration file passes through
It then removes this last two byte and configuration file is covered into original configuration file, and equipment is restarted, reach more newly-installed mesh
's.
Server end implementation method is described as follows：
1st, ftp data acquisition servo module mainly builds ftp server.By the way of device packets, 50 equipment
One group of account logs in, in case the data file under a ftp catalogue is excessive.It can be in Duo Tai using domain name mode game server
Different domain names is configured in data acquisition equipment, so as to the equipment connection that a certain ftp server is excessive.
2nd, data loading servo module is mainly the script of timing automatic running in server.These scripts are to each ftp
Catalogue under decompress the data file of upload, the implementor name copy compression data packet in the filename of compressed package is to accordingly
In the catalogue of name and carry out zip decompressions.The out document data record of decompression can be introduced directly into mysql with tool
Database.
Real-time communication servo module.The heartbeat packet that mainly server end sends for the timing of receiving device end, receives
After heartbeat packet, a return message package is replied.Message package is received when not needing to equipment work configuration update, return one is simple
Conditional code.When needing equipment update setting, the conditional code setting update mark of message package is returned, while encloses setting below
The download address of file.
1) effect of the invention is embodied in the following：Flexibility.Using two kinds of data collection strategies, to the terminal of acquisition
Data are stored and compress packing.Both strategies are fixed data item number and Fixed Time Interval respectively.Fixed data item
Number refers to reach certain data pick-up slip number and just preserves a file and compress upload, so easily according to equipment position
Bandwidth situation and flow of the people are suitably set.Such as it is high to bandwidth traffic requirement, the setting of item number is big, secondary to transmit
Number saves communication-cost less.For another example crowded place, setting fixed strip number uploads and item number sets some more compression ratios
Rate is also high.And Fixed Time Interval, refer to the time interval (such as 5 minutes) for reaching setting, just the institute of this time interval
There is gathered data to preserve a file and be packaged upload.This strategy is used in the place of stream of people's rareness, is suitble to judge device data
Whether upload is normal, because there is data packet upload in the set time, will not can't see upload data for a long time.
2) it is few to upload data volume, it is small to bandwidth requirement, while in turn ensure time granularity.The duplicate removal processing submodule of data
The Repeated m ac address dates that (such as 1 minute) in the right times that can be set captures are merged, are suitble to real under commercial terms
The not high demand of when property, and prevent from merging overlong time, detection time is inaccurate.Effectively reduce data volume again simultaneously.In addition,
Data volume is had compressed very much (actual measurement is 5 times or more average) again by the way that zip compress modes are average.
3) data transmission is uploaded the zip compressed packages that successfully just equipment is locally stored using ftp modes and deleted, if network
Perhaps communication condition is bad for interruption, and compression data packet can be remained stored in the storage inside of equipment or extrapolation storage card.
After network recovery or it can continue to upload.Therefore there is zmodem, reliability is high, and data do not allow the characteristics of easy to be lost.
4) Wifi probe devices send heartbeat packet by setting time interval, and server replys return message after receiving heartbeat packet
Packet.It can be used for record and detecting devices state.Setting time interval can meet using minimum time granularity.
5) server needs to update in equipment the message package returned during setting comprising download address is set, and equipment can be according to this
Address is configured update and sets more new function so as to fulfill automatic equipment.
Claims (2)
1. a kind of user terminal data acquisition system based on wifi detections, including wifi probe devices end and server end,
It is characterized in that, the wifi probe devices end, which acquires mac address dates and data are carried out compression, is uploaded to server end, by setting
Interval transmission heartbeat bag data fix time to server end, server end can reply heartbeat packet response message, and be supplied to equipment
End sets newer download service.
A kind of 2. user terminal data acquisition system based on wifi detections according to claim 1, which is characterized in that institute
State wifi probe devices end include data deduplication submodule, data record submodule, data packing block, data uploading module,
Real-time communication module and server, probe data pass sequentially through duplicate removal submodule, data record submodule, data packing block and
Data uploading module is transferred to server-side, real-time communication module also Connection Service device.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611201808.1A CN108243067A (en) | 2016-12-23 | 2016-12-23 | A kind of user terminal data acquisition system based on wifi detections |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611201808.1A CN108243067A (en) | 2016-12-23 | 2016-12-23 | A kind of user terminal data acquisition system based on wifi detections |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108243067A true CN108243067A (en) | 2018-07-03 |
Family
ID=62703293
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611201808.1A Pending CN108243067A (en) | 2016-12-23 | 2016-12-23 | A kind of user terminal data acquisition system based on wifi detections |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108243067A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109089301A (en) * | 2018-07-19 | 2018-12-25 | 深圳云盈网络科技有限公司 | network data processing system and method |
| CN109150617A (en) * | 2018-09-04 | 2019-01-04 | 上海瀚讯信息技术股份有限公司 | A kind of method of self-organizing network route planning and dynamic optimization |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102130950A (en) * | 2011-03-14 | 2011-07-20 | 中国科学技术大学苏州研究院 | Distributed monitoring system based on Hadoop cluster and monitoring method thereof |
| CN103051635A (en) * | 2012-12-28 | 2013-04-17 | 宁波新然电子信息科技发展有限公司 | Method of communication between streaming media server and terminal |
| CN104778642A (en) * | 2015-03-20 | 2015-07-15 | 北京邮电大学 | Data processing method, server and monitoring system for campus users based on WiFi |
| CN105204922A (en) * | 2014-06-30 | 2015-12-30 | 金电联行（北京）信息技术有限公司 | Collecting method of client terminal of data collecting platform |
| CN105813019A (en) * | 2016-03-08 | 2016-07-27 | 深圳市百米生活股份有限公司 | WIFI hotspot device-based positioning management method |
-
2016
- 2016-12-23 CN CN201611201808.1A patent/CN108243067A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102130950A (en) * | 2011-03-14 | 2011-07-20 | 中国科学技术大学苏州研究院 | Distributed monitoring system based on Hadoop cluster and monitoring method thereof |
| CN103051635A (en) * | 2012-12-28 | 2013-04-17 | 宁波新然电子信息科技发展有限公司 | Method of communication between streaming media server and terminal |
| CN105204922A (en) * | 2014-06-30 | 2015-12-30 | 金电联行（北京）信息技术有限公司 | Collecting method of client terminal of data collecting platform |
| CN104778642A (en) * | 2015-03-20 | 2015-07-15 | 北京邮电大学 | Data processing method, server and monitoring system for campus users based on WiFi |
| CN105813019A (en) * | 2016-03-08 | 2016-07-27 | 深圳市百米生活股份有限公司 | WIFI hotspot device-based positioning management method |
Non-Patent Citations (1)
| Title |
|---|
| 向庭勇;向庭波;: "用大数据技术构建公安wifi侦测系统的研究与应用" * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109089301A (en) * | 2018-07-19 | 2018-12-25 | 深圳云盈网络科技有限公司 | network data processing system and method |
| CN109089301B (en) * | 2018-07-19 | 2021-01-15 | 深圳云盈网络科技有限公司 | Network data processing system and method |
| CN109150617A (en) * | 2018-09-04 | 2019-01-04 | 上海瀚讯信息技术股份有限公司 | A kind of method of self-organizing network route planning and dynamic optimization |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103944744B (en) | Log collection method and system | |
| CN111291079A (en) | Data query method and device | |
| CN104246767B (en) | Telemetry system for cloud synchronization system | |
| CN106709043A (en) | Data synchronous loading method based on database log | |
| CN105391524B (en) | QAR datamation management method and system | |
| CN106330788B (en) | Message fragment transmission method and device | |
| US10681115B2 (en) | Multimedia data transmission method and device | |
| CN108243067A (en) | A kind of user terminal data acquisition system based on wifi detections | |
| CN109150617A (en) | A kind of method of self-organizing network route planning and dynamic optimization | |
| CN104618961A (en) | Single-channel TCP/ IP header compression method and system for intelligent power grid | |
| WO2021068891A1 (en) | Method, system, electronic device, and storage medium for storing and collecting temperature data | |
| CN110417892B (en) | Message analysis-based data replication link optimization method and device | |
| CN111182306A (en) | Video extraction method, system, terminal and storage medium for video compression | |
| CN106709053A (en) | Data slice management method and system | |
| CN108228476B (en) | Data acquisition method and device | |
| WO2016153377A1 (en) | Smart logging of trace data for storage systems | |
| CN102820982B (en) | Data transmission method and device | |
| CN205427832U (en) | Data storage capacity calamity device | |
| CN116991858A (en) | Acquisition method, acquisition device and data acquisition system for changed data | |
| CN111935237B (en) | Log processing method and system, electronic device and storage medium | |
| CN103049351B (en) | The log processing method of multivariate data server and device | |
| CN109620193A (en) | Blood pressure monitoring unit and system | |
| CN109947371B (en) | Data recording method, device, memory and T-BOX | |
| US11513769B2 (en) | Data acquisition system, input device, data acquisition apparatus, and data combining apparatus | |
| CN112783724A (en) | Log capture debugging method and system, intelligent device and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180703 |
|
| RJ01 | Rejection of invention patent application after publication |