CN110913012A - High-speed parallel data processing method based on agricultural Internet of things - Google Patents
High-speed parallel data processing method based on agricultural Internet of things Download PDFInfo
- Publication number
- CN110913012A CN110913012A CN201911236366.8A CN201911236366A CN110913012A CN 110913012 A CN110913012 A CN 110913012A CN 201911236366 A CN201911236366 A CN 201911236366A CN 110913012 A CN110913012 A CN 110913012A
- Authority
- CN
- China
- Prior art keywords
- data
- sensor
- fpga
- layer
- dictionary
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/12—Protocols specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/565—Conversion or adaptation of application format or content
- H04L67/5651—Reducing the amount or size of exchanged application data
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/04—Protocols for data compression, e.g. ROHC
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer Security & Cryptography (AREA)
- Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
A high-speed parallel data processing method based on an agricultural Internet of things aims to use an FPGA (field programmable gate array) as a controller of a network layer of the Internet of things, utilizes the characteristic that the FPGA processes data in parallel at a high speed, and uses the FPGA to receive data such as air temperature, soil humidity, soil pH value, illumination intensity and the like measured by sensors such as a temperature sensor, a humidity sensor, a soil pH value sensor, a light intensity sensor and the like at a high speed, and because a large amount of redundant data exist in the data packet format of each branch point and the data measured by the sensors are stable signals, the data packets among the branch points have large redundant data. The method comprises the steps of firstly, carrying out differential compression on data values of two adjacent moments of the same sensor to reduce the redundancy rate of the measured value of the same sensor, then, carrying out cross splicing on data packets of the measured data of different sensors, and then, carrying out two-stage parallel compression on the data by using an improved LZW data compression method to remove the data redundancy rate and improve the communication rate.
Description
Technical Field
The invention relates to the technical field of data processing of agricultural Internet of things, in particular to a high-speed parallel data processing method based on the agricultural Internet of things.
Background
The agricultural internet of things is characterized in that a large number of sensor nodes form a monitoring network, information is collected through various sensors, agricultural scientific management is achieved, farmers are helped to find problems in time, and the purposes of reasonably using agricultural resources, reducing production cost and improving yield and quality of agricultural products are achieved. Thus, agriculture will gradually shift from a manpower-centric, stand-alone machine-dependent production mode to an information and software-centric production mode, thereby using a wide variety of automated, intelligent, remotely-controlled production equipment.
The data in all aspects of agriculture are acquired in real time by establishing a wireless sensor network, so that various agricultural products are managed finely. However, the agricultural production environment is complex and the conditions are severe. The actual transmission speed of the wireless sensor is low, which causes data loss and is difficult to satisfy the real-time transmission of data. Therefore, a method for optimizing data redundancy of the agricultural internet of things is urgently needed to be provided so as to improve the data transmission speed of the internet of things and perform fine management on agriculture.
Disclosure of Invention
In order to solve the problems, the technical scheme of the invention is a processing method of high-speed parallel data based on an agricultural internet of things, the data of the internet of things is subjected to real-time lossless compression by utilizing the high-speed parallel data processing capability of an FPGA based on an LZW and differential dual lossless compression technology, the redundancy of communication data is reduced, the real-time transmission capability of the data of the internet of things is improved, and fine management is performed on agriculture, so that the purpose is achieved, the processing method of the high-speed parallel data based on the agricultural internet of things comprises the following steps:
step 1: the FPGA is used as a controller of a network layer of the Internet of things, and a zigbee wireless communication module is used as a network layer to be respectively in wireless communication with a perception layer and a platform layer;
step 2: determining the communication data packet formats of a network layer and a sensing layer and the network layer and the sensing layer, and carrying out address coding on different sensors of the sensing layer to distinguish data;
and step 3: a sensing layer is formed by a temperature sensor, a humidity sensor, a soil pH value sensor and a light intensity sensor, and all node sensors of the sensing layer are mutually independent to measure air temperature, soil humidity, soil pH value and illumination intensity parameters;
and 4, step 4: using a network layer controller FPGA to receive mass data frames of a sensing layer in parallel, carrying out difference on measured values of adjacent moments of the same sensor, carrying out first-stage compression on data, caching the data after difference into a fifo memory of the FPGA, and splicing data packets of the difference data measured by each sensor in the fifo memory so as to improve the compression ratio of subsequent data;
and 5: the dictionary query uses a hash table-based quick query method to establish a golden section hash function and a coupled data structure between dictionary variables and dictionary storage addresses and dictionary data;
step 6: the FPGA sets a value searching module, searches all possible values of data in the data packet in parallel, maps the data to a corresponding storage address through a hash function, serves as an initial dictionary of the LZW, and stores the data into a dual-port RAM of the FPGA;
and 7: performing LZW compression on the data stream in the fifo memory, if the numeric string does not exist in the dictionary in the process, updating the character string into the dictionary, and outputting a prefix code;
and 8: and (3) packaging the coded data of the compressed data according to the data format in the step (1), and transmitting the data to the platform layer through the wireless communication zigbee wireless communication module for data storage and analysis.
As a further improvement of the invention, in the step 2, the communication data packet formats of the network layer and the sensing layer and the network layer and the sensing layer are a frame header, an address, valid data, a check bit and a frame tail.
The invention relates to a high-speed parallel data processing method based on an agricultural Internet of things, which adopts the technical scheme and has the following beneficial effects:
1. the processing method for high-speed parallel data compression based on the agricultural Internet of things can greatly reduce the redundancy of the information of the Internet of things, improve the transmission rate of data communication and greatly reduce the data storage space.
2. Two-stage data lossless compression is used based on the FPGA, so that the low-cost low-speed wireless communication module can be used in the Internet of things, and the cost of a communication part is greatly reduced.
3. According to the method, the FPGA chip is used as a network layer controller of the Internet of things, and due to the characteristic that the FPGA processes data in parallel, one network layer fulcrum can control a plurality of sensing layer fulcrums simultaneously, so that the cost of the network layer is reduced.
4. The method has the advantages of high data transmission rate and low cost. The method can be applied to a large-scale Internet of things monitoring system in real time.
5. The method is high in data processing speed and can be used for a real-time video Internet of things monitoring system.
Drawings
FIG. 1 is a diagram of an agricultural Internet of things system of the present invention;
FIG. 2 is a data interaction diagram between a network layer and a perception layer according to the present invention;
FIG. 3 is a system block diagram of the difference module of the present invention;
FIG. 4 is a schematic diagram of data stitching according to the present invention;
FIG. 5 is a flow chart of LZW dictionary compression according to the present invention;
FIG. 6 is a block diagram of an LZW dictionary compression module according to the present invention.
Detailed Description
The invention is described in further detail below with reference to the following detailed description and accompanying drawings:
the technical scheme of the invention is a processing method of high-speed parallel data based on an agricultural Internet of things, which utilizes the high-speed parallel data processing capability of an FPGA based on a dual lossless compression technology of LZW and difference to perform real-time lossless compression on the Internet of things data, reduces the redundancy of communication data, improves the real-time transmission capability of the Internet of things data, and performs fine management on agriculture.
The agricultural internet mainly comprises a perception identification layer, a network construction layer and a platform management layer, as shown in figure 1.
On the sensing layer, the invention uses a series of sensors including a temperature sensor, a humidity sensor, a pressure sensor, a light intensity sensor and the like to monitor the climate condition of the farmland. The perception layer is formed by a plurality of sensor nodes, each sensor node is independent, and the sensor group of each node measures air temperature, soil humidity, soil pH value, illumination intensity and the like respectively. Each node receives the data and packages the data into a data format as follows: frame head, address, effective data, check bit and frame tail, then sending the data to network layer through zigbee wireless communication module.
The controller of the network layer uses a Cyclone IV series FPGA chip of altera corporation, has high performance and low price, and can realize high-speed parallel processing of data by two technologies of concurrency and pipelining. Due to the characteristic of parallel data processing of the FPGA in the network layer, the FPGA can receive data sent by a plurality of sensing layer nodes in parallel in real time as shown in fig. 2.
The controller of the network layer performs two-stage data compression of differential compression and LZW dictionary compression on the data sent by the sensing layer, so that the redundancy of the data is greatly reduced to increase the transmission rate of the data. The measured data of the sensors are mostly stable signals, and a large amount of redundant data exist among the measured data of the same sensor at different moments. Therefore, the first-stage compression of the invention uses differential compression, the first-stage compression can be carried out on the data by differentiating the measured data at adjacent moments of the sensors, and the network layer can differentiate the effective data of the data packets sent at adjacent moments of the same sensing layer to primarily reduce the redundancy of the data. After the data is compressed differentially, there is still a great redundancy between the data collected by different sensors. And performing secondary compression on the total data packets received by the network layer by using LZW dictionary compression. And (3) splicing the data packets subjected to the first-stage compression to increase the compression rate of the LZW, wherein the specific splicing mode is shown in fig. 4.
According to the invention, a golden section hash function is established between data and storage addresses in the dictionary compressed by the LZW data, a mapping relation is established between the storage data and the storage addresses, and the dictionary inquiring time in the LZW compression process is reduced.
The Hash function is formulated as:
wherein 2654435761 is 0-
Value is the output Value of the hash function, and Key is the input Value of the hash function, which correspond to the storage address and the storage data respectively.
The FPGA sends the spliced data packet to a value searching module and initializes an LZW dictionary; and then inputting the data stream into a compression module, wherein the compression module compares the data with the dictionary, if the dictionary does not exist, the data is mapped to the corresponding address of the dictionary through a hash module to perform dictionary compression, and prefix code codes are output, and the structure chart is shown in fig. 5.
The flow of the LZW algorithm is shown in FIG. 6, and the step 1: the initial dictionary initially contains all possible roots of data, while the current prefix P is empty; step 2: update current character (C) step 3: judging whether the affix-character string P + C is in the dictionary, and if yes, updating P = P + C; otherwise, outputting a code word representing the current prefix P to a code word stream, adding an affix-character string P + C to the dictionary, and updating P = C; and 4, step 4: judging whether code words exist in the code word stream to be decoded, if so, returning to the step 2; otherwise, the codeword representing the current prefix P is output to the stream of codewords.
The FPGA repacks the LZW output code, and the data packet format is as follows: frame header, address, valid data, check bit, frame tail. And sending the data to a platform layer for data storage and analysis through a zigbee wireless communication module.
The above description is only a preferred embodiment of the present invention, and is not intended to limit the present invention in any way, but any modifications or equivalent variations made according to the technical spirit of the present invention are within the scope of the present invention as claimed.
Claims (2)
1. A high-speed parallel data processing method based on an agricultural Internet of things is characterized by comprising the following steps:
step 1: the FPGA is used as a controller of a network layer of the Internet of things, and a zigbee wireless communication module is used as a network layer to be respectively in wireless communication with a perception layer and a platform layer;
step 2: determining the communication data packet formats of a network layer and a sensing layer and the network layer and the sensing layer, and carrying out address coding on different sensors of the sensing layer to distinguish data;
and step 3: a sensing layer is formed by a temperature sensor, a humidity sensor, a soil pH value sensor and a light intensity sensor, and all node sensors of the sensing layer are mutually independent to measure air temperature, soil humidity, soil pH value and illumination intensity parameters;
and 4, step 4: using a network layer controller FPGA to receive mass data frames of a sensing layer in parallel, carrying out difference on measured values of adjacent moments of the same sensor, carrying out first-stage compression on data, caching the data after difference into a fifo memory of the FPGA, and splicing data packets of the difference data measured by each sensor in the fifo memory so as to improve the compression ratio of subsequent data;
and 5: the dictionary query uses a hash table-based quick query method to establish a golden section hash function and a coupled data structure between dictionary variables and dictionary storage addresses and dictionary data;
step 6: the FPGA sets a value searching module, searches all possible values of data in the data packet in parallel, maps the data to a corresponding storage address through a hash function, serves as an initial dictionary of the LZW, and stores the data into a dual-port RAM of the FPGA;
and 7: performing LZW compression on the data stream in the fifo memory, if the numeric string does not exist in the dictionary in the process, updating the character string into the dictionary, and outputting a prefix code;
and 8: and (3) packaging the coded data of the compressed data according to the data format in the step (1), and transmitting the data to the platform layer through the wireless communication zigbee wireless communication module for data storage and analysis.
2. The agricultural internet of things-based high-speed parallel data processing method according to claim 1, characterized in that: and in the step 2, the formats of the communication data packets of the network layer and the sensing layer and the formats of the communication data packets of the network layer and the sensing layer are a frame header, an address, effective data, a check bit and a frame tail.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911236366.8A CN110913012A (en) | 2019-12-05 | 2019-12-05 | High-speed parallel data processing method based on agricultural Internet of things |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911236366.8A CN110913012A (en) | 2019-12-05 | 2019-12-05 | High-speed parallel data processing method based on agricultural Internet of things |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110913012A true CN110913012A (en) | 2020-03-24 |
Family
ID=69822674
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911236366.8A Pending CN110913012A (en) | 2019-12-05 | 2019-12-05 | High-speed parallel data processing method based on agricultural Internet of things |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110913012A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112100455A (en) * | 2020-08-18 | 2020-12-18 | 北京理工大学 | LZW dictionary searching method based on character string parallel search |
| CN112770286A (en) * | 2021-01-13 | 2021-05-07 | 深圳市木千科技有限公司 | Sensor data processing method and device and computer equipment |
| CN112800877A (en) * | 2021-01-15 | 2021-05-14 | 金陵科技学院 | Pest control method based on agricultural Internet of things |
| CN116260842A (en) * | 2023-02-21 | 2023-06-13 | 华智生物技术有限公司 | Internet of things data acquisition method, system, device and medium for greenhouse |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040077342A1 (en) * | 2002-10-17 | 2004-04-22 | Pantech Co., Ltd | Method of compressing sounds in mobile terminals |
| CN101211341A (en) * | 2006-12-29 | 2008-07-02 | 上海芯盛电子科技有限公司 | Image intelligent mode recognition and searching method |
| CN108873062A (en) * | 2018-05-08 | 2018-11-23 | 吉林大学 | A kind of Multi-encoder high-speed seismic data parallel lossless compression method based on FPGA |
| CN109199374A (en) * | 2018-10-15 | 2019-01-15 | 烟台羿中医疗科技有限公司 | A kind of multi-lead electrocardiogram (ECG) data recording device and method |
| CN109672449A (en) * | 2018-12-13 | 2019-04-23 | 武汉中元华电科技股份有限公司 | A kind of device and method fast implementing LZ77 compression based on FPGA |
| US20190196907A1 (en) * | 2019-03-05 | 2019-06-27 | Intel Corporation | Compression techniques for distributed data |
| CN110149990A (en) * | 2019-06-13 | 2019-08-23 | 华东师范大学 | Greenhouse plants growing environment regulating system based on ZigBee |
-
2019
- 2019-12-05 CN CN201911236366.8A patent/CN110913012A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040077342A1 (en) * | 2002-10-17 | 2004-04-22 | Pantech Co., Ltd | Method of compressing sounds in mobile terminals |
| CN101211341A (en) * | 2006-12-29 | 2008-07-02 | 上海芯盛电子科技有限公司 | Image intelligent mode recognition and searching method |
| CN108873062A (en) * | 2018-05-08 | 2018-11-23 | 吉林大学 | A kind of Multi-encoder high-speed seismic data parallel lossless compression method based on FPGA |
| CN109199374A (en) * | 2018-10-15 | 2019-01-15 | 烟台羿中医疗科技有限公司 | A kind of multi-lead electrocardiogram (ECG) data recording device and method |
| CN109672449A (en) * | 2018-12-13 | 2019-04-23 | 武汉中元华电科技股份有限公司 | A kind of device and method fast implementing LZ77 compression based on FPGA |
| US20190196907A1 (en) * | 2019-03-05 | 2019-06-27 | Intel Corporation | Compression techniques for distributed data |
| CN110149990A (en) * | 2019-06-13 | 2019-08-23 | 华东师范大学 | Greenhouse plants growing environment regulating system based on ZigBee |
Non-Patent Citations (2)
| Title |
|---|
| 江煜，许飞云，许丙胜: "声发射层析成像技术在混凝土结构中的应用", 《东南大学学报( 自然科学版)》 * |
| 韩九强，杨磊: "《数字图像处理——基于XAVIS组态软件》", 31 August 2018 * |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112100455A (en) * | 2020-08-18 | 2020-12-18 | 北京理工大学 | LZW dictionary searching method based on character string parallel search |
| CN112100455B (en) * | 2020-08-18 | 2022-09-20 | 北京理工大学 | LZW dictionary searching method based on character string parallel search |
| CN112770286A (en) * | 2021-01-13 | 2021-05-07 | 深圳市木千科技有限公司 | Sensor data processing method and device and computer equipment |
| CN112770286B (en) * | 2021-01-13 | 2023-11-17 | 深圳市木千科技有限公司 | Sensor data processing method and device and computer equipment |
| CN112800877A (en) * | 2021-01-15 | 2021-05-14 | 金陵科技学院 | Pest control method based on agricultural Internet of things |
| CN116260842A (en) * | 2023-02-21 | 2023-06-13 | 华智生物技术有限公司 | Internet of things data acquisition method, system, device and medium for greenhouse |
| CN116260842B (en) * | 2023-02-21 | 2024-01-23 | 华智生物技术有限公司 | Internet of things data acquisition method, system, device and medium for greenhouse |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110913012A (en) | High-speed parallel data processing method based on agricultural Internet of things | |
| Al-Qurabat et al. | Data traffic management based on compression and MDL techniques for smart agriculture in IoT | |
| CN110933146B (en) | Data conversion method and device and server | |
| M Al-Qurabat | A lightweight Huffman-based differential encoding lossless compression technique in IoT for smart agriculture | |
| CN104079488B (en) | Transmission equipment and method based on two layers of head compression of Ethernet | |
| CN108600192A (en) | A kind of DBC document analysis and message analysis method based on regular expression | |
| CN104200620A (en) | Building healthy remote monitoring system and building healthy remote monitoring method | |
| CN113094346A (en) | Big data coding and decoding method and device based on time sequence | |
| AU2016341126A1 (en) | System and method for data compression over a communication network | |
| CN110620637B (en) | Data decompression device and method based on FPGA | |
| Yuanbin et al. | A data compression algorithm based on adaptive Huffman code for wireless sensor networks | |
| CN104270829B (en) | A kind of downhole data collection and processing method based on compressed sensing | |
| CN108873062A (en) | A kind of Multi-encoder high-speed seismic data parallel lossless compression method based on FPGA | |
| CN100476797C (en) | Data lossless compression method for data transmission of remote monitoring system | |
| CN108173620A (en) | WSN abnormal datas based on compression network coding find network system realization | |
| CN101241499A (en) | Patricia tree rapid lookup method | |
| US11310353B2 (en) | Data transmission method for creating data structure facilitating data transmission and reception | |
| CN108429729B (en) | Data communication isolation system and isolation method in industrial big data acquisition environment | |
| Sheikh et al. | Data compression techniques for wireless sensor network | |
| CN101888303A (en) | Recording method of network traffic information and related device | |
| CN113329442A (en) | Universal multilink load compression and decompression method and system | |
| CN102394718A (en) | Sensing network data compression coding/decoding method | |
| CN114827287B (en) | Data compression method and system | |
| US6313763B1 (en) | Variable length data packet with adjustable length indicator | |
| CN111800825B (en) | Dynamic retransmission method of data in wireless sensor network based on compressed sensing |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200324 |
|
| RJ01 | Rejection of invention patent application after publication |