CN107070590B - WSN perception data distributed decoding method based on MapReduce - Google Patents
WSN perception data distributed decoding method based on MapReduce Download PDFInfo
- Publication number
- CN107070590B CN107070590B CN201611264864.XA CN201611264864A CN107070590B CN 107070590 B CN107070590 B CN 107070590B CN 201611264864 A CN201611264864 A CN 201611264864A CN 107070590 B CN107070590 B CN 107070590B
- Authority
- CN
- China
- Prior art keywords
- matrix
- wsn
- decoding
- data
- coding
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L1/00—Arrangements for detecting or preventing errors in the information received
- H04L1/004—Arrangements for detecting or preventing errors in the information received by using forward error control
- H04L1/0045—Arrangements at the receiver end
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L1/00—Arrangements for detecting or preventing errors in the information received
- H04L1/004—Arrangements for detecting or preventing errors in the information received by using forward error control
- H04L1/0056—Systems characterized by the type of code used
- H04L1/0057—Block codes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04W—WIRELESS COMMUNICATION NETWORKS
- H04W84/00—Network topologies
- H04W84/18—Self-organising networks, e.g. ad-hoc networks or sensor networks
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D30/00—Reducing energy consumption in communication networks
- Y02D30/70—Reducing energy consumption in communication networks in wireless communication networks
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Error Detection And Correction (AREA)
Abstract
The invention discloses a WSN perception data distributed decoding method based on MapReduce, which comprises the following steps: firstly, constructing a sparse matrix through a Reed-Solomon code (RS code) in a WSN (wireless sensor network), coding sensing data, and sending the coded data to a Hadoop cluster through an external network; then, the Hadoop cluster acquires coding vectors to form a coding matrix, and acquires coded data to form vectors to be decoded; then, the Hadoop cluster utilizes a MapReduce framework to perform inversion operation on the coding matrix to obtain a decoding matrix; and finally, the Hadoop cluster uses the decoding matrix and the received coded data to perform decoding calculation by using a MapReduce frame to obtain the original WSN sensing data. The invention can improve the reliability of the WSN and the energy utilization rate of the WSN network, and can realize the efficient and quick decoding of the coded data.
Description
Technical Field
The invention belongs to the technical field of cloud computing, and particularly relates to a WSN perception data distributed decoding method based on MapReduce.
Background
With the increasing maturity of Wireless Sensor Network (WSN) technology, the application field of WSN is also wider and wider. However, as the sensor nodes in the WSN are simple in design and the computing resources, the storage resources and the energy supply resources are limited, the communication distance of the nodes in the WSN is short, the network bandwidth is low, and the wireless communication link is unreliable. By using the network coding technology, the throughput of the WSN can be improved, the load of the WSN is improved, the bandwidth utilization rate of the WSN is improved, and the energy consumption of the WSN is reduced. With the continuous development of network coding technology, network coding is also applied in WSNs. The invention utilizes Reed-Solomon codes (RS codes) to construct a sparse matrix to encode the WSN sensing data, thereby improving the reliability and the energy utilization rate of the WSN.
Due to the use of network coding, time delay is generated in sink decoding, and data decoding in the wireless sensor network greatly increases the burden of the wireless sensor network, and particularly when the scale of the sensing data is large, the network performance is seriously affected.
Disclosure of Invention
The invention aims to provide an efficient and reliable WSN sensing data distributed decoding method based on MapReduce. Meanwhile, the cluster has strong computing power, so that efficient decoding of large-scale perception data can be realized.
The technical solution for realizing the purpose of the invention is as follows: a WSN perception data distributed decoding method based on MapReduce comprises the following steps:
step 1, constructing a sparse matrix through a Reed-Solomon code in a WSN, coding sensing data, and sending the coded data to a Hadoop cluster through an external network, wherein the method comprises the following steps:
step 1.1, generating matrix M of RS coden×kSub-matrix N of left-hand multiplication Mk×kObtaining a coding matrix D by the inverse matrix of the coding matrix D;
step 1.2, according to the multicast capacity k of the WSN network, k original data packets (F) are obtained1,F2,F3,…,Fk) Form an original data vector F ═ F1,F2,F3,…,Fk]T
Step 1.3, encoding an original data vector F by using a target sparse matrix D;
step 1.4, the WSN sends data to a Hadoop cluster through an external network;
step 2, the Hadoop cluster acquires the coding vectors to form a coding matrix, acquires the coded data to form a vector to be decoded, and comprises the following steps:
step 2.1, the Hadoop cluster receives the coded data and forms a vector F' to be decoded;
step 2.2, the Hadoop cluster acquires the coded vector of the WSN sensing data and forms a coding matrix D' by the coded vector;
step 3, the Hadoop cluster utilizes a MapReduce framework to perform inverse operation on the coding matrix to obtain a decoding matrix, and the method comprises the following steps:
step 3.1, block processing is carried out on the coding matrix D' until the divided sub-matrix blocks can be subjected to rapid LU decomposition on a single machine;
step 3.2, LU decomposition is carried out on the minimum subblock, and an inverse matrix of L, an inverse matrix of U and P are returned;
step 3.3, LU decomposition is carried out on the father block of the current sub-block, and an inverse matrix of L, an inverse matrix of U and P are returned;
step 3.4, repeating the step 3.3 until the coding matrix D' is subjected to LU decomposition, and returning to the inverse matrix of L, the inverse matrix of U and P;
step 3.5, calculating an inverse matrix D' of the encoding matrix by using the returned inverse matrix of L, the inverse matrix of U and P, namely a decoding matrix;
and 4, using a MapReduce frame by the Hadoop cluster, and using the decoding matrix and the received coded data to perform decoding calculation to obtain original WSN sensing data, wherein the method comprises the following steps:
step 4.1, storing the decoding matrix and the matrix to be decoded by using the HDFS, wherein the storage format is as follows: matrix row number, matrix column number, element value, and the element value in the matrix is 0;
step 4.2, the elements of the decoding matrix stored in the HDFS are stored as required into a key-value pair key ═ (i, k), k ═ 1, value ═ d' ", j, d ″", andi，j)；
step 4.2, the elements of the matrix to be decoded stored in the HDFS are stored as required into a key-value pair key ═ (i,1), i ═ 1,2, …, k, value ═ (' F ' ", i, F ″ ', andi×1)；
step 4.3, corresponding value in the same key is enteredAccumulating and summing after line multiplication to obtain original data Fi×1(if the value corresponding to the value is empty, not calculating).
Compared with the prior art, the invention has the following remarkable advantages: (1) the reliability and the energy utilization rate of the WSN can be improved by constructing a sparse matrix to encode the sensing data; (2) the decoding work of the coded data is placed in a Hadoop cluster, so that the calculation load of a WSN (wireless sensor network) can be reduced; (3) the Hadoop cluster has strong computing power and can deal with decoding work of large-scale data sets, so that the decoding efficiency is improved.
Drawings
FIG. 1 is a schematic diagram of the overall coordination framework of the present invention.
FIG. 2 is a flowchart of a WSN-aware data distributed decoding method based on MapReduce.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples.
With reference to fig. 2, the method for WSN-aware data distributed decoding based on MapReduce of the present invention includes the following steps:
step 1, constructing a sparse matrix through a Reed-Solomon code in a WSN, coding sensing data, and sending the coded data to a Hadoop cluster through an external network, wherein the method comprises the following steps:
step 1.1, generating matrix M of RS coden×kSub-matrix N of left-hand multiplication Mk×kObtaining a sparse matrix D by using the inverse matrix of the first matrix;
let M be an n × k RS code generation matrix, and since the RS code has MDS attributes, this matrix M also has MDS attributes. N is a k × k sub-matrix of the matrix M, so the target sparse matrix D can be obtained by the following transformation:
step 1.2, according to the multicast capacity k of the WSN network, k original data packets (F) are obtained1,F2,F3,…,Fk) Form an original data vector F ═ F1,F2,F3,…,Fk]T；
Step 1.3, encoding an original data vector F by using a target sparse matrix D;
step 2, the Hadoop cluster acquires the coding vectors to form a coding matrix, acquires the coded data to form a vector to be decoded, and comprises the following steps:
step 2.1, the Hadoop cluster receives the coded data, forms the coded data into a vector to be decoded, and sets the vector to be decoded as F ', wherein F' is expressed as follows:
F″＝[F1 F2 … Fk-2 C1 C2]T；
step 2.2, the Hadoop cluster acquires the coded vectors of the coded WSN sensing data, and the coded vectors form a coding matrix which is set as D ', and the assumption that D' is as follows:
step 3, the Hadoop cluster utilizes a MapReduce framework to perform inverse operation on the coding matrix to obtain a decoding matrix, and the method comprises the following steps:
step 3.1, the coding matrix D' is processed in a partitioning mode until the partitioned submatrix blocks can be rapidly decomposed into LU on a single machine, and the partitioned submatrix blocks are divided into D1,1Thus, sub-blocks are denoted by S1, S2;
step 3.2, LU decomposition is carried out on the minimum subblock, and an inverse matrix of L, an inverse matrix of U and P are returned, wherein LU decomposition is carried out on S2;
step 3.3, LU decomposition is carried out on the father block of the current sub-block, and an inverse matrix of L, an inverse matrix of U and P are returned, wherein LU decomposition is carried out on S1;
step 3.4, repeating the step 3.3 until the coding matrix D' is subjected to LU decomposition, and returning to the inverse matrix of L, the inverse matrix of U and P;
step 3.5, calculating an inverse matrix D' of the encoding matrix by using the returned inverse matrix of L, the inverse matrix of U and P, namely a decoding matrix;
and 4, using a MapReduce frame by the Hadoop cluster, and using the decoding matrix and the received coded data to perform decoding calculation to obtain original WSN sensing data, wherein the method comprises the following steps:
step 4.1, storing the decoding matrix and the matrix to be decoded by using the HDFS, wherein the storage format is as follows: matrix row number, matrix column number, element value, and the element value in the matrix is 0;
step 4.2, the elements of the decoding matrix stored in the HDFS are stored as required into the key-value pair key ═ i, k ═ 1, value ═ (' matrix name ', j, corresponding element value), and if the decoding matrix D ″ is used, the key-value pair can be represented as key ═ i, k ═ 1, value ═ (' D ' ", j, D ″", where k is 1, and k is (', D "")i,j)；
Step 4.2, the elements of the matrix to be decoded stored in the HDFS are stored as required into the key-value pair key ═ i,1, i ═ 1,2, …, k, value ═ ('matrix name', i, corresponding element value), and if the matrix to be decoded is F ″, the key-value pair can be represented as key ═ i,1, i ═ 1,2, …, k, value ═ ('F' ", i, F ″, and soi×1)；
Step 4.3, multiplying the corresponding values in the same key and then accumulating and summing to obtain original data Fi×1(if the value corresponding to the value is empty, not calculating).
In summary, the encoding data distributed decoding method based on MapReduce of the present invention performs data decoding in a Hadoop cluster, so as to reduce the load of the wireless sensor network. And parallelization of a perceptual data decoding algorithm is realized by using MapReduce so as to improve the problem of sink delay caused by perceptual data decoding. The network reliability and the energy utilization rate of the WSN are improved through a network coding technology, the resource advantages of the Hadoop cluster are fully utilized, and technical support is provided for realizing efficient and rapid decoding of coded data.
Claims (1)
1. A WSN perception data distributed decoding method based on MapReduce is characterized by comprising the following steps:
step 1, constructing a sparse matrix through a Reed-Solomon code in a WSN, coding sensing data, and sending the coded data to a Hadoop cluster through an external network;
step 1, constructing a sparse matrix through a Reed-Solomon code in the WSN, coding sensing data, and sending the coded data to a Hadoop cluster through an external network, wherein the method specifically comprises the following steps:
step 1.1, generating matrix M of RS coden×kSub-matrix N of left-hand multiplication Mk×kObtaining a coding matrix D by the inverse matrix of the coding matrix D;
step 1.2, according to the multicast capacity k of the WSN network, k original data packets (F) are obtained1,F2,F3,…,Fk) Form an original data vector F ═ F1,F2,F3,…,Fk]T；
Step 1.3, encoding an original data vector F by using a target sparse matrix D;
step 1.4, the WSN sends the encoded data to a Hadoop cluster through an external network;
step 2, the Hadoop cluster acquires coding vectors to form a coding matrix, and acquires coded data to form vectors to be decoded; the specific method in the step 2 comprises the following steps:
step 2.1, the Hadoop cluster receives the coded data and forms the coded data into a vector F to be decoded;
step 2.2, the Hadoop cluster acquires the coded vector of the WSN sensing data and forms a coding matrix D' by the coded vector;
step 3, the Hadoop cluster utilizes a MapReduce framework to perform inverse operation on the coding matrix to obtain a decoding matrix;
the method in step 3 specifically comprises the following steps:
step 3.1, block processing is carried out on the coding matrix D' until the divided sub-matrix blocks can be subjected to rapid LU decomposition on a single machine;
step 3.2, LU decomposition is carried out on the minimum subblock, and an inverse matrix of L, an inverse matrix of U and P are returned;
step 3.3, LU decomposition is carried out on the father block of the current sub-block, and an inverse matrix of L, an inverse matrix of U and P are returned;
step 3.4, repeating the step 3.3 until the coding matrix D' is subjected to LU decomposition, and returning to the inverse matrix of L, the inverse matrix of U and P;
step 3.5, calculating an inverse matrix D' of the encoding matrix by using the returned inverse matrix of L, the inverse matrix of U and P, namely a decoding matrix;
step 4, the Hadoop cluster performs decoding calculation by using the decoding matrix and the received coded data to obtain original WSN sensing data;
and 4, the Hadoop cluster performs decoding calculation by using the decoding matrix and the received encoded data to obtain original WSN sensing data, and the method specifically comprises the following steps:
step 4.1, storing the decoding matrix and the matrix to be decoded by using the HDFS, wherein the storage format is as follows: matrix row number, matrix column number, element value, and the element value in the matrix is 0;
step 4.2, the elements of the decoding matrix stored in the HDFS are stored as required into a key-value pair key ═ (i, k), k ═ 1, value ═ d ('d' ″, j, d ″).i,j)；
Step 4.2, the elements of the matrix to be decoded stored in the HDFS are stored as required into key-value pairs key ═ (i,1), i ═ 1,2, …, k, value ═ ('F' ", i, F ″)"i×1)；
Step 4.3, multiplying the corresponding values in the same key and then accumulating and summing to obtain original data Fi×1(ii) a If the value corresponding to the value is empty, the calculation is not carried out.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611264864.XA CN107070590B (en) | 2016-12-30 | 2016-12-30 | WSN perception data distributed decoding method based on MapReduce |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611264864.XA CN107070590B (en) | 2016-12-30 | 2016-12-30 | WSN perception data distributed decoding method based on MapReduce |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107070590A CN107070590A (en) | 2017-08-18 |
| CN107070590B true CN107070590B (en) | 2020-12-29 |
Family
ID=59624386
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611264864.XA Active CN107070590B (en) | 2016-12-30 | 2016-12-30 | WSN perception data distributed decoding method based on MapReduce |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107070590B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108092743B (en) * | 2017-12-15 | 2021-12-03 | 南京海道普数据技术有限公司 | Cluster-based network coded data packet decoding method and system for sensor network system |
| CN111610938B (en) * | 2020-05-29 | 2022-07-05 | 广东奥飞数据科技股份有限公司 | Distributed data code storage method, electronic device and computer readable storage medium |
| CN112532252B (en) * | 2020-11-24 | 2024-04-02 | 深圳市大数据研究院 | Encoding method, decoding method, electronic device, and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101840377A (en) * | 2010-05-13 | 2010-09-22 | 上海交通大学 | Data storage method based on RS (Reed-Solomon) erasure codes |
| CN104052576A (en) * | 2014-06-07 | 2014-09-17 | 华中科技大学 | Data recovery method based on error correcting codes in cloud storage |
| CN105635252A (en) * | 2015-12-23 | 2016-06-01 | 浪潮集团有限公司 | Erasure code redundant backup strategy of Hadoop distributed file system (HDFS) |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR102029285B1 (en) * | 2013-04-01 | 2019-10-07 | 한국전자통신연구원 | System and method for big data aggregaton in sensor network |
| CN103346864B (en) * | 2013-07-05 | 2017-04-12 | 哈尔滨工业大学深圳研究生院 | Data processing method and system suitable for wireless distributed perception system |
| CN105930384A (en) * | 2016-04-14 | 2016-09-07 | 南京理工大学 | Sensing cloud data storage system based on Hadoop system and implementation method thereof |
-
2016
- 2016-12-30 CN CN201611264864.XA patent/CN107070590B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101840377A (en) * | 2010-05-13 | 2010-09-22 | 上海交通大学 | Data storage method based on RS (Reed-Solomon) erasure codes |
| CN104052576A (en) * | 2014-06-07 | 2014-09-17 | 华中科技大学 | Data recovery method based on error correcting codes in cloud storage |
| CN105635252A (en) * | 2015-12-23 | 2016-06-01 | 浪潮集团有限公司 | Erasure code redundant backup strategy of Hadoop distributed file system (HDFS) |
Non-Patent Citations (2)
| Title |
|---|
| "基于Hadoop的海量传感数据管理系统";成飞龙;《中国优秀硕士学位论文全文数据库 信息科技辑》;20130516;第1-54页 * |
| "基于HDFS的优化数据冗余策略的研究";付园;《中国优秀硕士学位论文全文数据库 信息科技辑》;20140916;第1-42页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107070590A (en) | 2017-08-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107070590B (en) | WSN perception data distributed decoding method based on MapReduce | |
| CN102572435B (en) | Compressive sampling-based (CS-based) video coding/decoding system and method thereof | |
| WO2014154162A1 (en) | Channel encoding and decoding method and device | |
| US20180351693A1 (en) | Decoding method and apparatus in wireless communication system | |
| CN101350827B (en) | Method for compressing wavelet progressive data of wireless sensor network | |
| CN112449009B (en) | SVD-based communication compression method and device for Federal learning recommendation system | |
| CN105515590A (en) | Successive cancellation list polarization code decoding algorithm with effective low complexity based on random binary data flows and decoding structural frame thereof | |
| CN100414841C (en) | High-speed coding method of low density check code | |
| CN103546161A (en) | Lossless compression method based on binary processing | |
| CN102857760B (en) | Feedback-free code rate optimization distributed video encoding and decoding method and system | |
| Yue et al. | Communication-efficient federated learning via predictive coding | |
| CN109361492B (en) | High-performance decoding method combining physical layer network coding and polarization code | |
| CN103327530B (en) | A kind of data transmission method in wireless sensor network | |
| CN103368586A (en) | Deep space exploration multimedia service-oriented independent window unequal protective fountain coding method | |
| CN101431336B (en) | Search circuit and search method for decoding unit of low-density parity check code | |
| CN100557983C (en) | A kind of quasi-cyclic low-density parity check codes encoder and check digit generation method | |
| CN103001648B (en) | Based on the simple coding device and method of the quasi-cyclic LDPC code of FPGA | |
| CN112118074B (en) | Communication method and device | |
| CN105578183A (en) | Compression sensing video encoding and decoding method based on Gaussian mixture model (GMM) | |
| WO2020000490A1 (en) | Method and device for decoding polar code | |
| CN105791866A (en) | Video coding intermediate data obtaining method, device and system | |
| CN104301729A (en) | Code rate control method for non-feedback distributed video coding | |
| WO2023226689A1 (en) | Encoding method and apparatus, and decoding method and apparatus | |
| Akalu et al. | Design and performance analysis of energy efficient technique for wireless multimedia sensor networks using machine learning algorithm | |
| JP2013093672A (en) | Decoding apparatus, encoding/decoding system, encoding/decoding method and decoding program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |