CN110990489A - System and method for batch writing of sensing data into blocks - Google Patents
System and method for batch writing of sensing data into blocks Download PDFInfo
- Publication number
- CN110990489A CN110990489A CN201911273538.9A CN201911273538A CN110990489A CN 110990489 A CN110990489 A CN 110990489A CN 201911273538 A CN201911273538 A CN 201911273538A CN 110990489 A CN110990489 A CN 110990489A
- Authority
- CN
- China
- Prior art keywords
- data
- sensing
- sensing data
- cache queue
- queue
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 68
- 238000012217 deletion Methods 0.000 claims description 3
- 230000037430 deletion Effects 0.000 claims description 3
- 238000005516 engineering process Methods 0.000 description 2
- 238000007726 management method Methods 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 238000012550 audit Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000013479 data entry Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000000737 periodic effect Effects 0.000 description 1
- 230000000717 retained effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/50—Queue scheduling
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a system and a method for batch writing of sensing data into a block, which aim to improve the speed of writing data into the block, improve the efficiency and reduce the concurrency and the load of a block chain network during data writing. Specifically, the method includes a received data execution process and a timer execution process, wherein the received data execution process is a data batch writing process executed when the sensing data is received, and the timer execution process is a data batch writing process periodically executed at fixed time intervals.
Description
Technical Field
The invention relates to the technical field of Internet of things and block chains, in particular to a system and a method for writing sensing data into blocks in batches.
Background
The Internet of things has wide application in smart cities, intelligent transportation, industrial monitoring, food traceability and other fields, and is rapidly developed in recent years. The Internet of things connects any article with a network according to an agreed protocol through information sensing equipment such as radio frequency identification, an infrared sensor, a global positioning system, a laser scanner, a gas sensor and the like, and performs information exchange and communication, so that a network for intelligent identification, positioning, tracking, monitoring and management is realized.
Blockchains are a decentralized, distributed database technique that is participated in by nodes. The information recorded in the block chain network is recorded by all verification nodes together without any centralized node audit, and the recorded information has the characteristics of being incapable of being forged and being falsified. The internet of things and the block chain technology can realize effective combination of a physical world and an information world, and realize distributed management, credibility and traceability of information.
One feature of the internet of things is the wide application of various sensing technologies. Various types of sensors are deployed on the Internet of things, and each sensor is an information source. The sensor collects environmental information according to a certain frequency period and continuously updates data. The data generation speed is high, the concurrency is high, the storage requirement on the data is high, and the information writing performance of the current block chain cannot completely meet the requirement.
Therefore, in the face of massive internet-of-things sensing data, the writing efficiency of a block chain is low, the expansion is complex, the technical requirement is high, and a massive data fast writing method which is convenient to expand is urgently needed.
Disclosure of Invention
Aiming at the defects in the prior art, the invention provides a system and a method for sensing data batch writing in a block, which are used for improving the speed of data writing in the block, improving the efficiency and reducing the concurrency and the load of a block chain network when data is written.
Specifically, the invention provides a system and a method for sensing data batch writing blocks, wherein the method comprises a receiving data execution process and a timer execution process, the receiving data execution process is a data batch writing process executed when the sensing data is received, and the timer execution process is a data batch writing process periodically executed at fixed time intervals.
A system and a method for batch writing of sensing data into a block comprise a sensor and a data receiving server, wherein the data receiving server is used for executing a data receiving execution process, and the steps are as follows:
s1, establishing a buffer queue, and setting a threshold value of the number of the buffer queues;
s2, the data receiving server receives the sensing data collected by the sensor;
s3, setting a data locking semaphore of the buffer queue, and locking the data of the buffer queue;
s4, adding the sensing data into the buffer queue;
s5, judging whether the quantity of the sensing data in the cache queue meets the quantity threshold, if the quantity exceeds the quantity threshold, entering a step S6, otherwise, resetting the data locking semaphore of the cache queue, unlocking the data of the cache queue, and entering a step S2;
s6, generating batch sensing data according to the sensing data in the cache queue, sending the batch sensing data to a block chain network, emptying the sensing data in the cache queue, resetting the data locking semaphore of the cache queue, and unlocking the data in the cache queue.
Further, when the number threshold is not exceeded in step S5, the sensing data in the buffer queue is retained.
A system and a method for writing sensing data into a block in batch comprise a sensor, a data receiving server and a timer, wherein the data receiving server performs a periodic data receiving execution process through the timer after receiving the sensing data acquired by the sensor, and the method comprises the following steps:
s1, establishing a buffer queue, wherein the buffer queue is used for temporarily storing the sensing data;
s2, establishing an execution cycle through the timer, and executing the following steps only once in each execution cycle;
s3, setting a data locking semaphore of the buffer queue, and locking the data of the buffer queue;
s4, judging whether the sensing data exist in the cache queue, if so, entering the step S5, otherwise, resetting the data locking semaphore of the cache queue, unlocking the cache queue data, and ending the execution cycle;
s5, generating batch sensing data according to the sensing data in the cache queue, sending the batch sensing data to a block chain network, emptying the sensing data in the cache queue, resetting the data locking semaphore of the cache queue, and unlocking the data in the cache queue.
The steps of establishing a cache queue and adding the sensing data can be carried out orderly or unorderly; the quantity threshold may also be customized by the data receiving server itself.
Further, the batch of sensing data includes a merged attribute and an attribute list, the merged attribute is composed of the same attributes of the sensing data in the cache queue, and the attribute list is composed of the remaining attributes of the sensing data in the cache queue except for the merged attribute. Each record in the attribute list corresponds to one piece of sensing data.
Further, the system and method for writing the sensing data into the block in batch further comprise a block chain account book, and the block chain network writes the batch sensing data into a block structure of the block chain account book through a consensus process.
Further, after the data of the buffer queue is locked, the buffer queue only performs data addition or data deletion at the data locked position. And in addition to the operation processes of data addition or data deletion, all other operation processes of the buffer queue are only executed after the data locking semaphore is reset.
Furthermore, the sensing data in the buffer queue are arranged according to the sequence of entering the buffer queue.
Further, the data receiving server is a cloud server or an internet of things server; hardware servers may also be selected.
Further, the sensor includes a temperature sensor.
Further, the sensing data collected by the temperature sensor includes: time, area, temperature, and device number.
The invention has the beneficial effects that:
according to the technical scheme, the received sensing data is firstly added into the cache queue, then a plurality of pieces of sensing data in the cache queue are organized into sensing batch data, finally the sensing batch data are submitted to the block chain network, and the batch writing of the sensing data is realized through a consensus process. Compared with the prior art, the method has the obvious advantages that the blocks are written in batch after the sensing data are cached, so that frequent write-in operation and a consensus process when the data are written one by one are avoided, the write-in concurrency and the load of a block chain network are reduced, and the data write-in efficiency is improved.
Drawings
In order to more clearly illustrate the detailed description of the invention or the technical solutions in the prior art, the drawings that are needed in the detailed description of the invention or the prior art will be briefly described below. Throughout the drawings, like elements or portions are generally identified by like reference numerals. In the drawings, elements or portions are not necessarily drawn to scale.
FIG. 1 is a block flow diagram of an implementation of the receive data process of the present invention;
FIG. 2 is a flow chart of the timer execution process of the present invention.
Detailed Description
Embodiments of the present invention will be described in detail below with reference to the accompanying drawings. The following examples are only for illustrating the technical solutions of the present invention more clearly, and therefore are only examples, and the protection scope of the present invention is not limited thereby.
It is to be noted that, unless otherwise specified, technical or scientific terms used herein shall have the ordinary meaning as understood by those skilled in the art to which the invention pertains.
Example 1
The invention provides a system and a method for sensing data batch writing block, wherein the method comprises a receiving data execution process and a timer execution process, the receiving data execution process is a data batch writing process executed when sensing data is received, and the timer execution process is a data batch writing process periodically executed at fixed time intervals.
Fig. 1 is a flow chart of an execution process of receiving data in a system and a method for sensing a batch write of data into a block according to an embodiment of the present invention. The method specifically comprises the following basic steps:
establishing a buffer queue, and setting a number threshold of the buffer queue, wherein the step is generally completed by a data receiving server in advance.
ID | Date | Time | Region | DeviceID | Temp |
10003 | Date1 | Time3 | Region1 | Device1 | Temp3 |
TABLE 1
ID | Date | Time | Region | DeviceID | Temp |
10001 | Date1 | Time1 | Region1 | Device1 | Temp1 |
10002 | Date1 | Time2 | Region1 | Device1 | Temp2 |
10003 | Date1 | Time3 | Region1 | Device1 | Temp3 |
TABLE 2
And 105, generating batch sensing data according to the sensing data in the cache queue and submitting the batch sensing data to the blockchain network, writing the batch sensing data into a blockchain block structure of the blockchain account book through a consensus process by the blockchain network, and then emptying the data in the cache queue. The batch of sensing data comprises a combined attribute part and an attribute list part, wherein the combined attribute part is formed by combining a plurality of pieces of sensing data with repeated attributes with the same value; the attribute list part is formed by removing the residual attributes of the combined attributes of the sensing data, and each record in the attribute list corresponds to one piece of sensing data.
The merged attributes section in this embodiment includes two attributes, namely, Region and DeviceID, and the attributes of the attribute list section include ID, Date, Time, and Temp. The merged attribute part of the batch sensing data is Region1, Device1, and the attribute list part is shown in table 3.
ID | Date | Time | Temp |
10001 | Date1 | Time1 | Temp1 |
10002 | Date1 | Time2 | Temp2 |
10003 | Date1 | Time3 | Temp3 |
TABLE 3
And 106, resetting the data locking semaphore of the buffer queue, and unlocking the data of the buffer queue.
Example 2
The system and the method for batch writing of the sensing data into the block are based on the embodiment 1.
As shown in fig. 2, the timer execution process includes:
in step 201, execution of a timer clock cycle begins, where the timer clock cycle refers to a cycle executed according to a given cycle timing. The timer execution process is performed once every one minute, assuming that the set timer is executed once every 1 minute.
ID | Date | Time | Region | DeviceID | Temp |
10003 | Date1 | Time3 | Region1 | Device1 | Temp3 |
TABLE 4
The merged attributes section in this embodiment includes two attributes, namely, Region and DeviceID, and the attributes of the attribute list section include ID, Date, Time, and Temp. The merged attribute part of the batch sensing data is Region1, Device1, and the attribute list part is shown in table 5.
ID | Date | Time | Temp |
10001 | Date1 | Time1 | Temp1 |
TABLE 5
And step 205, resetting the data locking semaphore of the buffer queue, and unlocking the data of the buffer queue.
Finally, it should be noted that: the above embodiments are only used to illustrate the technical solution of the present invention, and not to limit the same; while the invention has been described in detail and with reference to the foregoing embodiments, it will be understood by those skilled in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; such modifications and substitutions do not depart from the spirit and scope of the present invention, and they should be construed as being included in the following claims and description.
Claims (10)
1. A system and a method for writing sensing data into a block in batch are characterized by comprising a sensor and a data receiving server, wherein the data receiving server is used for executing a data receiving execution process, and the steps are as follows:
s1, establishing a buffer queue, and setting a threshold value of the number of the buffer queues;
s2, the data receiving server receives the sensing data collected by the sensor;
s3, setting a data locking semaphore of the buffer queue, and locking the data of the buffer queue;
s4, adding the sensing data into the buffer queue;
s5, judging whether the quantity of the sensing data in the cache queue meets the quantity threshold, if the quantity exceeds the quantity threshold, entering a step S6, otherwise, resetting the data locking semaphore of the cache queue, unlocking the data of the cache queue, and entering a step S2;
s6, generating batch sensing data according to the sensing data in the cache queue, sending the batch sensing data to a block chain network, emptying the sensing data in the cache queue, resetting the data locking semaphore of the cache queue, and unlocking the data in the cache queue.
2. The system and method for sensing the writing of data to a block in bulk as claimed in claim 1, wherein: if the number threshold is not exceeded in step S5, retaining the sensing data in the buffer queue.
3. A system and a method for writing sensing data into a block in batch are characterized by comprising a sensor, a data receiving server and a timer, wherein after the data receiving server receives the sensing data collected by the sensor, a receiving data execution process containing a period is carried out through the timer, and the method comprises the following steps:
s1, establishing a buffer queue, wherein the buffer queue is used for temporarily storing the sensing data;
s2, establishing an execution cycle through the timer, and executing the following steps only once in each execution cycle;
s3, setting a data locking semaphore of the buffer queue, and locking the data of the buffer queue;
s4, judging whether the sensing data exist in the cache queue, if so, entering the step S5, otherwise, resetting the data locking semaphore of the cache queue, unlocking the cache queue data, and ending the execution cycle;
s5, generating batch sensing data according to the sensing data in the cache queue, sending the batch sensing data to a block chain network, emptying the sensing data in the cache queue, resetting the data locking semaphore of the cache queue, and unlocking the data in the cache queue.
4. A system and method for sensing the writing of data into a block in bulk according to any of claims 1 to 3, wherein: the batch sensing data comprises a merged attribute and an attribute list, wherein the merged attribute is composed of the same attributes of the sensing data in the cache queue, and the attribute list is composed of the rest attributes of the sensing data in the cache queue except the merged attribute.
5. A system and method for sensing the writing of data into a block in bulk according to any of claims 1 to 3, wherein: the block chain network writes the batch sensing data into a block structure of the block chain account book through a consensus process.
6. A system and method for sensing the writing of data into a block in bulk according to any of claims 1 to 3, wherein: after the data of the cache queue is locked, the cache queue only performs data addition or data deletion at the data locked position.
7. A system and method for sensing the writing of data into a block in bulk according to any of claims 1 to 3, wherein: and arranging the sensing data in the cache queue according to the sequence of entering the cache queue.
8. A system and method for sensing the writing of data into a block in bulk according to any of claims 1 to 3, wherein: the data receiving server is a cloud server or an Internet of things server.
9. A system and method for sensing the writing of data into a block in bulk according to any of claims 1 to 3, wherein: the sensor comprises a temperature sensor.
10. The system and method of claim 9, wherein the system comprises: the sensing data collected by the temperature sensor includes: time, area, temperature, and device number.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201911273538.9A CN110990489A (en) | 2019-12-12 | 2019-12-12 | System and method for batch writing of sensing data into blocks |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201911273538.9A CN110990489A (en) | 2019-12-12 | 2019-12-12 | System and method for batch writing of sensing data into blocks |
Publications (1)
Publication Number | Publication Date |
---|---|
CN110990489A true CN110990489A (en) | 2020-04-10 |
Family
ID=70092887
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201911273538.9A Pending CN110990489A (en) | 2019-12-12 | 2019-12-12 | System and method for batch writing of sensing data into blocks |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN110990489A (en) |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN105791439A (en) * | 2016-04-29 | 2016-07-20 | 中国农业银行股份有限公司 | Data processing method, data receiving apparatus and data processing system |
CN108805702A (en) * | 2018-04-19 | 2018-11-13 | 深圳市网心科技有限公司 | Transaction buffer/accelerated method based on block chain and block chain transaction processing system |
CN108876370A (en) * | 2018-06-12 | 2018-11-23 | 北京航空航天大学 | The architectural framework of the shared open data of transregional piece of chain under a kind of isomery multichain framework |
CN109636388A (en) * | 2018-12-07 | 2019-04-16 | 深圳市智税链科技有限公司 | Data processing method, device, medium and electronic equipment in block chain network |
CN110231992A (en) * | 2019-06-06 | 2019-09-13 | 深圳前海微众银行股份有限公司 | Distributed resource allocation method, device, equipment and readable storage medium storing program for executing |
-
2019
- 2019-12-12 CN CN201911273538.9A patent/CN110990489A/en active Pending
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN105791439A (en) * | 2016-04-29 | 2016-07-20 | 中国农业银行股份有限公司 | Data processing method, data receiving apparatus and data processing system |
CN108805702A (en) * | 2018-04-19 | 2018-11-13 | 深圳市网心科技有限公司 | Transaction buffer/accelerated method based on block chain and block chain transaction processing system |
CN108876370A (en) * | 2018-06-12 | 2018-11-23 | 北京航空航天大学 | The architectural framework of the shared open data of transregional piece of chain under a kind of isomery multichain framework |
CN109636388A (en) * | 2018-12-07 | 2019-04-16 | 深圳市智税链科技有限公司 | Data processing method, device, medium and electronic equipment in block chain network |
CN110231992A (en) * | 2019-06-06 | 2019-09-13 | 深圳前海微众银行股份有限公司 | Distributed resource allocation method, device, equipment and readable storage medium storing program for executing |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN109087090B (en) | Method and apparatus for tracking targets using trusted account book | |
US11176110B2 (en) | Data updating method and device for a distributed database system | |
CN109918229B (en) | Database cluster copy construction method and device in non-log mode | |
Wang et al. | A near-optimal protocol for the grouping problem in RFID systems | |
WO2014021978A4 (en) | Aggregating data in a mediation system | |
CN111066040A (en) | Inventory management system | |
CN102782679A (en) | Event information management system, event management method and program | |
CN105227662A (en) | Message treatment method, device and system | |
JP4848128B2 (en) | Database system, computer-executable method for database system, program and method for updating index table in database system | |
CN110990489A (en) | System and method for batch writing of sensing data into blocks | |
Fenga et al. | COVID‐19: Metaheuristic Optimization‐Based Forecast Method on Time‐Dependent Bootstrapped Data | |
CN107295059A (en) | The statistical system and method for service propelling amount | |
CN104750849A (en) | Method and system for maintaining tree structure-based directory relationship | |
CN108647243B (en) | Industrial big data storage method based on time series | |
CN105512313A (en) | Incremental data processing method and device | |
CN104700055A (en) | Method for detecting complex events on multi-probability RFID event flows | |
CN111078425B (en) | Message processing method and device, storage medium and electronic equipment | |
JP5308927B2 (en) | Device information management apparatus and method | |
CN110532108B (en) | Resume delivery task processing method, device, server and system | |
CN100357969C (en) | Automatic tracking system of goods and its method | |
US10715619B2 (en) | Cache management using a probabilistic data structure | |
CN111640027A (en) | Service data processing method, service data processing device, service processing device and electronic equipment | |
CN106649444B (en) | Method and device for CAE database system to acquire CAD data | |
Liu et al. | On fast and reliable missing event detection protocol for multitagged RFID systems | |
JP2021068365A5 (en) |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
RJ01 | Rejection of invention patent application after publication | ||
RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200410 |