CN110990489A - System and method for batch writing of sensing data into blocks - Google Patents

System and method for batch writing of sensing data into blocks Download PDF

Info

Publication number
CN110990489A
CN110990489A CN201911273538.9A CN201911273538A CN110990489A CN 110990489 A CN110990489 A CN 110990489A CN 201911273538 A CN201911273538 A CN 201911273538A CN 110990489 A CN110990489 A CN 110990489A
Authority
CN
China
Prior art keywords
data
sensing
sensing data
cache queue
queue
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201911273538.9A
Other languages
Chinese (zh)
Inventor
张重阳
林宇飞
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nanjing Ronglian Technology Co ltd
Original Assignee
Nanjing Ronglian Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nanjing Ronglian Technology Co ltd filed Critical Nanjing Ronglian Technology Co ltd
Priority to CN201911273538.9A priority Critical patent/CN110990489A/en
Publication of CN110990489A publication Critical patent/CN110990489A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G06F16/2455—Query execution
    • G06F16/24552—Database cache management
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L47/00—Traffic control in data switching networks
    • H04L47/50—Queue scheduling

Landscapes

  • Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computational Linguistics (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Computing Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention discloses a system and a method for batch writing of sensing data into a block, which aim to improve the speed of writing data into the block, improve the efficiency and reduce the concurrency and the load of a block chain network during data writing. Specifically, the method includes a received data execution process and a timer execution process, wherein the received data execution process is a data batch writing process executed when the sensing data is received, and the timer execution process is a data batch writing process periodically executed at fixed time intervals.

Description

System and method for batch writing of sensing data into blocks
Technical Field
The invention relates to the technical field of Internet of things and block chains, in particular to a system and a method for writing sensing data into blocks in batches.
Background
The Internet of things has wide application in smart cities, intelligent transportation, industrial monitoring, food traceability and other fields, and is rapidly developed in recent years. The Internet of things connects any article with a network according to an agreed protocol through information sensing equipment such as radio frequency identification, an infrared sensor, a global positioning system, a laser scanner, a gas sensor and the like, and performs information exchange and communication, so that a network for intelligent identification, positioning, tracking, monitoring and management is realized.
Blockchains are a decentralized, distributed database technique that is participated in by nodes. The information recorded in the block chain network is recorded by all verification nodes together without any centralized node audit, and the recorded information has the characteristics of being incapable of being forged and being falsified. The internet of things and the block chain technology can realize effective combination of a physical world and an information world, and realize distributed management, credibility and traceability of information.
One feature of the internet of things is the wide application of various sensing technologies. Various types of sensors are deployed on the Internet of things, and each sensor is an information source. The sensor collects environmental information according to a certain frequency period and continuously updates data. The data generation speed is high, the concurrency is high, the storage requirement on the data is high, and the information writing performance of the current block chain cannot completely meet the requirement.
Therefore, in the face of massive internet-of-things sensing data, the writing efficiency of a block chain is low, the expansion is complex, the technical requirement is high, and a massive data fast writing method which is convenient to expand is urgently needed.
Disclosure of Invention
Aiming at the defects in the prior art, the invention provides a system and a method for sensing data batch writing in a block, which are used for improving the speed of data writing in the block, improving the efficiency and reducing the concurrency and the load of a block chain network when data is written.
Specifically, the invention provides a system and a method for sensing data batch writing blocks, wherein the method comprises a receiving data execution process and a timer execution process, the receiving data execution process is a data batch writing process executed when the sensing data is received, and the timer execution process is a data batch writing process periodically executed at fixed time intervals.
A system and a method for batch writing of sensing data into a block comprise a sensor and a data receiving server, wherein the data receiving server is used for executing a data receiving execution process, and the steps are as follows:
s1, establishing a buffer queue, and setting a threshold value of the number of the buffer queues;
s2, the data receiving server receives the sensing data collected by the sensor;
s3, setting a data locking semaphore of the buffer queue, and locking the data of the buffer queue;
s4, adding the sensing data into the buffer queue;
s5, judging whether the quantity of the sensing data in the cache queue meets the quantity threshold, if the quantity exceeds the quantity threshold, entering a step S6, otherwise, resetting the data locking semaphore of the cache queue, unlocking the data of the cache queue, and entering a step S2;
s6, generating batch sensing data according to the sensing data in the cache queue, sending the batch sensing data to a block chain network, emptying the sensing data in the cache queue, resetting the data locking semaphore of the cache queue, and unlocking the data in the cache queue.
Further, when the number threshold is not exceeded in step S5, the sensing data in the buffer queue is retained.
A system and a method for writing sensing data into a block in batch comprise a sensor, a data receiving server and a timer, wherein the data receiving server performs a periodic data receiving execution process through the timer after receiving the sensing data acquired by the sensor, and the method comprises the following steps:
s1, establishing a buffer queue, wherein the buffer queue is used for temporarily storing the sensing data;
s2, establishing an execution cycle through the timer, and executing the following steps only once in each execution cycle;
s3, setting a data locking semaphore of the buffer queue, and locking the data of the buffer queue;
s4, judging whether the sensing data exist in the cache queue, if so, entering the step S5, otherwise, resetting the data locking semaphore of the cache queue, unlocking the cache queue data, and ending the execution cycle;
s5, generating batch sensing data according to the sensing data in the cache queue, sending the batch sensing data to a block chain network, emptying the sensing data in the cache queue, resetting the data locking semaphore of the cache queue, and unlocking the data in the cache queue.
The steps of establishing a cache queue and adding the sensing data can be carried out orderly or unorderly; the quantity threshold may also be customized by the data receiving server itself.
Further, the batch of sensing data includes a merged attribute and an attribute list, the merged attribute is composed of the same attributes of the sensing data in the cache queue, and the attribute list is composed of the remaining attributes of the sensing data in the cache queue except for the merged attribute. Each record in the attribute list corresponds to one piece of sensing data.
Further, the system and method for writing the sensing data into the block in batch further comprise a block chain account book, and the block chain network writes the batch sensing data into a block structure of the block chain account book through a consensus process.
Further, after the data of the buffer queue is locked, the buffer queue only performs data addition or data deletion at the data locked position. And in addition to the operation processes of data addition or data deletion, all other operation processes of the buffer queue are only executed after the data locking semaphore is reset.
Furthermore, the sensing data in the buffer queue are arranged according to the sequence of entering the buffer queue.
Further, the data receiving server is a cloud server or an internet of things server; hardware servers may also be selected.
Further, the sensor includes a temperature sensor.
Further, the sensing data collected by the temperature sensor includes: time, area, temperature, and device number.
The invention has the beneficial effects that:
according to the technical scheme, the received sensing data is firstly added into the cache queue, then a plurality of pieces of sensing data in the cache queue are organized into sensing batch data, finally the sensing batch data are submitted to the block chain network, and the batch writing of the sensing data is realized through a consensus process. Compared with the prior art, the method has the obvious advantages that the blocks are written in batch after the sensing data are cached, so that frequent write-in operation and a consensus process when the data are written one by one are avoided, the write-in concurrency and the load of a block chain network are reduced, and the data write-in efficiency is improved.
Drawings
In order to more clearly illustrate the detailed description of the invention or the technical solutions in the prior art, the drawings that are needed in the detailed description of the invention or the prior art will be briefly described below. Throughout the drawings, like elements or portions are generally identified by like reference numerals. In the drawings, elements or portions are not necessarily drawn to scale.
FIG. 1 is a block flow diagram of an implementation of the receive data process of the present invention;
FIG. 2 is a flow chart of the timer execution process of the present invention.
Detailed Description
Embodiments of the present invention will be described in detail below with reference to the accompanying drawings. The following examples are only for illustrating the technical solutions of the present invention more clearly, and therefore are only examples, and the protection scope of the present invention is not limited thereby.
It is to be noted that, unless otherwise specified, technical or scientific terms used herein shall have the ordinary meaning as understood by those skilled in the art to which the invention pertains.
Example 1
The invention provides a system and a method for sensing data batch writing block, wherein the method comprises a receiving data execution process and a timer execution process, the receiving data execution process is a data batch writing process executed when sensing data is received, and the timer execution process is a data batch writing process periodically executed at fixed time intervals.
Fig. 1 is a flow chart of an execution process of receiving data in a system and a method for sensing a batch write of data into a block according to an embodiment of the present invention. The method specifically comprises the following basic steps:
establishing a buffer queue, and setting a number threshold of the buffer queue, wherein the step is generally completed by a data receiving server in advance.
Step 101, a data receiving server receives sensing data collected by a sensor. For example, in power equipment detection, temperature sensors are deployed at equipment nodes and lines for periodically collecting temperature data. The currently received sensing data is shown in table 1, and includes collected data including Time (Time), Region, DeviceID (device number), Temp (temperature), where Time includes Date and Time. The ID in the data is a unique key of the data entry, i.e., uniquely identifies a piece of data.
ID Date Time Region DeviceID Temp
10003 Date1 Time3 Region1 Device1 Temp3
TABLE 1
Step 102, setting a data locking semaphore of a buffer queue, marking the buffer queue as possible data in the queue to be added or deleted, and continuing to execute other processes related to buffer queue operation after the semaphore is reset again, wherein the buffer queue is used for temporarily storing the received sensing data, and the data in the queue are arranged according to the sequence of entering the queue.
Step 103, adding the received sensing data into a buffer queue. For example, two pieces of sensing data with original IDs 10001 and 10002 in the buffer queue, and the data obtained by adding the received data with ID 10003 to the buffer list is shown in table 2.
ID Date Time Region DeviceID Temp
10001 Date1 Time1 Region1 Device1 Temp1
10002 Date1 Time2 Region1 Device1 Temp2
10003 Date1 Time3 Region1 Device1 Temp3
TABLE 2
Step 104, judging the quantity of the sensing data in the buffer queue, and if the quantity exceeds a given threshold value, starting to execute step 105; otherwise, step 106 is performed. Assuming that the given threshold is 2, since there are 3 pieces of sensing data in the current queue, the given threshold is exceeded, and thus step 105 is performed.
And 105, generating batch sensing data according to the sensing data in the cache queue and submitting the batch sensing data to the blockchain network, writing the batch sensing data into a blockchain block structure of the blockchain account book through a consensus process by the blockchain network, and then emptying the data in the cache queue. The batch of sensing data comprises a combined attribute part and an attribute list part, wherein the combined attribute part is formed by combining a plurality of pieces of sensing data with repeated attributes with the same value; the attribute list part is formed by removing the residual attributes of the combined attributes of the sensing data, and each record in the attribute list corresponds to one piece of sensing data.
The merged attributes section in this embodiment includes two attributes, namely, Region and DeviceID, and the attributes of the attribute list section include ID, Date, Time, and Temp. The merged attribute part of the batch sensing data is Region1, Device1, and the attribute list part is shown in table 3.
ID Date Time Temp
10001 Date1 Time1 Temp1
10002 Date1 Time2 Temp2
10003 Date1 Time3 Temp3
TABLE 3
And 106, resetting the data locking semaphore of the buffer queue, and unlocking the data of the buffer queue.
Example 2
The system and the method for batch writing of the sensing data into the block are based on the embodiment 1.
As shown in fig. 2, the timer execution process includes:
in step 201, execution of a timer clock cycle begins, where the timer clock cycle refers to a cycle executed according to a given cycle timing. The timer execution process is performed once every one minute, assuming that the set timer is executed once every 1 minute.
Step 202, setting a data locking semaphore of a buffer queue, marking the buffer queue as possible data in the queue to be added or deleted, and continuing to execute other processes related to buffer queue operation after the semaphore is reset again, wherein the buffer queue is used for temporarily storing the received sensing data, and the data in the queue are arranged according to the sequence of entering the queue.
Step 203, judging whether sensing data exist in the buffer queue, if so, starting to execute step 204; otherwise step 205 is performed. Assume that there is a piece of sensed data in the buffer queue as shown in table 4.
ID Date Time Region DeviceID Temp
10003 Date1 Time3 Region1 Device1 Temp3
TABLE 4
Step 204, generating batch sensing data and submitting the batch sensing data to a block chain network, and emptying a buffer queue, wherein the step is the same as the step 105 of the data receiving execution process.
The merged attributes section in this embodiment includes two attributes, namely, Region and DeviceID, and the attributes of the attribute list section include ID, Date, Time, and Temp. The merged attribute part of the batch sensing data is Region1, Device1, and the attribute list part is shown in table 5.
ID Date Time Temp
10001 Date1 Time1 Temp1
TABLE 5
And step 205, resetting the data locking semaphore of the buffer queue, and unlocking the data of the buffer queue.
Finally, it should be noted that: the above embodiments are only used to illustrate the technical solution of the present invention, and not to limit the same; while the invention has been described in detail and with reference to the foregoing embodiments, it will be understood by those skilled in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; such modifications and substitutions do not depart from the spirit and scope of the present invention, and they should be construed as being included in the following claims and description.

Claims (10)

1. A system and a method for writing sensing data into a block in batch are characterized by comprising a sensor and a data receiving server, wherein the data receiving server is used for executing a data receiving execution process, and the steps are as follows:
s1, establishing a buffer queue, and setting a threshold value of the number of the buffer queues;
s2, the data receiving server receives the sensing data collected by the sensor;
s3, setting a data locking semaphore of the buffer queue, and locking the data of the buffer queue;
s4, adding the sensing data into the buffer queue;
s5, judging whether the quantity of the sensing data in the cache queue meets the quantity threshold, if the quantity exceeds the quantity threshold, entering a step S6, otherwise, resetting the data locking semaphore of the cache queue, unlocking the data of the cache queue, and entering a step S2;
s6, generating batch sensing data according to the sensing data in the cache queue, sending the batch sensing data to a block chain network, emptying the sensing data in the cache queue, resetting the data locking semaphore of the cache queue, and unlocking the data in the cache queue.
2. The system and method for sensing the writing of data to a block in bulk as claimed in claim 1, wherein: if the number threshold is not exceeded in step S5, retaining the sensing data in the buffer queue.
3. A system and a method for writing sensing data into a block in batch are characterized by comprising a sensor, a data receiving server and a timer, wherein after the data receiving server receives the sensing data collected by the sensor, a receiving data execution process containing a period is carried out through the timer, and the method comprises the following steps:
s1, establishing a buffer queue, wherein the buffer queue is used for temporarily storing the sensing data;
s2, establishing an execution cycle through the timer, and executing the following steps only once in each execution cycle;
s3, setting a data locking semaphore of the buffer queue, and locking the data of the buffer queue;
s4, judging whether the sensing data exist in the cache queue, if so, entering the step S5, otherwise, resetting the data locking semaphore of the cache queue, unlocking the cache queue data, and ending the execution cycle;
s5, generating batch sensing data according to the sensing data in the cache queue, sending the batch sensing data to a block chain network, emptying the sensing data in the cache queue, resetting the data locking semaphore of the cache queue, and unlocking the data in the cache queue.
4. A system and method for sensing the writing of data into a block in bulk according to any of claims 1 to 3, wherein: the batch sensing data comprises a merged attribute and an attribute list, wherein the merged attribute is composed of the same attributes of the sensing data in the cache queue, and the attribute list is composed of the rest attributes of the sensing data in the cache queue except the merged attribute.
5. A system and method for sensing the writing of data into a block in bulk according to any of claims 1 to 3, wherein: the block chain network writes the batch sensing data into a block structure of the block chain account book through a consensus process.
6. A system and method for sensing the writing of data into a block in bulk according to any of claims 1 to 3, wherein: after the data of the cache queue is locked, the cache queue only performs data addition or data deletion at the data locked position.
7. A system and method for sensing the writing of data into a block in bulk according to any of claims 1 to 3, wherein: and arranging the sensing data in the cache queue according to the sequence of entering the cache queue.
8. A system and method for sensing the writing of data into a block in bulk according to any of claims 1 to 3, wherein: the data receiving server is a cloud server or an Internet of things server.
9. A system and method for sensing the writing of data into a block in bulk according to any of claims 1 to 3, wherein: the sensor comprises a temperature sensor.
10. The system and method of claim 9, wherein the system comprises: the sensing data collected by the temperature sensor includes: time, area, temperature, and device number.
CN201911273538.9A 2019-12-12 2019-12-12 System and method for batch writing of sensing data into blocks Pending CN110990489A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201911273538.9A CN110990489A (en) 2019-12-12 2019-12-12 System and method for batch writing of sensing data into blocks

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201911273538.9A CN110990489A (en) 2019-12-12 2019-12-12 System and method for batch writing of sensing data into blocks

Publications (1)

Publication Number Publication Date
CN110990489A true CN110990489A (en) 2020-04-10

Family

ID=70092887

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201911273538.9A Pending CN110990489A (en) 2019-12-12 2019-12-12 System and method for batch writing of sensing data into blocks

Country Status (1)

Country Link
CN (1) CN110990489A (en)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105791439A (en) * 2016-04-29 2016-07-20 中国农业银行股份有限公司 Data processing method, data receiving apparatus and data processing system
CN108805702A (en) * 2018-04-19 2018-11-13 深圳市网心科技有限公司 Transaction buffer/accelerated method based on block chain and block chain transaction processing system
CN108876370A (en) * 2018-06-12 2018-11-23 北京航空航天大学 The architectural framework of the shared open data of transregional piece of chain under a kind of isomery multichain framework
CN109636388A (en) * 2018-12-07 2019-04-16 深圳市智税链科技有限公司 Data processing method, device, medium and electronic equipment in block chain network
CN110231992A (en) * 2019-06-06 2019-09-13 深圳前海微众银行股份有限公司 Distributed resource allocation method, device, equipment and readable storage medium storing program for executing

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105791439A (en) * 2016-04-29 2016-07-20 中国农业银行股份有限公司 Data processing method, data receiving apparatus and data processing system
CN108805702A (en) * 2018-04-19 2018-11-13 深圳市网心科技有限公司 Transaction buffer/accelerated method based on block chain and block chain transaction processing system
CN108876370A (en) * 2018-06-12 2018-11-23 北京航空航天大学 The architectural framework of the shared open data of transregional piece of chain under a kind of isomery multichain framework
CN109636388A (en) * 2018-12-07 2019-04-16 深圳市智税链科技有限公司 Data processing method, device, medium and electronic equipment in block chain network
CN110231992A (en) * 2019-06-06 2019-09-13 深圳前海微众银行股份有限公司 Distributed resource allocation method, device, equipment and readable storage medium storing program for executing

Similar Documents

Publication Publication Date Title
CN109087090B (en) Method and apparatus for tracking targets using trusted account book
US11176110B2 (en) Data updating method and device for a distributed database system
CN109918229B (en) Database cluster copy construction method and device in non-log mode
Wang et al. A near-optimal protocol for the grouping problem in RFID systems
WO2014021978A4 (en) Aggregating data in a mediation system
CN111066040A (en) Inventory management system
CN102782679A (en) Event information management system, event management method and program
CN105227662A (en) Message treatment method, device and system
JP4848128B2 (en) Database system, computer-executable method for database system, program and method for updating index table in database system
CN110990489A (en) System and method for batch writing of sensing data into blocks
Fenga et al. COVID‐19: Metaheuristic Optimization‐Based Forecast Method on Time‐Dependent Bootstrapped Data
CN107295059A (en) The statistical system and method for service propelling amount
CN104750849A (en) Method and system for maintaining tree structure-based directory relationship
CN108647243B (en) Industrial big data storage method based on time series
CN105512313A (en) Incremental data processing method and device
CN104700055A (en) Method for detecting complex events on multi-probability RFID event flows
CN111078425B (en) Message processing method and device, storage medium and electronic equipment
JP5308927B2 (en) Device information management apparatus and method
CN110532108B (en) Resume delivery task processing method, device, server and system
CN100357969C (en) Automatic tracking system of goods and its method
US10715619B2 (en) Cache management using a probabilistic data structure
CN111640027A (en) Service data processing method, service data processing device, service processing device and electronic equipment
CN106649444B (en) Method and device for CAE database system to acquire CAD data
Liu et al. On fast and reliable missing event detection protocol for multitagged RFID systems
JP2021068365A5 (en)

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication
RJ01 Rejection of invention patent application after publication

Application publication date: 20200410