CN111078778A - High-frequency data warehousing system based on multithreading asynchronous communication - Google Patents
High-frequency data warehousing system based on multithreading asynchronous communication Download PDFInfo
- Publication number
- CN111078778A CN111078778A CN201911299108.4A CN201911299108A CN111078778A CN 111078778 A CN111078778 A CN 111078778A CN 201911299108 A CN201911299108 A CN 201911299108A CN 111078778 A CN111078778 A CN 111078778A
- Authority
- CN
- China
- Prior art keywords
- data
- module
- sensor group
- frequency
- database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- Software Systems (AREA)
- Arrangements For Transmission Of Measured Signals (AREA)
Abstract
The invention provides a high-frequency data warehousing system based on multithreading asynchronous communication. The system comprises a sensor group, a data processing module and a database; the sensor group is arranged on the motion carrier, and the data processing module and the database are arranged on the control box; the sensor group is connected with the data processing module through an optical fiber; and the data collected by the sensor group is written into the database after being processed by the data processing module. In the invention, the data receiving module asynchronously receives the data sent by the sensor group by adopting a communication technology, accurately receives the data in high-frequency communication, and avoids the problems of process blockage, data storage failure and the like caused by frequent database operation by introducing the data pool.
Description
Technical Field
The invention relates to the technical field of computer communication and software, in particular to a high-frequency data warehousing system based on multithreading asynchronous communication.
Background
In the underground space positioning algorithm, sensor data such as a gyroscope, an inclinometer, an accelerometer and the like are required to be utilized. The data frequency of the sensors is usually high and can reach 200 Hz. For such high-frequency data, although techniques such as asynchronous communication and data cache check can be adopted to ensure the correctness of the received data, in some service scenarios, not only the result data of the positioning needs to be put in storage, but also the sensor data needs to be written into the database at the same time. In such a scenario, due to high data frequency and more time consumption of database operation, frequent database operation may cause process blocking and data storage failure.
In view of the foregoing, there is a need for a high-frequency data warehousing system based on multithreaded asynchronous communication to solve the problems in the prior art.
Disclosure of Invention
The invention aims to provide a high-frequency data warehousing system based on multithreading asynchronous communication, which aims to solve the problem of high-frequency data warehousing.
In order to achieve the aim, the invention provides a high-frequency data storage system based on multithreading asynchronous communication, which comprises a sensor group, a data processing module and a database; the sensor group is arranged on the motion carrier, and the data processing module and the database are arranged on the control box; the sensor group is connected with the data processing module through an optical fiber; the data collected by the sensor group is written into a database after being processed by the data processing module;
the data processing module comprises a data receiving module, a sensing module, a state identifier, a data pool and a data library module; the data receiving module adopts a communication technology to asynchronously receive data sent by the sensor group; the sensing module is used for monitoring the state of the data pool; the state identifier is used for identifying the state of the moving carrier; the data pool is used for storing the data received by the data receiving module after the sensor is processed; and the data storage module stores the sensor data subjected to data cleaning into a database.
Preferably, the sensor set comprises one or more of a fiber optic gyroscope, an inclinometer and an accelerometer, and the data frequency is not lower than 100 Hz.
Preferably, the communication mode adopted by the data receiving module and the sensor group is asynchronous serial port communication or TPC/IP communication.
Preferably, the rule that the sensing module monitors the state of the data pool is that the number of data in the data pool reaches a threshold value or the time from the last data storage reaches the threshold value, and when one of the number of data in the data pool and the time from the last data storage reaches the threshold value, the data storage module is triggered.
Preferably, the state identifier identifies the state of the identified moving carrier, and the state is obtained through the background of the control box, wherein the state is a moving state or a static state.
Preferably, each piece of data stored in the data pool is an SQL statement.
Preferably, after the sensing module triggers the data entry module, all data are transmitted to the data entry module, and the data pool is emptied.
Preferably, when the state identifier is static, the data of the sensor group is cleaned, and the high-frequency data is cleaned into low-frequency data at a certain sampling interval.
Preferably, when the data warehousing module is called, a thread is separately opened up, and the data transmitted by the data pool is written into the database once by using database transactions.
The technical scheme of the invention has the following beneficial effects:
in the invention, the data receiving module asynchronously receives the data sent by the sensor group by adopting a communication technology, accurately receives the data in high-frequency communication, and avoids the problems of process blockage, data storage failure and the like caused by frequent database operation by introducing the data pool.
In addition to the objects, features and advantages described above, other objects, features and advantages of the present invention are also provided. The present invention will be described in further detail below with reference to the drawings.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this application, illustrate embodiments of the invention and, together with the description, serve to explain the invention and not to limit the invention. In the drawings:
FIG. 1 is a schematic structural view of the present invention;
the system comprises a sensor group 1, a sensor group 2, a data processing module 21, a data receiving module 22, an induction module 23, a state identifier 24, a data pool 25, a data storage module 3 and a database.
Detailed Description
Embodiments of the invention will be described in detail below with reference to the drawings, but the invention can be implemented in many different ways, which are defined and covered by the claims.
Example 1:
referring to fig. 1, a high-frequency data warehousing system based on multithreading asynchronous communication comprises a sensor group 1, a data processing module 2 and a database 3; the sensor group 1 is arranged on a motion carrier, and the data processing module 2 and the database 3 are arranged on a control box; the sensor group 1 is connected with the data processing module 2 through an optical fiber; the data collected by the sensor group 1 is processed by the data processing module 2 and then written into the database 3;
the data processing module 2 comprises a data receiving module 21, a sensing module 22, a state identifier 23, a data pool 24 and a data storage module 25; the data receiving module 21 asynchronously receives the data sent by the sensor group 1 by adopting a communication technology; the sensing module 22 is used for monitoring the state of the data pool 24; the state identifier 23 is used for identifying the state of the moving carrier; the data pool 24 is used for storing the data received by the data receiving module 21 after the sensor is processed; the data storage module 25 stores the sensor data subjected to data cleaning into the database 3.
The sensor group 1 comprises a fiber optic gyroscope, an inclinometer and an accelerometer sensor, and the data frequency is 200 Hz.
The data receiving module 21 and the sensor group 1 adopt asynchronous serial port communication, and the data processing module is written by C # programming language.
The rule for the sensing module 22 to monitor the state of the data pool 24 is that the number of data in the data pool reaches a threshold value or the time from the last data storage reaches a threshold value, and when one of the number of data in the data pool and the time reaches the threshold value, the sensing module triggers the data storage module 25. In this embodiment, the threshold of the number of data pieces is 2000, and the threshold of the warehousing time interval is 5 s.
The state of the moving carrier identified by the state identifier 23 is a moving state or a static state, and the state is acquired through the background of the control box.
Each piece of data stored by the data pool 24 is an SQL statement. On the basis of original data, a SQL inserting statement is formed after a timestamp, project information, a database table object and the like are added.
When the sensing module 22 triggers the data entry module 25, all data is transmitted to the data entry module 25, and the data pool is emptied. When the state identifier 23 is static, the data of the inclinometer and the accelerometer are cleaned, and the high-frequency data is cleaned into low-frequency data according to a certain sampling interval. Specifically, at the time t0, the state identifier is changed from motion to rest, at the time t1, the state identifier is changed from rest to motion, at the time t1, the data pool is cleaned, 1 piece of data is selected from the inclinometer and the accelerometer every 10 pieces of data according to the time sequence, the rest data are deleted from the data pool, and the gyroscope data are not changed.
When the data warehousing module 25 is called, a thread is separately opened up, and the data transmitted by the data pool 24 is written into the database once by using database transactions. The database adopts SQL Server 2012 database.
In the invention, the data receiving module asynchronously receives the data sent by the sensor group by adopting a communication technology, accurately receives the data in high-frequency communication, and avoids the problems of process blockage, data storage failure and the like caused by frequent database operation by introducing the data pool.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (9)
1. A high-frequency data warehousing system based on multithreading asynchronous communication is characterized by comprising a sensor group (1), a data processing module (2) and a database (3); the sensor group (1) is arranged on a motion carrier, and the data processing module (2) and the database (3) are arranged on a control box; the sensor group (1) is connected with the data processing module (2) through an optical fiber; data collected by the sensor group (1) are written into the database (3) after being processed by the data processing module (2);
the data processing module (2) comprises a data receiving module (21), a sensing module (22), a state identifier (23), a data pool (24) and a data library module (25); the data receiving module (21) adopts a communication technology to asynchronously receive the data sent by the sensor group (1); the sensing module (22) is used for monitoring the state of the data pool (24); the state identifier (23) is used for identifying the state of the moving carrier; the data pool (24) is used for storing the data which is received by the data receiving module (21) and processed by the sensor; the data storage module (25) stores the sensor data subjected to data cleaning into the database (3).
2. A multithreading asynchronous communication-based high-frequency data warehousing system as claimed in claim 1, wherein the sensor group (1) comprises one or more combinations of fiber optic gyroscopes, inclinometers and accelerometers, and the data frequency is not lower than 100 Hz.
3. The multithreading asynchronous communication-based high-frequency data warehousing system as claimed in claim 1, wherein the data receiving module (21) and the sensor group (1) adopt a communication mode of asynchronous serial communication or TPC/IP communication.
4. A multithreading asynchronous communication-based high-frequency data warehousing system as claimed in claim 1, wherein the rule that the sensing module (22) monitors the status of the data pool (24) is that the number of data in the data pool reaches a threshold value or the time from the last data warehousing reaches a threshold value, and when one of the two reaches the threshold value, the data warehousing module (25) is triggered.
5. The multithreading asynchronous communication-based high-frequency data warehousing system as claimed in claim 1, wherein the state of the moving carrier identified by the state identifier (23) is a moving state or a static state, and the state is obtained through a control box background.
6. The multithreading asynchronous communication-based high-frequency data warehousing system as claimed in claim 1, wherein each piece of data stored by the data pool (24) is an SQL statement.
7. A multithreading asynchronous communication-based high-frequency data warehousing system as claimed in claim 4, wherein when the sensing module (22) triggers the data warehousing module (25), all data is transmitted to the data warehousing module (25) and the data pool is emptied.
8. A multithreading asynchronous communication-based high-frequency data warehousing system as claimed in claim 5, wherein when the status flag (23) is static, the data of the sensor group (1) is cleaned, and the high-frequency data is cleaned into low-frequency data at a certain sampling interval.
9. The multithread asynchronous communication-based high-frequency data warehousing system as claimed in any one of claims 1 to 8, wherein the data warehousing module (25) opens up a thread separately when being called, and writes the data transmitted by the data pool (24) into the database at one time by using database transaction.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911299108.4A CN111078778A (en) | 2019-12-17 | 2019-12-17 | High-frequency data warehousing system based on multithreading asynchronous communication |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911299108.4A CN111078778A (en) | 2019-12-17 | 2019-12-17 | High-frequency data warehousing system based on multithreading asynchronous communication |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111078778A true CN111078778A (en) | 2020-04-28 |
Family
ID=70314914
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911299108.4A Pending CN111078778A (en) | 2019-12-17 | 2019-12-17 | High-frequency data warehousing system based on multithreading asynchronous communication |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111078778A (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090313310A1 (en) * | 2008-06-13 | 2009-12-17 | Timothy Thome | Managing a dynamic data pool on ad-hoc networks |
| CN105302692A (en) * | 2015-10-28 | 2016-02-03 | 联动优势科技有限公司 | Internal and external parameters based database connection pool monitoring and management method and apparatus |
| CN106209982A (en) * | 2016-06-24 | 2016-12-07 | 成都之达科技有限公司 | Tire pressure monitoring method based on wireless senser |
| CN107707619A (en) * | 2017-08-28 | 2018-02-16 | 北京小米移动软件有限公司 | Distributed storage method and device |
| CN108200166A (en) * | 2017-12-30 | 2018-06-22 | 广州柏颐信息科技有限公司 | A kind of Low-power-consumptiocontrol control method and device based on Sensor monitoring |
| CN109151762A (en) * | 2018-10-19 | 2019-01-04 | 海南易乐物联科技有限公司 | A kind of the asynchronous process system and processing method of high concurrent acquisition data |
| CN109508289A (en) * | 2018-10-25 | 2019-03-22 | 阿里巴巴集团控股有限公司 | Test method and device, electronic equipment |
| WO2019094721A2 (en) * | 2017-11-09 | 2019-05-16 | Strong Force Iot Portfolio 2016, Llc | Methods and systems for the industrial internet of things |
-
2019
- 2019-12-17 CN CN201911299108.4A patent/CN111078778A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090313310A1 (en) * | 2008-06-13 | 2009-12-17 | Timothy Thome | Managing a dynamic data pool on ad-hoc networks |
| CN105302692A (en) * | 2015-10-28 | 2016-02-03 | 联动优势科技有限公司 | Internal and external parameters based database connection pool monitoring and management method and apparatus |
| CN106209982A (en) * | 2016-06-24 | 2016-12-07 | 成都之达科技有限公司 | Tire pressure monitoring method based on wireless senser |
| CN107707619A (en) * | 2017-08-28 | 2018-02-16 | 北京小米移动软件有限公司 | Distributed storage method and device |
| WO2019094721A2 (en) * | 2017-11-09 | 2019-05-16 | Strong Force Iot Portfolio 2016, Llc | Methods and systems for the industrial internet of things |
| CN108200166A (en) * | 2017-12-30 | 2018-06-22 | 广州柏颐信息科技有限公司 | A kind of Low-power-consumptiocontrol control method and device based on Sensor monitoring |
| CN109151762A (en) * | 2018-10-19 | 2019-01-04 | 海南易乐物联科技有限公司 | A kind of the asynchronous process system and processing method of high concurrent acquisition data |
| CN109508289A (en) * | 2018-10-25 | 2019-03-22 | 阿里巴巴集团控股有限公司 | Test method and device, electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107077495B (en) | High performance transactions in a database management system | |
| US20140149368A1 (en) | Compressed Representation of a Transaction Token | |
| CN105956166B (en) | Database reading and writing method and device | |
| US6708288B1 (en) | Compiler-based checkpointing for support of error recovery | |
| CN103488690B (en) | Data integrated system and data integrating method | |
| US8386720B2 (en) | Method for allowing exclusive access to shared data | |
| US8762425B2 (en) | Managing a data structure | |
| US8250111B2 (en) | Automatic detection and correction of hot pages in a database system | |
| US20180322158A1 (en) | Changing concurrency control modes | |
| US8666981B2 (en) | Bottom-up optimistic latching method for index trees | |
| US10275323B2 (en) | Resilient programming frameworks for handling failures in parallel programs | |
| US8584123B2 (en) | Linking transactions | |
| US10268610B1 (en) | Determining whether a CPU stalling a current RCU grace period had interrupts enabled | |
| US20120303591A1 (en) | Managing rollback in a transactional memory environment | |
| EP2674868A1 (en) | Database update notification method | |
| US20180267840A1 (en) | Using Expedited Grace Periods To Short-Circuit Normal Grace-Period Computations | |
| CN111176850B (en) | Data pool construction method, device, server and medium | |
| US20110087722A1 (en) | Stitching transactions | |
| US20050240747A1 (en) | Memory leak detection | |
| CN111737661A (en) | Exception stack processing method, system, electronic device and storage medium | |
| US10915639B2 (en) | Staged dynamic taint flow inference | |
| US20120089577A1 (en) | Nondisruptive overflow avoidance of tuple validity timestamps in temporal database systems | |
| CN111078778A (en) | High-frequency data warehousing system based on multithreading asynchronous communication | |
| KR20170036000A (en) | Mutable chronologies for accommodation of randomly occurring event delays | |
| CN108614772A (en) | The framework and method of a kind of concurrently detection stacker buffer spilling |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |