CN106227899A - The storage of the big data of a kind of internet of things oriented and querying method - Google Patents
The storage of the big data of a kind of internet of things oriented and querying method Download PDFInfo
- Publication number
- CN106227899A CN106227899A CN201610797518.1A CN201610797518A CN106227899A CN 106227899 A CN106227899 A CN 106227899A CN 201610797518 A CN201610797518 A CN 201610797518A CN 106227899 A CN106227899 A CN 106227899A
- Authority
- CN
- China
- Prior art keywords
- data
- storage
- internet
- things
- structured
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 19
- 238000013500 data storage Methods 0.000 claims abstract description 17
- 238000005516 engineering process Methods 0.000 claims description 26
- 238000007405 data analysis Methods 0.000 claims description 12
- 238000007726 management method Methods 0.000 claims description 10
- 230000008447 perception Effects 0.000 claims description 8
- 238000004891 communication Methods 0.000 claims description 7
- 238000012544 monitoring process Methods 0.000 claims description 7
- 230000006870 function Effects 0.000 claims description 6
- 230000003139 buffering effect Effects 0.000 claims description 3
- 238000013144 data compression Methods 0.000 claims description 3
- 238000005192 partition Methods 0.000 claims description 3
- 238000012545 processing Methods 0.000 claims description 3
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 claims description 3
- 230000007547 defect Effects 0.000 abstract description 3
- 238000011161 development Methods 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 230000004927 fusion Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 239000000470 constituent Substances 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/33—Querying
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/568—Storing data temporarily at an intermediate stage, e.g. caching
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/568—Storing data temporarily at an intermediate stage, e.g. caching
- H04L67/5683—Storage of data provided by user terminals, i.e. reverse caching
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computational Linguistics (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention belongs to design and application of software technical field, be specifically related to storage and the querying method of the big data of a kind of internet of things oriented, its application suiting field, smart city is actual, has powerful application prospect.The method comprises the steps: step S1: gathered data by sensor device layer；Step S2: carry out data parsing by data parsing；Step S3: carry out data storage by data storage layer；Step S4: carry out data query by data query layer；Compared with prior art, the present invention provides a kind of high speed storing at Internet of Things big market demand FIELD Data and the method for inquiry, it is possible to overcome the shortcoming and defect based on traditional data library storage and inquiry.
Description
Technical Field
The invention belongs to the technical field of software design and application, and particularly relates to a storage and query method for big data of the Internet of things.
Background
Today is an era from the internet era to the internet of things, which is a revolutionary development of the information industry again after computers, internet and mobile communications. The internet of things is expanded and extended on the basis of the internet, and is widely applied to network fusion through communication perception technologies such as intelligent perception, recognition technology and pervasive computing. The appearance of the Internet of things provides technical support for innovation and development in the fields of smart cities and the like. With the continuous expansion of the application range of the technology of the internet of things, the data generated by the internet of things is expanded continuously, and the related data volume is huge in scale, so that the data cannot be captured, managed and processed in reasonable time through the current mainstream software tool, and useful information is provided based on the data, thereby forming the fusion of the internet of things era and the big data era. The big data of the Internet of things has the following characteristics. Firstly, the data volume in the internet of things is larger, one of the main characteristics of the internet of things is the massive property of nodes, except people and servers, articles, equipment, a sensor network and the like are all the constituent nodes of the internet of things, and the quantity and the scale of the nodes are far larger than those of the internet; meanwhile, the data generation frequency of the nodes of the Internet of things is far higher than that of the Internet, and if most of the sensing nodes are in a full-time working state, the data flow is continuous. Secondly, the data rate in the internet of things is higher, on one hand, the data volume in the internet of things inevitably requires a backbone network to gather more data, and the data transmission rate requirement is higher; on the other hand, since the internet of things is directly associated with the real physical world, in many cases, real-time access and control of corresponding nodes and devices are required, and thus a high data transmission rate is required to support corresponding real-time performance. Thirdly, data in the internet of things are more diversified, the application range related to the internet of things is wide, and the application range is not the application range of the internet of things from smart cities, smart traffic, smart logistics, commodity traceability, smart homes, smart medical treatment, security monitoring and the like; in different fields and different industries, application data of different types and different formats needs to be faced, so that the data diversity in the internet of things is more prominent.
The analysis shows that big data is a necessary key technology in the Internet of things, and the combination of the big data and the key technology can bring a better technical foundation for the development of the Internet of things system and application. Therefore, the technology and the method for internet of things big data storage query are urgently needed to solve the problems of internet of things big data storage and query through practice and application of smart pipe network projects, and the technology and the method are suitable for the application fields of internet of things such as smart pipe networks.
Disclosure of Invention
Technical problem to be solved
The technical problem to be solved by the invention is as follows: how to overcome the defects and shortcomings of storage and query based on the traditional database, and a method for storing and querying data at high speed in the field of big data application of the Internet of things is provided.
(II) technical scheme
In order to solve the technical problems, the invention provides a storage and query method for big data of the internet of things, which is characterized by comprising the following steps:
step S1: collecting data by a sensor device layer;
data sources in the application field of the Internet of things come from various professional sensor devices; the Internet of things perception sensor in the sensor equipment layer is responsible for sending an original protocol data packet to an upper layer for data analysis;
step S2: performing data analysis through data analysis;
because the data collected and reported by the internet of things perception sensor is data based on a specific protocol, the transmitted data belongs to an original data packet in the network communication layer, and cannot be directly pushed to a system application for direct application, and the original data message needs to be analyzed according to the communication protocol of the equipment to form formatted data; because different sensors have different purposes, the data format formed after the sensor data is analyzed is divided into structured data, semi-structured data and unstructured data; wherein,
structuring data: the data obtained when most sensor data are analyzed are structured data, the data comprise a temperature sensor, a data message reported at a certain moment is analyzed and then comprises equipment number, equipment manufacturer, equipment protocol, equipment operation condition, temperature data and time information for generating data, the formats of the information are all fixed and unchanged, therefore, a temperature sensor data table is created according to a traditional relational data processing mode, and the service information is stored according to corresponding field information;
semi-structured data: in some applications of the internet of things with specific requirements and specific scenes, although the data of some sensors are structured, the structure of the sensors is not permanently changed, and structural differences and variability exist; the data comprises data of professional monitoring sites of urban underground pipeline water supply, although the data is monitoring data of the same profession, the uploaded data is inconsistent in structure due to the fact that different devices are used by the profession, and the data belongs to semi-structured data;
unstructured data: the unstructured data is information whose content cannot be directly known, and includes image data, sound data, and video data; for an important application camera in the internet of things, data generated by the equipment are images, sounds and videos, and the data are difficult to store in a relational database for query and viewing;
step S3: performing data storage through a data storage layer;
the data storage layer is used for storing the data processed by the data analysis layer, and different storage strategies are adopted for solving the storage problem of data diversity aiming at different data types; storing structured data into a relational database, storing semi-structured data into a distributed database, and storing unstructured data into a DFS; wherein,
RDBMS: because the application range of the prior relational database technology is wider and the maturity is higher, the structured data is stored in a relational database RDBMS (relational database management system), including Oracle and MySQL;
distributed database: for semi-structured data, due to the characteristics of uncertainty and variability of the structure, the structural change is difficult to handle in the RDBMS technical system, but in the more popular big data technology, the distributed database technology based on column storage is suitable for scenes with uncertain and variable table structures; the method for storing the column names from the stored data is different from the RDBMS in the greatest way that the data is stored according to the columns, and the column storage has the greatest characteristic of facilitating storage of structured and semi-structured data and data compression, and has great IO (input/output) advantages for querying a certain column or a plurality of columns;
DFS: for unstructured data, if an RDBMS is used to store images, sounds and videos, it is a common practice to create a table containing three fields of numbers, content descriptions and content blobs, and the unstructured data is stored in the content blob fields, which is a great challenge for the storage of a large amount of unstructured data; aiming at the problem, a distributed file system DFS is adopted to store unstructured data into a file system, and a distributed technical architecture can well solve the storage problem of massive unstructured data;
step S4: data query is carried out through a data query layer;
the data query is based on the data storage of the lower layer, and provides a quick and efficient data query service for the application of the upper layer system; the fast indexing of mass data is solved by adopting an inquiry caching technology, access control is used for limiting the access authority of the data, and the access mode of the data adopts a form of issuing data service; the method specifically comprises the following steps: inquiring cache, access control and data service;
and (3) inquiring cache: due to the huge data volume of the Internet of things and the high frequency of data access of the system, the data are frequently read through the disk IO operation, but the disk IO speed is low, the efficiency is low, and the data reading efficiency is low; the cache technology can replace disk reading for memory data reading, and the reading speed of the memory data is far higher than that of the disk reading, so that the data reading efficiency is improved;
inquiring distributed caches and local caches of cache sub-clusters; because the bottom storage adopts a distributed technology, the query efficiency is influenced if the nodes are searched one by one when certain data is read. The distributed cache can read data with high performance, can dynamically expand cache nodes, can automatically discover and switch fault nodes, can automatically balance data partitions, can provide a graphical management interface for a user, and is very convenient to deploy and maintain; the local cache is used for dividing a local physical memory of a client into a part of space for buffering data written back to a server by the client, writing the data written back by the client into a hard disk of the server no longer firstly, but writing the data written back into the local write-back cache firstly, and writing the data back to the server when the cache space reaches a certain threshold value; after the local write-back cache function is provided, the read-write pressure and the network load of the server can be greatly reduced;
and (3) access control: the access control is a management function which is responsible for verifying and authorizing the access authority of the system when the application system sends a data access request;
data service: the data access interface is provided for the application system in a data service publishing mode, the interface realizes the consistency of the interface when accessing heterogeneous data, and a user does not need to care whether the data is stored in the RDBMS, the distributed database or the DFS.
(III) advantageous effects
Compared with the prior art, the invention provides a method for storing and inquiring data at high speed in the field of big data application of the Internet of things, which can overcome the defects and shortcomings of storage and inquiry based on the traditional database.
Drawings
Fig. 1 is a schematic diagram of the technical solution of the present invention.
Detailed Description
In order to make the objects, contents, and advantages of the present invention clearer, the following detailed description of the embodiments of the present invention will be made in conjunction with the accompanying drawings and examples.
In order to solve the problems in the prior art, the invention provides a storage and query method for big data of the internet of things, which specifically comprises the following steps as shown in fig. 1:
step S1: collecting data by a sensor device layer;
data sources in the application field of the Internet of things generally come from various professional sensor devices; taking the application of the comprehensive management of urban underground pipelines as an example, sensors installed on each professional pipeline, such as a temperature sensor, a pressure sensor, a flow sensor and the like, monitor the operation condition of each parameter index in the pipeline at any moment and upload data, and different devices adopt different protocols, so that the formats of data sources and the receiving modes of data are greatly different. The Internet of things perception sensor in the sensor equipment layer is responsible for sending an original protocol data packet to an upper layer for data analysis;
step S2: performing data analysis through data analysis;
because the data collected and reported by the internet of things perception sensor is data based on a specific protocol, the transmitted data belongs to an original data packet in the network communication layer, and cannot be directly pushed to a system application for direct application, and the original data message needs to be analyzed according to the communication protocol of the equipment to form formatted data; because different sensors have different purposes, the data format formed after the sensor data is analyzed can be divided into structured data, semi-structured data and unstructured data; wherein,
structuring data: data obtained when most sensor data are analyzed are structured data, the data comprise a temperature sensor, data messages reported at a certain moment are analyzed and then contain information such as equipment numbers, equipment manufacturers, equipment protocols, equipment operation conditions, temperature data, data generation time and the like, the formats of the information are fixed and unchanged, and therefore a temperature sensor data table is created according to a traditional relational data processing mode, and the service information is stored according to corresponding field information;
semi-structured data: in some applications of the internet of things with specific requirements and specific scenes, although the data of some sensors are structured, the structure of the sensors is not permanently changed, and structural differences and variability exist; the data comprises data of professional monitoring sites of urban underground pipeline water supply, although the data is monitoring data of the same profession, the uploaded data is inconsistent in structure due to the fact that different devices are used by the profession, and the data belongs to semi-structured data;
unstructured data: the unstructured data is information whose content cannot be directly known, and includes image data, sound data, and video data; for an important application camera in the internet of things, data generated by the equipment are images, sounds and videos, and the data are difficult to store in a relational database for query and viewing;
step S3: performing data storage through a data storage layer;
the data storage layer is used for storing the data processed by the data analysis layer, and different storage strategies are adopted for solving the storage problem of data diversity aiming at different data types; storing structured data into a relational database, storing semi-structured data into a distributed database, and storing unstructured data into a DFS; wherein,
RDBMS: because the application range of the prior relational database technology is wider and the maturity is higher, the structured data is stored in a relational database RDBMS (relational database management system), including Oracle and MySQL; in addition, in the face of the problem of big data of the Internet of things, namely the problem of huge data scale and magnitude, relational databases such as Oracle and MySQL also provide solutions such as distributed clusters;
distributed database: for semi-structured data, due to the characteristics of uncertainty and variability of the structure, the structural change is difficult to handle in the RDBMS technical system, but in the more popular big data technology, the distributed database technology based on column storage is suitable for scenes with uncertain and variable table structures; the method for storing the column names from the stored data is different from the RDBMS in the greatest way that the data is stored according to the columns, and the column storage has the greatest characteristic of facilitating storage of structured and semi-structured data and data compression, and has great IO (input/output) advantages for querying a certain column or a plurality of columns;
DFS: for unstructured data, if an RDBMS is used to store images, sounds and videos, it is a common practice to create a table containing three fields of numbers, content descriptions and content blobs, and the unstructured data is stored in the content blob fields, which is a great challenge for the storage of a large amount of unstructured data; aiming at the problem, a distributed file system DFS is adopted to store unstructured data into a file system, and a distributed technical architecture can well solve the storage problem of massive unstructured data;
step S4: data query is carried out through a data query layer;
the data query is based on the data storage of the lower layer, and provides a quick and efficient data query service for the application of the upper layer system; the fast indexing of mass data is solved by adopting an inquiry caching technology, access control is used for limiting the access authority of the data, and the access mode of the data adopts a form of issuing data service; the method specifically comprises the following steps: inquiring cache, access control and data service;
and (3) inquiring cache: due to the huge data volume of the Internet of things and the high frequency of data access of the system, the data are frequently read through the disk IO operation, but the disk IO speed is low, the efficiency is low, and the data reading efficiency is low; the cache technology can replace disk reading for memory data reading, and the reading speed of the memory data is far higher than that of the disk reading, so that the data reading efficiency is improved;
inquiring distributed caches and local caches of cache sub-clusters; because the bottom storage adopts a distributed technology, the query efficiency is influenced if the nodes are searched one by one when certain data is read. The distributed cache can read data with high performance, can dynamically expand cache nodes, can automatically discover and switch fault nodes, can automatically balance data partitions, can provide a graphical management interface for a user, and is very convenient to deploy and maintain; the local cache is used for dividing a local physical memory of a client into a part of space for buffering data written back to a server by the client, the technology writes the data written back by the client into the local write-back cache without writing the data into a hard disk of the server first, and writes the data back to the server when the cache space reaches a certain threshold value; after the local write-back cache function is provided, the read-write pressure and the network load of the server can be greatly reduced;
and (3) access control: the access control is a management function which is responsible for verifying and authorizing the access authority of the system when the application system sends a data access request;
data service: the data access interface is provided for the application system in a data service publishing mode, the interface realizes the consistency of the interface when accessing heterogeneous data, and a user does not need to care whether the data is stored in the RDBMS, the distributed database or the DFS.
The above description is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, several modifications and variations can be made without departing from the technical principle of the present invention, and these modifications and variations should also be regarded as the protection scope of the present invention.
Claims (1)
1. A big data storage and query method for the Internet of things is characterized by comprising the following steps:
step S1: collecting data by a sensor device layer;
data sources in the application field of the Internet of things come from various professional sensor devices; the Internet of things perception sensor in the sensor equipment layer is responsible for sending an original protocol data packet to an upper layer for data analysis;
step S2: performing data analysis through data analysis;
because the data collected and reported by the internet of things perception sensor is data based on a specific protocol, the transmitted data belongs to an original data packet in the network communication layer, and cannot be directly pushed to a system application for direct application, and the original data message needs to be analyzed according to the communication protocol of the equipment to form formatted data; because different sensors have different purposes, the data format formed after the sensor data is analyzed is divided into structured data, semi-structured data and unstructured data; wherein,
structuring data: the data obtained when most sensor data are analyzed are structured data, the data comprise a temperature sensor, a data message reported at a certain moment is analyzed and then comprises equipment number, equipment manufacturer, equipment protocol, equipment operation condition, temperature data and time information for generating data, the formats of the information are all fixed and unchanged, therefore, a temperature sensor data table is created according to a traditional relational data processing mode, and the service information is stored according to corresponding field information;
semi-structured data: in some applications of the internet of things with specific requirements and specific scenes, although the data of some sensors are structured, the structure of the sensors is not permanently changed, and structural differences and variability exist; the data comprises data of professional monitoring sites of urban underground pipeline water supply, although the data is monitoring data of the same profession, the uploaded data is inconsistent in structure due to the fact that different devices are used by the profession, and the data belongs to semi-structured data;
unstructured data: the unstructured data is information whose content cannot be directly known, and includes image data, sound data, and video data; for an important application camera in the internet of things, data generated by the equipment are images, sounds and videos, and the data are difficult to store in a relational database for query and viewing;
step S3: performing data storage through a data storage layer;
the data storage layer is used for storing the data processed by the data analysis layer, and different storage strategies are adopted for solving the storage problem of data diversity aiming at different data types; storing structured data into a relational database, storing semi-structured data into a distributed database, and storing unstructured data into a DFS; wherein,
RDBMS: because the application range of the prior relational database technology is wider and the maturity is higher, the structured data is stored in a relational database RDBMS (relational database management system), including Oracle and MySQL;
distributed database: for semi-structured data, due to the characteristics of uncertainty and variability of the structure, the structural change is difficult to handle in the RDBMS technical system, but in the more popular big data technology, the distributed database technology based on column storage is suitable for scenes with uncertain and variable table structures; the method for storing the column names from the stored data is different from the RDBMS in the greatest way that the data is stored according to the columns, and the column storage has the greatest characteristic of facilitating storage of structured and semi-structured data and data compression, and has great IO (input/output) advantages for querying a certain column or a plurality of columns;
DFS: for unstructured data, if an RDBMS is used to store images, sounds and videos, it is a common practice to create a table containing three fields of numbers, content descriptions and content blobs, and the unstructured data is stored in the content blob fields, which is a great challenge for the storage of a large amount of unstructured data; aiming at the problem, a distributed file system DFS is adopted to store unstructured data into a file system, and a distributed technical architecture can well solve the storage problem of massive unstructured data;
step S4: data query is carried out through a data query layer;
the data query is based on the data storage of the lower layer, and provides a quick and efficient data query service for the application of the upper layer system; the fast indexing of mass data is solved by adopting an inquiry caching technology, access control is used for limiting the access authority of the data, and the access mode of the data adopts a form of issuing data service; the method specifically comprises the following steps: inquiring cache, access control and data service;
and (3) inquiring cache: due to the huge data volume of the Internet of things and the high frequency of data access of the system, the data are frequently read through the disk IO operation, but the disk IO speed is low, the efficiency is low, and the data reading efficiency is low; the cache technology can replace disk reading for memory data reading, and the reading speed of the memory data is far higher than that of the disk reading, so that the data reading efficiency is improved;
inquiring distributed caches and local caches of cache sub-clusters; because the bottom storage adopts a distributed technology, the query efficiency is influenced if the nodes are searched one by one when certain data is read. The distributed cache can read data with high performance, can dynamically expand cache nodes, can automatically discover and switch fault nodes, can automatically balance data partitions, can provide a graphical management interface for a user, and is very convenient to deploy and maintain; the local cache is used for dividing a local physical memory of a client into a part of space for buffering data written back to a server by the client, writing the data written back by the client into a hard disk of the server no longer firstly, but writing the data written back into the local write-back cache firstly, and writing the data back to the server when the cache space reaches a certain threshold value; after the local write-back cache function is provided, the read-write pressure and the network load of the server can be greatly reduced;
and (3) access control: the access control is a management function which is responsible for verifying and authorizing the access authority of the system when the application system sends a data access request;
data service: the data access interface is provided for the application system in a data service publishing mode, the interface realizes the consistency of the interface when accessing heterogeneous data, and a user does not need to care whether the data is stored in the RDBMS, the distributed database or the DFS.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610797518.1A CN106227899A (en) | 2016-08-31 | 2016-08-31 | The storage of the big data of a kind of internet of things oriented and querying method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610797518.1A CN106227899A (en) | 2016-08-31 | 2016-08-31 | The storage of the big data of a kind of internet of things oriented and querying method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN106227899A true CN106227899A (en) | 2016-12-14 |
Family
ID=58075401
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610797518.1A Pending CN106227899A (en) | 2016-08-31 | 2016-08-31 | The storage of the big data of a kind of internet of things oriented and querying method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106227899A (en) |
Cited By (23)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106845202A (en) * | 2017-03-09 | 2017-06-13 | 北京旷视科技有限公司 | File access method, device and system for face identification system |
| CN107016056A (en) * | 2017-03-07 | 2017-08-04 | 西安电子科技大学 | The distributed memory system and method for magnanimity heterogeneous sensor data in a kind of Internet of Things |
| CN108197261A (en) * | 2017-12-30 | 2018-06-22 | 北京通途永久科技有限公司 | A kind of wisdom traffic operating system |
| CN108416067A (en) * | 2018-03-29 | 2018-08-17 | 重庆大学 | Mass data processing and the optimization of storing process execute evaluation method in industrial process |
| CN108536823A (en) * | 2018-04-10 | 2018-09-14 | 北京工业大学 | A kind of caching design and querying method of Internet of Things perception big data |
| CN108650292A (en) * | 2018-03-27 | 2018-10-12 | 吉旗（成都）科技有限公司 | A kind of internet of things equipment message can quick locating query storage method |
| CN108763534A (en) * | 2018-05-31 | 2018-11-06 | 北京百度网讯科技有限公司 | Method and apparatus for handling information |
| CN109104710A (en) * | 2018-06-29 | 2018-12-28 | 湖北海纳天鹰科技发展有限公司 | It is a kind of based on the air quality of NBiot network and MQTT agreement inquiry and distributing device |
| CN109224999A (en) * | 2018-09-29 | 2019-01-18 | 湖北航鹏化学动力科技有限责任公司 | Vibration mixing apparatus control method and system based on Internet of Things |
| CN109639790A (en) * | 2018-12-06 | 2019-04-16 | 上海美亦健健康管理有限公司 | A kind of distributed Internet of Things software architecture |
| CN110275771A (en) * | 2018-03-15 | 2019-09-24 | 中国移动通信集团有限公司 | A kind of method for processing business, Internet of Things billing infrastructure system and storage medium |
| CN110336851A (en) * | 2019-05-06 | 2019-10-15 | 腾讯科技（深圳）有限公司 | Access to content processing method, device, computer equipment and storage medium |
| CN110688391A (en) * | 2019-09-17 | 2020-01-14 | 中盈优创资讯科技有限公司 | Massive Internet of things terminal query system, cloud database and side controller |
| CN111177765A (en) * | 2020-01-06 | 2020-05-19 | 广州知弘科技有限公司 | Financial big data processing method, storage medium and system |
| CN111611266A (en) * | 2019-02-22 | 2020-09-01 | 通用电气公司 | Knowledge-driven joint big data query and analysis platform |
| CN111665773A (en) * | 2020-07-02 | 2020-09-15 | 长沙钛合电子设备有限公司 | Internet of things system and construction method |
| CN112214469A (en) * | 2020-10-29 | 2021-01-12 | 北京红山信息科技研究院有限公司 | Drive test data processing method, device, server and storage medium |
| CN112688921A (en) * | 2020-12-09 | 2021-04-20 | 浙江蓝卓工业互联网信息技术有限公司 | Industrial data acquisition system |
| CN113253685A (en) * | 2021-05-31 | 2021-08-13 | 航天中认软件测评科技（北京）有限责任公司 | Industrial data acquisition method, device, equipment and medium |
| CN113791729A (en) * | 2021-08-11 | 2021-12-14 | 合肥先进产业研究院 | Dynamic capacity expansion method for monitoring data storage of internet of things equipment |
| CN113806611A (en) * | 2020-06-17 | 2021-12-17 | 海信集团有限公司 | Method and equipment for storing search engine results |
| CN114162106A (en) * | 2021-12-24 | 2022-03-11 | 大秦铁路股份有限公司科学技术研究所 | Intelligent monitoring system and method for heavy-duty vehicle braking |
| CN116738157A (en) * | 2023-08-09 | 2023-09-12 | 柏森智慧空间科技集团有限公司 | Method for preprocessing data in property management platform |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103678665A (en) * | 2013-12-24 | 2014-03-26 | 焦点科技股份有限公司 | Heterogeneous large data integration method and system based on data warehouses |
| CN104410662A (en) * | 2014-10-23 | 2015-03-11 | 山东大学 | Parallel mass data transmitting middleware of Internet of things and working method thereof |
| CN105868395A (en) * | 2016-04-19 | 2016-08-17 | 武汉邮电科学研究院 | Event driven based smart city big data system and processing method |
| CN104820670B (en) * | 2015-03-13 | 2018-11-06 | 华中电网有限公司 | A kind of acquisition of power information big data and storage method |
-
2016
- 2016-08-31 CN CN201610797518.1A patent/CN106227899A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103678665A (en) * | 2013-12-24 | 2014-03-26 | 焦点科技股份有限公司 | Heterogeneous large data integration method and system based on data warehouses |
| CN104410662A (en) * | 2014-10-23 | 2015-03-11 | 山东大学 | Parallel mass data transmitting middleware of Internet of things and working method thereof |
| CN104820670B (en) * | 2015-03-13 | 2018-11-06 | 华中电网有限公司 | A kind of acquisition of power information big data and storage method |
| CN105868395A (en) * | 2016-04-19 | 2016-08-17 | 武汉邮电科学研究院 | Event driven based smart city big data system and processing method |
Cited By (30)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107016056A (en) * | 2017-03-07 | 2017-08-04 | 西安电子科技大学 | The distributed memory system and method for magnanimity heterogeneous sensor data in a kind of Internet of Things |
| CN106845202B (en) * | 2017-03-09 | 2020-06-02 | 北京旷视科技有限公司 | File access method, device and system for face recognition system |
| CN106845202A (en) * | 2017-03-09 | 2017-06-13 | 北京旷视科技有限公司 | File access method, device and system for face identification system |
| CN108197261A (en) * | 2017-12-30 | 2018-06-22 | 北京通途永久科技有限公司 | A kind of wisdom traffic operating system |
| CN110275771A (en) * | 2018-03-15 | 2019-09-24 | 中国移动通信集团有限公司 | A kind of method for processing business, Internet of Things billing infrastructure system and storage medium |
| CN110275771B (en) * | 2018-03-15 | 2021-12-14 | 中国移动通信集团有限公司 | Service processing method, Internet of things charging infrastructure system and storage medium |
| CN108650292A (en) * | 2018-03-27 | 2018-10-12 | 吉旗（成都）科技有限公司 | A kind of internet of things equipment message can quick locating query storage method |
| CN108416067A (en) * | 2018-03-29 | 2018-08-17 | 重庆大学 | Mass data processing and the optimization of storing process execute evaluation method in industrial process |
| CN108536823A (en) * | 2018-04-10 | 2018-09-14 | 北京工业大学 | A kind of caching design and querying method of Internet of Things perception big data |
| CN108536823B (en) * | 2018-04-10 | 2022-02-15 | 北京工业大学 | Cache design and query method for sensing big data of Internet of things |
| CN108763534B (en) * | 2018-05-31 | 2019-10-18 | 北京百度网讯科技有限公司 | Method and apparatus for handling information |
| CN108763534A (en) * | 2018-05-31 | 2018-11-06 | 北京百度网讯科技有限公司 | Method and apparatus for handling information |
| CN109104710A (en) * | 2018-06-29 | 2018-12-28 | 湖北海纳天鹰科技发展有限公司 | It is a kind of based on the air quality of NBiot network and MQTT agreement inquiry and distributing device |
| CN109224999A (en) * | 2018-09-29 | 2019-01-18 | 湖北航鹏化学动力科技有限责任公司 | Vibration mixing apparatus control method and system based on Internet of Things |
| CN109639790A (en) * | 2018-12-06 | 2019-04-16 | 上海美亦健健康管理有限公司 | A kind of distributed Internet of Things software architecture |
| CN111611266A (en) * | 2019-02-22 | 2020-09-01 | 通用电气公司 | Knowledge-driven joint big data query and analysis platform |
| CN110336851A (en) * | 2019-05-06 | 2019-10-15 | 腾讯科技（深圳）有限公司 | Access to content processing method, device, computer equipment and storage medium |
| CN110688391A (en) * | 2019-09-17 | 2020-01-14 | 中盈优创资讯科技有限公司 | Massive Internet of things terminal query system, cloud database and side controller |
| CN111177765A (en) * | 2020-01-06 | 2020-05-19 | 广州知弘科技有限公司 | Financial big data processing method, storage medium and system |
| CN113806611A (en) * | 2020-06-17 | 2021-12-17 | 海信集团有限公司 | Method and equipment for storing search engine results |
| CN111665773B (en) * | 2020-07-02 | 2021-10-26 | 长沙钛合电子设备有限公司 | Internet of things system and construction method |
| CN111665773A (en) * | 2020-07-02 | 2020-09-15 | 长沙钛合电子设备有限公司 | Internet of things system and construction method |
| CN112214469A (en) * | 2020-10-29 | 2021-01-12 | 北京红山信息科技研究院有限公司 | Drive test data processing method, device, server and storage medium |
| CN112688921A (en) * | 2020-12-09 | 2021-04-20 | 浙江蓝卓工业互联网信息技术有限公司 | Industrial data acquisition system |
| CN113253685A (en) * | 2021-05-31 | 2021-08-13 | 航天中认软件测评科技（北京）有限责任公司 | Industrial data acquisition method, device, equipment and medium |
| CN113253685B (en) * | 2021-05-31 | 2021-09-24 | 航天中认软件测评科技（北京）有限责任公司 | Industrial data acquisition method, device, equipment and medium |
| CN113791729A (en) * | 2021-08-11 | 2021-12-14 | 合肥先进产业研究院 | Dynamic capacity expansion method for monitoring data storage of internet of things equipment |
| CN114162106A (en) * | 2021-12-24 | 2022-03-11 | 大秦铁路股份有限公司科学技术研究所 | Intelligent monitoring system and method for heavy-duty vehicle braking |
| CN114162106B (en) * | 2021-12-24 | 2024-05-03 | 大秦铁路股份有限公司科学技术研究所 | Intelligent monitoring system and method for braking of heavy-duty vehicle |
| CN116738157A (en) * | 2023-08-09 | 2023-09-12 | 柏森智慧空间科技集团有限公司 | Method for preprocessing data in property management platform |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106227899A (en) | The storage of the big data of a kind of internet of things oriented and querying method | |
| US9367574B2 (en) | Efficient query processing in columnar databases using bloom filters | |
| US10812589B2 (en) | Storage architecture for heterogeneous multimedia data | |
| US11093466B2 (en) | Incremental out-of-place updates for index structures | |
| Hu et al. | Toward scalable systems for big data analytics: A technology tutorial | |
| US9432430B1 (en) | Event stream processing device providing real-time incident identification | |
| CN103116661B (en) | A kind of data processing method of database | |
| CN105354151B (en) | Cache management method and equipment | |
| US11429630B2 (en) | Tiered storage for data processing | |
| US10061834B1 (en) | Incremental out-of-place updates for datasets in data stores | |
| US10860562B1 (en) | Dynamic predicate indexing for data stores | |
| CN105791431A (en) | On-line distributed monitoring video processing task scheduling method and device | |
| CN102880854B (en) | Distributed processing and Hash mapping-based outdoor massive object identification method and system | |
| CN102722405B (en) | Counting method and system in high concurrent and multithreaded application | |
| Mitra et al. | Stash: Fast hierarchical aggregation queries for effective visual spatiotemporal explorations | |
| Hoque et al. | Disk layout techniques for online social network data | |
| CN102012946A (en) | High-efficiency safety monitoring video/image data storage method | |
| US10776368B1 (en) | Deriving cardinality values from approximate quantile summaries | |
| Ma et al. | A retrieval optimized surveillance video storage system for campus application scenarios | |
| Yu et al. | A bigtable/mapreduce-based cloud infrastructure for effectively and efficiently managing large-scale sensor networks | |
| KR102002360B1 (en) | Method and Apparatus of building NoSQL database for signal processing | |
| Colace et al. | Pervasive systems architecture and the main related technologies | |
| Lu et al. | Research on Cassandra data compaction strategies for time-series data | |
| Lian et al. | Sql or nosql? which is the best choice for storing big spatio-temporal climate data? | |
| Shona et al. | A survey on the data management in IoT |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20161214 |