CN105930384A - Sensing cloud data storage system based on Hadoop system and implementation method thereof - Google Patents
Sensing cloud data storage system based on Hadoop system and implementation method thereof Download PDFInfo
- Publication number
- CN105930384A CN105930384A CN201610232078.5A CN201610232078A CN105930384A CN 105930384 A CN105930384 A CN 105930384A CN 201610232078 A CN201610232078 A CN 201610232078A CN 105930384 A CN105930384 A CN 105930384A
- Authority
- CN
- China
- Prior art keywords
- data
- cloud
- sensing
- hadoop
- real time
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2471—Distributed queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/283—Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
Abstract
The invention discloses a sensing cloud data storage system based on the Hadoop system and an implementation method thereof. The sensing cloud data storage system based on the Hadoop system is provided by analyzing the features and application scenarios of Storm framework, HBase distributed data table, and distributed data warehouse Hive under the Hadoop framework and in combination with the sensing clouds technology formed by wireless sensor networks. Through analyzing three common application scenarios under the Hadoop system, a data cloud gateway, formed by upper computer software, data buffer pool MYSQL database and task scheduling scripts, is used as a data communication bridge between the sensing cloud and a storage system. The scheduling scripts are responsible for front end data dissemination and response by pushing daily real-time data to the Storm framework cluster and writing the real-time data into the HBase table in real time; historical data is operated by ETL data through being extracted from the buffer pool and loaded into database of the Hive. The sensing cloud data storage system has clear structure and good level expansibility and load balancing stability; modules have distinct functions; and the system has been on stable operation and provides data storage support for other systems.
Description
Technical field
The present invention relates to the sensing cloud field of data storage systems of Hadoop system, specifically a kind of based on Hadoop race large-scale sensor safe data cloud storage system and its implementation.
Technical background
Wireless sensor network is the most ripe and universal, creates substantial amounts of sensing data.Traditional wireless sensor network gradually combines with remote internet communication technology and Hadoop technology in development, forms wireless sensing cloud system.Hadoop is a kind of distributed storage with high reliability and good autgmentability and calculates platform, can form the storage such as cluster service data file and calculate on the basis of the most cheap hardware device.Based on Hadoop theory and technology, a series of Hadoop system product for different scenes is had been developed.Such as distributed data base HBase, Distributed Data Warehouse Hive, the platform Storm of calculating in real time etc. of class Hadoop.
Hadoop system has been provided with the multiple application being adapted to different application scene, and these application all possess large-scale data management and the feature of Distributed Calculation, and certain applications are used in every profession and trade.But, owing to the sensing cloud genera of wireless sensor network composition is in emerging data source, the most universal and generation mass data, therefore this technology is integrated also in experimental stage with what Hadoop system was applied.When processing the data that sensing cloud is data source, many number systems still use these part data of traditional relational data library storage.
Processing the data that sensing cloud produces, traditional method uses data gateway to receive data, and stores relevant database, disclosure satisfy that sensing cloud data storage and the requirements for access of system demonstration in a short time.But, owing to sensing cloud data generation frequency is high, speed is fast.The accumulating rate of sensing data amount is surprising.Along with the rising of data volume, the storage of relevant database and access performance can be gradually lowered, system adapts to less able.It addition, traditional database autgmentability is poor, run the relatively costly of laggard row data base extension in system.Relatively, the core concept of autgmentability and reliability Hadoop just, sensing cloud and the application of Hadoop system are carried out the integrated data storage processing that disclosure satisfy that needed for rapid data accumulation and extension demand.
Summary of the invention
It is an object of the invention to provide a kind of based on Hadoop system Hive, the sensing cloud data-storage system implementation method of HBase, Storm application, thus tackle data storage autgmentability and the durability requirements that wireless sensor network data amount Rapid Accumulation produces.
1, the technical solution realizing the object of the invention is: a kind of sensing cloud data-storage system based on Hadoop system, system is with data cloud gateway as core, and the sensing cloud, data cloud gateway, distributed data base HBase, the Storm framework of real time data calculating, Hive data warehouse and the storage of bottom Hadoop foundation distributing that are made up of wireless sensor network calculate supports that platform 6 part is constituted；
Sensing cloud, is used for gathering environmental data；It is made up of wireless sensor network, network comprises the sensor group equipment of the quantity of illumination of acquisition equipment collection point, aerial temperature and humidity and soil temperature and humidity and carbon dioxide content；
Data cloud gateway, is made up of upper computer software, Data buffer MYSQL database and the task scheduling script run on it；Wherein, upper computer software receives that sensing cloud produces and be transferred to data cloud gateway by GPRS module real time data, and then resolve the IP packet comprising sensing cloud environment data, obtain sensing concrete nodal information and environmental sensor data in cloud and be stored in Data buffer MYSQL database；
The Storm framework that real time data calculates, it is achieved real time data is to the write operation of distributed data base HBase；
Distributed data base HBase, is used for storing real time environmental data, with sensor name as key, sensor real time data for value, and with key for standard partitioned storage key-value pair to HBase real time data table in；
Data warehouse Hive, it is achieved the storage of historical data；In data cloud gateway, the environmental data that host computer resolves is temporarily stored in Buffer Pool, when data accumulation is to specifying duration, and task scheduling script performs ETL operation, historical data is deposited in the corresponding table in data Hive warehouse；
The storage of Hadoop foundation distributing calculates supports that platform provides support service.
The present invention compared with prior art, its remarkable advantage: 1, the present invention by wireless sensor network composition sensing cloud, set up communication link with data gateway, it is ensured that sensing cloud data transmit sustainedly and stably from data source to Buffer Pool；2, the present invention is centered by data cloud gateway, integrated real-time Computational frame Storm based on Hadoop system, it is ensured that the write of real time data and process operational stability and low time delay, improves the response speed to real time data demand；3, the present invention is by the HBase tables of data storage real time data under Hadoop system, supports high speed writein and the read operation of Storm.Extending space is provided for sensing cloud data transmission bauds；4, the present invention is around the Data buffer of cloud data gateway, uses the Hive application under Hadoop system as data warehouse as history data store target.Achieve the extensive storage capacity autgmentability sensing cloud data, improve the reliability of magnanimity sensing data storage.
Accompanying drawing explanation
Fig. 1 is present invention sensing based on Hadoop system cloud data-storage system.
Fig. 2 is present invention sensing based on Hadoop system cloud data-storage system flow chart of data processing figure.
Detailed description of the invention
By analyzing Storm framework, HBase distributed data table, the feature of Distributed Data Warehouse Hive and the respective application scenarios under Hadoop framework, the sensing cloud that combining with wireless sensor network is formed, the present invention proposes a kind of sensing cloud data-storage system based on Hadoop system.The present invention, by the analysis of the common application scenarios of these three under Hadoop system, uses by upper computer software, and the data cloud gateway of Data buffer MYSQL database and task scheduling script composition is as the data communication bridge of sensing cloud to storage system.Real time data on the same day is write HBase table by being pushed to Storm framework cluster by scheduling script in real time, is responsible for front end data and issues and response；Historical data passes through ETL data manipulation, extracts and be loaded into the data warehouse of Hive from Buffer Pool.
1, the system of the present invention is with data cloud gateway as core, the sensing cloud being made up of wireless sensor network, data cloud gateway, distributed data base HBase, the Storm framework that real time data calculates, data warehouse Hive application and the storage of bottom Hadoop foundation distributing calculate supports that platform 6 part is constituted；
2, sensing cloud is the data production source of native system.Combine GPRS Radio Transmission Technology by common wireless sensor network to constitute.Use when native system realizes and communicate between SMAC protocol realization sensing cloud network node.Node edge disposes GPRS module, by the data of sensor acquisition with 5 seconds time delays to the IP packet of data cloud gateway transmission data message.
3, the upper computer software that data cloud gateway is operated above by it, Data buffer MYSQL database and task scheduling script are for constituting part.Data cloud gateway is to link up Distributed Storage and process the bridge of function and sensing cloud data source, is that native system realizes data receiver from sensing cloud data source and carries out the core of history data store and real time data respectively.Its effect is: run host computer, with GPRS module communication, receives the real time data of self-sensing cloud and stores Data buffer.Data are carried out the real-time and distributed storage of historical data and process by scheduling script respectively.
It is as follows that the host computer related in 3 realizes flow process:
The first step: run TCP thread and carry out specifying intercepting of COM1.
Second step: set up communication link with the sensing cloud GPRS module asking on port to be connected, receives the sensing data packet of self-sensing cloud.
3rd step: the packet received is unpacked operation by host computer.Obtain the detailed data information of each node and sensor thereof, and be stored in Data buffer.Buffer Pool uses MYSQL mini-relational type data base to undertake.It is responsible for the sensing data that interim storing and resolving packet obtains.
It is as follows that the scheduling script related in 3 realizes flow process:
The first step: be daily that the unit real time data respectively to Buffer Pool carries out push operation, historical data carried out data pick-up, changes and load operation.
For the real time data processing in the first step, its step is as follows:
Step 3.1: for the sensing data parsed the same day, scheduling script is hour to carry out push operation for granularity；
Step 3.2: real time data pushes HBase platform and carries out distributed storage, treats that Storm framework processes and calls；
Step 3.3:Storm framework reads real time data, the real-time computation requests of response Web site server from HBase database table.As calculated when previous hour interior quantity of illumination average, medial humidity tendency etc. in a day.
4, the storage of real time data and calculating are respectively adopted under Hadoop system HBase distributed data base and the distributed real-time calculating platform of Storm realize.HBase is the distributed non-relational database towards row, uses<key, value>key-value pair based on column storage.Its good real-time support and row storage mode adapt to sensor coordination and the real-time data memory scene of sensing cloud data model.Native system is dispatched script and stores real-time sensing cloud data by the hour.With sensor name as key, sensor real time data for value, and with key for standard partitioned storage key-value pair to HBase table in.
5, real-time data memory realizes write in real time by Storm in HBase.Storm undertakes real time data write HBase table in the present system and reads the function that HBase table data calculate in real time.Storm is real-time Computational frame based on Hadoop system, is different from the batch facility of Hadoop.This framework realizes flow data in the present system and processes, and is used in low time delay, quickly the front end of the process in real time web site requests scene of response.
Flow process is realized alternately for the Storm framework mentioned in 5 as follows with HBase:
The first step: the scheduling script of data cloud gateway have submitted N bar record and is updated behaviour and does；
Second step: N bar is divided into 10 parts, every part of N/10 bar；
3rd step: each Java Virtual Machine JVM example can build a thread pool having 10 threads, by ThreadLocal, each thread in thread pool can safeguard that a Connection connects；
4th step: thread can carry out row from increasing to this N/10 data order of oneself, it is achieved the connection pool of HTable table in HBase is shared the optimization of connection.Alleviate the pressure of cluster management and load balancing operation.
6, the Distributed Data Warehouse Hive under history data store uses Hadoop system combines ETL data processing operation and realizes.Hive possesses good data management level, distributed autgmentability and reliability.ETL operation relies on script to realize extracting the sensing cloud data of proxima luce (prox. luc) from the MYSQL Data buffer of data cloud gateway, and changes according to Hive data structure, and then is loaded in the table that this data warehouse is corresponding, as the storage target of historical data.The Deby data base of the metadata storage Indexed Dependencies clustered deploy(ment) of Hive, its physical store relies on Hadoop distributed memory system HDFS, and storing process relies on the Distributed Calculation engine MapReduce of Hadoop.
It is as follows that history data store in 6 realizes flow process:
The first step: the relational data table of storage in ETL data manipulation script extraction cloud gateway buffers pond MYSQL.ETL is data pick-up, conversion, the overall process of loading.In the present system by the Metadata Extraction of MYSQL to Hive；
Second step: metadata sets up Metadata View The in the built-in metadatabase Deby of Hive；
The data file of the 3rd step: ETL extraction stores in distributed file system HDFS of Hadoop platform；
4th step: starting the MapReduce computing engines of Hadoop platform, set up corresponding data table and load the meta data file being stored in HDFS in Hive, storing process completes a map-reduce operation.
4, relate in 5,6 undertake real time data and application service that history data store processes belongs to Hadoop system.Hadoop is the distributed system architecture being made up of a large amount of computers, has preferable extensibility.Native system mainly uses distributed file system HDFS and two basic function module of Distributed Calculation engine MapReduce of this framework.Along with the development of Hadoop framework, its ecosphere is the most perfect.Native system employs the Hadoop system periphery software Storm framework supporting that real-time streams calculates, and supports HBase tables of data and the data warehouse Hive application of the storage of train value data.
The system architecture of 1 couple of present invention is described further below in conjunction with the accompanying drawings.
Sensing cloud (1) is that the data of this system produce source, wireless sensor network integrated GPRS communication module formed.Sensing cloud is internal uses SMAC agreement intercommunication, and the sensing data collected by each node (2) to via node (3), and then is encapsulated data into IP packet by GPRS module by SMAC protocol transmission, is sent to long-range cloud data gateway (4).
Data cloud gateway (4) is responsible for reception and is carried out the packet of self-sensing cloud (1), receive IP by its upper computer software run (5) to be grouped, parse corresponding nodal information and sensing data in sensing cloud, this data correspondence is stored in the database Buffer Pool (6) on gateway.When scheduling script checks integral point non-zero points, it is unit by the hour by real time data, is pushed in the cluster (7) of Storm framework.Storm to specifications in 5 flow processs that realize related to data are write in real time HBase tables of data (8).The now storage of real time data has write, and the real time data being stored in HBase can carry out front end for Website server (9) and call and realize visual control process.This part mainly undertakes the storing process of real time data write HBase table.
The upper scheduling script run of data cloud gateway (4) checks zero point.Call ETL data manipulation script, start the day data of working as in extracted data Buffer Pool (6), be loaded in Hive Distributed Data Warehouse (10) after conversion data form.Hive data warehouse uses built-in Deby to store index of metadata, the distributed file system HDFS module that its physical store provides based on Hadoop cluster (11), the Distributed Calculation engine MapReduce operation that process provides based on Hadoop cluster (11) from the beginning of the year realizes.This part is mainly responsible for the historical data storing process to data warehouse.
Sensing cloud data-storage system flow chart of data processing as based on Hadoop system in Fig. 2.
Sensing cloud production environment data, environmental data is packaged into IP packet, is transferred to data cloud gateway by GPRS module.Data cloud gateway receives IP packet, and the data of parsing are stored in the MYSQL Data buffer operated on data cloud gateway.After task scheduling script judges that system time arrives zero point, start ETL historical data extraction script, in the data of extraction Buffer Pool to Hive data warehouse.If system time is the integral point time of non-zero points, pushes data into Storm cluster and carry out data cleansing, and then store in distributed data base HBase of real-time response.
Claims (8)
1. a sensing cloud data-storage system based on Hadoop system, it is characterized in that: system is with data cloud gateway as core, and the sensing cloud, data cloud gateway, distributed data base HBase, the Storm framework of real time data calculating, Hive data warehouse and the storage of bottom Hadoop foundation distributing that are made up of wireless sensor network calculate supports that platform 6 part is constituted；
Sensing cloud, is used for gathering environmental data；It is made up of wireless sensor network, network comprises the sensor group equipment of the quantity of illumination of acquisition equipment collection point, aerial temperature and humidity and soil temperature and humidity and carbon dioxide content；
Data cloud gateway, is made up of upper computer software, Data buffer MYSQL database and the task scheduling script run on it；Wherein, upper computer software receives that sensing cloud produces and be transferred to data cloud gateway by GPRS module real time data, and then resolve the IP packet comprising sensing cloud environment data, obtain sensing concrete nodal information and environmental sensor data in cloud and be stored in Data buffer MYSQL database；
The Storm framework that real time data calculates, it is achieved real time data is to the write operation of distributed data base HBase；
Distributed data base HBase, is used for storing real time environmental data, with sensor name as key, sensor real time data for value, and with key for standard partitioned storage key-value pair to HBase real time data table in；
Data warehouse Hive, it is achieved the storage of historical data；In data cloud gateway, the environmental data that host computer resolves is temporarily stored in Buffer Pool, when data accumulation is to specifying duration, and task scheduling script performs ETL operation, historical data is deposited in the corresponding table in data Hive warehouse；
The storage of Hadoop foundation distributing calculates supports that platform provides support service.
Sensing cloud data-storage system based on Hadoop system the most according to claim 1, it is characterized in that: described Storm framework is that the real time data of class Hadoop framework calculates platform, for processing real-time stream, the environmental data that host computer resolves IP packet acquisition is written in distributed data base HBase in real time.
Sensing cloud data-storage system based on Hadoop system the most according to claim 1, it is characterised in that: the real time data in described distributed data base HBase is used for providing visual presentation.
Sensing cloud data-storage system based on Hadoop system the most according to claim 1, it is characterized in that: the storage of described Hadoop foundation distributing calculates the file storage basis that the HDFS function of support platform is Hive data warehouse and HBase distributed data base, and its MapReduce function is the practical operation engine of Hive and HBase.
5. the implementation method of a sensing cloud data-storage system based on Hadoop system, it is characterised in that:
(1) the sensing cloud being made up of wireless sensor network obtains the quantity of illumination of collecting device point, air and soil temperature and humidity environmental data, is wirelessly transmitted to data cloud gateway by its GPRS module；
(2) data cloud gateway is made up of upper computer software, Data buffer MYSQL database and task scheduling script；Host computer receives the IP grouped data of self-sensing cloud, obtains sensing in cloud concrete nodal information and environmental sensor data and be stored in the Data buffer that MYSQL database is served as after parsing；
(3) detection operation is carried out during the scheduling script integral point of data cloud gateway；Dispatch ETL data pick-up script during zero point and be organized in data cloud gateway buffers pond temporary this period environmental data carrying out self-sensing cloud to data warehouse Hive, during non-zero points, propelling data is to Storm framework, it is achieved real time data is to the write operation of distributed data base HBase.
The implementation method of sensing cloud data-storage system based on Hadoop system the most according to claim 5, it is characterised in that: use described in step (3) Storm framework realize the real time data write operation to distributed data base HBase to realize flow process as follows:
The first step: the scheduling script of data cloud gateway have submitted N bar record and is updated behaviour and does；
Second step: N bar is divided into 10 parts, every part of N/10 bar；
3rd step: each Java Virtual Machine JVM example builds a thread pool having 10 threads, by ThreadLocal, each thread in thread pool safeguards that a Connection connects；
4th step: thread can carry out row from increasing to this N/10 data order of oneself, it is achieved the connection pool of HTable table in HBase is shared the optimization of connection.
The implementation method of sensing cloud data-storage system based on Hadoop system the most according to claim 5, it is characterised in that: this period environmental data carrying out self-sensing cloud temporary during ETL data pick-up script is organized in data cloud gateway buffers pond described in step (3) to realize flow process as follows:
The first step: daily respectively the real time data of Buffer Pool is carried out push operation for unit；
Second step: historical data is carried out data pick-up, changes and loads operation.
8. according to the implementation method of the sensing cloud data-storage system based on Hadoop system described in claim 5 or 7, it is characterised in that: described in the first step described in step (3) and second step in real time and the concretely comprising the following steps of historical data process:
Step 1: for the sensing data parsed the same day, scheduling script is hour to carry out push operation for granularity；
Step 2: real time data is inserted HBase data base and carried out distributed storage, treats that Storm framework processes and calls；
The table that step 3:Storm framework is corresponding from HBase distributed data base reads real time data, the real-time computation requests of response Web site server.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610232078.5A CN105930384A (en) | 2016-04-14 | 2016-04-14 | Sensing cloud data storage system based on Hadoop system and implementation method thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610232078.5A CN105930384A (en) | 2016-04-14 | 2016-04-14 | Sensing cloud data storage system based on Hadoop system and implementation method thereof |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN105930384A true CN105930384A (en) | 2016-09-07 |
Family
ID=56838140
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610232078.5A Pending CN105930384A (en) | 2016-04-14 | 2016-04-14 | Sensing cloud data storage system based on Hadoop system and implementation method thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105930384A (en) |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105938487A (en) * | 2016-04-14 | 2016-09-14 | 北京思特奇信息技术股份有限公司 | Application method and apparatus based on Hbase connection pool |
| CN106445790A (en) * | 2016-10-12 | 2017-02-22 | 北京集奥聚合科技有限公司 | Counting and account-checking method and device used in distributed real-time computing system |
| CN106951442A (en) * | 2017-02-15 | 2017-07-14 | 中国保险信息技术管理有限责任公司 | Data interactive method and device between a kind of heterogeneous database |
| CN107070590A (en) * | 2016-12-30 | 2017-08-18 | 南京海道普数据技术有限公司 | The distributed coding/decoding method of WSN perception datas based on MapReduce |
| CN107229673A (en) * | 2017-04-20 | 2017-10-03 | 努比亚技术有限公司 | Method for writing data, Hbase terminals and the storage medium of Hbase databases |
| CN108268565A (en) * | 2017-01-04 | 2018-07-10 | 北京京东尚科信息技术有限公司 | Method and system based on data warehouse processing user browsing behavior data |
| CN108446145A (en) * | 2018-03-21 | 2018-08-24 | 苏州提点信息科技有限公司 | A kind of distributed document loads MPP data base methods automatically |
| PL423219A1 (en) * | 2017-10-20 | 2019-04-23 | Politechnika Slaska Im Wincent | Method for extraction and transformation of stream-oriented measuring data, using the parallel computing |
| CN110113398A (en) * | 2019-04-23 | 2019-08-09 | 安徽云融信息技术有限公司 | A kind of large-scale wireless sensing network data-storage system based on HBase |
| CN110837509A (en) * | 2019-11-08 | 2020-02-25 | 深圳市彬讯科技有限公司 | Method, device, equipment and storage medium for scheduling dependence |
| CN111131014A (en) * | 2020-01-22 | 2020-05-08 | 北方工业大学 | Internet of things gateway |
| CN112256226A (en) * | 2020-10-22 | 2021-01-22 | 广州市金其利信息科技有限公司 | Wireless screen expansion method and system |
| CN112700622A (en) * | 2020-12-21 | 2021-04-23 | 中铁二院工程集团有限责任公司 | Storm-based railway geological disaster monitoring big data preprocessing method and system |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104184785A (en) * | 2013-09-12 | 2014-12-03 | 中国林业科学研究院资源信息研究所 | Forest Internet of Things system based on cloud platform |
| CN104601665A (en) * | 2014-12-22 | 2015-05-06 | 西安电子科技大学 | System and method for real-time cloud simulation on Internet of things sensing device |
| US20160092548A1 (en) * | 2014-09-26 | 2016-03-31 | Oracle International Corporation | System and method for consistent reads between tasks in a massively parallel or distributed database environment |
-
2016
- 2016-04-14 CN CN201610232078.5A patent/CN105930384A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104184785A (en) * | 2013-09-12 | 2014-12-03 | 中国林业科学研究院资源信息研究所 | Forest Internet of Things system based on cloud platform |
| US20160092548A1 (en) * | 2014-09-26 | 2016-03-31 | Oracle International Corporation | System and method for consistent reads between tasks in a massively parallel or distributed database environment |
| CN104601665A (en) * | 2014-12-22 | 2015-05-06 | 西安电子科技大学 | System and method for real-time cloud simulation on Internet of things sensing device |
Cited By (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105938487A (en) * | 2016-04-14 | 2016-09-14 | 北京思特奇信息技术股份有限公司 | Application method and apparatus based on Hbase connection pool |
| CN106445790A (en) * | 2016-10-12 | 2017-02-22 | 北京集奥聚合科技有限公司 | Counting and account-checking method and device used in distributed real-time computing system |
| CN107070590A (en) * | 2016-12-30 | 2017-08-18 | 南京海道普数据技术有限公司 | The distributed coding/decoding method of WSN perception datas based on MapReduce |
| CN108268565B (en) * | 2017-01-04 | 2020-11-03 | 北京京东尚科信息技术有限公司 | Method and system for processing user browsing behavior data based on data warehouse |
| CN108268565A (en) * | 2017-01-04 | 2018-07-10 | 北京京东尚科信息技术有限公司 | Method and system based on data warehouse processing user browsing behavior data |
| CN106951442A (en) * | 2017-02-15 | 2017-07-14 | 中国保险信息技术管理有限责任公司 | Data interactive method and device between a kind of heterogeneous database |
| CN107229673A (en) * | 2017-04-20 | 2017-10-03 | 努比亚技术有限公司 | Method for writing data, Hbase terminals and the storage medium of Hbase databases |
| PL423219A1 (en) * | 2017-10-20 | 2019-04-23 | Politechnika Slaska Im Wincent | Method for extraction and transformation of stream-oriented measuring data, using the parallel computing |
| CN108446145A (en) * | 2018-03-21 | 2018-08-24 | 苏州提点信息科技有限公司 | A kind of distributed document loads MPP data base methods automatically |
| CN110113398A (en) * | 2019-04-23 | 2019-08-09 | 安徽云融信息技术有限公司 | A kind of large-scale wireless sensing network data-storage system based on HBase |
| CN110837509A (en) * | 2019-11-08 | 2020-02-25 | 深圳市彬讯科技有限公司 | Method, device, equipment and storage medium for scheduling dependence |
| CN111131014A (en) * | 2020-01-22 | 2020-05-08 | 北方工业大学 | Internet of things gateway |
| CN112256226A (en) * | 2020-10-22 | 2021-01-22 | 广州市金其利信息科技有限公司 | Wireless screen expansion method and system |
| CN112700622A (en) * | 2020-12-21 | 2021-04-23 | 中铁二院工程集团有限责任公司 | Storm-based railway geological disaster monitoring big data preprocessing method and system |
| CN112700622B (en) * | 2020-12-21 | 2022-05-17 | 中铁二院工程集团有限责任公司 | Storm-based railway geological disaster monitoring big data preprocessing method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105930384A (en) | Sensing cloud data storage system based on Hadoop system and implementation method thereof | |
| CN110784419B (en) | Method and system for visualizing professional railway electric service data | |
| US20220156319A1 (en) | Systems and methods for video archive and data extraction | |
| US11064053B2 (en) | Method, apparatus and system for processing data | |
| CN109951463A (en) | A kind of Internet of Things big data analysis method stored based on stream calculation and novel column | |
| CN104252536B (en) | A kind of internet log data query method and device based on hbase | |
| CN108021809A (en) | A kind of data processing method and system | |
| CN103761309A (en) | Operation data processing method and system | |
| CN107332719A (en) | A kind of method that daily record is analyzed in real time in CDN system | |
| CN103268336A (en) | Fast data and big data combined data processing method and system | |
| CN106951552A (en) | A kind of user behavior data processing method based on Hadoop | |
| CN104778225A (en) | Method for synchronizing data in unstructured data multi-storage system | |
| CN111258978B (en) | Data storage method | |
| CN107343021A (en) | A kind of Log Administration System based on big data applied in state's net cloud | |
| CN103260050A (en) | Video-on-demand system based on Google App Engine Cloud platform | |
| CN105338113A (en) | Multi-platform data interconnected system for sharing urban data resources | |
| CN103207920A (en) | Parallel metadata acquisition system | |
| CN109947729B (en) | Real-time data analysis method and device | |
| CN109063158A (en) | A kind of method, equipment, system and the medium of the inquiry of website visiting ranking information | |
| CN104699757A (en) | Distributed network information acquisition method in cloud environment | |
| CN108573029A (en) | A kind of method, apparatus and storage medium obtaining network access relational data | |
| CN103034650A (en) | System and method for processing data | |
| Buddhika et al. | Living on the edge: Data transmission, storage, and analytics in continuous sensing environments | |
| Luo et al. | Big-data analytics: challenges, key technologies and prospects | |
| CN106570151A (en) | Data collection processing method and system for mass files |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20160907 |