CN105138673A - Mass storage system and method based on Internet of Things - Google Patents
Mass storage system and method based on Internet of Things Download PDFInfo
- Publication number
- CN105138673A CN105138673A CN201510566048.3A CN201510566048A CN105138673A CN 105138673 A CN105138673 A CN 105138673A CN 201510566048 A CN201510566048 A CN 201510566048A CN 105138673 A CN105138673 A CN 105138673A
- Authority
- CN
- China
- Prior art keywords
- data
- node
- stored
- flow process
- gateway
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
- G06F16/1824—Distributed file systems implemented using Network-attached Storage [NAS] architecture
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a mass storage system and method based on the Internet of Things. The system comprises a plurality of high-speed caching nodes used for storing hot spot data, a plurality of database nodes and a distributed file system, wherein each database node comprises a main server and a plurality of backup servers and is used for storing moderate data, and the distributed file system is used for storing cold data. The method comprises the steps that the hot spot data are stored in the high-speed caching nodes; the moderate data are horizontally split through a gateway Id and are stored in the main servers and the backup servers of the database nodes respectively; the cold data are stored in the distributed file system; the processes are not subjected to a sequence and can also be carried out simultaneously. The mass storage system and method based on the Internet of Things support horizontal extension of mass data, support horizontal extension of data processing and are very high in storage and processing speed.
Description
Technical field
The present invention relates to technology of Internet of things field, particularly a kind of mass storage system (MSS) based on Internet of Things and storage means.
Background technology
Internet of Things be one based on information carriers such as internet, conventional telecommunications nets, allow the network that all general physics object implementatio8 that can be independently addressable interconnect.
Gateway is also known as gateway, protocol converter.Gateway, on the level of the transport layer to realize network interconnection, is the most complicated network interconnection apparatus, network interconnection, only for network interconnection that two upper-layer protocols are different.Things-internet gateway is generally used for the protocol conversion between Internet of Things and internet.Under 100,000 gateways and 100,000 App user's scenes, p.s., gateway reported 1167 next states to upgrade, and triggered 3333 interlocks p.s..In 100,000 gateway situations, within 1 year, just have the data of 36T, data volume is very huge, and along with the development of business, also going from strength to strength, data processing amount is very large.The horizontal extension of existing data-storage system not supported data, the horizontal extension of not supported data process, processing power is very limited, and processing speed is slow.
Summary of the invention
For defect of the prior art, the object of this invention is to provide a kind of mass storage system (MSS) based on Internet of Things and storage means.
The present invention is achieved by the following technical solutions:
The invention provides a kind of mass storage system (MSS) based on Internet of Things, it comprises: multiple cache nodes, for storing hot spot data;
Multiple database node, each described database node comprises a master server and multiple standby server, for storing gentle data;
Distributed file system, for storing cold data.
Preferably, described cache nodes is Redis node.
Preferably, described database node is MySQL node.
Preferably, described distributed computational nodes is HDinsight node.
The present invention also provides a kind of mass memory method based on Internet of Things, and it comprises following flow process:
F11: hot spot data is stored in cache nodes;
F12: gentle data carry out horizontal cutting by gateway Id, is stored in a master server in each database node and multiple standby server respectively;
F13: cold data are stored in distributed file system;
Above-mentioned flow process is order in no particular order, also can carry out simultaneously.
Preferably, described flow process F11 is specially: hot spot data is stored in Redis node by consistent hashing algorithm.
Preferably, described flow process F12 is specially: gentle data carry out horizontal cutting by gateway Id, the equipment relevant with gateway is also according to the horizontal cutting of gateway Id, and user and App do not carry out cutting, stores portion respectively in a master server in each database node and multiple standby server.
Preferably, described flow process F12 also comprises: database node carries out subregion according to per schedule time and deposits.
Preferably, described flow process F13 is specially: cold data realize MapReduce algorithm by HDinsight, are stored in the blockblob of Azure.
Compared to prior art, the present invention has the following advantages:
Mass storage system (MSS) storage means based on Internet of Things provided by the invention, in conjunction with Redis high-speed cache, the horizontal cutting of Mysql database, and HDinsight realizes Distributed Calculation, thus solve the mass memory problem of Internet of Things;
(1), Redis cache nodes can not shut down linear expansion, newly-increased or reduce node, do not affect the data of buffer memory;
(2), because pipelined data is all relevant with gateway, both ensure that the validity of cutting, also achieve the simplicity of cutting;
3), cold Data distribution8 formula statistic algorithm and real-time system separately, thus ensure real-time system performance.
Certainly, implement arbitrary product of the present invention might not need to reach above-described all advantages simultaneously.
Accompanying drawing explanation
Below in conjunction with accompanying drawing, embodiments of the present invention are described further:
Fig. 1 is the Organization Chart of Redis node of the present invention;
Fig. 2 is the Organization Chart of database node of the present invention;
Fig. 3 is the process flow diagram of distributed statistic algorithm of the present invention.
Symbol description:
1-Redis node, 2-database node;
11-the one Redis node, 12-the 2nd Redis node;
21-first database node, 22-second database node, 23-the 3rd database node;
The main Redis node of 111-first, 112-first for Redis node, the main Redis node of 121-second, 122-second is for Redis node;
211-first master server, 212-first is for server, and 221-second master server, 222-second is for server, and 231-the 3rd master server, 232-the 3rd is for server.
Embodiment
Elaborate to embodiments of the invention below, the present embodiment is implemented under premised on technical solution of the present invention, give detailed embodiment and concrete operating process, but protection scope of the present invention is not limited to following embodiment.
Embodiment 1:
Composition graphs 1-Fig. 2, the present embodiment describes the mass storage system (MSS) based on Internet of Things of the present invention in detail, and it comprises: multiple Redis node, and each Redis node realizes host stores key-value, stores for hot spot data; Client, by consistent hashing algorithm, is distributed on different nodes data cached, and ensures newly-increased or reduce node, does not affect the data of buffer memory.When machine delayed by main frame, standby host is from moving up, and as shown in Figure 1, this sentences two Redis nodes 1 for example to its structure, be respectively a Redis node 11 and the 2nd Redis node 12, wherein: a Redis node 11 comprises a Redis host node 111 and a Redis slave node 112; 2nd Redis node 12 comprises the 2nd Redis host node 121 and the 2nd Redis slave node 122.APP does consistent hashing algorithm to key, navigates to concrete Redis node, then just can store data.Simultaneously Redis host node can synchronous a data to slave node.
Multiple database node 2, each database node comprises a master server and multiple standby server, for storing gentle data; All pipelined data are all relevant with gateway, so carry out cutting according to gateway ID to all pipelined data, other metadata, ensure that each database has portion, reduce the complicacy of inquiry polymerization greatly; Namely the single-point problem solved between master server and standby server, read data is read from standby server, its structure as shown in Figure 2, this sentences three database nodes 2 for example, be respectively the first database node 21, second database node 22 and the 3rd database node, first database node 21 comprises one first master server 211 and three first for server 212, second database node 22 comprises one second master server 221 and three second for server 222, and the 3rd database node 23 comprises one the 3rd master server 231 and three the 3rd for server 232.APP according to gateway ID, Hash to concrete a certain node database.Master/slave data storehouse in node realizes data syn-chronization by the backup scenario of database oneself, and data query and some aminated polyepichlorohydrin are realized by Open-Source Tools " amoeba ".
Distributed file system, for storing cold data; MapReduce algorithm is realized by HDinsight.
Embodiment 2:
The storage scheme of composition graphs 1-Fig. 3, the present embodiment describes the mass memory method based on Internet of Things of the present invention in detail, and it comprises following flow process:
F11: hot spot data is stored in cache nodes.
F12: gentle data carry out horizontal cutting by gateway Id, is stored in a master server in each database node and multiple standby server respectively;
F13: cold data are stored in distributed file system;
Above-mentioned flow process is order in no particular order, also can carry out simultaneously.
Wherein:
F11: the last state of each equipment of buffer memory, the metadata of each equipment user of buffer memory;
F12: connect master server and read binary log, extract the row insertion event, the decoded events that occur on described master server, extract data, and be appended in a text in distributed file system;
F13: the data are for the previous period assigned to different settings, carries out different MapReduce by product demand and calculates, obtain Output rusults, comprising: piecewise analysis when equipment runs, user use the large data such as device analysis.
Above specific embodiments of the invention are described.It is to be appreciated that the present invention is not limited to above-mentioned particular implementation, those skilled in the art can make various distortion or amendment within the scope of the claims, and this does not affect flesh and blood of the present invention.
Claims (9)
1. based on a mass storage system (MSS) for Internet of Things, it is characterized in that, comprising: multiple cache nodes, for storing hot spot data;
Multiple database node, each described database node comprises a master server and multiple standby server, for storing gentle data;
Distributed file system, for storing cold data.
2. mass storage system (MSS) according to claim 1, is characterized in that, described cache nodes is Redis node.
3. mass storage system (MSS) according to claim 1, is characterized in that, described database node is MySQL node.
4. mass storage system (MSS) according to claim 1, is characterized in that, described distributed file system is AzureBlockBlob.
5. based on a mass memory method for Internet of Things, it is characterized in that, comprise following flow process:
F11: hot spot data is stored in cache nodes;
F12: gentle data carry out horizontal cutting by gateway Id, is stored in a master server in each database node and multiple standby server respectively;
F13: cold data are stored in distributed file system;
Above-mentioned flow process is order in no particular order, also can carry out simultaneously.
6. mass memory method according to claim 5, is characterized in that, described flow process F11 is specially: hot spot data is stored in Redis node by consistent hashing algorithm.
7. mass memory method according to claim 5, it is characterized in that, described flow process F12 is specially: gentle data carry out horizontal cutting by gateway Id, the equipment relevant with gateway is also according to the horizontal cutting of gateway Id, user and App do not carry out cutting, store portion respectively in a master server in each database node and multiple standby server.
8. mass memory method according to claim 7, is characterized in that, described flow process F12 also comprises: database node carries out subregion according to per schedule time and deposits.
9. mass memory method according to claim 5, is characterized in that, described flow process F13 is specially: cold data realize MapReduce algorithm by HDinsight, are stored in the blockblob of Azure.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510566048.3A CN105138673A (en) | 2015-09-08 | 2015-09-08 | Mass storage system and method based on Internet of Things |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510566048.3A CN105138673A (en) | 2015-09-08 | 2015-09-08 | Mass storage system and method based on Internet of Things |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN105138673A true CN105138673A (en) | 2015-12-09 |
Family
ID=54724020
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510566048.3A Pending CN105138673A (en) | 2015-09-08 | 2015-09-08 | Mass storage system and method based on Internet of Things |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105138673A (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106341492A (en) * | 2016-10-19 | 2017-01-18 | 杭州赛客网络科技有限公司 | Distributed memory data storing and reading method based on cloud communication service |
| CN106776955A (en) * | 2016-12-02 | 2017-05-31 | 北京中交兴路车联网科技有限公司 | A kind of method and device of data storage and search |
| CN106933493A (en) * | 2015-12-30 | 2017-07-07 | 伊姆西公司 | Method and apparatus for caching disk array dilatation |
| CN107341267A (en) * | 2017-07-24 | 2017-11-10 | 郑州云海信息技术有限公司 | A kind of distributed file system access method and platform |
| CN108170824A (en) * | 2018-01-05 | 2018-06-15 | 马上消费金融股份有限公司 | A kind of date storage method based on SQL, device, equipment and storage medium |
| CN109710182A (en) * | 2018-12-14 | 2019-05-03 | 北京向上一心科技有限公司 | Market data processing method, device and controller and medium |
| CN109933575A (en) * | 2019-02-28 | 2019-06-25 | 鲁东大学 | The storage method and device of monitoring data |
| CN114518848A (en) * | 2022-02-15 | 2022-05-20 | 北京百度网讯科技有限公司 | Hierarchical storage system, and method, apparatus, device, and medium for processing storage data |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103268336A (en) * | 2013-05-13 | 2013-08-28 | 刘峰 | Fast data and big data combined data processing method and system |
| CN103631924A (en) * | 2013-12-03 | 2014-03-12 | Tcl集团股份有限公司 | Application method and system for distributive database platform |
| CN104331505A (en) * | 2014-11-20 | 2015-02-04 | 合一网络技术(北京)有限公司 | Distributed acquisition and storage-based monitoring system |
| CN104850572A (en) * | 2014-11-18 | 2015-08-19 | 中兴通讯股份有限公司 | HBase non-primary key index building and inquiring method and system |
-
2015
- 2015-09-08 CN CN201510566048.3A patent/CN105138673A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103268336A (en) * | 2013-05-13 | 2013-08-28 | 刘峰 | Fast data and big data combined data processing method and system |
| CN103631924A (en) * | 2013-12-03 | 2014-03-12 | Tcl集团股份有限公司 | Application method and system for distributive database platform |
| CN104850572A (en) * | 2014-11-18 | 2015-08-19 | 中兴通讯股份有限公司 | HBase non-primary key index building and inquiring method and system |
| CN104331505A (en) * | 2014-11-20 | 2015-02-04 | 合一网络技术(北京)有限公司 | Distributed acquisition and storage-based monitoring system |
Non-Patent Citations (1)
| Title |
|---|
| 朱亚兴等: "基于Redis+MySQL+MongoDB存储架构应用", 《微型机与应用》 * |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106933493A (en) * | 2015-12-30 | 2017-07-07 | 伊姆西公司 | Method and apparatus for caching disk array dilatation |
| CN106933493B (en) * | 2015-12-30 | 2020-04-24 | 伊姆西Ip控股有限责任公司 | Method and equipment for capacity expansion of cache disk array |
| CN106341492A (en) * | 2016-10-19 | 2017-01-18 | 杭州赛客网络科技有限公司 | Distributed memory data storing and reading method based on cloud communication service |
| CN106776955A (en) * | 2016-12-02 | 2017-05-31 | 北京中交兴路车联网科技有限公司 | A kind of method and device of data storage and search |
| CN107341267A (en) * | 2017-07-24 | 2017-11-10 | 郑州云海信息技术有限公司 | A kind of distributed file system access method and platform |
| CN108170824A (en) * | 2018-01-05 | 2018-06-15 | 马上消费金融股份有限公司 | A kind of date storage method based on SQL, device, equipment and storage medium |
| CN109710182A (en) * | 2018-12-14 | 2019-05-03 | 北京向上一心科技有限公司 | Market data processing method, device and controller and medium |
| CN109933575A (en) * | 2019-02-28 | 2019-06-25 | 鲁东大学 | The storage method and device of monitoring data |
| CN109933575B (en) * | 2019-02-28 | 2021-04-27 | 鲁东大学 | Monitoring data storage method and device |
| CN114518848A (en) * | 2022-02-15 | 2022-05-20 | 北京百度网讯科技有限公司 | Hierarchical storage system, and method, apparatus, device, and medium for processing storage data |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105138673A (en) | Mass storage system and method based on Internet of Things | |
| US10331641B2 (en) | Hash database configuration method and apparatus | |
| US20140188870A1 (en) | Lsm cache | |
| WO2017201970A1 (en) | Branch base database system and routing method therefor | |
| Varvello et al. | On the design and implementation of a wire-speed pending interest table | |
| CN105472056B (en) | DNS recursion server is layered caching method and system | |
| CN104104717A (en) | Inputting channel data statistical method and device | |
| CN104699723A (en) | Data exchange adapter and system and method for synchronizing data among heterogeneous systems | |
| CN105302830B (en) | Map tile caching method and device | |
| CN104657401A (en) | Web cache updating method | |
| Gupta et al. | In-memory database systems-a paradigm shift | |
| CN103442036A (en) | System integrating design development, post production and data storage and based on cloud platform | |
| CN106210117A (en) | A kind of high performance service framework realized by high in the clouds caching | |
| US20160224641A1 (en) | Obtaining incremental updates from a database using a partial query | |
| CN104933054B (en) | The URL storage methods and device of cache resource file, cache server | |
| CN111506604A (en) | Method, apparatus and computer program product for accessing data | |
| CN104636395A (en) | Count processing method and device | |
| CN103580991B (en) | The method for uploading and equipment of a kind of Email attachment | |
| CN103209119A (en) | Low-power-consumption embedding type cloud intelligent gateway | |
| CN105989065B (en) | Flash data processing method and system | |
| CN103516765A (en) | Storage method and system of social networking service background data | |
| KR101830504B1 (en) | In-Memory DB Connection Support Type Scheduling Method and System for Real-Time Big Data Analysis in Distributed Computing Environment | |
| Riaz et al. | Filtering the big data based on volume, variety and velocity by using Kalman filter recursive approach | |
| CN104794158A (en) | Domain name data repeated detection and fast index method in boundscript window | |
| CN105740442A (en) | Data storage method in HBase |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: Binjiang District West Street seven a gate village in Hangzhou City, Zhejiang province 310051 7-49 Building No. 1 room 1018 Applicant after: THE VANE TECHNOLOGY COMPANY LIMITED Address before: 314001, 11 floor, building 3288, Zhongshan West Road, Xiuzhou New District, Zhejiang, Jiaxing, 2 Applicant before: THE VANE TECHNOLOGY COMPANY LIMITED |
|
| COR | Change of bibliographic data | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20151209 |
|
| WD01 | Invention patent application deemed withdrawn after publication |