CN102404406A - Massive real-time data synchronization system based on private cloud storage - Google Patents
Massive real-time data synchronization system based on private cloud storage Download PDFInfo
- Publication number
- CN102404406A CN102404406A CN2011103942415A CN201110394241A CN102404406A CN 102404406 A CN102404406 A CN 102404406A CN 2011103942415 A CN2011103942415 A CN 2011103942415A CN 201110394241 A CN201110394241 A CN 201110394241A CN 102404406 A CN102404406 A CN 102404406A
- Authority
- CN
- China
- Prior art keywords
- data
- time data
- real
- privately owned
- cloud storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a massive real-time data synchronization system based on private cloud storage, which makes up the defects that the existing real-time data synchronization memory system causes processing delay, memory overflow and the like after receiving real-time data owning to different library table structures and extracting method of attributes in each special topic database and the like, particularly in the invention, the massive real-time data synchronization system also contributes to the aspect of extracting the attribute information and synchronously storing into multiple databases after analyzing the massive data by utilizing the private cloud storage. The massive real-time data synchronization system at least comprises a set of complete and closed data transmission network, a set of data stream receiving and sending interfaces, a set of private cloud storage devices, a set of permanent storage devices and a set of normal distributed database systems. The massive real-time data synchronization system is suitable for the applications for extracting all attribute values of massive real-time data and synchronously storing the attribute values into a plurality of target special topic databases.
Description
Technical field
The present invention relates to a kind of data-storage system, particularly a kind of method of data synchronization that utilizes Network storage technology to realize based on the computer documents of privately owned cloud storage.
Background technology
Instantiation research platform along with data flow model: the construction of " digital ocean " aid decision-making system and perfect; Current scientific research personnel and businessization user of service are not only that oceanographic data " digitlization " is stored to the demand of aid decision-making system; This builds the phase I at " digital ocean " and accomplishes; In the research and development and businessization use of " digital ocean " aid decision-making system; Scientific research personnel and Service User have proposed some new problems; To load balancing property and the reliability of mass data memory space dynamic allocation, and can practice thrift human and financial resources more such as how, because in real work to guarantee its storage system; Server cluster is very big with the electric weight that cooling is consumed; And the real-time Monitoring Data in ocean does storage endlessly from gather the oceanographic station of observation station under sending to separately, and oceanographic station is the ocean central station with real-time data memory and under being transferred to, and each ocean central station is sent to the sea area central station after the real-time data memory the most at last and does storage and data mining.In the real work, sea area central station server extracts the different attribute of each bar data, concrete property value is stored in the thematic data base of different business demand, rather than whole piece storage data.Because oceanographic data has the characteristics of magnanimity, multi-source, isomery; When real time data is continual when transferring to the sea area central station; The server that receives data will occur calculating that warehouse-in postpones even owing to do not stop calculating memory space and produced the operating system deadlock and finally cause crashing, had a strong impact on the professional progress of ocean central station.Such as in real work; Ocean observation station with frequency be Monitoring Data that to send a size in per 10 minutes be 1MB to the ocean central station; Suppose that National Bureau of Oceanography's sea area central station receives the real-time Monitoring Data that 15 ocean central stations (each ocean central station is made up of 15 stations) send simultaneously; And when server cluster the time in the memory allocated space, some servers still need in the face of the heavier request of load such as the inquiry of carrying out computation-intensive, database access etc.; And most of server only the low weight request of load is such as only need reading a html page or very simply calculate, the load lack of uniformity of this server cluster will cause individual servers to crash even cause the operating efficiency of whole digital ocean system to reduce.Yet above hypothesis has also only been considered the stock number that data are spent that each station is only sent; In fact ocean observation station is gathered various oceanographic datas and these data is divided into some and is sent to the ocean central station and stores, and the load pressure of sea area server cluster is well imagined.So meeting the storage dynamic load leveling storage problem of the oceanographic data of actual demand is badly in need of solving.
Privately owned storage cloud only provides corresponding stores service to limited users; Use the user of stores service not need to understand the detail that " cloud " formed,, and corresponding strategy is provided as long as know corresponding interface; Remaining work is transferred to " cloud " and is accomplished; This " cloud " just provides memory space, and data wherein can accomplish to visit at any time, just as the local storage of visit.Then under the situation that does not influence the user, a lot of additional functions is provided as high in the clouds, has made cloud become efficient, reliable, safe storage pool.With regard to lower global storage cost, using the benefit of cloud memory technology is conspicuous to unstructured data.The cloud memory technology is to be the basis with the service, need not to buy, administer and maintain storage hardware equipment, and the characteristics that privately owned cloud storage has are: 1. unified management; 2. be easy to realize centralized backup and disaster tolerance; 3. be easy to expansion, upgrading conveniently; 4. saving cost, green energy conservation.
Along with oceanographic data transmission continuous upgrading of special line and improving constantly of network computer performance; The oceanographic data transmission network has possessed the condition that makes up privately owned cloud storage oceanographic data; Structure comprises based on the advantage of the privately owned cloud storage of digital ocean transfer of data special line: 1. improve flexibility, comprise the remarkable shortening internal transmission response time; 2. realize higher efficient through improving resource utilization, comprise significantly energy savings; 3. make full use of the idle hardware and software of network internal, when promoting availability, the increase of farthest controlling cost; 4. utilize brand-new data sync warehouse-in system to optimize to improve data management mechanism, a large amount of memory spaces that growth capacity and the load of privately owned cloud memory technology through eliminating the conventional data storage expection disposed are to reach data space near 100% utilance.Except above four advantages mentioning, the storage of privately owned cloud also has extensibility, transparent support basis ability and solves characteristics such as the peak load ability is strong.
Summary of the invention
The present invention utilizes oceanographic data transmission private line network; Make up the privately owned cloud storage system of an ocean transmission; The storage of privately owned cloud be not only transmit through the uniform data interface with the storage of real time data function in brought into play effect; The more important thing is that this storage system will optimize existing real time data multiple database memory mechanism, can realize during central station, utilizing privately owned cloud storage system that the attribute of every new data is stored and analyzed to the sea area when real-time Data Transmission; According to target thematic data base different service demand, the extraction correlation attribute value of automated intelligent is gone into library storage respectively.
Description of drawings
Fig. 1 is the frame diagram of the multiple database synchro system of disclosed privately owned cloud storage.
Fig. 2 is disclosed privately owned cloud storage system workflow diagram.
Embodiment
Like Fig. 1; Operation principle based on the multiple database synchronous updating system of privately owned cloud storage is to utilize the transfer of data private line network to make up privately owned cloud storage system; When ocean observation station collects new real-time marine monitoring data; Data store privately owned high in the clouds into through unified data stream receiving port; The design philosophy of data stream interface is to utilize programming technique design one cover software interface, and purpose is to evade can't being transferred to privately owned high in the clouds between the different oceanographic data acquisition hardwares and can't each property value being synchronized to the defective in each database through behind the privately owned cloud storage optimization.After real-time data memory arrives privately owned high in the clouds; Real time data will be stored in privately owned cloud storage device; The metadata analysis module will be analyzed according to attribute real time data; And with Agent the return value of the attribute demand of target thematic data base is mated, extract behind the property value through the data flow transmission interface synchronously to each target thematic data base.Expired afterwards real time data will in order to discharge the elastic storage space of privately owned cloud, be optimized its operating efficiency by the regular unloading of system to permanent storage device.
Like Fig. 2; This privately owned cloud module is made up of four parts: 1. data map interface, 2. memory device, 3. metadata analysis module 4. Agent layer (Agent); And the realization in the privately owned cloud storage of elasticity is to rely on some virtual machine server that really are in operation to distribute to user's virtual machine; This user has whole access rights, comprises the administrator right to this virtual machine.
Because the user is in the on-premise network program, most applications can realize an operational effect through the collaborative work of a plurality of instances.The privately owned cloud storage system of elasticity has been realized the transmission between instance and the Internet in the internal structure through the IP address of inside; Make the application program of user's operation between different instances, can call and handle the communication resource each other, improved operating efficiency greatly.In order the real-time Monitoring Data in ocean to be utilized the cloud storage realize load balancing memory allocated space; And a property value is stored to the purpose in each ocean thematic data base in the real time parsing data flow, and the present invention has designed the workflow of a perfect privately owned cloud storage system.The working mechanism and the communication mechanism of various piece are set forth as follows:
(1) data map interface
This interface is to flow to the line data mapping around how solving the real-time Monitoring Data in ocean, and stores corresponding thematic data base in mapping.The design philosophy of this interface is to utilize programming technique design one cover software interface, and purpose is to evade can't being transferred to privately owned high in the clouds between the different oceanographic data acquisition hardwares and can't each property value being synchronized to the defective in the thematic data base through behind the privately owned cloud storage optimization.
(2) memory device
The memory device layer is that cloud is stored most basic part, and memory device is the computer at all working of forming privately owned cloud.The inner memory device of wherein privately owned cloud storage links together through oceanographic data transmission private line network.After the new real time data of station collection; Hardware devices such as the buoy of the station, transducer will utilize the data map interface that data are sent to privately owned high in the clouds through the transfer of data special line; Privately owned high in the clouds at first receiving real-time data is stored in the buffer memory; Then according to the real-time storage state of the virtual server in current each privately owned cloud storage system and current deliberated index COMPREHENSIVE CALCULATING and memory allocated spaces such as computational load state; With the purpose of realization elastic storage, and in accumulation layer, real time data is stored.
Another purpose of design of this memory device layer is to realize the logic virtual management of memory device, the functions such as condition monitoring of hardware device.This compares with existing privately owned cloud management level, has increased more monitoring content to guarantee the steady load property of working computer in the privately owned cloud, has realized dynamic load leveling memory allocated mechanism.Management level realize the collaborative work between a plurality of memory devices in the cloud storage through technology such as cluster, distributed file systems, and a plurality of memory devices can externally be provided with a kind of service, and more greatly stronger better data access performance is provided.
(3) data analysis module
After utilizing privately owned high in the clouds that real time data is stored; System will use the metadata analysis technology that the attribute of real time data is analyzed classification; System begins real time data top layer attribute to analyze with top-down, finally gives Agent layer (Agent) through the unified XML delivered.
(4) Agent layer (Agent)
In Agent layer; The demand service attribute of target thematic data base is analyzed and obtained to the business demand to each database of system intelligence; And feed back to memory device; Memory device will extract the different attribute value of each bar real time data according to the business demand of concrete each database, if do not find corresponding property value, system then gives tacit consent to and dates back to former data analysis module and continue to search.The final data map interface that passes through has really been realized the synchronous of real time data synchronously to each database.
In sum; The present invention is based on the stores synchronized problem that privately owned cloud memory technology solves real time data; Because special real-time, magnanimity property and the isomerism of marine monitoring data, when storing in the distributed data base after using conventional method that each bar attribute of real time data is extracted respectively, tend to cause the damaged of real time data property value; The user uses these missing datas to cause great difficulty to work, traditional marine monitoring method of data synchronization can't this difficult problem of solution.To this defective; The present invention proposes a kind of brand-new multiple database synchronous updating method based on privately owned cloud storage; Utilize the computer in transfer of data private line network and the network to make up privately owned cloud storage; Utilize privately owned cloud storage and administrative skill that the ocean real-time data base is stored; And according to target thematic data base different service demand, design Agent and metadata analysis module are carried out the intelligence coupling and are extracted property value, finally are synchronized in the target thematic data base through the property value of data map interface with each bar real time data.
The above is merely illustrative, but not is restricted.This description should not be construed as limitation of the present invention, and any spirit of the present invention and category of not breaking away from all should be contained within the application range
Claims (5)
1. based on the multiple database Design of Simultaneous System of privately owned cloud storage, it is characterized in that, should include:
The data transmission network of the complete closure of one cover, the transmitting real-time data of be used for carrying out at a high speed, maintaining secrecy;
One sets of data stream receives, transmission interface, is used for the unified real-time data memory that receives station transmission in privately owned cloud storage device, and final result is synchronized in the target thematic data base;
The privately owned cloud storage device of one cover; This device links to each other with the distributed station through the data flow receiving interface; Be responsible for the real time data of storage, and data are analyzed, extract its corresponding attribute according to the demand of each target thematic data base and distribute stores synchronized by station transmission;
One cover permanent storage device its objective is the marine monitoring data that storage is expired, to discharge the elastic storage space of privately owned cloud.
2. data transmission network according to claim 1 is characterized in that this network has high bandwidth, and network internal has 24 hours not out-of-work computers more than at least three.
3. data flow according to claim 1 receives, transmission interface, it is characterized in that being responsible in real-time Data Transmission to the privately owned cloud storage device with each station, and the property value that extracts is stored to the target thematic data base synchronously.
4. privately owned cloud storage device according to claim 1; It is characterized in that having made up privately owned cloud and carry out the storage of real time data; Simultaneously according to the different demands of target thematic data base, analyze and extract corresponding data attribute value and store to the target thematic data base synchronously.
5. permanent storage device according to claim 1 is characterized in that the marine monitoring data of using relatively inexpensive large capacity disc array stores expired.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2011103942415A CN102404406A (en) | 2011-12-02 | 2011-12-02 | Massive real-time data synchronization system based on private cloud storage |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2011103942415A CN102404406A (en) | 2011-12-02 | 2011-12-02 | Massive real-time data synchronization system based on private cloud storage |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN102404406A true CN102404406A (en) | 2012-04-04 |
Family
ID=45886186
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN2011103942415A Pending CN102404406A (en) | 2011-12-02 | 2011-12-02 | Massive real-time data synchronization system based on private cloud storage |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102404406A (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103516759A (en) * | 2012-06-28 | 2014-01-15 | 中兴通讯股份有限公司 | Cloud system resource management method, cloud call center seat management method and cloud system |
| CN103595735A (en) * | 2012-08-15 | 2014-02-19 | 福建天晴数码有限公司 | A method for a private cloud storage system based on household devices |
| CN103973586A (en) * | 2014-05-14 | 2014-08-06 | 江苏省无线电科学研究所有限公司 | Method for achieving automatic meteorological station wireless networking load balance |
| CN104301421A (en) * | 2014-10-24 | 2015-01-21 | 浪潮集团有限公司 | Cloud calculation real-time data synchronization method |
| CN108710533A (en) * | 2012-04-23 | 2018-10-26 | 谷歌有限责任公司 | The shared file preserved with synchronous electronics |
| CN108833600A (en) * | 2018-09-13 | 2018-11-16 | 湖北鑫恒福科技发展有限公司 | A kind of method and device that cloud storage data are synchronous |
| CN114401317A (en) * | 2022-03-25 | 2022-04-26 | 山东省计算中心（国家超级计算济南中心） | Ocean buoy-oriented multipoint cooperative active cache networking method and system |
| US12086109B2 (en) | 2012-04-23 | 2024-09-10 | Google Llc | Sharing and synchronizing electronically stored files |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101902498A (en) * | 2010-07-02 | 2010-12-01 | 广州鼎甲计算机科技有限公司 | Network technology based storage cloud backup method |
| CN101986663A (en) * | 2010-11-29 | 2011-03-16 | 北京卓微天成科技咨询有限公司 | OTP-based cloud storage data storing method, device and system |
-
2011
- 2011-12-02 CN CN2011103942415A patent/CN102404406A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101902498A (en) * | 2010-07-02 | 2010-12-01 | 广州鼎甲计算机科技有限公司 | Network technology based storage cloud backup method |
| CN101986663A (en) * | 2010-11-29 | 2011-03-16 | 北京卓微天成科技咨询有限公司 | OTP-based cloud storage data storing method, device and system |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108804213B (en) * | 2012-04-23 | 2022-04-19 | 谷歌有限责任公司 | Sharing and synchronizing electronically stored files |
| CN108710533A (en) * | 2012-04-23 | 2018-10-26 | 谷歌有限责任公司 | The shared file preserved with synchronous electronics |
| CN108804213A (en) * | 2012-04-23 | 2018-11-13 | 谷歌有限责任公司 | The shared file preserved with synchronous electronics |
| CN108710533B (en) * | 2012-04-23 | 2022-04-19 | 谷歌有限责任公司 | Sharing and synchronizing electronically stored files |
| US12086109B2 (en) | 2012-04-23 | 2024-09-10 | Google Llc | Sharing and synchronizing electronically stored files |
| CN103516759B (en) * | 2012-06-28 | 2018-11-09 | 中兴通讯股份有限公司 | Cloud system method for managing resource, cloud call center are attended a banquet management method and cloud system |
| CN103516759A (en) * | 2012-06-28 | 2014-01-15 | 中兴通讯股份有限公司 | Cloud system resource management method, cloud call center seat management method and cloud system |
| CN103595735A (en) * | 2012-08-15 | 2014-02-19 | 福建天晴数码有限公司 | A method for a private cloud storage system based on household devices |
| CN103973586A (en) * | 2014-05-14 | 2014-08-06 | 江苏省无线电科学研究所有限公司 | Method for achieving automatic meteorological station wireless networking load balance |
| CN104301421A (en) * | 2014-10-24 | 2015-01-21 | 浪潮集团有限公司 | Cloud calculation real-time data synchronization method |
| CN108833600A (en) * | 2018-09-13 | 2018-11-16 | 湖北鑫恒福科技发展有限公司 | A kind of method and device that cloud storage data are synchronous |
| CN114401317A (en) * | 2022-03-25 | 2022-04-26 | 山东省计算中心（国家超级计算济南中心） | Ocean buoy-oriented multipoint cooperative active cache networking method and system |
| CN114401317B (en) * | 2022-03-25 | 2022-07-05 | 山东省计算中心（国家超级计算济南中心） | Ocean buoy-oriented multipoint cooperative active cache networking method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102404406A (en) | Massive real-time data synchronization system based on private cloud storage | |
| CN109740037B (en) | Multi-source heterogeneous flow state big data distributed online real-time processing method and system | |
| CN105335513B (en) | A kind of distributed file system and file memory method | |
| CN105117497B (en) | Ocean big data principal and subordinate directory system and method based on Spark cloud network | |
| CN102694868A (en) | Cluster system implementation and task dynamic distribution method | |
| CN107766402A (en) | A kind of building dictionary cloud source of houses big data platform | |
| CN107341205A (en) | A kind of intelligent distribution system based on big data platform | |
| CN101902497B (en) | Cloud computing based internet information monitoring system and method | |
| CN105005570A (en) | Method and apparatus for mining massive intelligent power consumption data based on cloud computing | |
| CN104778188A (en) | Distributed device log collection method | |
| CN105808358B (en) | A kind of data dependence thread packet mapping method for many-core system | |
| CN111586091A (en) | Edge computing gateway system for realizing computing power assembly | |
| CN103106152A (en) | Data scheduling method based on gradation storage medium | |
| CN103281351A (en) | Cloud service platform for high-efficiency remote sensing data processing and analysis | |
| CN111198918B (en) | Data processing system based on big data platform and link optimization method | |
| CN102480489A (en) | Logging method and device used in distributed environment | |
| CN109344207A (en) | Incorporate frequency spectrum big data platform based on Beidou scanning | |
| CN102843420A (en) | Fuzzy division based social network data distribution system | |
| CN107679127A (en) | Point cloud information parallel extraction method and its system based on geographical position | |
| CN201726426U (en) | Internet information monitoring system based on cloud computing | |
| CN109635161B (en) | Cross-domain allopatric collaborative design method in mass data environment | |
| CN105354757A (en) | Electric power data integration processing system | |
| CN106161056B (en) | The distributed caching O&M method and device of preiodic type data | |
| CN106294526B (en) | A kind of mass small documents moving method in hierarchical stor | |
| CN106227465A (en) | A kind of data placement method of ring structure |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C02 | Deemed withdrawal of patent application after publication (patent law 2001) | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20120404 |