CN105224256A - A kind of storage system - Google Patents
A kind of storage system Download PDFInfo
- Publication number
- CN105224256A CN105224256A CN201510671810.4A CN201510671810A CN105224256A CN 105224256 A CN105224256 A CN 105224256A CN 201510671810 A CN201510671810 A CN 201510671810A CN 105224256 A CN105224256 A CN 105224256A
- Authority
- CN
- China
- Prior art keywords
- share directory
- node
- management node
- computing node
- log information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of storage system, comprise management node and at least one computing node, wherein, described management node comprises share directory, described share directory shares to each described computing node by NFS, is mounted with described share directory in each described computing node; Each described computing node is stored to described share directory by running the calculating log information generated in the high-performance calculation of FVCOM, or the calculating log information needed for reading in described share directory.Compared with prior art, the application carries out other calculating again without the need to being stored to by the calculating log information of generation after in the memory node of copy mechanism, but under directly calculating log information being stored to the share directory shared by NFS, calculating log information needed for simultaneously can reading from share directory, the processing speed calculating log information is improved significantly, saves the time further and improve system performance.
Description
Technical field
The present invention relates to technical field of memory, more particularly, relate to a kind of storage system.
Background technology
The exploration of the mankind to terrestrial climate enters new era, and large data are helping that researchist stores, treatment and analysis climatic data, shows the heroic undertaking of " prediction is following ".For marine forecasting, numerical weather forecast is carried out to relevant marine site, realize predicting accurately, reduce Oceanic disasters and bring economic loss to be a vital task of marine forecasting.But because numerical forecasting cleanliness is subject to the impact of factors, as field data, mode parameter, computing velocity, storage mode etc., its complicacy proposes challenge to large-scale data management work, especially very high to the requirement of storage mode, the feature of the large market demand of demand fulfillment Meteorological Field.
FVCOM is non-structure triangular mesh framework, limited bulk, Free Surface, three-dimensional original equation ocean numerical model.It contains multiple physics, water quality, ecological computing module.FVCOM model is based on Fortran90/95 standard, and realize calculating parallelization under the framework of MPI (MessagePassingInterface), parallel Fast simulation can be realized on the high-performance computer of shared drive and the many computing nodes of distributed memory.
And for running the key factor that the storage of the calculating log information generated in the high-performance calculation of FVCOM model and reading are the high-performance calculations ensureing trouble-free operation FVCOM model, in prior art, normally used storage system comprises management node, memory node and computing node, calculation task is issued to computing node by management node, computing node runs the high-performance calculation of the FVCOM model corresponding with calculation task, and calculating log information is stored to the memory node of copy mechanism, after storing successfully, carry out subsequent operation again.But be stored to time and the system performance of the process need at substantial of the memory node of copy mechanism by calculating log information, this is also unfavorable for the execution efficiency of whole system.
In sum, in prior art, the process calculating log information is existed and expend time in and problem that system performance is more serious.
Summary of the invention
The object of this invention is to provide a kind of storage system, to solve expending time in and problem that system performance is more serious of existing for the process calculating log information in prior art.
To achieve these goals, the invention provides following technical scheme:
A kind of storage system, comprise management node and at least one computing node, wherein, described management node comprises share directory, and described share directory shares to each described computing node by NFS, is mounted with described share directory in each described computing node;
Each described computing node is stored to described share directory by running the calculating log information generated in the high-performance calculation of FVCOM, or the calculating log information needed for reading in described share directory.
Preferably, described storage system comprises multiple memory node, is all stored in the described memory node of copy mechanism for the source data of running the high-performance calculation of FVCOM and the target data that utilizes described source data to run the correspondence that the high-performance calculation of FVCOM obtains.
Preferably, described storage system also comprises management node for subsequent use, described management node for subsequent use be used for described management node break down cannot normally work time, the work replacing described management node to complete described management node need completing.
Preferably, carry module is included in each described computing node, described carry module is used for determining described share directory by scanning, described share directory is mounted in corresponding computing node, and the generation pass of the calculating log information run in the high-performance calculation of FVCOM is appointed as described share directory.
Preferably, described management node comprises sharing module, described sharing module is used for setting up specified file, is in the described specified file of shared order write of share directory, and described share directory is shared to each described computing node by the catalogue determined in described management node.
Preferably, described management node comprises automatic activation module, and described automatic activation module is used for automatically indicating described sharing module that described share directory is shared to each described computing node when described management node start.
Preferably, described sharing module comprises delete cells, and described delete cells is used for when there is described specified file in described management node, is deleted by described specified file, and indicates described sharing module to set up new specified file.
A kind of storage system provided by the invention, comprise management node and at least one computing node, wherein, described management node comprises share directory, described share directory shares to each described computing node by NFS, is mounted with described share directory in each described computing node; Each described computing node is stored to described share directory by running the calculating log information generated in the high-performance calculation of FVCOM, or the calculating log information needed for reading in described share directory.Compared with prior art, the application carries out other calculating again without the need to being stored to by the calculating log information of generation after in the memory node of copy mechanism, but under directly calculating log information being stored to the share directory shared by NFS, calculating log information needed for simultaneously can reading from share directory, the processing speed calculating log information is improved significantly, saves the time further and improve system performance.
Accompanying drawing explanation
In order to be illustrated more clearly in the embodiment of the present invention or technical scheme of the prior art, be briefly described to the accompanying drawing used required in embodiment or description of the prior art below, apparently, accompanying drawing in the following describes is only embodiments of the invention, for those of ordinary skill in the art, under the prerequisite not paying creative work, other accompanying drawing can also be obtained according to the accompanying drawing provided.
The structural representation of a kind of storage system that Fig. 1 provides for the embodiment of the present invention.
Embodiment
Below in conjunction with the accompanying drawing in the embodiment of the present invention, be clearly and completely described the technical scheme in the embodiment of the present invention, obviously, described embodiment is only the present invention's part embodiment, instead of whole embodiments.Based on the embodiment in the present invention, those of ordinary skill in the art, not making the every other embodiment obtained under creative work prerequisite, belong to the scope of protection of the invention.
Embodiments provide a kind of storage system, comprise management node and at least one computing node, the quantity of computing node can be determined according to actual needs, when the quantity of computing node is 1, the structural representation of storage system as shown in Figure 1, comprises management node 11 and computing node 12.Wherein, management node comprises share directory, and share directory shares to each computing node by NFS, is mounted with share directory in each computing node;
Each computing node is stored to share directory by running the calculating log information generated in the high-performance calculation of FVCOM, or the calculating log information needed for reading in share directory.
It should be noted that, NFS (NetworkFileSystem, network file system(NFS)), is the one in the file system of FreeBSD support, and its allows between the computing machine in network by TCP/IP network shared resource.In the application of NFS, the client application of local NFS can read and write the file be positioned on far-end nfs server pellucidly, just as access local file.In the application, share directory is shared to each computing node by NFS by management node, and computing node can be made directly to access share directory.
The application carries out other calculating again without the need to being stored to by the calculating log information of generation after in the memory node of copy mechanism, but under directly calculating log information being stored to the share directory shared by NFS, calculating log information needed for simultaneously can reading from share directory, thus, decrease the request process time delay calculating log information in high-performance calculation process, the processing speed calculating log information is improved significantly, save the time and improve system performance, reaching the object of the execution efficiency promoting whole system.
In a kind of storage system that above-described embodiment provides, multiple memory node can be comprised, be all stored in the described memory node of copy mechanism for the source data of running the high-performance calculation of FVCOM and the target data that utilizes described source data to run the correspondence that the high-performance calculation of FVCOM obtains.
The concrete quantity of memory node can be determined according to actual needs, and it can form distributed memory system.Distributed memory system by distributed data storage in different memory nodes, by adopting the mode of distributed storage, the high bandwidth that data are transmitted can be provided, to ensure between different memory nodes, or between memory node and computing node, or can quick data transfering between memory node and management node, improve the execution efficiency of whole storage system.
It should be noted that, in a kind of storage system that the embodiment of the present invention provides, run the calculating log information generated in the high performance computation process of FVCOM only by the process of NFS mode, and other any data are not processed.Therefore, source data corresponding for the above-mentioned high performance computation with running FVCOM and target data are all stored in the memory node of copy mechanism, be stored in two memory nodes by same source data and target data, thus, when arbitrary memory node breaks down and then cause the loss of data or damage, the data that can lost by acquisition in another memory node or be damaged, with fast quick-recovery data, ensure that the security of data, and the stability of whole storage system.
Wherein, management node is as initiation point, the step that its controlling calculation node runs the high-performance calculation of FVCOM can comprise: management node sends calculation task to computing node, computing node obtains the source data corresponding with calculation task by corresponding memory node, high-performance calculation is carried out to source data, and the target data calculated is sent in memory node stores, target data is synchronized to another memory node of copy mechanism by memory node, with the consistance of data in two memory nodes ensureing copy mechanism.
In a kind of storage system that above-described embodiment provides, described storage system can also comprise management node for subsequent use, described management node for subsequent use be used for described management node break down cannot normally work time, the work replacing described management node to complete described management node need completing.
Thus, when management node can normally work, management node for subsequent use is idle management node, does not carry out any work; And when management node breaks down and normally cannot complete related work, replace management node to complete the work completed needed for it by management node for subsequent use, thus, ensure that the normal operation of whole storage system.
In addition, in order to ensure that a kind of storage system that the embodiment of the present invention provides can realize above-mentioned functions, need to carry out relevant configuration to computing node and management node, specifically, management node can comprise sharing module, described sharing module is used for setting up specified file, is in the described specified file of shared order write of share directory, and described share directory is shared to each described computing node by the catalogue determined in described management node.
Management node realizes share directory to share to each computing node by sharing module, and wherein, by sharing module, first, need to set up specified file on management node mu01, specific implementation statement can be :/etc/exports; Secondly, need to be that the shared order of share directory writes in described specified file by the catalogue determined in described management node, specific implementation statement can be: vim/etc/exports/ ,/share* (rw, sync, no_root_squash); Finally, activate NFS and realize described share directory to share to each described computing node, specific implementation statement can be: [rootmu01 ~] #servicenfsrestart.And before this, determine that the concrete statement realizing management node share directory being shared to each computing node can be: ibmu01:/share.
In addition, described management node can also comprise automatic activation module, and described automatic activation module is used for automatically indicating described sharing module that described share directory is shared to each described computing node when described management node start.
When after management node start, need again to complete the work that sharing module completes, so that share directory is shared with each computing node, and be that management node arranges automatic activation module, can after management node start, instruction sharing module completes related work automatically, and share directory is shared with each computing node, and without the need to staff's manual operation, improve the work efficiency of whole system.Its specific implementation statement can be: [rootmu01 ~] #chkconfignfson.
It should be noted that, when specified file exists in management node, the information stored in specified file causes harmful effect to system, sharing module can comprise delete cells, delete cells is used for when there is specified file in management node, specified file is deleted, and indicates sharing module to set up new specified file, to realize the configuration of server smoothly.
All carry module can be comprised in each computing node, carry module is used for determining described share directory by scanning, described share directory is mounted in corresponding computing node, and the generation pass of the calculating log information run in the high-performance calculation of FVCOM is appointed as described share directory.
It should be noted that, carry module is the module for realizing share directory to carry out carry, by carry module, first, needs to determine share directory by scanning, concrete realize statement and can be: showmount-emu01; Secondly, share directory is mounted in corresponding computing node, concrete realize statement and can be: mount-tnfs-overs=3ibmu01:/share/share; Then, amendment runs the generation pass of the calculating log information generated in the high-performance calculation of FVCOM, the generation pass running the calculating log information generated in the high-performance calculation of FVCOM is appointed as the share directory of carry, specific implementation statement can be: INFOFILE=../../../share/runlog_sdw, and namely wherein/share catalogue is share directory; Finally preserve and exit.
Because FVCOM model can be deployed in distributed memory system, reading and the write of its data are all under the share directory of distributed memory system, do not need to specify the input file catalogue INPDIR in configuration file sdw_run.dat and output file catalogue OUTDIR.
In addition, the order of carry shared file can be write on each computing node /etc/rc.local file in so that after start-up simulation node automatic carry share directory.
By the above-mentioned relevant configuration for computing node and management node, ensure that the work well of a kind of storage system that the embodiment of the present invention provides.
To the above-mentioned explanation of the disclosed embodiments, those skilled in the art are realized or uses the present invention.To be apparent for a person skilled in the art to the multiple amendment of these embodiments, General Principle as defined herein can without departing from the spirit or scope of the present invention, realize in other embodiments.Therefore, the present invention can not be restricted to these embodiments shown in this article, but will meet the widest scope consistent with principle disclosed herein and features of novelty.
Claims (7)
1. a storage system, is characterized in that, comprises management node and at least one computing node, wherein, described management node comprises share directory, and described share directory shares to each described computing node by NFS, is mounted with described share directory in each described computing node;
Each described computing node is stored to described share directory by running the calculating log information generated in the high-performance calculation of FVCOM, or the calculating log information needed for reading in described share directory.
2. system according to claim 1, it is characterized in that, described storage system comprises multiple memory node, is all stored in the described memory node of copy mechanism for the source data of running the high-performance calculation of FVCOM and the target data that utilizes described source data to run the correspondence that the high-performance calculation of FVCOM obtains.
3. system according to claim 2, it is characterized in that, described storage system also comprises management node for subsequent use, described management node for subsequent use be used for described management node break down cannot normally work time, the work replacing described management node to complete described management node need completing.
4. system according to claim 1, it is characterized in that, carry module is included in each described computing node, described carry module is used for determining described share directory by scanning, described share directory is mounted in corresponding computing node, and the generation pass of the calculating log information run in the high-performance calculation of FVCOM is appointed as described share directory.
5. system according to claim 4, it is characterized in that, described management node comprises sharing module, described sharing module is used for setting up specified file, be in the described specified file of shared order write of share directory by the catalogue determined in described management node, and described share directory is shared to each described computing node.
6. system according to claim 5, it is characterized in that, described management node comprises automatic activation module, and described automatic activation module is used for automatically indicating described sharing module that described share directory is shared to each described computing node when described management node start.
7. system according to claim 6, it is characterized in that, described sharing module comprises delete cells, and described delete cells is used for when there is described specified file in described management node, described specified file is deleted, and indicates described sharing module to set up new specified file.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510671810.4A CN105224256A (en) | 2015-10-13 | 2015-10-13 | A kind of storage system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510671810.4A CN105224256A (en) | 2015-10-13 | 2015-10-13 | A kind of storage system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN105224256A true CN105224256A (en) | 2016-01-06 |
Family
ID=54993262
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510671810.4A Pending CN105224256A (en) | 2015-10-13 | 2015-10-13 | A kind of storage system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105224256A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106027595A (en) * | 2016-04-25 | 2016-10-12 | 乐视控股（北京）有限公司 | Access log processing method and system for CDN node |
| CN107733667A (en) * | 2016-08-10 | 2018-02-23 | 北京京东尚科信息技术有限公司 | A kind of blog management method and system |
| CN108984345A (en) * | 2018-07-11 | 2018-12-11 | 吉林吉大通信设计院股份有限公司 | A kind of big data backup method based on virtual share directory |
| CN109086134A (en) * | 2018-07-19 | 2018-12-25 | 郑州云海信息技术有限公司 | A kind of operation method and device of deep learning operation |
| CN109587205A (en) * | 2017-09-29 | 2019-04-05 | 成都华为技术有限公司 | The creation hanging method and relevant device of share directory |
| CN110865989A (en) * | 2019-11-22 | 2020-03-06 | 浪潮电子信息产业股份有限公司 | Business processing method for large-scale computing cluster |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1567248A (en) * | 2003-07-01 | 2005-01-19 | 华为技术有限公司 | Log recording method and system for application program |
| CN102541994A (en) * | 2011-11-18 | 2012-07-04 | 浪潮电子信息产业股份有限公司 | Method for deploying and sharing NFS (network file system) automatically and quickly |
| CN102902590A (en) * | 2012-09-03 | 2013-01-30 | 南京师范大学 | Parallel digital terrain analysis-oriented massive DEM (Digital Elevation Model) deploying and scheduling method |
| CN103377069A (en) * | 2012-04-26 | 2013-10-30 | 东方有线网络有限公司 | Novel method for achieving virtual calculation deployment based on writing of log files into plug-in network attached storage (NAS) |
| CN103631623A (en) * | 2013-11-29 | 2014-03-12 | 浪潮(北京)电子信息产业有限公司 | Method and device for allocating application software in trunking system |
-
2015
- 2015-10-13 CN CN201510671810.4A patent/CN105224256A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1567248A (en) * | 2003-07-01 | 2005-01-19 | 华为技术有限公司 | Log recording method and system for application program |
| CN102541994A (en) * | 2011-11-18 | 2012-07-04 | 浪潮电子信息产业股份有限公司 | Method for deploying and sharing NFS (network file system) automatically and quickly |
| CN103377069A (en) * | 2012-04-26 | 2013-10-30 | 东方有线网络有限公司 | Novel method for achieving virtual calculation deployment based on writing of log files into plug-in network attached storage (NAS) |
| CN102902590A (en) * | 2012-09-03 | 2013-01-30 | 南京师范大学 | Parallel digital terrain analysis-oriented massive DEM (Digital Elevation Model) deploying and scheduling method |
| CN103631623A (en) * | 2013-11-29 | 2014-03-12 | 浪潮(北京)电子信息产业有限公司 | Method and device for allocating application software in trunking system |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106027595A (en) * | 2016-04-25 | 2016-10-12 | 乐视控股（北京）有限公司 | Access log processing method and system for CDN node |
| CN107733667A (en) * | 2016-08-10 | 2018-02-23 | 北京京东尚科信息技术有限公司 | A kind of blog management method and system |
| CN107733667B (en) * | 2016-08-10 | 2021-05-25 | 北京京东尚科信息技术有限公司 | Log management method and system |
| CN109587205A (en) * | 2017-09-29 | 2019-04-05 | 成都华为技术有限公司 | The creation hanging method and relevant device of share directory |
| CN109587205B (en) * | 2017-09-29 | 2021-07-16 | 成都华为技术有限公司 | Shared directory creation mounting method and related equipment |
| CN108984345A (en) * | 2018-07-11 | 2018-12-11 | 吉林吉大通信设计院股份有限公司 | A kind of big data backup method based on virtual share directory |
| CN108984345B (en) * | 2018-07-11 | 2020-06-23 | 吉林吉大通信设计院股份有限公司 | Big data backup method based on virtual shared directory |
| CN109086134A (en) * | 2018-07-19 | 2018-12-25 | 郑州云海信息技术有限公司 | A kind of operation method and device of deep learning operation |
| CN110865989A (en) * | 2019-11-22 | 2020-03-06 | 浪潮电子信息产业股份有限公司 | Business processing method for large-scale computing cluster |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105224256A (en) | A kind of storage system | |
| US8949182B2 (en) | Continuous and asynchronous replication of a consistent dataset | |
| US10284623B2 (en) | Optimized browser rendering service | |
| CN107450961A (en) | A kind of distributed deep learning system and its building method, method of work based on Docker containers | |
| US10713330B2 (en) | Optimized browser render process | |
| US9405579B2 (en) | Seamless extension of local computing power | |
| US11307900B2 (en) | Adjustment of the number of central processing units to meet performance requirements of an I/O resource | |
| US10656852B2 (en) | Performing preprocessing operations in anticipation of log file writes | |
| CN104504147A (en) | Resource coordination method, device and system for database cluster | |
| Pérez-Arteaga et al. | Cost comparison of lambda architecture implementations for transportation analytics using public cloud software as a service | |
| CN104580422A (en) | Cluster rendering node data access method based on shared cache | |
| CN107786670A (en) | A kind of system and method that data and cloud storage R. concomitans are locally stored | |
| US11308119B2 (en) | Replicating large statements with low latency | |
| US9665308B2 (en) | Optimization of the copying of change recorded data by using spare flash capacity | |
| CN103810258A (en) | Data aggregation scheduling method based on data warehouse | |
| Yang et al. | On construction of the air pollution monitoring service with a hybrid database converter | |
| CN105094761A (en) | Data storage method and device | |
| EP3396553B1 (en) | Method and device for processing data after restart of node | |
| CN106201784B (en) | A kind of data back up method and system based on git | |
| CN104281486A (en) | Processing method and device of VM (virtual machine) | |
| CN110046132B (en) | Metadata request processing method, device, equipment and readable storage medium | |
| US11170000B2 (en) | Parallel map and reduce on hash chains | |
| CN111949370A (en) | Method, system, equipment and medium for cleaning docker mirror image | |
| RU2591020C1 (en) | Method of monitoring user program progress, running on computational nodes of computer system | |
| CN103970618A (en) | Technical method for applying no-backup disaster recovery based on cloud computing platform |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20160106 |
|
| RJ01 | Rejection of invention patent application after publication |