CN102880658A - Distributed file management system based on seismic data processing - Google Patents
Distributed file management system based on seismic data processing Download PDFInfo
- Publication number
- CN102880658A CN102880658A CN2012103170363A CN201210317036A CN102880658A CN 102880658 A CN102880658 A CN 102880658A CN 2012103170363 A CN2012103170363 A CN 2012103170363A CN 201210317036 A CN201210317036 A CN 201210317036A CN 102880658 A CN102880658 A CN 102880658A
- Authority
- CN
- China
- Prior art keywords
- file
- data
- read
- burst
- interface
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Abstract
The invention discloses a distributed file management system based on seismic data processing, which includes data read-write interfaces, a file management server and communication modules, wherein the communication modules are provided with interfaces for information receiving and transmission, which are utilized by the data read-write interfaces and the file management server. The invention has the benefits as follows: the distributed file system principle is adopted to automatically slice and manage files, so as to realize the high-speed paralleling I/O and provide simple and efficient file management; users only simply call the corresponding interface to realize operation for data, and files can be sliced and managed completely and automatically through programs without manual intervention; the problem of competition conflict that a plurality of computation nodes access the same file on a unified magnetic disc array under the cluster environment can be solved; and the method for slicing the files fundamentally solves the problem of sharp performance reduction of I/O caused by the file locking protection of a file system.
Description
Technical field
The present invention relates to a kind of distributed file management system that processes based on geological data.
Background technology
Distributed file system (Distributed File System) refers to that the physical store resource of file system management not necessarily directly is connected on the local node, but links to each other with node by computer network.
Along with the continuous acceleration of human information process, the explosive growth that traditional storage organization can't the supported data amount.Large data storage and processing have become a study hotspot of computer nowadays science.When the processing that relates to large data, the I/O performance of traditional file systems has become the bottleneck of restriction whole system, but fault-tolerant and scaling problem is also severeer.And the appearance of distributed file system is just solving this problem, and is fault-tolerant in the I/O performance, and convenient all acquisition of expansibility greatly promotes.
Under the overall background of cloud computing, to the harshness requirement of large data and the day by day universalness of parallel computation, so that distributed file system has obtained using more widely.
The research boom of distributed file system starts from the GFS file system of google.Google FileSystem(GFS) is the proprietary distributed file system based on Linux that Google company develops in order to satisfy its demand.
GFS is an extendible distributed file system, is used for application large-scale, distributed, that mass data is conducted interviews.It runs on the cheap common hardware, but fault tolerance can be provided.It provides overall performance higher service can for a large amount of users.GFS with respect to the maximum characteristics of traditional distributed file system be it mistake as normal but not unusually process, a kind of efficiently and reliably fault tolerant mechanism is provided.
The node of GFS is divided into two classes, and one is host node, and one is back end, adopts simple C/S model operation.
The data of storing in GFS are divided into the piece of fixed size, are stored in respectively on the different memory nodes.Host node is in charge of the information of All Files piece, is responsible for the memory load of each back end of balance.Master server is not stored actual chunk data usually, but the storage metadata relevant with chunk data, such data such as mapping form being are are being read and write " snapshot " that specific long data block or tracking copy chunk data with 64 label mappings to file, chunk data copy position, which process of chunk data position and composition thereof.All these metadata periodically receive the renewal (" heartbeat message ") that comes from each data block server by master server and keep last state.When the user needs visit data, by GFS provide interface operate, interface is obtained the position of concrete data block by the database of access to netwoks host node, by the corresponding data of Network Capture.
Prior art related to the present invention comprises:
（1）Hadoop:
Because google does not announce the source code that its GFS realizes, hadoop is in the industry cycle wide-spread as the specific implementation of three pieces of important cloud computing papers (mapreduce, gfs, bigtable) of google, has also obtained very extensive and important application.
Hadoop is a distributed system architecture, is developed by the Apache foundation.The user can be in the situation that do not understand distributed bottom details, the exploitation distributed program.Take full advantage of power high-speed computation and the storage of cluster.Hadoop has realized a distributed file system (Hadoop Distributed FileSystem), is called for short HDFS.HDFS has the characteristics of high fault tolerance, and design is used for being deployed on the general commercial hardware.And its data of providing high transmission rates to visit application program, being fit to those has the application program of super large data set.HDFS has relaxed the requirement of POSIX, the data in the form access file system that can flow.
For external client, HDFS is just as a traditional hierarchical file system.Can create, delete, move or Rename file, etc.But the framework of HDFS is based on one group of specific node and makes up.These nodes comprise NameNode, and it provides Metadata Service in HDFS inside; DataNode, it provides storage block for HDFS.Owing to only there being a NameNode, so there is Single Point of Faliure, must guarantee the high availability of NameNode, this also is the implementation of GFS.
The file that is stored among the HDFS is divided into piece, then these pieces is copied to (DataNode) in a plurality of computing machines.This and traditional RAID framework differ widely.Block size (being generally 64MB) and the number of blocks that copies are determined by client computer when creating file.NameNode can control the All Files operation.All communications of HDFS inside are measured ICP/IP protocol all.
(2) disk array:
Disk array be by a lot of cheap, capacity is less, stability is higher, the slower disk of speed, is combined into a large-scale disk group, utilizes indivedual disks to provide data to produce the whole disk system usefulness of addition effect promoting.Utilize simultaneously this technology, data are cut into many sections, leave in respectively on each hard disk.Disk array can also utilize the idea of parity check (Parity Check), and in array during arbitrary hard disk failure, but sense data still when data reconstruction, is inserted data in the new hard disk after as calculated again.RAID significantly improves the data throughout (Throughput) of storage system by simultaneously storage and reading out data on a plurality of disks.In RAID, can allow simultaneously the transmission of datas of a lot of disc drivers, and these disc drivers logically are disc drivers, so use RAID can reach the speed of several times of single disc drivers, tens times even hundreds of times.
(3) Linux cluster high-performance calculation:
Cluster is in recent years main a kind of HPC hardware: cluster (cluster) is exactly the set of one group of MPP (massively parallel processor (MPP)).Processor in the cluster is commonly called node, and it has oneself CPU, internal memory, operating system, I/O subsystem, and can communicate with other nodes.There are at present a lot of places all to use common workstation operation Linux and other Open Source Code Software to serve as node in the cluster.
High-performance calculation (High-Performance Computing) is a branch of computer science, and it is devoted to develop supercomputer, research parallel algorithm and exploitation related software.High-performance calculation is mainly studied like an elephant weather forecast, terrain analysis, Storage and Processing mass data, data mining, image processing etc.
In the processing of geological data, single geological data file can be up to hundreds of G, and the data processing method of traditional individual node needs a couple of days, even the data processing to a file can be finished in tens of sky.For this reason, Data Segmentation must be processed on a plurality of nodes concurrently.Ideally, if the nodes that participates in simultaneously processing is n, then the overall operation time can shorten to 1/n, but actual run time is far above ideal value.Reason be for realize a plurality of nodes concurrent reading and concurrent writing, remove to read same file by different nodes according to different document misregistration amounts, in the situation that the processing node number is more, the situation of comparatively serious contention access has occured, file system has caused greatly expense to locking of file, situation when I/O speed is read far below single node, the I/O time becomes the serious bottleneck of system.
If be a plurality of son files with the individual data file division artificially, allow different computing nodes read different file fragmentations, just can avoid the situation of contention access to occur.Although this mode has solved the reading and writing of files conflict, but introduced simultaneously a lot of problems: at first each file needs to carry out artificially burst, and geological data has its intrinsic form, need to guarantee that each burst includes complete minimal processing unit (road or road collection), can't realize burst with simple file division; Secondly because the file behind the burst is numerous, bring greatly confusion can for the management of file itself, bring difficulty also for writing of application program.To need a large amount of manual operations in order addressing these problems, not only to have increased the weight of file management personnel, developer's burden, also so that the reduction of the reliability of system itself.
Summary of the invention
In order to overcome the above-mentioned shortcoming of prior art, the invention provides a kind of distributed file management system that processes based on geological data, adopt the principle of distributed file system, automatically finish burst and the management of file, with realization Parallel I/O at a high speed, and provide succinct efficiently file management.The user only need call the various operations that corresponding interface just can complete paired data simply, and is automatically finished by program fully the burst of file and management be need not artificial intervention.
The technical solution adopted for the present invention to solve the technical problems is: a kind of distributed file management system that processes based on geological data, comprise data read and write interface, document management server and communication module, communication module provides the interface of messaging for data read and write interface and document management server, wherein:
The structure of described data read and write interface comprises Business Logic, file read-write module, file die section and local file information cache, is achieved as follows function:
(1) initialization storage file;
(2) obtain file fragmentation information;
(3) obtain complete raw data file;
(4) deletion/updating file;
The structure of described document management server comprises that Business treatment, geological data cut apart module, file storage module, data base read-write interface and management by district module, is achieved as follows function:
(1) the specific form according to geological data carries out burst to large data, guarantees that each burst comprises complete geological data unit;
(2) original document that adds is carried out the burst storage;
(3) with file fragmentation information write into Databasce;
(4) provide file fragmentation information for data read and write interface;
The structure of described communication module comprises document management server information inquiry module, message sequence module, Socket manager and TCPSocket/UDPSocket, is achieved as follows function:
(1) provide the current file server to move the network information of node;
(2) provide transmission, accept the interface of message;
(3) provide the ability of serializing, unserializing.
Compared with prior art, good effect of the present invention is: solved under cluster environment the mutual competitions and conflicts of identical file on the unified disk array of a plurality of computing node access; Fundamentally avoided because the problem that the I/O performance that file system causes the locking protection of file declines to a great extent by the method that makes file fragmentation.The present invention has following advantage:
(1) is engaged in the distributed-computation program of moving in the group system, each program can be processed separately a burst of a large data file, but not read single data file with other node competition, the program that greatly reduces is in the time of I/O operation cost, in the situation that n computing node parallel computation, make program operation T.T. close to ideal value 1/n, solved the problem of I/O bottleneck, bring into play to greatest extent the usefulness of parallel computation.
(2) adopt unified document management server.When the benefit that the acquisition file fragmentation brings, the burden of bringing for fear of Fragmentation, burst information administrative institute, by introducing document management server, the file fragmentation management of robotization is provided centered by document management server, all work is finished automatically by document management server and data read and write interface, whole process need not manpower intervention, greatly reduces the use difficulty of system and the human cost of maintenance.
(3) adopt unified communication module interface, greatly reduced the difficulty of communications portion program development, unified communication construction also makes program have better unitarity, is convenient to maintenance and the upgrading in later stage.
Description of drawings
Examples of the present invention will be described by way of reference to the accompanying drawings, wherein:
Fig. 1 is system communication structural representation of the present invention;
Fig. 2 is the structural representation of data read and write interface;
Fig. 3 is the structural representation of document management server;
Fig. 4 is the structural representation of communication module.
Embodiment
Application scenarios of the present invention is the enterprise-level group system, has a plurality of computing nodes, and each node is connected by express network, can reach very high transmission bandwidth.All data are all deposited on the disk array, and all computing nodes all addressable disk array carry out data read-write operation.
Each computing node can read one or more data file bursts by data read and write interface and process.Task Assigning and collection are finished by unified distributed computing framework, and the present invention for this reason reading and writing data of application framework provides support.
A kind of distributed file management system that processes based on geological data as shown in Figure 1, comprising: data read and write interface, document management server and communication module; Wherein:
The higher level operation that the data operating interface that described data read and write interface uses to the user provides the basic operation of reading and writing of files and reads slicing files, obtains the file fragmentation data, and realize mutual by communication module and document management server.
Described document management server is the chief executive of distributed file system, be responsible for burst information and the deposit position of storage file, finish the initial burst work to file, acceptance is from the query requests of user interface, return the corresponding fileinfo of user, and communicate by communication module and user program.
Described communication module provides reliable and stable communication mechanism for data read and write interface and document management server, and communication mode uses message passing mechanism, so that good solution coupling and asynchronous performance to be provided.
Because the user program in the group system may be positioned on the different nodes with document management server, must introduce inter-process communication mechanisms.Message-Passing Model is all adopted in communication among the present invention, sends message by Ethernet and realizes communication process.All communications in the system are finished by communication module, and communication module provides the interface of messaging for data read and write interface and document management server.
Data read and write interface is filed a request to document management server to send message mode, and document management server returns corresponding fileinfo, and data read and write interface obtains just can carrying out the operation of file behind the corresponding information (name of file fragmentation and store path).
Be stored in each distributed node the data distributed earth different from other distributed document, in application scenarios of the present invention, all public documents all leave in the disk array, and each computing node can be accessed the file in the disk array as the access local file.
All for directly reading from disk array, each process can use the file system interface of operating system to carry out data manipulation to the mode of all process access files.
One, data read and write interface:
Data read and write interface provides the interface of the operation file burst that is simple and easy to usefulness for the user.The user only just can finish all operations to file by data read and write interface, and need not be again by other external means.Data read and write interface offers the user in the mode in storehouse.
As shown in Figure 2, the structure of data read and write interface comprises:
(1) Business Logic: the operation to the user is classified, and carries out corresponding operation;
(2) file read-write module: according to the file fragmentation information of obtaining, from disk array, obtain corresponding data file;
(3) file die section: obtain all file fragmentation information, read each file fragmentation and merging, realize the amalgamation to burst;
(4) local file information cache: will be buffered in this locality with the burst information of obtaining from document management server, so that the user also can reduce the read latch that causes because of the access file management server in the situation that the repeated multiple times same slice that reads reduces pressure to document management server.
The function that data read and write interface is realized comprises:
(1) initialization storage file:
Original data file needed to be stored to first among the distributed file management system before using, and the distributed file management system will finish each function of its burst and information storage automatically.
1) the initialization memory interface of user's calling data read-write interface, the parameter of calling are specific address, the burst requirement of data file on disk array;
2) the data call interface is filled up to relevant parameter in the message, and the calling communication module sends message to document management server;
3) communication module sends to the communication module of file manager after with message sequence;
4) file manager receives that message obtains the parameter of initial burst, according to the peculiar form of geological data file is carried out burst, and is the specific file of this document creation, stores putting into unified file after the data number behind the burst;
5) information (comprising the absolute path of filename, burst quantity, file fragmentation storage etc.) with file fragmentation stores into and carries out lasting storage in the database;
6) inform the performance that the data read and write interface burst is preserved by the mode that sends message to user program, data read and write interface returns to the user with the result again, and the initialization procedure of file is finished.
(2) obtain file fragmentation information:
The operation that the user carries out when using some or a plurality of burst of file is as follows:
1) user calls the burst scope of obtaining burst interface, specified file name, wanting to obtain;
2) data read and write interface is encapsulated as message with user's request, mails to document management server by communication module;
3) document management server receives message, and message is resolved, and obtains required parameter;
4) management by district module is read and write out corresponding file fragmentation information by the data base read-write interface from database;
5) document management server is sent back to data read and write interface with the information that obtains;
6) data read and write interface is read one or more bursts according to the burst information of receiving by the file read-write module, by user interface data is returned to the user.
(3) obtain complete raw data file:
Can call specified interface when need to be transferred, copy in raw data, all bursts are carried out amalgamation, obtain complete raw data file, concrete grammar is as follows:
1) user calls and returns the source document interface, the name of presenting a paper;
2) document management server is received request, and all burst information of specified file are sent to data read and write interface;
3) the file die section is collected all burst information, calls the new file of file read-write module creation, RNTO raw data file name, and read successively slicing files and write in the file of new establishment according to burst information;
4) amalgamation returns to the user with absolute path and the filename of new files after complete.
(4) deletion/updating file:
1) user interface is submitted deletion/update request to;
2) document management server receives deletion/update request;
3) the burst information administration module is read corresponding burst information;
4) finish deletion to file/renewal operation by the file read-write module according to concrete slicing files information;
5) operating result is sent back to data read and write interface by message;
6) data read and write interface returns to the user with the result after resolving message.
Two, document management server:
Document management server is unique supvr of All Files, has comprised the information of all current files, and all operations to file all need to obtain information from document management server at first.
As shown in Figure 3, the structure of document management server comprises:
(1) Business treatment: to the classification of service request, existing class of business comprises: initialization storing data files, the burst information of obtaining specified file, renewal specified file, deletion specified file;
(2) geological data is cut apart module: determine the side-play amount of cutting apart of file by the road, guarantee the integrality of minimum data unit;
Geological data has its structured features: as all being to store by the road.The road is the smallest element data structure of geological data storage.On the basis, road, can be a set with the track data merger with certain same alike result, such as CMP road collection, CRP road collection, big gun record etc.
Each processing corresponds to a functional module in the processing procedure of geological data.Functional module is by the road processing or by the road process of aggregation according to the characteristics selection of calculation task in the geological data processing procedure.The minimum treat data cell of functional module is the data partitioning unit of this module minimum.As, when the processing capacity module was pressed processing, the road was minimum data partitioning unit; When the processing capacity module was pressed the collection processing, the road set was minimum data partitioning unit.
The type that this requires to need the user to specify before file is carried out burst cut apart also can be carried out polytype cutting apart (but can increase disk storage space) simultaneously.
(3) file storage module: the file system interface of call operation system carries out read-write, deletion, the renewal operation of file;
(4) data base read-write interface:
The machine if document management server is delayed because document management server is unique supvr of All Files information, if with all deposit data in internal memory, will lose all fileinfos and can't recover.For the save data of persistence, data need to be stored in the database, and database provides the information storage mechanism of high efficient and reliable.The data base read-write interface provides the ability of accessing database for the management by district module.
(5) management by district module:
Access to burst information manages.The management by district module is responsible for all relevant affairs of burst, and coordinative file memory module, geological data are cut apart the function of module, and calling data storehouse read-write interface visits burst information.
The function that document management server is realized comprises:
(1) the specific form according to geological data carries out burst to large data, guarantees that each burst comprises complete geological data unit;
(2) original document that adds is carried out the burst storage;
(3) file fragmentation information is carried out lasting storage (write into Databasce), guarantee that the document management server machine of delaying restarts rear burst information and can not lose;
(4) provide file fragmentation information for data read and write interface;
Three, communication module:
Communication module has reduced as each needs the process of network service and has write separately the overhead of communication cryptology for each process in the system provides unified communication solution, has reduced the difficulty of program development, also is easier to realize the unification of communication mode.
As shown in Figure 4, the structure of communication module comprises:
(1) document management server information inquiry module:
Document management server can write oneself current network information after the startup among the configuration file, configuration file can leave the assigned address on the disk array in, document management server information inquiry module just can obtain the current location of document management server after reading this configuration file, thereby communicates by letter with document management server.
(2) message sequence module:
All data in transmission over networks must exist with the form of data stream.The message sequence module is carried out the serializing operation with user message, thereby adapts to the needs of Internet Transmission.When accepting message, need to carry out the unserializing operation to the data that receive equally, thus the message object that obtains receiving.
(3) Socket manager:
The Socket manager is responsible for the socket object of current use is managed (because can exist a plurality of socket to connect, communicate by letter such as time between document management server and a plurality of user programs), the basic network operations such as inquiry to destination node, TCP establishment of connection are provided.
（4）TCPSocket/UDPSocket:
To the encapsulation of operating system socket object, the interface that socket is operated is provided, make things convenient for the unified management of socket manager.
The function that communication module realizes comprises:
(1) provide the current file server to move the network information (ip, port etc.) of node;
(2) provide transmission, accept the interface of message;
(3) provide the ability of serializing, unserializing.
Claims (5)
1. distributed file management system that processes based on geological data, it is characterized in that: comprise data read and write interface, document management server and communication module, communication module provides the interface of messaging for data read and write interface and document management server, wherein:
The structure of described data read and write interface comprises Business Logic, file read-write module, file die section and local file information cache, is achieved as follows function:
(1) initialization storage file;
(2) obtain file fragmentation information;
(3) obtain complete raw data file;
(4) deletion/updating file;
The structure of described document management server comprises that Business treatment, geological data cut apart module, file storage module, data base read-write interface and management by district module, is achieved as follows function:
(1) the specific form according to geological data carries out burst to large data, guarantees that each burst comprises complete geological data unit;
(2) original document that adds is carried out the burst storage;
(3) with file fragmentation information write into Databasce;
(4) provide file fragmentation information for data read and write interface;
The structure of described communication module comprises document management server information inquiry module, message sequence module, Socket manager and TCPSocket/UDPSocket, is achieved as follows function:
(1) provide the current file server to move the network information of node;
(2) provide transmission, accept the interface of message;
(3) provide the ability of serializing, unserializing.
2. the distributed file management system that processes based on geological data according to claim 1, it is characterized in that: the step of described initialization storage file is:
1) the initialization memory interface of user's calling data read-write interface, the parameter of calling are specific address, the burst requirement of data file on disk array;
2) the data call interface is filled up to relevant parameter in the message, and the calling communication module sends message to document management server;
3) communication module sends to the communication module of file manager after with message sequence;
4) file manager receives that message obtains the parameter of initial burst, according to the form of geological data file is carried out burst, and is that this document creates file, stores putting into this document folder after the data number behind the burst;
5) information of file fragmentation is stored into carry out lasting storage in the database;
6) inform the performance that the data read and write interface burst is preserved by the mode that sends message to user program, data read and write interface returns to the user with the result again, and the initialization procedure of file is finished.
3. the distributed file management system that processes based on geological data according to claim 1, it is characterized in that: the described step of obtaining file fragmentation information is:
1) user calls the burst scope of obtaining burst interface, specified file name and wanting to obtain;
2) data read and write interface is encapsulated as message with user's request, mails to document management server by communication module;
3) document management server receives message, and message is resolved, and obtains required parameter;
4) management by district module is read and write out corresponding file fragmentation information by the data base read-write interface from database;
5) document management server is sent back to data read and write interface with the information that obtains;
6) data read and write interface is read one or more bursts according to the burst information of receiving by the file read-write module, by user interface data is returned to the user.
4. the distributed file management system that processes based on geological data according to claim 1, it is characterized in that: the described step of obtaining complete raw data file is:
1) user calls and returns the source document interface, the name of presenting a paper;
2) document management server is received request, and all burst information of specified file are sent to data read and write interface;
3) the file die section is collected all burst information, calls the new file of file read-write module creation, and RNTO raw data file name is read successively slicing files and write in the file of new establishment according to burst information;
4) amalgamation absolute path and the filename that will newly create file after complete returns to the user.
5. the distributed file management system that processes based on geological data according to claim 1, it is characterized in that: the step of described deletion/updating file is:
1) user interface is submitted deletion/update request to;
2) document management server receives deletion/update request;
3) the burst information administration module is read corresponding burst information;
4) finish deletion to file/renewal operation by the file read-write module according to concrete slicing files information;
5) operating result is sent back to data read and write interface by message;
6) data read and write interface returns to the user with the result after resolving message.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210317036.3A CN102880658B (en) | 2012-08-31 | 2012-08-31 | Distributed file management system based on seismic data process |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210317036.3A CN102880658B (en) | 2012-08-31 | 2012-08-31 | Distributed file management system based on seismic data process |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN102880658A true CN102880658A (en) | 2013-01-16 |
| CN102880658B CN102880658B (en) | 2016-11-16 |
Family
ID=47481984
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201210317036.3A Expired - Fee Related CN102880658B (en) | 2012-08-31 | 2012-08-31 | Distributed file management system based on seismic data process |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102880658B (en) |
Cited By (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102682078A (en) * | 2012-03-20 | 2012-09-19 | 浪潮电子信息产业股份有限公司 | Method for automatically and rapidly deploying NFS (network file system) sharing |
| CN103559036A (en) * | 2013-11-04 | 2014-02-05 | 北京中搜网络技术股份有限公司 | Data batch processing system and method based on Hadoop |
| CN104750657A (en) * | 2013-12-31 | 2015-07-01 | 中国石油化工股份有限公司 | Numerical simulation redundancy parallel computing method applicable to fracture-cavity type structure carbonate reservoirs |
| WO2016086649A1 (en) * | 2014-12-04 | 2016-06-09 | 中兴通讯股份有限公司 | Method, client, and storage medium for reading, writing, deleting, and querying a file |
| CN106383886A (en) * | 2016-09-21 | 2017-02-08 | 深圳市博瑞得科技有限公司 | Big data distribution type programming framework-based big data pre-statistics system and method |
| CN106547837A (en) * | 2016-10-13 | 2017-03-29 | 广西电网有限责任公司电力科学研究院 | A kind of distributed file system and its data documents disposal method |
| CN110019008A (en) * | 2017-11-03 | 2019-07-16 | 北京金山安全软件有限公司 | Data storage method and device |
| CN110321374A (en) * | 2018-10-23 | 2019-10-11 | 开采夫(杭州)科技有限公司 | Standard file I/O operations system and method based on distributed network |
| CN110320553A (en) * | 2018-03-30 | 2019-10-11 | 中国石油化工股份有限公司 | The distributed implementation method that seismic properties are sought |
| CN111026751A (en) * | 2019-11-22 | 2020-04-17 | 北京金山云网络技术有限公司 | Processing method, device and system of distributed table and electronic equipment |
| CN111258748A (en) * | 2018-12-03 | 2020-06-09 | 中国移动通信集团上海有限公司 | Distributed file system and control method |
| CN111696589A (en) * | 2020-06-04 | 2020-09-22 | 华夏吉泰(北京)科技有限公司 | Mass seismic data storage equipment and seismic professional data management system |
| CN111767264A (en) * | 2019-04-02 | 2020-10-13 | 中国石油化工股份有限公司 | Distributed storage method and data reading method based on geological information coding |
| CN112632005A (en) * | 2019-10-08 | 2021-04-09 | 中国石油化工股份有限公司 | Seismic data calculation method and system based on MPI |
| CN112685047A (en) * | 2020-12-31 | 2021-04-20 | 杭州当虹科技股份有限公司 | Rapid analysis system based on large file |
| CN113448946A (en) * | 2021-07-05 | 2021-09-28 | 星辰天合（北京）数据科技有限公司 | Data migration method and device and electronic equipment |
| CN116080423A (en) * | 2023-04-03 | 2023-05-09 | 电子科技大学 | Cluster unmanned vehicle energy supply system based on ROS and execution method thereof |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040210948A1 (en) * | 2003-01-25 | 2004-10-21 | Huazhon University Of Science Technology | Video splitting and distributed placement scheme for clustered video servers |
| CN102088491A (en) * | 2011-02-01 | 2011-06-08 | 西安建筑科技大学 | Distributed storage oriented cloud storage security architecture and data access method thereof |
| CN102281321A (en) * | 2011-04-25 | 2011-12-14 | 程旭 | Data cloud storage partitioning and backup method and device |
-
2012
- 2012-08-31 CN CN201210317036.3A patent/CN102880658B/en not_active Expired - Fee Related
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040210948A1 (en) * | 2003-01-25 | 2004-10-21 | Huazhon University Of Science Technology | Video splitting and distributed placement scheme for clustered video servers |
| CN102088491A (en) * | 2011-02-01 | 2011-06-08 | 西安建筑科技大学 | Distributed storage oriented cloud storage security architecture and data access method thereof |
| CN102281321A (en) * | 2011-04-25 | 2011-12-14 | 程旭 | Data cloud storage partitioning and backup method and device |
Non-Patent Citations (1)
| Title |
|---|
| 杨进: ""地震数据处理中的并行计算技术研究"", 《中国硕士论文全文数据库》 * |
Cited By (23)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102682078A (en) * | 2012-03-20 | 2012-09-19 | 浪潮电子信息产业股份有限公司 | Method for automatically and rapidly deploying NFS (network file system) sharing |
| CN103559036A (en) * | 2013-11-04 | 2014-02-05 | 北京中搜网络技术股份有限公司 | Data batch processing system and method based on Hadoop |
| CN104750657A (en) * | 2013-12-31 | 2015-07-01 | 中国石油化工股份有限公司 | Numerical simulation redundancy parallel computing method applicable to fracture-cavity type structure carbonate reservoirs |
| WO2016086649A1 (en) * | 2014-12-04 | 2016-06-09 | 中兴通讯股份有限公司 | Method, client, and storage medium for reading, writing, deleting, and querying a file |
| CN106383886A (en) * | 2016-09-21 | 2017-02-08 | 深圳市博瑞得科技有限公司 | Big data distribution type programming framework-based big data pre-statistics system and method |
| CN106547837A (en) * | 2016-10-13 | 2017-03-29 | 广西电网有限责任公司电力科学研究院 | A kind of distributed file system and its data documents disposal method |
| CN110019008A (en) * | 2017-11-03 | 2019-07-16 | 北京金山安全软件有限公司 | Data storage method and device |
| CN110320553A (en) * | 2018-03-30 | 2019-10-11 | 中国石油化工股份有限公司 | The distributed implementation method that seismic properties are sought |
| CN110321374A (en) * | 2018-10-23 | 2019-10-11 | 开采夫(杭州)科技有限公司 | Standard file I/O operations system and method based on distributed network |
| CN110321374B (en) * | 2018-10-23 | 2022-03-25 | 开采夫(杭州)科技有限公司 | Standard file IO operating system and method based on distributed network |
| CN111258748A (en) * | 2018-12-03 | 2020-06-09 | 中国移动通信集团上海有限公司 | Distributed file system and control method |
| CN111258748B (en) * | 2018-12-03 | 2023-09-05 | 中国移动通信集团上海有限公司 | Distributed file system and control method |
| CN111767264A (en) * | 2019-04-02 | 2020-10-13 | 中国石油化工股份有限公司 | Distributed storage method and data reading method based on geological information coding |
| CN112632005A (en) * | 2019-10-08 | 2021-04-09 | 中国石油化工股份有限公司 | Seismic data calculation method and system based on MPI |
| CN112632005B (en) * | 2019-10-08 | 2024-01-23 | 中国石油化工股份有限公司 | MPI-based seismic data calculation method and system |
| CN111026751A (en) * | 2019-11-22 | 2020-04-17 | 北京金山云网络技术有限公司 | Processing method, device and system of distributed table and electronic equipment |
| CN111026751B (en) * | 2019-11-22 | 2024-02-09 | 北京金山云网络技术有限公司 | Distributed form processing method, device and system and electronic equipment |
| CN111696589A (en) * | 2020-06-04 | 2020-09-22 | 华夏吉泰(北京)科技有限公司 | Mass seismic data storage equipment and seismic professional data management system |
| CN112685047A (en) * | 2020-12-31 | 2021-04-20 | 杭州当虹科技股份有限公司 | Rapid analysis system based on large file |
| CN113448946A (en) * | 2021-07-05 | 2021-09-28 | 星辰天合（北京）数据科技有限公司 | Data migration method and device and electronic equipment |
| CN113448946B (en) * | 2021-07-05 | 2024-01-12 | 北京星辰天合科技股份有限公司 | Data migration method and device and electronic equipment |
| CN116080423A (en) * | 2023-04-03 | 2023-05-09 | 电子科技大学 | Cluster unmanned vehicle energy supply system based on ROS and execution method thereof |
| CN116080423B (en) * | 2023-04-03 | 2023-06-27 | 电子科技大学 | Cluster unmanned vehicle energy supply system based on ROS and execution method thereof |
Also Published As
| Publication number | Publication date |
|---|---|
| CN102880658B (en) | 2016-11-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102880658A (en) | Distributed file management system based on seismic data processing | |
| US10700991B2 (en) | Multi-cluster resource management | |
| US10764045B2 (en) | Encrypting object index in a distributed storage environment | |
| US20210357361A1 (en) | Snapshot-enabled storage system implementing algorithm for efficient reading of data from stored snapshots | |
| US20210019063A1 (en) | Utilizing data views to optimize secure data access in a storage system | |
| US10659225B2 (en) | Encrypting existing live unencrypted data using age-based garbage collection | |
| AU2016405587B2 (en) | Splitting and moving ranges in a distributed system | |
| WO2020204880A1 (en) | Snapshot-enabled storage system implementing algorithm for efficient reclamation of snapshot storage space | |
| US8862617B2 (en) | System and method for replicating objects in a distributed storage system | |
| CN106294870B (en) | Object-based distribution cloud storage method | |
| US11113155B1 (en) | Archiving and restoration of distributed database log records | |
| CN105677251B (en) | Storage system based on Redis cluster | |
| US20210055885A1 (en) | Enhanced data access using composite data views | |
| US10650027B2 (en) | Access accelerator for active HBase database regions | |
| US20230362248A1 (en) | Tiered Queuing System | |
| US11314459B2 (en) | Distributed metadata management in a distributed storage system | |
| CN106953910A (en) | A kind of Hadoop calculates storage separation method | |
| CN103595799A (en) | Method for achieving distributed shared data bank | |
| US9578120B1 (en) | Messaging with key-value persistence | |
| Venner et al. | Pro apache hadoop | |
| US11868334B2 (en) | Integrating change tracking of storage objects of a distributed object storage database into a distributed storage system | |
| CN113965582B (en) | Mode conversion method and system, and storage medium | |
| Donnelly et al. | Confuga: scalable data intensive computing for POSIX Workflows | |
| CN106161526A (en) | A kind of distributed storage technology based on file system | |
| US20240104081A1 (en) | Integrating change tracking of storage objects of a distributed object storage database into a distributed storage system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20161116 Termination date: 20190831 |
|
| CF01 | Termination of patent right due to non-payment of annual fee |