CN102880658B - Distributed file management system based on seismic data process - Google Patents
Distributed file management system based on seismic data process Download PDFInfo
- Publication number
- CN102880658B CN102880658B CN201210317036.3A CN201210317036A CN102880658B CN 102880658 B CN102880658 B CN 102880658B CN 201210317036 A CN201210317036 A CN 201210317036A CN 102880658 B CN102880658 B CN 102880658B
- Authority
- CN
- China
- Prior art keywords
- file
- data
- interface
- read
- burst
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of distributed file management system based on seismic data process, including data read and write interface, document management server and communication module, communication module provides the interface of messaging for data read and write interface and document management server.The positive effect of the present invention is: uses the principle of distributed file system, is automatically performed burst and the management of file, and to realize Parallel I/O at a high speed, and offer is concisely and efficiently file management.User only need to call simply corresponding interface just can with the various operations of complete paired data, and completely completed the burst to file and management by Automatic Program and without artificial intervention.Solve under cluster environment, the mutual competitions and conflicts of identical file on multiple calculating node visit integrated disc arrays；The problem that the I/O performance caused the locking protection of file due to file system declines to a great extent fundamentally is avoided by the method making file fragmentation.
Description
Technical field
The present invention relates to a kind of distributed file management system based on seismic data process.
Background technology
Distributed file system (Distributed File System) refers to the physical store of file system management
Resource is not necessarily directly connected on the local node, but is connected with node by computer network.
Along with the continuous acceleration of human information process, traditional storage organization cannot support the quick-fried of data volume
Fried formula increases.The storage of big data and process have become a study hotspot of computer nowadays science.Relating to
And when arriving the process to big data, the I/O performance of traditional file systems has become the bottleneck limiting whole system,
Fault-tolerant and expansible problem is the most more acute.And the appearance of distributed file system is just solving this problem,
In I/O performance, fault-tolerant, convenient all the obtaining of expansibility greatly promotes.
Under the overall background of cloud computing, rigors and the becoming increasingly popular of parallel computation to big data,
Distributed file system is obtained more be widely applied.
The research boom of distributed file system starts from the GFS file system of google.Google File
System (GFS) is based on Linux the proprietary distribution that Google company develops to meet its demand
Formula file system.
GFS is an extendible distributed file system, for large-scale, distributed, to a large amount of numbers
According to the application conducted interviews.It runs on cheap common hardware, but can provide fault tolerance.It can
To provide, to substantial amounts of user, the service that overall performance is higher.GFS is relative to traditional distributed file system
Maximum feature be it mistake as normal rather than abnormal process, it is provided that one is efficiently and reliably
Fault tolerant mechanism.
The node of GFS is divided into two classes, and one is host node, and one is back end, uses simple C/S
Mode operation.
In GFS, the data of storage are divided into the block of fixed size, are respectively stored in different memory nodes
On.Host node is responsible for the information of All Files block, is responsible for balancing the storage load of each back end.
Master server does not the most store the chunk data of reality, but stores the metadata relevant to chunk data,
Such data such as mapping form is by 64 label mappings to chunk data position and the file of composition thereof, big
Blocks of data locations of copies, which process are reading and writing specific long data block or tracking replicates chunk data
" snapshot ".All these metadata are periodically received by master server to be come from each data block server
Renewal (" heartbeat message ") keep last state.When user needs to access data, provided by GFS
Interface operate, interface by network access host node data base, obtain concrete data block
Position, by the corresponding data of Network Capture.
Prior art related to the present invention includes:
(1)Hadoop:
Do not announce the source code that its GFS realizes due to google, hadoop is important as google tri-
Cloud computing paper (mapreduce, gfs, bigtable) implement the most wide-spread, also
Arrive widely important application.
Hadoop is a distributed system architecture, Apache foundation develop.User can be
In the case of not knowing about distributed low-level details, develop distributed program.Make full use of the power of cluster at a high speed
Computing and storage.Hadoop achieves a distributed file system (Hadoop Distributed File
System), it is called for short HDFS.HDFS has the feature of high fault tolerance, and is designed to be deployed in commonly
In commercial hardware.And it provides high transmission rates to carry out the data of access application, being suitable for those has super large
The application program of data set.HDFS relaxes the requirement of POSIX, can access file system in the form of streaming
In data.
For external client, HDFS is just as a traditional hierarchical file system.Can create, delete
Remove, move or Rename file, etc..But the framework of HDFS is based on one group of specific node structure
Build.These nodes include NameNode, and it provides Metadata Service inside HDFS；DataNode,
It provides memory block for HDFS.Owing to only existing a NameNode, so there is Single Point of Faliure,
Must assure that the high availability of NameNode, this is also the implementation of GFS.
The file being stored in HDFS is divided into block, is then copied in multiple computer by these blocks
(DataNode).This differs widely with traditional RAID framework.The size (usually 64MB) of block
Determined by client computer when creating file with the number of blocks replicated.NameNode can control All Files behaviour
Make.All communications within HDFS are all based on the ICP/IP protocol of standard.
(2) disk array:
Disk array be by a lot of cheap, capacity is less, stability is higher, the slower disk of speed, be combined into
One large-scale disk group, utilizes indivedual disk to provide data the produced addition whole disk system of effect promoting
Usefulness.Utilize this technology simultaneously, data are cut into many sections, leaves in respectively on each hard disk.
Disk array can also utilize the idea of parity check (Parity Check), arbitrary hard disk failure in array
Time, still can read data, when data reconstruction, again insert in new hard disk after data are computed.RAID
The data throughout of storage system is greatly improved by storing and read data the most simultaneously
(Throughput).In RAID, a lot of disc driver can be allowed to transmit data simultaneously, and these magnetic
Disk drive is the most again a disc driver, drives so using RAID can reach single disk
Dynamic several times of device, tens times even speed of hundreds of times.
(3) Linux cluster high-performance calculation:
Cluster is main a kind of HPC hardware: cluster (cluster) is exactly one group
The set of MPP (massively parallel processor (MPP)).Processor in cluster is commonly called node, and it has certainly
Oneself CPU, internal memory, operating system, I/O subsystem, and can communicate with other nodes.
There is a lot of place all to use common work station to run Linux at present and other Open Source Code Software are served as
Node in cluster.
High-performance calculation (High-Performance Computing) is a branch of computer science, and it causes
Power is in exploitation supercomputer, research parallel algorithm and exploitation related software.High-performance calculation is mainly studied such as
As weather forecast, terrain analysis, store and process mass data, data mining, image procossing etc..
In the process of geological data, single seismic data acquisition may be up to hundreds of G, traditional single joint
The data processing method of point needs a couple of days, and the most tens of skies can complete the data to a file and process.
Process concurrently for that purpose it is necessary to data are segmented on multiple node.Ideally, if join simultaneously
Be n with the nodes processed, then total runtime can shorten to 1/n, but actual run time is the highest
In ideal value.Reason be in order to realize multiple node concurrent reading and concurrent writing, by different nodes according to difference
Document misregistration amount go read same file, process nodes more in the case of, there occurs the tightest
The situation of the contention access of weight, file system causes greatly expense to the locking of file, makes I/O speed
Situation when reading far below single node, the I/O time becomes the severe bottleneck of system.
If being multiple subfile by individual data file division artificially, different calculating nodes is allowed to read difference
File fragmentation, it is possible to avoid the situation of contention access to occur.Although this mode solves reading and writing of files
Conflict, but has been simultaneously introduced a lot of problem: the most each file is required for carrying out artificially burst, and earthquake
Data have the form that it is intrinsic, therefore, to assure that each burst include complete minimal processing unit (road or
Road collection), it is impossible to realize burst by simple file division；Secondly because the file after burst is numerous, can give
The management of file itself brings greatly chaotic, brings difficulty also to writing of application program.In order to solve
These problems need substantial amounts of manual operation, have not only increased the weight of the burden of file management personnel, developer,
Also the reliability making system itself reduces.
Summary of the invention
In order to overcome the disadvantages mentioned above of prior art, the invention provides a kind of based on seismic data process point
Cloth file management system, uses the principle of distributed file system, is automatically performed burst and the management of file,
To realize Parallel I/O at a high speed, and offer is concisely and efficiently file management.User only need to call phase simply
The interface answered just can with the various operations of complete paired data, and be completed to divide file by Automatic Program completely
Sheet and management and without artificial intervention.
The technical solution adopted for the present invention to solve the technical problems is: a kind of based on seismic data process point
Cloth file management system, including data read and write interface, document management server and communication module, communicate mould
The interface of block offer messaging is for data read and write interface and document management server, wherein:
The structure of described data read and write interface includes Business Logic, file read-write module, file die section
With local file information cache, it is achieved following function:
(1) storage file is initialized；
(2) file fragmentation information is obtained；
(3) complete raw data file is obtained；
(4) delete/update file；
The structure of described document management server includes that Business treatment, geological data segmentation module, file are deposited
Storage module, data base read-write interface and management by district module, it is achieved following function:
(1) according to the specific form of geological data, big data are carried out burst, it is ensured that each burst comprises
Complete seismic data unit；
(2) original document added is carried out burst storage；
(3) by file fragmentation information write into Databasce；
(4) file fragmentation information is provided for data read and write interface；
The structure of described communication module include document management server information inquiry module, message sequence module,
Socket manager and TCPSocket/UDPSocket, it is achieved following function:
(1) current file server is provided to run the network information of node；
(2) transmission, the interface of reception message are provided；
(3) serializing, the ability of unserializing are provided.
Compared with prior art, the positive effect of the present invention is: solve under cluster environment, multiple calculating
The mutual competitions and conflicts of identical file on node visit integrated disc array；By make the method for file fragmentation from
Fundamentally avoid that the I/O performance caused the locking protection of file due to file system declines to a great extent asks
Topic.Present invention have the advantage that
(1) coordinating the distributed-computation program run in group system, each program can be with individual processing
One burst of one large data file, rather than read single data file with the competition of other node, greatly
Reduce greatly the time that program spends in I/O operation, in the case of calculating nodal parallel calculating at n,
Make program operation total time close to ideal value 1/n, the problem solving I/O bottleneck, send out to greatest extent
Wave the usefulness of parallel computation.
(2) unified document management server is used.When obtaining the benefit that file fragmentation is brought, for
Fragmentation, burst information is avoided to manage the burden brought, by introducing document management server, with
The file fragmentation providing automatization centered by document management server manages, and all of work is by file management
Server and data read and write interface are automatically performed, omnidistance without manpower intervention, greatly reduce the use of system
Difficulty and the human cost of maintenance.
(3) use unified communication module interface, greatly reduce the difficulty of communications portion program development,
Unified communication construction also makes program have more preferable uniformity, it is simple to the maintenance in later stage and upgrading.
Accompanying drawing explanation
Examples of the present invention will be described by way of reference to the accompanying drawings, wherein:
Fig. 1 is the system communication structure schematic diagram of the present invention；
Fig. 2 is the structural representation of data read and write interface；
Fig. 3 is the structural representation of document management server；
Fig. 4 is the structural representation of communication module.
Detailed description of the invention
The application scenarios of the present invention is enterprise-level group system, has multiple calculating node, and each node is by height
Speed network connects, and can reach the highest transmission bandwidth.All data are all deposited on disk array, institute
There is calculating node all to may have access to disk array and carry out data read-write operation.
Each calculating node can be read at one or more data file burst by data read and write interface
Reason.Distribution and the collection of process task are completed by unified distributed computing framework, and the present invention applies frame for this
The reading and writing data of frame provides to be supported.
A kind of distributed file management system based on seismic data process, as it is shown in figure 1, include: data are read
Write interface, document management server and communication module；Wherein:
The data operating interface that described data read and write interface uses to user provide reading and writing of files basic operation and
Read slicing files, the higher level operation of acquisition file fragmentation data, and by communication module and file management
Server realizes mutual.
Described document management server is the chief executive of distributed file system, is responsible for dividing of storage file
Sheet information and deposit position, complete the work of the initial burst to file, and accepting the inquiry from user interface please
Ask, return the corresponding fileinfo of user, and communicated with user program by communication module.
Described communication module provides reliable and stable communication equipment for data read and write interface and document management server
System, communication mode uses message passing mechanism, to provide good decoupling and asynchronous performance.
Owing to the user program in group system and document management server are likely located on different nodes, must
Inter-process communication mechanisms must be introduced.Communication in the present invention all uses Message-Passing Model, is sent out by Ethernet
Send message to realize communication process.All communications in system are completed by communication module, and communication module provides
The interface of messaging is for data read and write interface and document management server.
Data read and write interface is filed a request to document management server to send message mode, file-management services
Device returns corresponding fileinfo, and data read and write interface obtains corresponding information (name of file fragmentation and storage
Path) after just can carry out the operation of file.
Data are stored in each distributed node in a distributed manner different from other distributed document, at this
In the application scenarios of invention, all of public documents all leave in disk array, and each calculating node can
To access the file in disk array as accessing local file.
It is all directly to read from disk array that all of process accesses the mode of file, and each process can make
Data manipulation is carried out with the file system interface of operating system.
One, data read and write interface:
Data read and write interface provides the user the interface of operation file burst easy to use.User is only by number
The all operations to file just can be completed according to read-write interface, and need not be again by other external means.Number
In the way of storehouse, user it is supplied to according to read-write interface.
As in figure 2 it is shown, the structure of data read and write interface includes:
(1) Business Logic: the operation to user is classified, and perform corresponding operation；
(2) file read-write module: according to the file fragmentation information obtained, obtains corresponding from disk array
Data file；
(3) file die section: obtain all of file fragmentation information, read each file fragmentation and merge,
Realize the split to burst；
(4) local file information cache: the burst information to obtain from document management server is buffered in this
Ground so that user reduces the pressure to document management server in the case of repeated multiple times reading same slice,
The read latch caused because accessing document management server can also be reduced.
The function that data read and write interface realizes includes:
(1) initialization storage file:
Original data file needs first to be stored among distributed file management system before the use,
Distributed file management system will be automatically performed each function of its burst and information storage.
1) user calls the initialization memory interface of data read and write interface, and the parameter called is that data file is at magnetic
Absolute address on disk array, burst requirement；
2) relevant parameter is filled up in message by data read and write interface, and calling communication module is to file-management services
Device sends message；
3) communication module will be sent to the communication module of file manager after message sequence；
4) file manager receives message and obtains the parameter of initial burst, according to the peculiar form pair of geological data
File carries out burst, and is the specific file of this document creation, puts into after the data number after burst
Unified file stores；
5) information of file fragmentation (is included the absolute path that filename, burst quantity, file fragmentation store
Deng) storage in data base, carry out lasting storage；
6) inform that what data read and write interface burst preserved completes feelings by the way of message by sending to user program
Condition, data read and write interface returns result to user again, and the initialization procedure of file completes.
(2) file fragmentation information is obtained:
The operation that user is carried out when using some or multiple burst of file is as follows:
1) user calls and obtains burst interface, specified file name, the burst scope to be obtained；
2) request of user is encapsulated as message by data read and write interface, mails to file management by communication module and takes
Business device；
3) document management server receives message, resolves message, it is thus achieved that required parameter；
4) management by district module is read and write out corresponding file fragmentation from data base by data base read-write interface and is believed
Breath；
5) information of acquirement is sent back to data read and write interface by document management server；
6) data read and write interface is read one or more according to the file fragmentation information received by file read-write module
Burst, returns data to user by user interface.
(3) complete raw data file is obtained:
Specified interface can be called initial data needs are transferred, copy when, all bursts are carried out
Split, obtains complete raw data file, and concrete grammar is as follows:
1) user calls return original document interface, name of presenting a paper；
2) document management server receives request, and the All Files burst information of specified file is sent to data
Read-write interface；
3) file die section collects all of file fragmentation information, calls file read-write module creation new
File, RNTO raw data file name, and sequential read out slicing files according to file fragmentation information and write
Enter in newly created file；
4) after split, absolute path and the filename of new files are returned to user.
(4) deletion/renewal file:
1) user interface submission deletion/more newly requested；
2) document management server reception deletion/more newly requested；
3) management by district module reads corresponding burst information；
4) deletion to file/renewal operation is completed by file read-write module according to concrete file fragmentation information；
5) operating result is sent back to data read and write interface by message；
6) data read and write interface returns result to user after resolving message.
Two, document management server:
Document management server is unique manager of All Files, contains the information of all current files,
All operations to file are required for obtaining information from document management server at first.
As it is shown on figure 3, the structure of document management server includes:
(1) Business treatment: the classification to service request, existing class of business includes: initialize storage
Data file, the burst information of acquisition specified file, renewal specified file, deletion specified file；
(2) geological data segmentation module: determine the segmentation side-play amount of file by road, it is ensured that minimum data unit
Integrity；
Geological data has its structured features: as Dou Shian road stores.Road is the minimum of geological data storage
Metadata structure.Can be a set by the track data merger with certain same alike result on the basis of road, as
CMP road collection, CRP road collection, big gun record etc..
In the processing procedure of geological data, each process corresponds to a functional module.At geological data
During reason, functional module selects Shi An road process or press process of aggregation according to the feature of the task of calculating.Merit
The minimum treat data cell of energy module is the data partitioning unit that this module is minimum.As, when processing function mould
When Kuai An road processes, road is minimum data partitioning unit；When processing functional module and pressing collection process, road
Set is minimum data partitioning unit.
This requirement needed user to specify the type of segmentation before file is carried out burst, it is possible to carry out many simultaneously
The segmentation (but disk storage space can be increased) of type.
(3) file storage module: the file system interface of call operation system carry out the read-write of file, deletion,
Update operation；
(4) data base read-write interface:
The machine if document management server is delayed, owing to document management server is unique management of All Files information
Person, if leaving in internal memory by all data, will lose all of fileinfo and cannot recover.In order to hold
The preservation data of longization, data need to be stored in data base, and data base provides the information of high efficient and reliable
Memory mechanism.Data base read-write interface provides the ability accessing data base for management by district module.
(5) management by district module:
Access to burst information is managed.Management by district module is responsible for all affairs that burst is relevant, association
Adjust the function of file storage module, geological data segmentation module, and call data base read-write interface and access point
Sheet information.
The function that document management server realizes includes:
(1) according to the specific form of geological data, big data are carried out burst, it is ensured that each burst comprises
Complete seismic data unit；
(2) original document added is carried out burst storage；
(3) file fragmentation information is carried out lasting storage (write into Databasce), it is ensured that file-management services
The device machine of delaying is restarted rear burst information and will not be lost；
(4) file fragmentation information is provided for data read and write interface；
Three, communication module:
Communication module provides unified communication solution for each process in system, decreases as each need
The process wanting network service individually writes the overhead of communication cryptology, reduces the difficulty of program development, also
It is more easily implemented the unification of communication mode.
As shown in Figure 4, the structure of communication module includes:
(1) document management server information inquiry module:
The current network information of oneself can be write among configuration file by document management server upon actuation, joins
Putting the appointment position that file can leave on disk array, document management server information inquiry module reads should
Just can obtain the current location of document management server after configuration file, thus communicate with document management server.
(2) message sequence module:
All data in transmission over networks must be presented in data stream.Message sequence module will be used
Family message carries out serializing operation, thus adapts to the needs of network transmission.It is right to need also exist for when accepting message
The data received carry out unserializing operation, thus obtain the message object received.
(3) Socket manager:
Socket manager is responsible for being managed currently used socket object (because can exist multiple
Socket connects, and communicates while between document management server with multiple user programs), it is provided that to mesh
The inquiry of node, the basic network operation such as TCP establishment of connection.
(4)TCPSocket/UDPSocket:
Encapsulation to operating system socket object, it is provided that the interface operating socket, facilitates socket
The unified management of manager.
The function that communication module realizes includes:
(1) current file server is provided to run the network information (ip, port etc.) of node；
(2) offer sends, accepts the interface of message；
(3) serializing, the ability of unserializing are provided.
Claims (4)
1. a distributed file management system based on seismic data process, it is characterised in that: include data
Read-write interface, document management server and communication module, communication module provides the interface of messaging for data
Read-write interface and document management server use, wherein:
The structure of described data read and write interface includes Business Logic, file read-write module, file die section
With local file information cache, it is achieved following function:
(1) storage file is initialized；
(2) file fragmentation information is obtained；
(3) complete raw data file is obtained；
(4) delete/update file；
The structure of described document management server includes that Business treatment, geological data segmentation module, file are deposited
Storage module, data base read-write interface and management by district module, it is achieved following function:
(1) according to the specific form of geological data, big data are carried out burst, it is ensured that each burst comprises
Complete seismic data unit；
(2) original document added is carried out burst storage；
(3) by file fragmentation information write into Databasce；
(4) file fragmentation information is provided for data read and write interface；
The structure of described communication module include document management server information inquiry module, message sequence module,
Socket manager and TCPSocket/UDPSocket, it is achieved following function:
(1) current file server is provided to run the network information of node；
(2) transmission, the interface of reception message are provided；
(3) serializing, the ability of unserializing are provided；
All of public documents all leave in disk array；
Described acquisition file fragmentation information is:
1) user calls and obtains burst interface, specified file name and the burst scope to be obtained；
2) request of user is encapsulated as message by data read and write interface, mails to file management by communication module and takes
Business device；
3) document management server receives message, resolves message, it is thus achieved that required parameter；
4) management by district module reads corresponding file fragmentation from data base by data base read-write interface and believes
Breath；
5) information of acquirement is sent back to data read and write interface by document management server；
6) data read and write interface is read one or more according to the file fragmentation information received by file read-write module
Burst, returns data to user by user interface.
Distributed file management system based on seismic data process the most according to claim 1, it is special
Levy and be: the described storage file that initializes is:
1) user calls the initialization memory interface of data read and write interface, and the parameter called is that data file is at magnetic
Absolute address on disk array, burst requirement；
2) relevant parameter is filled up in message by data read and write interface, and calling communication module is to file-management services
Device sends message；
3) communication module will be sent to the communication module of file manager after message sequence；
4) file manager receives message and obtains the parameter of initial burst, according to the form of geological data to file
Carry out burst, and create file for this document, will put into after the data number after burst in this document folder
Row storage；
5) file fragmentation information is stored data base carries out lasting storage；
6) inform that what data read and write interface burst preserved completes feelings by the way of message by sending to user program
Condition, data read and write interface returns result to user again, and the initialization procedure of file completes.
Distributed file management system based on seismic data process the most according to claim 1, it is special
Levy and be: the complete raw data file of described acquisition is:
1) user calls return original document interface, name of presenting a paper；
2) document management server receives request, and the All Files burst information of specified file is sent to data
Read-write interface；
3) file die section collects all of file fragmentation information, calls file read-write module creation new
File, RNTO raw data file name, sequential read out slicing files according to file fragmentation information and write
In newly created file；
4) after split, absolute path and the filename of newly created file are returned to user.
Distributed file management system based on seismic data process the most according to claim 1, it is special
Levy and be: described deletion/renewal file is:
1) user interface submission deletion/more newly requested；
2) document management server reception deletion/more newly requested；
3) management by district module reads corresponding file fragmentation information；
4) deletion to file/renewal operation is completed by file read-write module according to concrete file fragmentation information；
5) operating result is sent back to data read and write interface by message；
6) data read and write interface returns result to user after resolving message.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210317036.3A CN102880658B (en) | 2012-08-31 | 2012-08-31 | Distributed file management system based on seismic data process |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210317036.3A CN102880658B (en) | 2012-08-31 | 2012-08-31 | Distributed file management system based on seismic data process |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN102880658A CN102880658A (en) | 2013-01-16 |
| CN102880658B true CN102880658B (en) | 2016-11-16 |
Family
ID=47481984
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201210317036.3A Expired - Fee Related CN102880658B (en) | 2012-08-31 | 2012-08-31 | Distributed file management system based on seismic data process |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102880658B (en) |
Families Citing this family (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102682078A (en) * | 2012-03-20 | 2012-09-19 | 浪潮电子信息产业股份有限公司 | Method for automatically and rapidly deploying NFS (network file system) sharing |
| CN103559036A (en) * | 2013-11-04 | 2014-02-05 | 北京中搜网络技术股份有限公司 | Data batch processing system and method based on Hadoop |
| CN104750657A (en) * | 2013-12-31 | 2015-07-01 | 中国石油化工股份有限公司 | Numerical simulation redundancy parallel computing method applicable to fracture-cavity type structure carbonate reservoirs |
| CN105718484A (en) * | 2014-12-04 | 2016-06-29 | 中兴通讯股份有限公司 | File writing method, file reading method, file deletion method, file query method and client |
| CN106383886B (en) * | 2016-09-21 | 2019-08-30 | 深圳市博瑞得科技有限公司 | A kind of big data based on the distributed programmed frame of big data is united system and method in advance |
| CN106547837A (en) * | 2016-10-13 | 2017-03-29 | 广西电网有限责任公司电力科学研究院 | A kind of distributed file system and its data documents disposal method |
| CN110019008A (en) * | 2017-11-03 | 2019-07-16 | 北京金山安全软件有限公司 | Data storage method and device |
| CN110320553A (en) * | 2018-03-30 | 2019-10-11 | 中国石油化工股份有限公司 | The distributed implementation method that seismic properties are sought |
| CN110321374B (en) * | 2018-10-23 | 2022-03-25 | 开采夫(杭州)科技有限公司 | Standard file IO operating system and method based on distributed network |
| CN111258748B (en) * | 2018-12-03 | 2023-09-05 | 中国移动通信集团上海有限公司 | Distributed file system and control method |
| CN111767264A (en) * | 2019-04-02 | 2020-10-13 | 中国石油化工股份有限公司 | Distributed storage method and data reading method based on geological information coding |
| CN112632005B (en) * | 2019-10-08 | 2024-01-23 | 中国石油化工股份有限公司 | MPI-based seismic data calculation method and system |
| CN111026751B (en) * | 2019-11-22 | 2024-02-09 | 北京金山云网络技术有限公司 | Distributed form processing method, device and system and electronic equipment |
| CN111696589A (en) * | 2020-06-04 | 2020-09-22 | 华夏吉泰(北京)科技有限公司 | Mass seismic data storage equipment and seismic professional data management system |
| CN112685047A (en) * | 2020-12-31 | 2021-04-20 | 杭州当虹科技股份有限公司 | Rapid analysis system based on large file |
| CN113448946B (en) * | 2021-07-05 | 2024-01-12 | 北京星辰天合科技股份有限公司 | Data migration method and device and electronic equipment |
| CN116080423B (en) * | 2023-04-03 | 2023-06-27 | 电子科技大学 | Cluster unmanned vehicle energy supply system based on ROS and execution method thereof |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102088491A (en) * | 2011-02-01 | 2011-06-08 | 西安建筑科技大学 | Distributed storage oriented cloud storage security architecture and data access method thereof |
| CN102281321A (en) * | 2011-04-25 | 2011-12-14 | 程旭 | Data cloud storage partitioning and backup method and device |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1195274C (en) * | 2003-01-25 | 2005-03-30 | 华中科技大学 | Division distributed storage method of program source based on cluster video server |
-
2012
- 2012-08-31 CN CN201210317036.3A patent/CN102880658B/en not_active Expired - Fee Related
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102088491A (en) * | 2011-02-01 | 2011-06-08 | 西安建筑科技大学 | Distributed storage oriented cloud storage security architecture and data access method thereof |
| CN102281321A (en) * | 2011-04-25 | 2011-12-14 | 程旭 | Data cloud storage partitioning and backup method and device |
Non-Patent Citations (1)
| Title |
|---|
| "地震数据处理中的并行计算技术研究";杨进;《中国硕士论文全文数据库》;20081231;论文的第四章、第五章 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN102880658A (en) | 2013-01-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102880658B (en) | Distributed file management system based on seismic data process | |
| US11438413B2 (en) | Intelligent data storage and management for cloud computing | |
| US10700991B2 (en) | Multi-cluster resource management | |
| AU2016405587B2 (en) | Splitting and moving ranges in a distributed system | |
| US20180011874A1 (en) | Peer-to-peer redundant file server system and methods | |
| US11237750B2 (en) | Dynamic volume replication factor adjustment | |
| DE102021113808A1 (en) | Handling replications between different networks | |
| US10740005B1 (en) | Distributed file system deployment on a data storage system | |
| US11797491B2 (en) | Inofile management and access control list file handle parity | |
| US20200174671A1 (en) | Bucket views | |
| CN106294870B (en) | Object-based distribution cloud storage method | |
| US11797213B2 (en) | Freeing and utilizing unused inodes | |
| US11449260B2 (en) | Persistent hole reservation | |
| CN102833580A (en) | High-definition video application system and method based on infiniband | |
| US8701113B2 (en) | Switch-aware parallel file system | |
| US20200301948A1 (en) | Timestamp consistency for synchronous replication | |
| CN105468296B (en) | Nothing based on virtual platform shares memory management method | |
| CN105022779A (en) | Method for realizing HDFS file access by utilizing Filesystem API |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20161116 Termination date: 20190831 |
|
| CF01 | Termination of patent right due to non-payment of annual fee |