CN108062308A - A kind of method and system of Distributed Storage - Google Patents
A kind of method and system of Distributed Storage Download PDFInfo
- Publication number
- CN108062308A CN108062308A CN201610975124.0A CN201610975124A CN108062308A CN 108062308 A CN108062308 A CN 108062308A CN 201610975124 A CN201610975124 A CN 201610975124A CN 108062308 A CN108062308 A CN 108062308A
- Authority
- CN
- China
- Prior art keywords
- distributed
- data
- memory database
- storage
- distributed storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of method and system of Distributed Storage, this method includes：New data information writes the memory database in real time；The memory database and distributed data base collectively form the framework of distributed storage；Update the distributed storage of the memory database and distributed data base；The distributed storage that the memory database and distributed data base are formed is timed update.Technical solution of the present invention optimizes the flow that high-volume data are stored in distributed memory system, enhance the adaptibility to response to the storage of high-volume data, extend the application scenarios under distributed storage architecture, reduce the cost of overall technical architecture, the efficiency of distributed library storage is improved, has adapted to database technology and the demand of market fast development.
Description
Technical field
The present invention relates to information technology field, more particularly to a kind of method and system of Distributed Storage.
Background technology
Internet development is rapid, it has also become the important component of people's life and the important load of human civilization communication development
Body penetrates into economic, politics, culture, social life various aspects, changes the manner of intercourse and the mode of thinking of people.No matter
It is either in social life field in industrial circle, the change that internet gives people is huge.
As the core of internet data record, database also penetrates into the various aspects of society, is widely developed
And application.Database is the set for the related data got up according to certain structure and regular weaves, is from the overall point of view
It establishes, tissue, description and storage is carried out by certain data model.Its structure based on the associate naturally between data, so as to
All necessary access paths are provided, and data have whole structuring no longer for a certain application, but towards full tissue
Feature.
Database be used for data storage and processing, storage be belong to enterprise, operating divisions, organization and individual it is related
The set of data is the general data processing system of a unit or an application field.Data in database are for crowd
Multi-user shares its information and establishes, and has had been extricated from the limitation and restriction of specific procedure.Different users can be by each
From usage use the data in database；Multiple users can simultaneously in shared data bank data resource, i.e., different use
Family can access the same data in database simultaneously.Data sharing not only meets each user and the information content is wanted
It asks, while also meets the requirement of each user-to-user information communication.
Database is important e-sourcing, is the information technology for adapting to internet development, has the characteristics that：
1), the data in the structured database of data are not disorderly and unsystematic, not the least concerned, they have certain
Institutional framework, the data for belonging to identity set have similar feature.
2), data tool sharing.Between each department of a unit, there is substantial amounts of duplicate messages.Use number
It seeks to be managed collectively these information according to the purpose in storehouse, reduces redundancy, each department is made to enjoy identical data jointly.
3), the independence of data.The independence of data refers to the independence between data record and data management software.Data
And its structure should have independence, without that should go to change application program.
4), the integrality of data.The integrality of data refers to the correctness for ensureing data in database.It may cause data
There are many incorrect reason, and data base management system manages them by being checked data character.
5), the flexibility of data.Data base management system is not that data are simply accumulated, it is in the base of recording data information
There is many management functions, such as input, output, inquiry, edit-modify on plinth.
6), the security of data.According to the responsibility of user, the people of different stage has database different permissions, number
It can be ensured that the security of data according to base management system.
With the extensive use of database technology, information content increases has reached unprecedented height with spread speed.Letter
Variation, the fast development of information source of media types are ceased, the quantity for all making information data is increasing, and the storage of data is carried
Higher requirement is gone out.In order to tackle the challenge that data storage proposes, distributed storage is to slow down the effective ways of storage pressure.
Distributed memory system is that data are disperseed the storage mode being stored in more independent equipment.Traditional network
Storage system stores all data using the storage server concentrated, storage server become system performance bottleneck and can
By property and the focus of security, it is impossible to meet the needs of Mass storage application.Distributed network storage system is using expansible
System structure, share storage load using more storage servers, using location server position storage information, it is not only carried
The high reliability of system, availability and access efficiency, are also easy to extend.
The distributed storage Technical Architecture of mainstream includes at present：
1), C/S frameworks.The file that the system calling that client is provided using native operating sys-tern manages remote server
System carries out transparent access, and client is not aware that the physical location of file, also known as remotely accesses model.Its Typical Representative is Sun
The Network File System (Network File System NFS) of Microsystem.
The Technical Architecture is characterized in：A) NFS is the distributed storage architecture of the architecture of typical C/S frameworks；b)
Model is remotely accessed, is different from upload/download model；C) realized by remote procedure call；D) file system model refers to
POSIXAPI is realized.
2) storage SAN frameworks, are shared.Storage area network SAN is one and is used between application server and storage resource
The dedicated high performance network system, provided between multiple host and multiple storage devices logical between any two node
Believe passage.
The Technical Architecture is characterized in：A) server shares storage SAN；B) MDC manages metadata；C) SAN shared files
System；D) performance and capacity can individually extend；E) of high cost, scale is limited.
3), the distributed structure/architecture based on cluster.The mainstream framework of this current distributed storage of storage architecture, usually by member
Data and both data are independent, i.e. control stream is separated with data flow, so as to obtain higher set expandability and I/O
Concurrency.Its Typical Representative is GoogleFS.
The Technical Architecture is characterized in：A) distributed file system；B) the direct-connected respective memory node of server；C) MDS is managed
Manage metadata；D) RAID, volume management, file system three unification；E) performance and capacity extend simultaneously, and scale can be very big.
4), P2P symmetrical expressions architecture.This is a kind of decentralization based on peer to peer technology, holosymmetric framework, is set
It is to position position of the file in memory node using uniformity Hash consistent hash algorithms to count thought, so as to take
Disappeared the role of metadata server.Ideally, this model eliminate the performance bottleneck of metadata, Single Point of Faliure,
A series of relevant issues such as data consistency, set expandability significantly improve, and system concurrency and performance will realize linear expansion
Increase.The Typical Representative of the Technical Architecture is Glusterfs.
The Technical Architecture is characterized in：A) non-stop layer structure, complete peer-to-peer structure；B) P2P file system is needed to support；
C) built based on Chord DHT；D) meta data server is not required；E) block or file can be based on；F) availability faces huge choose
War.
Existing Distributed Storage technical solution mainly considers the storage of magnanimity big data, to meet big data quantity
The demand of storage.But the storage condition under database high-volume storage condition it is not suitable for solving, it is necessary to optimize to database
High-volume data read-write efficiency, to adapt to be widely applied scene.
The content of the invention
The present invention provides a kind of method and system of Distributed Storage, optimizes high-volume data and is stored in distribution
Flow in storage system enhances the adaptibility to response to the storage of high-volume data, extends answering under distributed storage architecture
With scene, the cost of overall technical architecture is reduced, the efficiency of distributed library storage is improved, has adapted to database technology and city
The fast-developing demand in field.
Technical scheme provides a kind of method of Distributed Storage, comprises the following steps：
New data information writes memory database；
Update the distributed storage of the memory database and distributed data base.
Further, new data information writes the memory database in real time, whenever reading new data information, writes in real time
Enter the memory database.
Further, the memory database and distributed data base collectively form the framework of distributed storage.
Further, the distributed storage that the memory database and distributed data base are formed is timed update, often
Every 20ms updates once.
Further, after the completion of the distributed storage update, the memory database preserves the data no more than 200
Information.
Further, the data message beyond 200 is stored in the distributed data base.
Further, the data of the memory database take the principle of first in first out.
Technical scheme additionally provides a kind of system of Distributed Storage, including：Administrative unit, memory number
According to storehouse and distributed data base, wherein,
Administrative unit carries out distributed for the update of managing internal memory database and distributed data base and to data message
Storage；
Memory database and distributed data base perform update for storing data message.
Further, new data message writes the memory database, the memory database and distributed number in real time
The distributed storage formed according to storehouse takes timing to update.
Further, the data of the memory database take the principle of first in first out；
After the completion of the distributed storage update, the memory database preserves the data message no more than 200.
Technical solution of the present invention optimizes the flow that high-volume data are stored in distributed memory system, enhances to big
The adaptibility to response of batch data storage, extends the application scenarios under distributed storage architecture, reduces overall technical architecture
Cost improves the efficiency of distributed library storage, has adapted to database technology and the demand of market fast development.
Other features and advantages of the present invention will be illustrated in the following description, also, partly becomes from specification
It obtains it is clear that being understood by implementing the present invention.The purpose of the present invention and other advantages can be by the explanations write
Specifically noted structure is realized and obtained in book, claims and attached drawing.
Below by drawings and examples, technical scheme is described in further detail.
Description of the drawings
Attached drawing is used for providing a further understanding of the present invention, and a part for constitution instruction, the reality with the present invention
Example is applied together for explaining the present invention, is not construed as limiting the invention.In the accompanying drawings：
Fig. 1 is the method flow diagram of Distributed Storage in the embodiment of the present invention one；
Fig. 2 is the system construction drawing of Distributed Storage in the embodiment of the present invention one.
Specific embodiment
The preferred embodiment of the present invention is illustrated below in conjunction with attached drawing, it should be understood that preferred reality described herein
It applies example to be merely to illustrate and explain the present invention, be not intended to limit the present invention.
Fig. 1 is the method flow diagram of Distributed Storage in the embodiment of the present invention one.As shown in Figure 1, the flow includes
Following steps：
Step 101, new data information write-in memory database.
New data information writes memory database in real time, and internal storage data is write in real time whenever reading new data information
Storehouse.
The distributed storage of step 102, update memory database and distributed data base.
Memory database and distributed data base collectively form the framework of distributed storage.
The distributed storage that memory database and distributed data base are formed is timed update, updates one every 20ms
It is secondary.
After the completion of distributed storage update, memory database preserves the data message no more than 200；
Data message beyond 200 is stored in distributed data base.
The data of memory database take the principle of first in first out, i.e., the data message write at first, at first transfer storage
In distributed data base.
In order to realize above method flow, the present embodiment additionally provides a kind of system of Distributed Storage, and Fig. 2 is this
The system construction drawing of Distributed Storage in inventive embodiments one.As shown in Fig. 2, the system includes：It is administrative unit 201, interior
Deposit data storehouse 202 and distributed data base 203, wherein,
Administrative unit carries out distributed for the update of managing internal memory database and distributed data base and to data message
Storage；
Memory database and distributed data base collectively form distributed storage, for storing data message, and perform more
Newly.
After new data message arrives, memory database is write in real time.
The distributed storage that memory database and distributed data base are formed takes the newer pattern of timing.
After the completion of distributed storage update, memory database preserves the data message no more than 200；
The transfer that the data of memory database take the principle of first in first out to carry out data message stores.
Technical solution of the present invention optimizes the flow that high-volume data are stored in distributed memory system, enhances to big
The adaptibility to response of batch data storage, extends the application scenarios under distributed storage architecture, reduces overall technical architecture
Cost improves the efficiency of distributed library storage, has adapted to database technology and the demand of market fast development.
It should be understood by those skilled in the art that, the embodiment of the present invention can be provided as method, system or computer program
Product.Therefore, the reality in terms of complete hardware embodiment, complete software embodiment or combination software and hardware can be used in the present invention
Apply the form of example.Moreover, the computer for wherein including computer usable program code in one or more can be used in the present invention
The shape for the computer program product that usable storage medium is implemented on (including but not limited to magnetic disk storage and optical memory etc.)
Formula.
The present invention be with reference to according to the method for the embodiment of the present invention, the flow of equipment (system) and computer program product
Figure and/or block diagram describe.It should be understood that it can be realized by computer program instructions every first-class in flowchart and/or the block diagram
The combination of flow and/or box in journey and/or box and flowchart and/or the block diagram.These computer programs can be provided
The processor of all-purpose computer, special purpose computer, Embedded Processor or other programmable data processing devices is instructed to produce
A raw machine so that the instruction performed by computer or the processor of other programmable data processing devices is generated for real
The device for the function of being specified in present one flow of flow chart or one box of multiple flows and/or block diagram or multiple boxes.
These computer program instructions, which may also be stored in, can guide computer or other programmable data processing devices with spy
Determine in the computer-readable memory that mode works so that the instruction generation being stored in the computer-readable memory includes referring to
Make the manufacture of device, the command device realize in one flow of flow chart or multiple flows and/or one box of block diagram or
The function of being specified in multiple boxes.
These computer program instructions can be also loaded into computer or other programmable data processing devices so that counted
Series of operation steps is performed on calculation machine or other programmable devices to generate computer implemented processing, so as in computer or
The instruction offer performed on other programmable devices is used to implement in one flow of flow chart or multiple flows and/or block diagram one
The step of function of being specified in a box or multiple boxes.
Obviously, various changes and modifications can be made to the invention without departing from essence of the invention by those skilled in the art
God and scope.In this way, if these modifications and changes of the present invention belongs to the scope of the claims in the present invention and its equivalent technologies
Within, then the present invention is also intended to comprising including these modification and variations.
Claims (10)
- A kind of 1. method of Distributed Storage, which is characterized in that comprise the following steps：New data information writes memory database；Update the distributed storage of the memory database and distributed data base.
- 2. according to the method described in claim 1, it is characterized in that, new data information writes the memory database in real time, often When reading new data information, the memory database is write in real time.
- 3. according to the method described in claim 1, it is characterized in that, the memory database and distributed data base collectively form The framework of distributed storage.
- 4. the method according to claim 1 or 3, which is characterized in that the memory database and distributed data base are formed Distributed storage be timed update, every 20ms update once.
- 5. according to the method described in claim 1, it is characterized in that, the distributed storage update after the completion of, the memory number The data message no more than 200 is preserved according to storehouse.
- 6. the method according to claim 1 or 6, which is characterized in that be stored in the distribution beyond the data messages of 200 Formula database.
- 7. according to the method described in claim 6, it is characterized in that, the data of the memory database take the original of first in first out Then.
- 8. a kind of system of Distributed Storage, which is characterized in that including：Administrative unit, memory database and distributed number According to storehouse, wherein,Administrative unit carries out distributed storage for the update of managing internal memory database and distributed data base and to data message；Memory database and distributed data base perform update for storing data message.
- 9. system according to claim 8, which is characterized in that new data message writes the memory database in real time, The distributed storage that the memory database and distributed data base are formed takes timing to update.
- 10. system according to claim 8, which is characterized in that further comprise：The data of the memory database take the principle of first in first out；After the completion of the distributed storage update, the memory database preserves the data message no more than 200.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610975124.0A CN108062308A (en) | 2016-11-07 | 2016-11-07 | A kind of method and system of Distributed Storage |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610975124.0A CN108062308A (en) | 2016-11-07 | 2016-11-07 | A kind of method and system of Distributed Storage |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108062308A true CN108062308A (en) | 2018-05-22 |
Family
ID=62136560
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610975124.0A Pending CN108062308A (en) | 2016-11-07 | 2016-11-07 | A kind of method and system of Distributed Storage |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108062308A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110598059A (en) * | 2019-09-16 | 2019-12-20 | 北京百度网讯科技有限公司 | Database operation method and device |
-
2016
- 2016-11-07 CN CN201610975124.0A patent/CN108062308A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110598059A (en) * | 2019-09-16 | 2019-12-20 | 北京百度网讯科技有限公司 | Database operation method and device |
| CN110598059B (en) * | 2019-09-16 | 2022-07-05 | 北京百度网讯科技有限公司 | Database operation method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102855294B (en) | Intelligent hash data layout method, cluster storage system and method thereof | |
| Padhy et al. | RDBMS to NoSQL: reviewing some next-generation non-relational database’s | |
| US20170308558A1 (en) | Local Key-Value Database Synchronization | |
| CN104603762B (en) | The method and system for supporting to access the coordination of the shared storage of file system using the automatic calibration of parallel file access protocol and metadata management | |
| US10157214B1 (en) | Process for data migration between document stores | |
| CN106775446A (en) | Based on the distributed file system small documents access method that solid state hard disc accelerates | |
| CN104881466B (en) | The processing of data fragmentation and the delet method of garbage files and device | |
| CN102968498A (en) | Method and device for processing data | |
| US9330271B1 (en) | Fine-grained access control for synchronized data stores | |
| KR20210075845A (en) | Native key-value distributed storage system | |
| CN106990915A (en) | A kind of SRM method based on storage media types and weighting quota | |
| CN104281717B (en) | A kind of method for setting up magnanimity ID mapping relations | |
| CN103559229A (en) | Small file management service (SFMS) system based on MapFile and use method thereof | |
| CN103595799A (en) | Method for achieving distributed shared data bank | |
| CN109542861A (en) | File management method, device and system | |
| CN104536908B (en) | A kind of magnanimity small records efficient storage management method towards unit | |
| CN103559247B (en) | A kind of data service handling method and device | |
| CN103473258A (en) | Cloud storage file system | |
| CN109062935A (en) | A kind of method and apparatus of file operation | |
| CN109460406A (en) | A kind of data processing method and device | |
| CN107506466A (en) | A kind of small documents storage method and system | |
| CN108153759A (en) | A kind of data transmission method of distributed data base, middle tier server and system | |
| CN103685342A (en) | Personal cloud data storage center and cloud data storage method | |
| US10521398B1 (en) | Tracking version families in a file system | |
| CN108062308A (en) | A kind of method and system of Distributed Storage |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| WD01 | Invention patent application deemed withdrawn after publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20180522 |