CN104360820A - High-speed big data storage method - Google Patents
High-speed big data storage method Download PDFInfo
- Publication number
- CN104360820A CN104360820A CN201410552499.7A CN201410552499A CN104360820A CN 104360820 A CN104360820 A CN 104360820A CN 201410552499 A CN201410552499 A CN 201410552499A CN 104360820 A CN104360820 A CN 104360820A
- Authority
- CN
- China
- Prior art keywords
- storage
- performance
- data
- big data
- optimization
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a high-speed big data storage method, and relates to the technical field of data storage. The high-speed big data storage method is characterized in that a storage optimization method comprises the following steps: firstly promoting physical properties of a storage device, and replacing a mechanical hard disk in conversion equipment with a solid-state disk; designing and optimizing a system structure: optimizing the system logical structure and data flow paths, and adopting an RAID1 plus 0 technology to greatly improve the system performance; finally, optimizing I/O (Input/Output) loads, and improving the storage performance under a certain storage mode. According to the high-speed big data storage method, the problem that the big data is stored slowly at present is solved; by optimizing hardware equipment and software programs, the storage speed is greatly increased and the problem of storage channel blockage is avoided.
Description
Technical field:
The present invention relates to technical field of data storage, be specifically related to a kind of large date storage method at a high speed.
Background technology:
Data storage is the information needing in the temporary file that produces in process of data stream or process to search.Data are recorded on computer-internal or exterior storage medium in a certain format.Data storage will be named, and this name will reflect the composition implication of information characteristics.Data stream reflects the data flowed in system, shows the feature of dynamic data; Data store data static in reflection system, show the feature of static data.
Conventional storage medium is Disk and tape.Data storage organization mode is different because of storage medium.Data only file mode access in order on tape; Disk then can adopt sequential access or direct access method by request for utilization.Data storage method and data file are organized closely related, and its key is corresponding relation between the logical and physical sequential that foundation is recorded, and determines memory address, to improve data access speed.
But current memory device is when storing in the face of mass data, and there will be system cloud gray model slow, storage speed slows down greatly, can cause system crash time serious.
Summary of the invention:
The object of this invention is to provide a kind of large date storage method at a high speed, it stores problem slowly for current large data, by the equipment of optimizing hardware and software program, greatly improves and carrys out storage speed, avoiding the problem of memory channel obturation.
In order to solve the problem existing for background technology, the present invention is by the following technical solutions: its storage optimization method: first to the lifting of storage device physical performance, in conversion equipment, mechanical hard disk is replaced by solid state hard disc; Then the design and optimization of system architecture, it comprises the optimization of system logic structure and data flow passage, adopts RAID1+0 technology greatly can improve system performance; Finally be optimized for I/O load, improve the memory property under particular memory pattern.
The method that described I/O load is optimized: a, physical features according to storage medium, optimization data access process is the effective ways improving single equipment performance, data access technology, is used for reducing the access delay across magnetic track; Concurrent and the load balance of b, I/O, I/O balance is that I/O request is evenly distributed in multiple memory device, thus avoid the overall performance due to some memory device overload two influential system, meanwhile, the throughput of system can be improved further by the concurrent access of memory device; C, data buffer storage and look ahead, within the storage system, utilize the locality of data in space and the locality of time design caching mechanism, a kind of effective ways of conventional raising computer system performance, utilize the difference of different storage device between performance and capacity, use upper strata high-speed processing apparatus as the buffer memory medium of lower floor's low speed storage device, reach the object improving system performance, the every one deck simultaneously sent out in storage system also uses caching technology all in a large number.
The present invention has following beneficial effect: it stores problem slowly for current large data, by the equipment of optimizing hardware and software program, greatly improves and carrys out storage speed, avoiding the problem of memory channel obturation.
Embodiment:
This embodiment is by the following technical solutions: its storage optimization method: first to the lifting of storage device physical performance, in conversion equipment, mechanical hard disk is replaced by solid state hard disc; Then the design and optimization of system architecture, it comprises the optimization of system logic structure and data flow passage, adopts RAID1+0 technology greatly can improve system performance; Finally be optimized for I/O load, improve the memory property under particular memory pattern.
The method that described I/O load is optimized: a, physical features according to storage medium, optimization data access process is the effective ways improving single equipment performance, data access technology, is used for reducing the access delay across magnetic track; Concurrent and the load balance of b, I/O, I/O balance is that I/O request is evenly distributed in multiple memory device, thus avoid the overall performance due to some memory device overload two influential system, meanwhile, the throughput of system can be improved further by the concurrent access of memory device; C, data buffer storage and look ahead, within the storage system, utilize the locality of data in space and the locality of time design caching mechanism, a kind of effective ways of conventional raising computer system performance, utilize the difference of different storage device between performance and capacity, use upper strata high-speed processing apparatus as the buffer memory medium of lower floor's low speed storage device, reach the object improving system performance, the every one deck simultaneously sent out in storage system also uses caching technology all in a large number.
The present invention has following beneficial effect: it stores problem slowly for current large data, by the equipment of optimizing hardware and software program, greatly improves and carrys out storage speed, avoiding the problem of memory channel obturation.
Claims (2)
1. the large date storage method of high speed, is characterized in that its storage optimization method: first to the lifting of storage device physical performance, in conversion equipment, mechanical hard disk is replaced by solid state hard disc; Then the design and optimization of system architecture, it comprises the optimization of system logic structure and data flow passage, adopts RAID1+0 technology greatly can improve system performance; Finally be optimized for I/O load, improve the memory property under particular memory pattern.
2. a kind of large date storage method at a high speed according to claim 1, it is characterized in that the method that described I/O load is optimized: (a), physical features according to storage medium, optimization data access process is the effective ways improving single equipment performance, data access technology, is used for reducing the access delay across magnetic track; Concurrent and the load balance of (b), I/O, I/O balance is that I/O request is evenly distributed in multiple memory device, thus avoid the overall performance due to some memory device overload two influential system, meanwhile, the throughput of system can be improved further by the concurrent access of memory device; (c), data buffer storage and look ahead, within the storage system, utilize the locality of data in space and the locality of time design caching mechanism, a kind of effective ways of conventional raising computer system performance, utilize the difference of different storage device between performance and capacity, use upper strata high-speed processing apparatus as the buffer memory medium of lower floor's low speed storage device, reach the object improving system performance, the every one deck simultaneously sent out in storage system also uses caching technology all in a large number.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410552499.7A CN104360820A (en) | 2014-10-16 | 2014-10-16 | High-speed big data storage method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410552499.7A CN104360820A (en) | 2014-10-16 | 2014-10-16 | High-speed big data storage method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN104360820A true CN104360820A (en) | 2015-02-18 |
Family
ID=52528084
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410552499.7A Pending CN104360820A (en) | 2014-10-16 | 2014-10-16 | High-speed big data storage method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104360820A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105373352A (en) * | 2015-12-04 | 2016-03-02 | 中国科学技术大学 | Bandwidth comparison-based check updating mode selection method during disk array capacity expansion |
-
2014
- 2014-10-16 CN CN201410552499.7A patent/CN104360820A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105373352A (en) * | 2015-12-04 | 2016-03-02 | 中国科学技术大学 | Bandwidth comparison-based check updating mode selection method during disk array capacity expansion |
| CN105373352B (en) * | 2015-12-04 | 2018-05-29 | 中国科学技术大学 | Checksum update mode selection method during the disk array dilatation compared based on bandwidth |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20180356993A1 (en) | Optimized data placement for individual file accesses on deduplication-enabled sequential storage systems | |
| CN102156738B (en) | Method for processing data blocks, and data block storage equipment and system | |
| CN103049397B (en) | A kind of solid state hard disc inner buffer management method based on phase transition storage and system | |
| CN102023931B (en) | Self-adaption cache pre-fetching method | |
| CN104360963B (en) | A kind of isomery mixing internal memory method and apparatus calculated towards internal memory | |
| CN103885728A (en) | Magnetic disk cache system based on solid-state disk | |
| CN107885456A (en) | Reduce the conflict that I/O command accesses NVM | |
| CN102117248A (en) | Caching system and method for caching data in caching system | |
| CN105574104A (en) | LogStructure storage system based on ObjectStore and data writing method thereof | |
| CN104267912A (en) | NAS (Network Attached Storage) accelerating method and system | |
| CN104503923B (en) | A kind of asymmetric disk array cache dispatching method | |
| CN102945207A (en) | Cache management method and system for block-level data | |
| US8856442B2 (en) | Method for volume management | |
| CN103049220A (en) | Storage control method, storage control device and solid-state storage system | |
| CN109388346B (en) | Data dropping method and related device | |
| CN102637147A (en) | Storage system using solid state disk as computer write cache and corresponding management scheduling method | |
| CN102981783A (en) | Cache accelerating method based on Nand Flash | |
| CN103916459A (en) | Big data filing and storing system | |
| CN104317770A (en) | Data storage structure and data access method for multiple core processing system | |
| CN105224473A (en) | The update method that a kind of solid state hard disc is data cached and device | |
| CN103198021A (en) | Method for improving solid state disk data transmission efficiency | |
| CN102609486A (en) | Data reading/writing acceleration method of Linux file system | |
| CN104834478A (en) | Data writing and reading method based on heterogeneous hybrid storage device | |
| CN104793892B (en) | A method of accelerating disk stochastic inputs output (IO) read-write | |
| CN103500147A (en) | Embedded and layered storage method of PB-class cluster storage system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20150218 |
|
| WD01 | Invention patent application deemed withdrawn after publication |