CN108255422A - A kind of storage method and storage device - Google Patents
A kind of storage method and storage device Download PDFInfo
- Publication number
- CN108255422A CN108255422A CN201711457721.5A CN201711457721A CN108255422A CN 108255422 A CN108255422 A CN 108255422A CN 201711457721 A CN201711457721 A CN 201711457721A CN 108255422 A CN108255422 A CN 108255422A
- Authority
- CN
- China
- Prior art keywords
- file
- storage
- stored
- storage unit
- memory cell
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
- G06F16/137—Hash-based
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0625—Power saving in storage systems
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention provides a kind of storage method and storage device, including：It is in advance storage unit by least one computer installation；The corresponding storage information of each described storage unit is determined respectively, wherein, the storage information, including storage location and remaining space；Receive the file to be stored that user sends；Determine the fileinfo of the file to be stored, wherein, the fileinfo, including sending position, file size；According to the fileinfo and each described storage information, the corresponding designated memory cell of the file to be stored is determined；It will be in file to be stored storage to the designated memory cell.This programme can reduce the cost of memory technology.
Description
Technical field
The present invention relates to network communication technology field, more particularly to a kind of storage method and storage device.
Background technology
Nowadays, file is uploaded, is downloaded already not rare, need file is supported to upload in more and more applications, under
The services such as load, storage.The storage mode of resource is varied, how to choose a kind of storage side with High Availabitity and low cost
Formula becomes these application program matters of utmost importance.
At present, it provides the enterprise of application program storage file to the user, when providing storage service for each user, needs
Buy the storage device of special object storage software and large capacity.
But with the development of internet, the big expansion of data set of user, while service and body of the user to memory technology
It tests and requires higher and higher, this then needs enterprise to buy the storage device that more storage efficiencies are high, capacity is big, therefore causes to store skill
Art it is of high cost.
Invention content
An embodiment of the present invention provides a kind of storage method and storage devices, can reduce the cost of memory technology.
In a first aspect, a kind of storage method, including：
It is in advance storage unit by least one computer installation；
The corresponding storage information of each described storage unit is determined respectively, wherein, the storage information, including storing position
It puts and remaining space；
Receive the file to be stored that user sends；
Determine the fileinfo of the file to be stored, wherein, the fileinfo, including sending position, file size；
According to the fileinfo and each described storage information, determine that corresponding specify of the file to be stored is deposited
Storage unit；
It will be in file to be stored storage to the designated memory cell.
Preferably, described in advance by least one computer installation for after storage unit, described according to the text
Part information and each described storage information, before determining the corresponding designated memory cell of the file to be stored, further
Including：
Each described storage unit is connected by preset interchanger；
Determine the corresponding election factor of each described storage unit, wherein, the election factor, including IP
Any one or more of location, CPU core number, memory size or custom list number；
The target election factor corresponding with preset election regulation is elected from each election factor；
Determine that the corresponding Destination Storage Unit of the target election factor is master；
It is described according to the fileinfo and each described storage information, determine the corresponding finger of the file to be stored
Determine storage unit, including：
By the master according to the file to be stored and each described storage information, the text to be stored is determined
The corresponding designated memory cell of part.
Preferably, it is described according to the file to be stored and each described storage information, it determines and the text to be stored
The corresponding designated memory cell of part, including：
By the transmission position position corresponding with storage unit each described in the fileinfo
Information is compared, and determines the Destination Storage Unit nearest with the transmission positional distance；
Determine the file size in the fileinfo is deposited described in whether corresponding less than the target designating unit
Store up the remaining space in information；
If so, determine that the Destination Storage Unit is the corresponding designated memory cell of the file to be stored.
Preferably, it is determined and the text to be stored according to the fileinfo and each described storage information described
After the corresponding designated memory cell of part, described by file to be stored storage to it in the designated memory cell
Before, further comprise：
Calculate the corresponding cryptographic Hash of the file to be stored；
Determine the designated memory cell it is corresponding cached at least one deposited in cryptographic Hash, if exist with it is described
Cryptographic Hash it is identical deposited cryptographic Hash；
If so, it caches the fileinfo, described deposited cryptographic Hash and the corresponding file of cryptographic Hash of having deposited exists
Position in the designated memory cell；
Otherwise, it performs in the storage to the designated memory cell by the file to be stored, and caches the institute of calculating
State cryptographic Hash, the file to be stored position and the fileinfo in the designated memory cell.
Preferably, described in advance by least one computer installation for after storage unit, it is described determine respectively it is every
Before the corresponding storage information of one storage unit, further comprise：
For storage unit each described, the corresponding configuration file of the storage unit is set；
According to the configuration file, at least one tar packets are created in the storage unit；
Determine the corresponding memory space of each described tar packet；
After the determining corresponding designated memory cell of file to be stored, described by the text to be stored
Before in part storage to the designated memory cell, further comprise：
The corresponding memory space of each described tar packet in the designated memory cell, determine with
The corresponding specified tar packets of fileinfo；
In the storage to the designated memory cell by the file to be stored, including：
It will be in file to be stored storage to the corresponding specified tar packets of the designated memory cell.
Preferably, before the file to be stored sent in the reception user, further comprise：
Build open protocol OAuth authentication services；
At least one user identity is set；
Each described user identity of setting is bound with the OAuth authentication services；
Receive the identity information that user sends；
It determines to whether there is the mesh identical with the identity information of reception at least one user identity of setting
User identity is marked, if so, performing the file to be stored for receiving user and sending, otherwise, is sent to the user without storage
Permission.
Second aspect, an embodiment of the present invention provides a kind of storage device, including：
Setting unit, in advance by least one computer installation be storage unit；
Processing unit stores letter for determining that each described storage unit of the setting unit setting is corresponding respectively
Breath, wherein, the storage information, including storage location and remaining space；Determine that the receiving unit receives described to be stored
The fileinfo of file, wherein, the fileinfo, including sending position, file size；According to the fileinfo and each
A storage information, determines the corresponding designated memory cell of the file to be stored
Receiving unit, for receiving the file to be stored of user's transmission；
Buffer unit, the file to be stored storage for the receiving unit to be received are determined to the processing unit
The designated memory cell in.
Preferably, the setting unit, for connecting each described storage unit by preset interchanger；
The processing unit, for determining the corresponding election factor of each described storage unit, wherein, the choosing
The factor is lifted, including any one or more of IP address, CPU core number, memory size or custom list number；From each described
The target election factor corresponding with preset election regulation is elected in the election factor；Determine that the target election factor pair should
Destination Storage Unit be master；By the master according to the file to be stored and each described storage information,
Determine the corresponding designated memory cell of the file to be stored.
Preferably, the processing unit, for by the transmission position in the fileinfo with being deposited described in each
The corresponding location information of storage unit is compared, and determines that the target storage nearest with the transmission positional distance is single
Member；Determine whether the file size in the fileinfo is less than the corresponding storage information of the target designating unit
In remaining space；If so, determine that the Destination Storage Unit is the corresponding designated memory cell of the file to be stored.
Preferably, the processing unit is further used for calculating the corresponding cryptographic Hash of the file to be stored；It determines described
Designated memory cell is corresponding at least one having deposited in cryptographic Hash of having cached, if there is the Hash identical with the cryptographic Hash
Value；
The buffer unit is further used for, when there is the cryptographic Hash identical with the cryptographic Hash, caching the file
Information described has deposited cryptographic Hash and the position for having deposited the corresponding file of cryptographic Hash in the designated memory cell；It is no
Then, it performs in the storage to the designated memory cell by the file to be stored, and caches the cryptographic Hash of calculating, institute
State file to be stored position and fileinfo in the designated memory cell.
Preferably, the setting unit is further used for, for each described storage unit, setting the storage unit
Corresponding configuration file；According to the configuration file, at least one tar packets are created in the storage unit；
The processing unit is further used for determining the corresponding memory space of each described tar packet；According to described
The corresponding memory space of each described tar packet in designated memory cell, determines opposite with the fileinfo
The specified tar packets answered；
The buffer unit, for the specified storage for determining file to be stored storage to the processing unit
In the corresponding specified tar packets of unit.
Preferably, the setting unit is further used for building open protocol OAuth authentication services；It sets at least one
User identity；Each described user identity of setting is bound with the OAuth authentication services；
The processing unit is further used for whether there is and reception at least one user identity for determining setting
The identical target user's identity of the identity information, it is no if so, perform the file to be stored for receiving user and sending
Then, it is sent to the user without storage permission.
In embodiments of the present invention, it by being storage unit by each computer installation, can be sent receiving user
File to be stored when, being spliced to form big Logical Disk and store the text to be stored of user using the hard disk in each computer
Part is deposited further according to corresponding specify of storage information selection of the transmission position of file to be stored, file size and storage unit
Storage unit reduces the network requirement of memory technology with this and meets the storage condition of file to be stored, and due to the valency of hard disk
Lattice are relatively low, therefore can reduce the cost of memory technology.
Description of the drawings
In order to illustrate more clearly about the embodiment of the present invention or technical scheme of the prior art, to embodiment or will show below
There is attached drawing needed in technology description to be briefly described, it should be apparent that, the accompanying drawings in the following description is the present invention
Some embodiments, for those of ordinary skill in the art, without creative efforts, can also basis
These attached drawings obtain other attached drawings.
Fig. 1 is a kind of flow chart for storage method that one embodiment of the invention provides；
Fig. 2 is the flow chart for another storage method that one embodiment of the invention provides；
Fig. 3 is a kind of structure diagram for storage device that one embodiment of the invention provides.
Specific embodiment
Purpose, technical scheme and advantage to make the embodiment of the present invention are clearer, below in conjunction with the embodiment of the present invention
In attached drawing, the technical solution in the embodiment of the present invention is clearly and completely described, it is clear that described embodiment is
Part of the embodiment of the present invention, instead of all the embodiments, based on the embodiments of the present invention, those of ordinary skill in the art
The all other embodiments obtained under the premise of creative work is not made, shall fall within the protection scope of the present invention.
As shown in Figure 1, an embodiment of the present invention provides a kind of storage method, including：
Step 101：It is in advance storage unit by least one computer installation；
Step 102：The corresponding storage information of each described storage unit is determined respectively, wherein, the storage information, packet
Include storage location and remaining space；
Step 103：Receive the file to be stored that user sends；
Step 104：Determine the fileinfo of the file to be stored, wherein, the fileinfo, including send position,
File size；
Step 105：According to the fileinfo and each described storage information, determine that the file to be stored is corresponding
Designated memory cell；
Step 106：It will be in file to be stored storage to the designated memory cell.
In embodiments of the present invention, it by being storage unit by each computer installation, can be sent receiving user
File to be stored when, being spliced to form big Logical Disk and store the text to be stored of user using the hard disk in each computer
Part is deposited further according to corresponding specify of storage information selection of the transmission position of file to be stored, file size and storage unit
Storage unit reduces the network requirement of memory technology with this and meets the storage condition of file to be stored, and due to the valency of hard disk
Lattice are relatively low, therefore can reduce the cost of memory technology.
In an embodiment of the present invention, described in advance by least one computer installation for after storage unit, in institute
It states according to the fileinfo and each described storage information, determines the corresponding designated memory cell of the file to be stored
Before, further comprise：
Each described storage unit is connected by preset interchanger；
Determine the corresponding election factor of each described storage unit, wherein, the election factor, including IP
Any one or more of location, CPU core number, memory size or custom list number；
The target election factor corresponding with preset election regulation is elected from each election factor；
Determine that the corresponding Destination Storage Unit of the target election factor is master；
It is described according to the fileinfo and each described storage information, determine the corresponding finger of the file to be stored
Determine storage unit, including：
By the master according to the file to be stored and each described storage information, the text to be stored is determined
The corresponding designated memory cell of part.
In embodiments of the present invention, each storage unit is connected by interchanger, master clusters can be established as collection
The control centre of group, and according to the corresponding election factor of each storage unit and preset election regulation, from storage unit
Master is elected to come to other Storage Unit Managements, and Single Point of Faliure can be effectively prevent by establishing master clusters, so as to real
Now improve the purpose of redundancy ability.
In an embodiment of the present invention, it is described according to the file to be stored and each described storage information, determine with
The corresponding designated memory cell of file to be stored, including：
By the transmission position position corresponding with storage unit each described in the fileinfo
Information is compared, and determines the Destination Storage Unit nearest with the transmission positional distance；
Determine the file size in the fileinfo is deposited described in whether corresponding less than the target designating unit
Store up the remaining space in information；
If so, determine that the Destination Storage Unit is the corresponding designated memory cell of the file to be stored.
In embodiments of the present invention, it determines to deposit with nearest, enough the specified of remaining space of the distance between file to be stored
Space is stored up, not only can expend the excessive time to avoid when file to be stored is stored into storage unit farther out, realization carries
The purpose of the speed of height storage file to be stored, moreover it is possible to achieve the purpose that reasonable distribution remaining space.
Wherein, in cache file information, the sending time, file name and user's body of file to be stored can also be cached
Part.
In an embodiment of the present invention, described according to the fileinfo and each described storage information, determine with
After the corresponding designated memory cell of file to be stored, file to be stored storage is deposited to described specify described
Before in storage unit, further comprise：
Calculate the corresponding cryptographic Hash of the file to be stored；
Determine the designated memory cell it is corresponding cached at least one deposited in cryptographic Hash, if exist with it is described
Cryptographic Hash it is identical deposited cryptographic Hash；
If so, it caches the fileinfo, described deposited cryptographic Hash and the corresponding file of cryptographic Hash of having deposited exists
Position in the designated memory cell；
Otherwise, it performs in the storage to the designated memory cell by the file to be stored, and caches the institute of calculating
State cryptographic Hash, the file to be stored position and the fileinfo in the designated memory cell.
It in embodiments of the present invention, can be by file to be stored with breathing out by calculating the corresponding cryptographic Hash of file to be stored
Uncommon value forms a kind of mapping relations, forms object storage, to ensure to dodge the functions such as biography, breakpoint transmission, when presence and cryptographic Hash phase
With when having deposited cryptographic Hash, you can determine exist the file identical with file to be stored, there is no need to repeat store, ensure one
Only storage is primary in the memory unit for file, so as to save the memory space of storage unit to the full extent.
In an embodiment of the present invention, described in advance by least one computer installation for after storage unit, in institute
It states before determining the corresponding storage information of each described storage unit respectively, further comprises：
For storage unit each described, the corresponding configuration file of the storage unit is set；
According to the configuration file, at least one tar packets are created in the storage unit；
Determine the corresponding memory space of each described tar packet；
After the determining corresponding designated memory cell of file to be stored, described by the text to be stored
Before in part storage to the designated memory cell, further comprise：
The corresponding memory space of each described tar packet in the designated memory cell, determine with
The corresponding specified tar packets of fileinfo；
In the storage to the designated memory cell by the file to be stored, including：
It will be in file to be stored storage to the corresponding specified tar packets of the designated memory cell.
In embodiments of the present invention, at least one tar packets are created in the memory unit, can receive file to be stored
When, according to the file size of file to be stored and the memory space of each tar packets, reasonably specified tar packets are selected, so as to reach
The purpose in reasonable distribution space, and by tar packets store user send multiple files to be stored when, can only merge file and
It does not compress, can not only reduce essence of the quantity of file to be stored without changing file to be stored, can also work as business needs
When, directly decompress tar packets reduction authentic document.
Wherein, it when storing file to be stored to tar packets, needs to store the data flow of file to be stored to preset
In database, stored further according to the form of tar packets, by the metadata information in file to be stored as unit of 512B, root
In top of file write-in tar packets according to the file size generation 512B sizes of file to be stored, less than one 512B of tail of file
It is incomplete with space during unit, and storage file to be stored can be to avoid the decompression of influence tar packets as unit of 512B.
In an embodiment of the present invention, before the file to be stored sent in the reception user, further comprise：
Build open protocol OAuth authentication services；
At least one user identity is set；
Each described user identity of setting is bound with the OAuth authentication services；
Receive the identity information that user sends；
It determines to whether there is the mesh identical with the identity information of reception at least one user identity of setting
User identity is marked, if so, performing the file to be stored for receiving user and sending, otherwise, is sent to the user without storage
Permission.
In embodiments of the present invention, by pre-setting RESTfulAPI, to provide distribution to the user, with the collection of extension
Group's service, so as to simplify the operation of user, and by the user identity of the OAUTH certifications built and setting (for example, user name and close
Code) it is bound, it can be sent out and the corresponding token of the identity information of user when the identity information of user is verified
Service is accessed, and in the file to be stored for receiving user's transmission, access service using token sends to complete storage user
File to be stored, without knowing the username and password of user, so as to achieve the purpose that improve user information safety.
In order to more clearly illustrate technical scheme of the present invention and advantage, with preset user identity a and user identity
B and computer j is set as storage unit j, computer s is set as storage unit s, computer k is set as storage unit k,
And by storage unit k elections for master, a kind of storage method provided in an embodiment of the present invention to be described in detail, such as
Shown in Fig. 2, specifically it may comprise steps of：
Step 201：According to storage unit j, storage unit s, the corresponding election factors of storage unit k and preset choosing
Rule is lifted, storage unit k is as master for election.
Specifically, by the computer j with file system, the computer s with file system and with file system
Computer k is set as storage unit, then connects storage unit j, storage unit s and storage unit k by interchanger and establish
Control centre of the master clusters as cluster, i.e., according to storage unit j, storage unit s and the corresponding choosings of storage unit k
It lifts the factor and preset election regulation elects master, entire storage system is managed by master, establishes master clusters
Single Point of Faliure can be effectively prevent, so as to achieve the purpose that improve redundancy ability, and according to the corresponding election of storage unit
The factor determines that master can be according to the distance of IP address, the size of CPU core number, memory size or custom list size
Any one or more of determine master.
For example, using the computer j with file system as storage unit j and by the meter with file system
Calculation machine s connects storage unit j and storage unit s as storage unit s, then by interchanger；
According to CPU of the CPU core number of the corresponding election factors of storage unit j for the corresponding election factor of 4G, storage unit s
Check figure is 3G, the CPU core number of the corresponding election factors of storage unit k is 8G and what election regulation was big for CPU core number is elected to
Master determines storage unit k as master, and management node set in master, by master management storage unit j and
Storage unit s.
Step 202：Corresponding tar packets are respectively created in storage unit j and storage unit s by master.
Specifically, master is according to storage unit j and the corresponding configuration files of storage unit s, to establish at least one
A tar packets are fast as file, so that when storing the file that user sends, according to the size of file and the size reasonable of tar packets
The corresponding tar packets of selection, and in storage unit tar packets can storage file up to merge not compress, can both reduce
The quantity of file does not change the essence of file again.
For example, in the corresponding configuration files of storage unit j in save as 500G, the corresponding configuration texts of storage unit s
400G is saved as in part, master creates the tar packets j1 that memory space is 200G in storage unit j and memory space is
The tar packet j2 of 300G create the tar packets s that memory space is 400G in storage unit s.
Step 203：The corresponding storage information of storage unit j and storage unit s is determined by master respectively.
Specifically, the storage location of storage unit, remaining sky can be included in the corresponding storage information of each storage unit
Between, the memory space of each tar packet, the position of tar packets can also be included.
For example, the storage location of storage unit j is Beijing, remaining space 150G, i.e. tar packets j1 has been filled with, tar
The remaining space for wrapping j2 is 150G；
The storage location of storage unit s is Shandong, remaining space 400G, i.e. the remaining space of tar packets s is 400G.
Step 204：OAuth authentication services are built, and user identity a and user identity b and OAuth authentication services are carried out
Binding.
Specifically, build OAuth authentication services can to avoid third party's program obtain user user information (for example, with
Name in an account book and password), so as to ensure the safety of user information.
For example, the user name a in user identity a and password a and OAuth authentication services are bound, by user
User name b and password b and OAuth authentication services in identity b are bound.
Step 205：Receive the identity information that user sends.
Specifically, user if you need to storage file and/or downloads file, can pass through pre-set restfulAPI oneself
Identity information be verified as validated user, it is and if only if that could be operated after in authentication corresponding with identity information
Access authorization for resource.
For example, it is user name a and password a to receive user by the identity information that restfulAPI is sent.
Step 206：It determines to whether there is and identity information same target user identity in user identity a and user identity b.
Specifically, verifying the identity information of user can determine whether the user has the permission for operating a certain resource, from
And it can ensure the safety of the file in storage unit.
For example, by identity information for user name a respectively in the user name a and user identity b in user identity a
User name b is compared, and determines there are user name a, and the password a password as mono- corresponding with identity information in user identity a
It causes, it is thus determined that the validated user of the user.
Step 207：When there is the target user identity identical with identity information, the text to be stored that user sends is received
Part.
Specifically, when it is validated user to determine user, the file to be stored that user sends is received.
For example, the file to be stored that user is sent by restfulAPI, the corresponding file of file to be stored are received
The transmission position of information is Beijing, file size 2G.
Step 208：By master according to the corresponding fileinfo of file to be stored and storage unit j and storage unit s
Corresponding storage information, determines designated memory cell.
Specifically, the restfulAPI that master can be called according to user determines the file to be stored that user sends
The fileinfos such as position, file size, sending time, user identity, file name are sent, are deposited further according to storage unit is corresponding
The storage information such as storage space is put, remaining space, to determine the designated memory cell of file to be stored.
For example, master according to the transmission position of file to be stored be Beijing, storage unit j storage location be north
Capital, storage unit s storage location for Shandong, determine the distance between storage unit j and file to be stored recently, then determine to deposit
The remaining space of storage unit j is 150G, can leave the file to be stored of 2G, it is thus determined that storage unit j is file to be stored
Corresponding designated memory cell.
Step 209：Calculate the cryptographic Hash of file to be stored.
Specifically, it when calculating the cryptographic Hash of file to be stored, can be counted by sha1, sha256 or sha512 algorithm
It calculates.
For example, cryptographic Hash 23 is calculated by sha1 in file to be stored.
Step 210：Determine designated memory cell it is corresponding cached deposited in cryptographic Hash whether there is and cryptographic Hash phase
Same has deposited cryptographic Hash, if so, performing step 213, otherwise performs step 211.
Specifically, after the cryptographic Hash of file to be stored is calculated, it is thus necessary to determine that whether there is phase in designated memory cell
Same cryptographic Hash, if it is, illustrating there is the file identical with this document in designated memory cell, there is no need to store one again
All over file to be stored, so as to largely save the space of storage unit, if there is no identical cryptographic Hash, then really
Determine in designated memory cell to store the file identical with file to be stored, it is therefore desirable to store file to be stored.
Step 211：It will be in file to be stored storage to the tar packets in specified buffer unit.
Specifically, according to the memory space of each tar packets and the file size of file to be stored in storage unit, to treat
Storage file selects rational tar packets.
For example, the file size of file to be stored is 2G, and the tar packets j1 in storage unit j has been filled with, tar packets j2
Remaining space for 150G, therefore will be in file to be stored storage to tar packets j2.
Step 212：File to be stored storage is cached to position, corresponding cryptographic Hash and the file letter in designated memory cell
Breath.
Specifically, it after in by file to be stored storage to the tar packets in designated memory cell, needs to cache text to be stored
Position, cryptographic Hash and file size that part is stored send the files such as position, sending time, file name, user identity
Information, so that user can provide a user corresponding file when searching required file according to fileinfo.
Step 213：It caches the corresponding fileinfo of file to be stored and has deposited the corresponding file of cryptographic Hash in specified storage
Position in unit.
Specifically, it is identical with the cryptographic Hash of file to be stored when having deposited cryptographic Hash when determining to exist, it is determined that specified to deposit
There is the file identical with file to be stored content in storage unit, therefore, need to only cache sending time, the file of file to be stored
Title, file size, sends the fileinfos such as position, storage location at user identity, without repeating storage, so as to most
The space of storage unit has been saved in big degree.
As shown in figure 3, an embodiment of the present invention provides a kind of storage device, including：
Setting unit 301, in advance by least one computer installation be storage unit；
Processing unit 302, for determining that each described storage unit that the setting unit 301 is set is corresponding respectively
Information is stored, wherein, the storage information, including storage location and remaining space；Determine that the described of the reception of receiving unit 303 is treated
The fileinfo of storage file, wherein, the fileinfo, including sending position, file size；According to the fileinfo and
Each described storage information, determines the corresponding designated memory cell of the file to be stored
Receiving unit 303, for receiving the file to be stored of user's transmission；
Buffer unit 304, the file to be stored storage for the receiving unit 303 to be received handle list to described
In the determining designated memory cell of member 302.
In an embodiment of the present invention, the setting unit is connected for passing through preset interchanger and is deposited described in each
Storage unit；
The processing unit, for determining the corresponding election factor of each described storage unit, wherein, the choosing
The factor is lifted, including any one or more of IP address, CPU core number, memory size or custom list number；From each described
The target election factor corresponding with preset election regulation is elected in the election factor；Determine that the target election factor pair should
Destination Storage Unit be master；By the master according to the file to be stored and each described storage information,
Determine the corresponding designated memory cell of the file to be stored.
In an embodiment of the present invention, the processing unit, for by the transmission position in the fileinfo with
The corresponding location information of each described storage unit is compared, and determines nearest with the transmission positional distance
Destination Storage Unit；Determine whether the file size in the fileinfo is less than the corresponding institute of the target designating unit
State the remaining space in storage information；If so, determine that the Destination Storage Unit is corresponding specified for the file to be stored
Storage unit.
In an embodiment of the present invention, the processing unit is further used for calculating the corresponding Kazakhstan of the file to be stored
Uncommon value；Determine the designated memory cell it is corresponding cached at least one deposited in cryptographic Hash, if exist and the Kazakhstan
The identical cryptographic Hash of uncommon value；
The buffer unit is further used for, when there is the cryptographic Hash identical with the cryptographic Hash, caching the file
Information described has deposited cryptographic Hash and the position for having deposited the corresponding file of cryptographic Hash in the designated memory cell；It is no
Then, it performs in the storage to the designated memory cell by the file to be stored, and caches the cryptographic Hash of calculating, institute
State file to be stored position and fileinfo in the designated memory cell.
In an embodiment of the present invention, the setting unit is further used for for each described storage unit, setting
The corresponding configuration file of the storage unit；According to the configuration file, at least one tar is created in the storage unit
Packet；
The processing unit is further used for determining the corresponding memory space of each described tar packet；According to described
The corresponding memory space of each described tar packet in designated memory cell, determines opposite with the fileinfo
The specified tar packets answered；
The buffer unit, for the specified storage for determining file to be stored storage to the processing unit
In the corresponding specified tar packets of unit.
In an embodiment of the present invention, the setting unit is further used for building open protocol OAuth authentication services；
At least one user identity is set；Each described user identity of setting is bound with the OAuth authentication services；
The processing unit is further used for whether there is and reception at least one user identity for determining setting
The identical target user's identity of the identity information, it is no if so, perform the file to be stored for receiving user and sending
Then, it is sent to the user without storage permission.
The each embodiment of the present invention at least has the advantages that：
1st, in an embodiment of the present invention, by being storage unit by each computer installation, user can received
During the file to be stored of transmission, being spliced to form big Logical Disk and wait to deposit to store user using the hard disk in each computer
File is stored up, corresponding finger is selected further according to the storage information of the transmission position of file to be stored, file size and storage unit
Determine storage unit, the network requirement of memory technology is reduced with this and meet the storage condition of file to be stored, and due to hard disk
Price it is relatively low, therefore the cost of memory technology can be reduced.
2nd, in an embodiment of the present invention, each storage unit is connected by interchanger, master clusters work can be established
It is single from storage for the control centre of cluster, and according to the corresponding election factor of each storage unit and preset election regulation
Master is elected in member to come to other Storage Unit Managements, and Single Point of Faliure can be effectively prevent by establishing master clusters, from
And realize the purpose for improving redundancy ability.
3rd, in an embodiment of the present invention, the finger of nearest, the enough remaining spaces of the distance between file to be stored is determined
Determine memory space, not only can expend the excessive time to avoid when file to be stored is stored into storage unit farther out, it is real
Now improve the purpose of the speed of storage file to be stored, moreover it is possible to achieve the purpose that reasonable distribution remaining space.
It 4th, in an embodiment of the present invention, can be by file to be stored by calculating the corresponding cryptographic Hash of file to be stored
A kind of mapping relations are formed with cryptographic Hash, form object storage, to ensure to dodge the functions such as biography, breakpoint transmission, when presence and Hash
It is worth identical when having deposited cryptographic Hash, you can determine there is the file identical with file to be stored, there is no need to repeat to store, ensure
Only storage is primary in the memory unit for one file, so as to save the memory space of storage unit to the full extent.
5th, in an embodiment of the present invention, at least one tar packets are created in the memory unit, can receive it is to be stored
During file, according to the file size of file to be stored and the memory space of each tar packets, reasonably specified tar packets are selected, so as to
When achieving the purpose that reasonable distribution space, and multiple files to be stored that user sends are stored by tar packets, it can only merge text
Part can not only reduce essence of the quantity of file to be stored without changing file to be stored, can also work as business without compressing
When needing, tar packets reduction authentic document is directly decompressed.
6th, in an embodiment of the present invention, by pre-setting RESTfulAPI, to provide distribution to the user, with extension
Cluster service, so as to simplify user operation, and by the OAUTH certifications built with setting user identity (for example, user name
And password) bound, it can be sent out corresponding with the identity information of user when the identity information of user is verified
Token accesses service, and in the file to be stored for receiving user's transmission, and access service using token uses to complete storage
The file to be stored that family is sent, without knowing the username and password of user, so as to reach the safety for improving user information
Purpose.
It should be noted that herein, such as first and second etc relational terms are used merely to an entity
Or operation is distinguished with another entity or operation, is existed without necessarily requiring or implying between these entities or operation
Any actual relationship or order.Moreover, term " comprising ", "comprising" or its any other variant be intended to it is non-
It is exclusive to include, so that process, method, article or equipment including a series of elements not only include those elements,
But also it including other elements that are not explicitly listed or further includes solid by this process, method, article or equipment
Some elements.In the absence of more restrictions, the element limited by sentence " including one ", is not arranged
Except in the process, method, article or apparatus that includes the element also in the presence of other identical factor.
It is last it should be noted that：The foregoing is merely presently preferred embodiments of the present invention, is merely to illustrate the skill of the present invention
Art scheme, is not intended to limit the scope of the present invention.Any modification for being made all within the spirits and principles of the present invention,
Equivalent replacement, improvement etc., are all contained in protection scope of the present invention.
Claims (10)
1. a kind of storage method, which is characterized in that including：
It is in advance storage unit by least one computer installation；
Determine the corresponding storage information of each described storage unit respectively, wherein, the storage information, including storage location and
Remaining space；
Receive the file to be stored that user sends；
Determine the fileinfo of the file to be stored, wherein, the fileinfo, including sending position, file size；
According to the fileinfo and each described storage information, determine that the corresponding specified storage of the file to be stored is single
Member；
It will be in file to be stored storage to the designated memory cell.
2. storage method according to claim 1, which is characterized in that
Described in advance by least one computer installation for after storage unit, described according to the fileinfo and each
A storage information, before determining the corresponding designated memory cell of the file to be stored, further comprises：
Each described storage unit is connected by preset interchanger；
Determine the corresponding election factor of each described storage unit, wherein, the election factor, including IP address, CPU
Any one or more of check figure, memory size or custom list number；
The target election factor corresponding with preset election regulation is elected from each election factor；
Determine that the corresponding Destination Storage Unit of the target election factor is master；
It is described according to the fileinfo and each described storage information, determine that the file to be stored is corresponding and specified deposit
Storage unit, including：
By the master according to the file to be stored and each described storage information, the file to be stored phase is determined
Corresponding designated memory cell.
3. storage method according to claim 1, which is characterized in that
It is described according to the file to be stored and each described storage information, determine and the corresponding finger of the file to be stored
Determine storage unit, including：
By the transmission position location information corresponding with storage unit each described in the fileinfo
It is compared, determines the Destination Storage Unit nearest with the transmission positional distance；
Determine whether the file size in the fileinfo is less than the corresponding storage letter of the target designating unit
Remaining space in breath；
If so, determine that the Destination Storage Unit is the corresponding designated memory cell of the file to be stored；
And/or
Described according to the fileinfo and each described storage information, determine and the corresponding finger of the file to be stored
After determining storage unit, it is described by file to be stored storage to before in the designated memory cell, further comprise：
Calculate the corresponding cryptographic Hash of the file to be stored；
Determine the designated memory cell it is corresponding cached at least one deposited in cryptographic Hash, if exist and the Hash
It is worth and identical has deposited cryptographic Hash；
If so, the fileinfo, cryptographic Hash and the corresponding file of cryptographic Hash of having deposited deposited are cached described
Position in designated memory cell；
Otherwise, it performs in the storage to the designated memory cell by the file to be stored, and caches the Kazakhstan of calculating
Uncommon value, the file to be stored position and fileinfo in the designated memory cell.
4. storage method according to claim 1, which is characterized in that
Described in advance by least one computer installation for after storage unit, each described storage is determined respectively described
Before the corresponding storage information of unit, further comprise：
For storage unit each described, the corresponding configuration file of the storage unit is set；
According to the configuration file, at least one tar packets are created in the storage unit；
Determine the corresponding memory space of each described tar packet；
After the determining corresponding designated memory cell of file to be stored, the file to be stored is deposited described
Before storing up in the designated memory cell, further comprise：
The corresponding memory space of each described tar packet in the designated memory cell, determine with it is described
The corresponding specified tar packets of fileinfo；
In the storage to the designated memory cell by the file to be stored, including：
It will be in file to be stored storage to the corresponding specified tar packets of the designated memory cell.
5. according to the storage method any in Claims 1-4, which is characterized in that
Before the file to be stored sent in the reception user, further comprise：
Build open protocol OAuth authentication services；
At least one user identity is set；
Each described user identity of setting is bound with the OAuth authentication services；
Receive the identity information that user sends；
It determines to use with the presence or absence of the target identical with the identity information of reception at least one user identity of setting
Family identity if so, performing the file to be stored for receiving user and sending, otherwise, is sent to the user without storage permission.
6. a kind of storage device, which is characterized in that including：
Setting unit, in advance by least one computer installation be storage unit；
Processing unit, for determining the corresponding storage information of each described storage unit of setting unit setting respectively,
Wherein, the storage information, including storage location and remaining space；Determine the text for the file to be stored that receiving unit receives
Part information, wherein, the fileinfo, including sending position, file size；It is deposited according to the fileinfo and each
Information is stored up, determines the corresponding designated memory cell of the file to be stored
Receiving unit, for receiving the file to be stored of user's transmission；
Buffer unit, the institute that the file to be stored storage for the receiving unit to be received is determined to the processing unit
It states in designated memory cell.
7. storage device according to claim 6, which is characterized in that
The setting unit, for connecting each described storage unit by preset interchanger；
The processing unit, for determining the corresponding election factor of each described storage unit, wherein, the election because
Son, including any one or more of IP address, CPU core number, memory size or custom list number；From each election
The target election factor corresponding with preset election regulation is elected in the factor；Determine the corresponding mesh of the target election factor
Mark storage unit is master；By the master according to the file to be stored and each described storage information, determine
The corresponding designated memory cell of file to be stored.
8. storage device according to claim 6, which is characterized in that
The processing unit, for the transmission position in the fileinfo and storage unit each described is right respectively
The location information answered is compared, and determines the Destination Storage Unit nearest with the transmission positional distance；Determine the text
Whether the file size in part information is less than the remaining space in the corresponding storage information of the target designating unit；
If so, determine that the Destination Storage Unit is the corresponding designated memory cell of the file to be stored；
And/or
The processing unit is further used for calculating the corresponding cryptographic Hash of the file to be stored；Determine that the specified storage is single
Member is corresponding at least one having deposited in cryptographic Hash of having cached, if there is the cryptographic Hash identical with the cryptographic Hash；
The buffer unit, be further used for when exist the cryptographic Hash identical with the cryptographic Hash when, cache the fileinfo,
It is described to have deposited cryptographic Hash and the position for having deposited the corresponding file of cryptographic Hash in the designated memory cell；Otherwise, it holds
Row it is described by file to be stored storage in the designated memory cell, and cache the cryptographic Hash of calculating, described treat
Storage file position and fileinfo in the designated memory cell.
9. storage device according to claim 6, which is characterized in that
The setting unit is further used for, for each described storage unit, setting the corresponding configuration of the storage unit
File；According to the configuration file, at least one tar packets are created in the storage unit；
The processing unit is further used for determining the corresponding memory space of each described tar packet；According to described specified
The corresponding memory space of each described tar packet in storage unit, determines corresponding with the fileinfo
Specified tar packets；
The buffer unit, for the designated memory cell for determining file to be stored storage to the processing unit
In the corresponding specified tar packets.
10. according to the storage device any in claim 6 to 9, which is characterized in that
The setting unit is further used for building open protocol OAuth authentication services；At least one user identity is set；It will
Each described user identity of setting is bound with the OAuth authentication services；
The processing unit is further used at least one user identity for determining setting with the presence or absence of the institute with reception
The identical target user's identity of identity information is stated, if so, the file to be stored for receiving user and sending is performed, otherwise, to
The user is sent without storage permission.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711457721.5A CN108255422A (en) | 2017-12-28 | 2017-12-28 | A kind of storage method and storage device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711457721.5A CN108255422A (en) | 2017-12-28 | 2017-12-28 | A kind of storage method and storage device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108255422A true CN108255422A (en) | 2018-07-06 |
Family
ID=62724247
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711457721.5A Pending CN108255422A (en) | 2017-12-28 | 2017-12-28 | A kind of storage method and storage device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108255422A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111787070A (en) * | 2020-06-10 | 2020-10-16 | 俞力奇 | Equipment end resource management method |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101316273A (en) * | 2008-05-12 | 2008-12-03 | 华中科技大学 | Distributed safety memory system |
| CN102591594A (en) * | 2012-02-10 | 2012-07-18 | 杭州华三通信技术有限公司 | Data processing method and equipment |
| CN103970852A (en) * | 2014-05-06 | 2014-08-06 | 浪潮电子信息产业股份有限公司 | Data deduplication method of backup server |
| CN104767822A (en) * | 2015-04-21 | 2015-07-08 | 成都影泰科技有限公司 | Data storage method based on version |
| CN105279047A (en) * | 2015-11-02 | 2016-01-27 | 广东欧珀移动通信有限公司 | Data backup device and method and data recovery method and device |
| CN106572153A (en) * | 2016-10-21 | 2017-04-19 | 乐视控股（北京）有限公司 | Data storage method and device of cluster |
-
2017
- 2017-12-28 CN CN201711457721.5A patent/CN108255422A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101316273A (en) * | 2008-05-12 | 2008-12-03 | 华中科技大学 | Distributed safety memory system |
| CN102591594A (en) * | 2012-02-10 | 2012-07-18 | 杭州华三通信技术有限公司 | Data processing method and equipment |
| CN103970852A (en) * | 2014-05-06 | 2014-08-06 | 浪潮电子信息产业股份有限公司 | Data deduplication method of backup server |
| CN104767822A (en) * | 2015-04-21 | 2015-07-08 | 成都影泰科技有限公司 | Data storage method based on version |
| CN105279047A (en) * | 2015-11-02 | 2016-01-27 | 广东欧珀移动通信有限公司 | Data backup device and method and data recovery method and device |
| CN106572153A (en) * | 2016-10-21 | 2017-04-19 | 乐视控股（北京）有限公司 | Data storage method and device of cluster |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111787070A (en) * | 2020-06-10 | 2020-10-16 | 俞力奇 | Equipment end resource management method |
| CN111787070B (en) * | 2020-06-10 | 2022-07-12 | 俞力奇 | Equipment end resource management method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105247529B (en) | The synchronous voucher hash between directory service | |
| CN103338252B (en) | Realizing method of distributed database concurrence storage virtual request mechanism | |
| CN105025053A (en) | Distributed file upload method based on cloud storage technology and system | |
| CN103338249B (en) | Caching method and device | |
| CN109857710A (en) | File memory method and terminal device | |
| CN109769028A (en) | Redis cluster management method, device, equipment and readable storage medium storing program for executing | |
| CN110336891A (en) | Data cached location mode, equipment, storage medium and device | |
| CN103944988A (en) | Repeating data deleting system and method applicable to cloud storage | |
| CN108008918A (en) | Data processing method, memory node and distributed memory system | |
| CN103873507A (en) | Data block uploading and storing system and method | |
| CN106850842A (en) | A kind of download of file, method for uploading and device | |
| US10067719B1 (en) | Methods and systems for storing and accessing data in a distributed data storage system | |
| CN108776758A (en) | The block level data De-weight method of dynamic ownership management is supported in a kind of storage of mist | |
| CN105408871A (en) | Method and apparatus for providing redundant data access | |
| CN109951514A (en) | Document handling method, system and computer equipment based on cloud storage | |
| CN111490963A (en) | Data processing method, system, equipment and storage medium based on QUIC protocol stack | |
| CN109842652A (en) | A kind of method for uploading of file, terminal, Cloud Server and computer storage medium | |
| CN110489696A (en) | Buffering updating method, device and electronic equipment, storage medium | |
| US20080270483A1 (en) | Storage Management System | |
| CN107633102A (en) | A kind of method, apparatus, system and equipment for reading metadata | |
| CN106980618B (en) | File storage method and system based on MongoDB distributed cluster architecture | |
| CN103258389B (en) | The method of self-aided terminal upload file, system and self-aided terminal | |
| CN105656794B (en) | Data distributing method, device and computer readable storage medium | |
| CN110413588A (en) | Distributed objects storage method, device, computer equipment and storage medium | |
| CN108255422A (en) | A kind of storage method and storage device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180706 |
|
| RJ01 | Rejection of invention patent application after publication |