CN107092598A - The management method and device of data storage location information - Google Patents
The management method and device of data storage location information Download PDFInfo
- Publication number
- CN107092598A CN107092598A CN201610088248.7A CN201610088248A CN107092598A CN 107092598 A CN107092598 A CN 107092598A CN 201610088248 A CN201610088248 A CN 201610088248A CN 107092598 A CN107092598 A CN 107092598A
- Authority
- CN
- China
- Prior art keywords
- data storage
- location information
- storage location
- data
- state
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
This application discloses a kind of management method of data storage location information and device, this method includes：Internal storage state is preserved according to preset rules persistence, wherein, the internal storage state includes data storage location information and the status information of data storage device；After restarting, internal storage state is returned to the internal storage state that persistence is preserved before restarting；Recover the access of the data storage device of normal condition according to the status information of the data storage device.The used time that data in EMS memory storage location information is rebuild when meta data server is restarted is shortened, the duration that distributed file system can not be serviced externally is reduced, the availability and O＆M of system is improved.
Description
Technical field
The application belongs to field of cloud computer technology, specifically, is related to a kind of data storage location information
Management method and device.
Background technology
In large-scale distributed file system, in order to reach the mesh in terms of fault-tolerant, availability, performance
, a kind of general design be by file cutting be a series of data block, and multiple different pieces of informations storage
Multiple copies of same data block are stored in equipment (Data Server).The position of these data trnascriptions
Information (i.e. the storage location of data storage device where copy) is particularly significant, and they are clients
(Client) addressing foundation during data is read and write, is also to carry out data distribution control, copy amount management
Information source.Generally, the meta data server (Meta Server) of file system can be in system
Locations of copies information is stored and managed.
Existing technical scheme is mainly：All data are safeguarded in the internal storage structure of meta data server
The positional information of copy, does not carry out persistent storage to it.When the system is operated, meta data server with
Constantly communicated between multiple data storage devices, persistently to obtain and update the position of the copy in meta data server
Confidence ceases, the current newest locations of copies of record；Client accesses meta data server to obtain " reading "
Latest copy position needed for required locations of copies, or " writing ".When cold open occurs for meta data server
It is dynamic or when restarting (such as because failure or upgrading need), due to the data volatility of internal memory, it
Preceding locations of copies information is all lost, meta data server weight directly from all data storage devices
Newly obtain current locations of copies information.Obtained in all locations of copies information of a data storage device
Before taking, the data storage device is considered as " in recovery " state, and meta data server does not allow to it
Conduct interviews.Mass data storage device is in " in recovery " during state, and distributed file system is not right
Outer offer service.
Current data copy is reacquired when meta data server is restarted from all data storage devices
The used time of positional information is longer.Because system can not be serviced externally in this period so that the failure of system
Recovery time and upgrading used time are elongated, so as to reduce the availability of system, O＆M.
The content of the invention
In view of this, this application provides a kind of management method of data storage location information and device, with
Solve meta data server and restart the time that the used time is longer, causes distributed file system not service externally
Longer technical problem.
In order to solve the above-mentioned technical problem, this application discloses a kind of manager of data storage location information
Method, including：Internal storage state is preserved according to preset rules persistence, wherein, the internal storage state includes number
According to storage location information and the status information of data storage device；After restarting, internal storage state is recovered
The internal storage state that persistence is preserved to before restarting；Recovered just according to the status information of the data storage device
The access of the data storage device of normal state.
In order to solve the above-mentioned technical problem, disclosed herein as well is a kind of management of data storage location information
Method, including：Internal storage state is preserved according to preset rules persistence, wherein, the internal storage state includes
The status information of data storage location information and data storage device；When the disabled data storage position of appearance
When confidence ceases, for data corresponding with the disabled data storage location information, the number is judged
Whether it is more than or equal to pre-determined threshold according to the quantity of available data storage location information；When the data can
When the quantity of data storage location information is more than or equal to pre-determined threshold, the disabled number is deleted
According to storage location information；When the quantity of the available data storage location information of the data is less than pre-determined threshold
When, preserve the disabled data storage location information.
In order to solve the above-mentioned technical problem, disclosed herein as well is a kind of management of data storage location information
Device, including：Persistence module, for preserving internal storage state according to preset rules persistence, wherein,
The internal storage state includes data storage location information and the status information of data storage device；Internal memory recovers
Module, the internal storage state that preceding persistence is preserved is restarted for after restarting, internal storage state to be returned to；
Recovery module is accessed, the data for recovering normal condition according to the status information of the data storage device
The access of storage device.
In order to solve the above-mentioned technical problem, disclosed herein as well is a kind of management of data storage location information
Device, including：Persistence module, for preserving internal storage state according to preset rules persistence, wherein,
The internal storage state includes data storage location information and the status information of data storage device；Judge mould
Block, for when there is disabled data storage location information, for being deposited with the disabled data
Store up the corresponding data of positional information, judge the available data storage location information of the data quantity whether
More than or equal to pre-determined threshold；Removing module, for when the available data storage location information of the data
Quantity when being more than or equal to pre-determined threshold, delete the disabled data storage location information；Preserve
Module, for when the quantity of the available data storage location information of the data is less than pre-determined threshold, protecting
Deposit the disabled data storage location information.
Compared with prior art, the application can be obtained including following technique effect：Shorten metadata clothes
Business think highly of when opening rebuild data in EMS memory storage location information used time, reduce distributed file system without
The duration that method is externally serviced, improves the availability and O＆M of system.
Certainly, implementing any product of the application must be not necessarily required to while reaching all the above skill
Art effect.
Brief description of the drawings
Accompanying drawing described herein is used for providing further understanding of the present application, constitutes one of the application
Point, the schematic description and description of the application is used to explain the application, does not constitute to the application's
It is improper to limit.In the accompanying drawings：
Fig. 1 is a kind of flow chart of the management method of data storage location information of the embodiment of the present application；
Fig. 2 is the structural representation of the distributed file system of the embodiment of the present application；
Fig. 3 is a kind of block diagram of the managing device of data storage location information of the embodiment of the present application；
Fig. 4 is a kind of block diagram of the managing device of data storage location information of the embodiment of the present application.
Embodiment
Presently filed embodiment is described in detail below in conjunction with drawings and Examples, thereby to the application
How application technology means can fully understand to solve technical problem and reach the implementation process of technology effect
And implement according to this.
The embodiment of the present application, meta data server carries out persistence preservation to memory data structure, so that will
The status information of data storage location information and data storage device in internal memory is preserved, and is occurring weight
Qi Shi, the internal storage state preserved by persistence recovers data storage location information and set according to data storage
Standby status information recovers the access to the data storage device in normal condition, can shorten metadata
The used time of data in EMS memory storage location information is rebuild during Server Restart, event occurs in meta data server
Under the scene of barrier, failure recovery time can be shortened, system availability is improved；In meta data server liter
Under the scene of level, updating speed can be accelerated, shorten the risk exposure that upgrading is brought, so that lifting system
Reliability.
Fig. 1 is a kind of management method for data storage location information that the embodiment of the present application is provided, it is adaptable to
Meta data server in distributed file system, meta data server is used for the storage position of management data block
Status information of confidence breath and data storage device etc.；Data storage device is used for the sheet of management data block
Ground copy, it is allowed to which client is written and read operation to managed data trnascription, and a meta data server can
To manage substantial amounts of data storage device, such as one meta data server can manage 5000 numbers simultaneously
According to storage device；Client is used to carry out read operation to the data trnascription that data storage device is preserved or write
Operation.As shown in figure 1, this method comprises the following steps S101-S103.
S101, internal storage state is preserved according to preset rules persistence, wherein, internal storage state is deposited including data
Store up positional information and the status information of data storage device.
S102, after restarting, internal storage state is returned to the internal storage state that persistence is preserved before restarting；
S103, recovers the data storage device of normal condition according to the status information of the data storage device
Access.
Wherein, preset rules can be predetermined period, periodically carry out persistence preservation to internal storage state；
Can after the default operation amount for data storage location information is reached, start to internal memory shape
State carries out persistence preservation.
Data storage location information is the physical storage locations of data in a data storage device, can be included
The network address information of data storage device is (for example, tcp://10.1.1.1:And data storage device 10200)
Internal disk mark (for example, Disk1, Disk2).
The status information of data storage device includes being in normal condition and abnormal condition, and normal condition is
Refer to data storage device and do not update the data storage location information and the number preserved to meta data server
According to that can be accessed, abnormal condition refers to that data storage device is updated the data to meta data server
Storage location information or data storage device have been off (as shut down).
Internal storage data can be cleared during due to restarting, and meta data server preserves internal storage state persistence,
Carried out persistently equivalent to the status information to data storage location information therein and data storage device
Change backup.After meta data server is restarted, the internal storage state before restarting can be returned to, is also just recovered
The status information of data storage location information and data storage device.For the state letter of data storage device
The data storage device of normal condition is in breath, its data storage location information can be directly multiplexed, that
, the data storage device in normal condition need not be again to the storage of meta data server synchrodata
Positional information, also avoids it and is introduced into " in recovery " state, the data storage of these normal conditions
Equipment can just start to receive the read-write operation for data, so that when shortening meta data server and restarting
The used time of data in EMS memory storage location information is rebuild, reducing distributed file system can not externally service
Duration, improve system availability and O＆M.
Distributed file system as shown in Figure 2, including meta data server 20 and data storage
Equipment 21.The file preserved in the distributed system is stored in many numbers respectively in the form of data block
According in storage device 21, and each data block includes multiple copies, and each copy is also stored in respectively
, can to ensure that each data block has multiple copies in system in different data storage devices 21
For reading, to improve the fault-tolerance of system.The copy amount of each data block can preset setting, for example
Each data block includes three copies.
In the internal memory of meta data server, concordance list can be set up preserve respectively data storage location information and
The status information of data storage device.Preserve data storage location information concordance list in, with inside with
Machine is designated index key, and the list formed using the corresponding all data storage location information of data block is rope
Draw value.Data storage location information can include the network address and the disk mark of data storage device, example
Such as " tcp://172.168.2.113:11170, Disk2 ", wherein 172.168.2.113:11170 represent number
According to the network address information of storage device, Disk2 is represented to be identified as " Disk2 " inside data storage device
Disk.
Meta data server can determine predetermined number (such as three when preserving data block according to preset rules
It is individual) be used for preserve the data storage location information of the data block, can be according to each data storage device
The information such as status information, disk residual capacity, network site distribution determine.For example, selection is in
The data storage device of normal condition to guarantee to preserve successfully, disk residual capacity it is more so that each
Load balancing between data storage device, and be distributed in different networks to prevent because network is asked as far as possible
Inscribe and each data storage location can not be all accessed.
In the status information concordance list for preserving data storage device, can with the address of data storage device or
Person's name is referred to as index key, and index value is used as using status information.For example, representing that equipment is in not with 00
Upstate, represents that equipment is in normal condition with 01, represents that equipment is in recovery state with 10, its
In, recovery state be after meta data server is restarted, data storage device to meta data server again
State during synchrodata storage location information.In addition, the status information of data storage device can also be wrapped
Include the state of each disk, capacity, copy amount and one in the synchronous copy amount crossed recently
Or it is multiple.For example, Disk1：1,10GB, 1500,500；" 1 " represent disk Disk1 be in can
With state, disk Disk1 capacity is 10GB, and the copy amount preserved is 1500, synchronous recently
The copy amount crossed is 500.
Meta data server is according to prefixed time interval or default Operation Log (operation log)
The factor such as increment be confirmed whether the overall persistence for needing to carry out an internal storage state.
Meta data server is to the information to be persisted (data storage location information and data storage device
Status information etc.) locking protection is carried out, prevent it to be in inconsistent shape by modification during persistence
State；Then any status buffer data in internal memory is emptied, it is to avoid state is imperfect.Carrying out
After locking protection, any modification operation for meta data server data will be temporarily barred.
The internal storage state that meta data server will be persisted is written in the first file of local disk,
It can directly be write among Metadata Service process, bifurcated (fork) function wound can also be utilized
A subprocess is built to complete write-in.First file can be snapshot document, such as checkpoint
(checkpoint) file.
, can be with logarithm because data storage location information has more duplicate contents during write-in
It is compressed according to the address information data of storage device.For example it is stored in the number of same data storage device
There is the address information of identical data storage device according to block, retain an address information and believe the address
Breath correspondence different data block title and disk offset, to accelerate writing speed, reduce persistence and preserve
Time, improve persistence preserve efficiency.
After the write-in of first file is finished, meta data server is discharged to the locking of the information persisted, this
When normal is recovered to the data modification operation of meta data server.
In addition to carrying out persistence preservation to internal storage state, meta data server persistence is preserved for number
Operated according to the modification of storage location information, modification operation is written to the second file.Second file can be with
It is Operation Log (operation log) file.
According to Operation Log and the fast quick-recovery internal storage state of snapshot document when meta data server is restarted.
Meta data server reads current newest (nearest apart from current time) snapshot document to rebuild internal memory
Then all Operation Logs after the snapshot document are re-executed so that internal memory by data structure
The state that state is returned to before restarting.
Meta data server obtains the status information of data storage device from the internal storage state of recovery, for
The data storage device of different conditions carries out different State Transferrings：A, the data for normal condition are deposited
Equipment is stored up, " is recovered due to that can be multiplexed the data storage location information of persistence, therefore need not enter
In " state, directly maintain " normal " state；B, for state non-" normal ", but to
(there are the data of heartbeat connection in the data storage device of meta data server synchrodata storage location information
Storage device) for, the data storage location information in persistence file is likely to outmoded inaccurate,
Therefore the preferential acquisition that data storage location information is carried out to it, " in recovery " is set to by its state,
After the completion of be reconverted into " normal " state；C, the number connected for state non-" normal " and Lungs from Non-Heart-Beating
According to storage device, obtained without locations of copies, maintain " unavailable " state constant.
After the completion of said process, the data storage device of normal condition can just start to receive client to depositing
Have accessed for data is stored up, file system can start external offer service, without waiting for all
Data storage device completes data storage location acquisition of information, and the file system service break period is contracted significantly
It is short.
Restarting for meta data server is likely to occur in meta data server failure and needs what is restarted
Under scene, it is also possible to occur in the case where carrying out the scene that system upgrade and needing is restarted.In these scenes
In, it can act as the recovery time that can shorten meta data server and distributed file system stopping pair
The effect of outer service time, improves system availability.
Persistence operation of the meta data server to internal storage state can also be accomplished by the following way.Will be interior
Deposit the nonvolatile memory (Non-Volatile that state mapping in real time is saved in meta data server
Memory, NVM) in, for example, non-volatile dual inline memory module (NV-
Dual-Inline-Memory-Modules, NV-DIMM) and nand-type (Nand) flash memory, three-dimensional friendship
Crunode (3D Xpoint) flash memory, phase change memory medium (Phase-Change Memory, PCM) etc.
Deng.The status information of data storage location information and data storage device so in internal storage state information
Change can be preserved in real time.Meta data server is when restarting according to nonvolatile memory
The data recovery internal storage state of middle preservation, obtains the status information of data storage device, makes normal condition
Data storage device recovers read operation and write operation of the client to data, so that distributed file system
Recover externally service, the recovery time and distributed file system for shortening meta data server stop external
Service time.
Meta data server recovers internal storage state, and the data storage device of normal condition starts to receive visitor
After family end is to the access of data, locations of copies information of the meta data server to all data storage devices
Carry out one and take turns quick obtaining.During acquisition still can according to the data storage device determined in above-mentioned steps state
Information, preferentially to being obtained in the data storage device for recovering state.Then to other normal conditions
Data storage device obtained, due to after the completion of persistence on data storage device data storage position
Confidence breath there may exist change, can so correct the data storage location information of persistence preservation to most
New state.
After the completion of the restarting of meta data server, it can still continue to keep between all data storage devices
The synchronization of data storage location information, and persistence preservation is carried out to internal storage state according to preset rules.
Meta data server not just safeguards current newest data storage location information, disabled when running into
(for example, the data storage device where some data storage location information stops during data storage location information
Machine), for the data block corresponding to the disabled data storage location information, if currently should
The available copies quantity of data block only has two, less than default value (such as three), Metadata Service
Device can't delete the disabled data storage location information, but by the data storage location information mark
It is designated as " unavailable ".Until there is the corresponding new available data storage location information of the data block to occur
(for example, distributed file system has carried out data duplication) so that available data storage location information
Quantity when being more than or equal to preset value, the data storage location for just deleting labeled " unavailable " is believed
Breath.This is done to there are enough data block copies in guarantee system, loss of data is prevented.For example,
One data block for presetting three copies currently has three available data storage location information A, B, C.
If any one or more appearance are unavailable in A, B, C, they can all be retained, because now
Available data storage location information is less than three, if now also deposited in the presence of other disabled data
Positional information E, F etc. is stored up, E and F can be also retained；But if there is a data available storage position
Confidence ceases D, while B becomes unavailable, then B can be deleted, because now available data are deposited
Storage positional information has reached three including A, C, D.
Therefore, meta data server internal storage state includes a number of disabled data storage location
Information, and these are marked as disabled data storage location information and believed for the copy history of data block
The inquiry of breath can bring help.All data storage location information in meta data server internal memory
(including being marked as the data storage location information of " unavailable ") can by client by order come
Inquiry, while they can be recorded to persistence file (snapshot document, the inspection of meta data server in the lump
Make an inventory of file etc.) in, this provides the record of historical information for the data storage location information of data block,
Reduce the inquiry difficulty of data storage location information relevant issues.
Client can be initiated to obtain the request of readable data storage location information, member to meta data server
When data server is to client returned data storage location information, above-mentioned be marked as " unavailable "
Data storage location information can also be returned together., can be preferentially using just in the fault tolerant mechanism of client
Normal data storage location information；When all data storage location information of the data block are equal " unavailable "
When, client can be to marking to be unavailable " data storage location information read operation is carried out to attempt to read
The data block is taken, to increase the possibility for reading the data block, this can be improved under extreme scenes is
System availability.
The historical information record of the data storage location information of data block is preserved, the distributed text of raising is also helped
The diagnosticability of part system.For example, when the copy for a data block occur all is lost, can pass through
The historical information record of the data storage location information of the data block is inquired about, the data gone wrong are navigated to
Storage device, assist support personnel recovers the data block from the data storage device, prevents data thorough
Lose.
Meta data server in normal operation, disabled data can be also preserved in the manner described above
Storage location information.Meta data server is carried out after persistence preservation to internal storage state, for disabled
Data storage location information, can judge the quantity of the available data storage location information of respective data blocks is
It is no to be less than predetermined number, it will continue to preserve disabled data storage location if less than predetermined number,
With the historical information for the data storage location information for preserving the data block, inquired about for client, when available
The quantity of data storage location information when recovering to more than or equal to predetermined number, then by disabled number
According to storage location information deletion.
It is the device embodiment of the application below, for performing the present processes embodiment.
Fig. 3 is a kind of managing device for data storage location information that the embodiment of the present application passes through, positioned at member
Data server side, including：
Persistence module 30, for preserving internal storage state according to preset rules persistence, wherein, it is described
Internal storage state includes data storage location information and the status information of data storage device；
Internal memory recovery module 31, for after restarting, internal storage state being returned to and restarts preceding persistence
The internal storage state of preservation；
Recovery module 32 is accessed, for recovering normal shape according to the status information of the data storage device
The access of the data storage device of state.
In one embodiment, the persistence module 30 further comprises：
Persistence submodule, for according to predetermined period or according to the behaviour for data storage location information
Make quantity and internal storage state persistence is saved as into the first file, and by repairing for data storage location information
Change operation persistence to preserve to the second file；
The internal memory recovery module 31 further comprises：
Recover submodule, the first file for after restarting, reading nearest apart from current time is laid equal stress on
It is new to perform the modification operation that second file is stored in after first file generated, recover extremely
Internal storage state before restarting.
The persistence submodule further comprises：
Trigger element, for according to predetermined period or according to the operand for data storage location information
Amount triggering carries out persistence preservation to internal storage state；
Compression unit, processing is compressed for the repeated data to the data storage location information；
Writing unit, for the data storage location information after compression to be written into the first file.
In one embodiment, the managing device of the data storage location information further comprises：
Update module, for storage location information to be obtained and updated the data from data storage device.
The access recovery module 32 further comprises：
Recover submodule, the access of the data storage device for recovering normal condition；
Submodule is marked, for the data storage device that will be connected in abnormal condition and with heartbeat
, will be in abnormal condition and without the data storage device mark of heartbeat connection labeled as state is recovered
It is designated as down state；
The update module further comprises：
First updates submodule, for preferentially being obtained simultaneously from the data storage device for being marked as recovery state
Update the data storage location information；
Second updates submodule, the data storage position of the data storage device for being marked as recovery state
Put after information updating finishes, obtained from the data storage device of normal condition and update the data storage location letter
Breath.
In one embodiment, the managing device of the data storage location information further comprises：
Judge module, for when occur disabled data storage location information when, for it is disabled
The corresponding data of data storage location information, judging the quantity of the available data storage location information of data is
It is no to be more than or equal to pre-determined threshold；
Removing module, for being preset when the quantity of the available data storage location information of data is more than or equal to
During thresholding, disabled data storage location information is deleted；
Preserving module, is less than pre-determined threshold for the quantity when the available data storage location information of data
When, preserve disabled data storage location information.
In one embodiment, the managing device of the data storage location information further comprises：
Request receiving module, for receiving the data storage location information inquiring request from client；
Information return module, for returning to corresponding data storage location information to client, the number of return
Include disabled data storage location information according to storage location information.
Fig. 4 is a kind of managing device for data storage location information that the embodiment of the present application is provided, positioned at member
Data server side, including：
Persistence module 30, for preserving internal storage state according to preset rules persistence, wherein, it is described
Internal storage state includes data storage location information and the status information of data storage device；
Judge module 33, for when occur disabled data storage location information when, for can not
The corresponding data of data storage location information, judge the number of the available data storage location information of data
Whether amount is more than or equal to pre-determined threshold；
Removing module 34, for being more than or equal to when the quantity of the available data storage location information of data
During pre-determined threshold, disabled data storage location information is deleted；
Preserving module 35, is less than pre- gating for the quantity when the available data storage location information of data
In limited time, disabled data storage location information is preserved.
In one embodiment, the managing device of the data storage location information further comprises：
Request receiving module, for receiving the data storage location information inquiring request from client；
Information return module, for returning to corresponding data storage location information to client, the number of return
Include disabled data storage location information according to storage location information.
In a typical configuration, computing device include one or more processors (CPU), input/
Output interface, network interface and internal memory.
Internal memory potentially includes the volatile memory in computer-readable medium, random access memory
And/or the form, such as read-only storage (ROM) or flash memory (flash such as Nonvolatile memory (RAM)
RAM).Internal memory is the example of computer-readable medium.
Computer-readable medium includes permanent and non-permanent, removable and non-removable media can be by
Any method or technique come realize information store.Information can be computer-readable instruction, data structure,
The module of program or other data.The example of the storage medium of computer includes, but are not limited to phase transition internal memory
(PRAM), static RAM (SRAM), dynamic random access memory (DRAM),
Other kinds of random access memory (RAM), read-only storage (ROM), electrically erasable
Read-only storage (EEPROM), fast flash memory bank or the read-only storage of other memory techniques, read-only optical disc
Device (CD-ROM), digital versatile disc (DVD) or other optical storages, magnetic cassette tape, magnetic
The storage of band magnetic rigid disk or other magnetic storage apparatus or any other non-transmission medium, can be with available for storage
The information being accessed by a computing device.Defined according to herein, computer-readable medium does not include non-temporary
The data-signal and carrier wave of computer readable media (transitory media), such as modulation.
Some vocabulary have such as been used to censure specific components among specification and claim.This area skill
Art personnel are, it is to be appreciated that hardware manufacturer may call same component with different nouns.This explanation
Book and claim be not in the way of the difference of title is used as differentiation component, but with component in function
On difference be used as the criterion of differentiation.Such as " the bag in specification in the whole text and claim mentioned in
Containing " it is an open language, therefore " include but be not limited to " should be construed to." substantially " refer to receive
Error range in, those skilled in the art can solve the technical problem in the range of certain error,
Basically reach the technique effect.In addition, " coupling " one word is herein comprising any directly and indirect electrical
Coupling means.Therefore, if a first device is coupled to a second device described in text, described is represented
One device can directly be electrically coupled to the second device, or by other devices or couple means indirectly
It is electrically coupled to the second device.Specification subsequent descriptions are to implement the better embodiment of the application,
So the description is the model that the application is not limited to for the purpose of the rule for illustrating the application
Enclose.The protection domain of the application is worked as to be defined depending on the appended claims person of defining.
It should also be noted that, term " comprising ", "comprising" or its any other variant be intended to it is non-
It is exclusive to include, so that commodity or system including a series of key elements not only will including those
Element, but also other key elements including being not expressly set out, or also include for this commodity or be
The intrinsic key element of system.In the absence of more restrictions, limited by sentence "including a ..."
Key element, it is not excluded that also there is other identical element in the commodity or system including the key element.
Some preferred embodiments of the application have shown and described in described above, but as it was previously stated, should manage
Solution the application is not limited to form disclosed herein, is not to be taken as the exclusion to other embodiment,
And available for it is various other combination, modification and environment, and can in invention contemplated scope described herein,
It is modified by the technology or knowledge of above-mentioned teaching or association area.And those skilled in the art are carried out changes
Dynamic and change does not depart from spirit and scope, then all should the application appended claims protection
In the range of.
Claims (18)
1. a kind of management method of data storage location information, it is characterised in that including：
Internal storage state is preserved according to preset rules persistence, wherein, the internal storage state includes data storage
The status information of positional information and data storage device；
After restarting, internal storage state is returned to the internal storage state that persistence is preserved before restarting；
Recover the visit of the data storage device of normal condition according to the status information of the data storage device
Ask.
2. according to the method described in claim 1, it is characterised in that described lasting according to preset rules
Changing preservation internal storage state includes：
According to predetermined period or according to the operation amount for data storage location information by internal storage state
Persistence saves as the first file, and the modification operation persistence for data storage location information is preserved
To the second file；
The internal storage state bag that internal storage state is returned to persistence preservation before restarting after restarting
Include：
After restarting, read first file nearest apart from current time and re-execute described first
The modification operation of second file is stored in after file generated, recovers the internal memory shape to before restarting
State.
3. method according to claim 2, it is characterised in that it is described according to predetermined period or
Internal storage state persistence is saved as by the first file according to the operation amount for data storage location information
Including：
Triggered according to predetermined period or according to the operation amount for data storage location information to internal memory
State carries out persistence preservation；
Repeated data to the data storage location information is compressed processing；
Data storage location information after compression is written to the first file.
4. according to the method described in claim 1, it is characterised in that methods described also includes：
Obtained from data storage device and update the data storage location information.
5. method according to claim 4, it is characterised in that described according to the data storage
The access that the status information of equipment recovers the data storage device of normal condition includes：
Recover the access of the data storage device of normal condition；
The data storage device connected in abnormal condition and with heartbeat is labeled as recovery state,
The data storage device connected in abnormal condition and without heartbeat is labeled as down state；
It is described obtained from data storage device and update the data storage location information include：
It is preferential to be obtained from the data storage device for being marked as recovery state and update the data storage location letter
Breath；
Be marked as the data storage device of recovery state data storage location information updating finish after, from
The data storage device of normal condition obtains and updates the data storage location information.
6. method according to claim 4, it is characterised in that methods described also includes：
When there is disabled data storage location information, for the disabled data storage position
Confidence ceases corresponding data, judges whether the quantity of the available data storage location information of the data is more than
Or equal to pre-determined threshold；
When the quantity of the available data storage location information of the data is more than or equal to pre-determined threshold, delete
Except the disabled data storage location information；
When the quantity of the available data storage location information of the data is less than pre-determined threshold, preserve described
Disabled data storage location information.
7. method according to claim 6, it is characterised in that methods described also includes：
Receive the data storage location information inquiring request from client；
Corresponding data storage location information is returned to client, the data storage location information of return includes
Disabled data storage location information.
8. a kind of management method of data storage location information, it is characterised in that including：
Internal storage state is preserved according to preset rules persistence, wherein, the internal storage state includes data storage
The status information of positional information and data storage device；
When there is disabled data storage location information, for the disabled data storage position
Confidence ceases corresponding data, judges whether the quantity of the available data storage location information of the data is more than
Or equal to pre-determined threshold；
When the quantity of the available data storage location information of the data is more than or equal to pre-determined threshold, delete
Except the disabled data storage location information；
When the quantity of the available data storage location information of the data is less than pre-determined threshold, preserve described
Disabled data storage location information.
9. method according to claim 8, it is characterised in that methods described also includes：
Receive the data storage location information inquiring request from client；
Corresponding data storage location information is returned to client, the data storage location information of return includes
Disabled data storage location information.
10. a kind of managing device of data storage location information, it is characterised in that including：
Persistence module, for preserving internal storage state according to preset rules persistence, wherein, the internal memory
State includes data storage location information and the status information of data storage device；
Internal memory recovery module, preceding persistence preservation is restarted for after restarting, internal storage state to be returned to
Internal storage state；
Recovery module is accessed, for recovering normal condition according to the status information of the data storage device
The access of data storage device.
11. device according to claim 10, it is characterised in that the persistence module includes：
Persistence submodule, for according to predetermined period or according to the behaviour for data storage location information
Make quantity and internal storage state persistence is saved as into the first file, and by repairing for data storage location information
Change operation persistence to preserve to the second file；
The internal memory recovery module includes：
Recover submodule, the first file for after restarting, reading nearest apart from current time is laid equal stress on
It is new to perform the modification operation that second file is stored in after first file generated, recover extremely
Internal storage state before restarting.
12. device according to claim 10, it is characterised in that the persistence submodule bag
Include：
Trigger element, for according to predetermined period or according to the operand for data storage location information
Amount triggering carries out persistence preservation to internal storage state；
Compression unit, processing is compressed for the repeated data to the data storage location information；
Writing unit, for the data storage location information after compression to be written into the first file.
13. device according to claim 10, it is characterised in that described device also includes：
Update module, for storage location information to be obtained and updated the data from data storage device.
14. device according to claim 13, it is characterised in that the access recovery module bag
Include：
Recover submodule, the access of the data storage device for recovering normal condition；
Submodule is marked, for the data storage device that will be connected in abnormal condition and with heartbeat
, will be in abnormal condition and without the data storage device mark of heartbeat connection labeled as state is recovered
It is designated as down state；
The update module includes：
First updates submodule, for preferentially being obtained simultaneously from the data storage device for being marked as recovery state
Update the data storage location information；
Second updates submodule, the data storage position of the data storage device for being marked as recovery state
Put after information updating finishes, obtained from the data storage device of normal condition and update the data storage location letter
Breath.
15. device according to claim 13, it is characterised in that described device also includes：
Judge module, for when occur disabled data storage location information when, for it is described can not
The corresponding data of data storage location information, judge the available data storage location information of the data
Quantity whether be more than or equal to pre-determined threshold；
Removing module, for being more than or equal to when the quantity of the available data storage location information of the data
During pre-determined threshold, the disabled data storage location information is deleted；
Preserving module, is less than pre- gating for the quantity when the available data storage location information of the data
In limited time, the disabled data storage location information is preserved.
16. device according to claim 15, it is characterised in that described device also includes：
Request receiving module, for receiving the data storage location information inquiring request from client；
Information return module, for returning to corresponding data storage location information to client, the number of return
Include disabled data storage location information according to storage location information.
17. a kind of managing device of data storage location information, it is characterised in that including：
Persistence module, for preserving internal storage state according to preset rules persistence, wherein, the internal memory
State includes data storage location information and the status information of data storage device；
Judge module, for when occur disabled data storage location information when, for it is described can not
The corresponding data of data storage location information, judge the available data storage location information of the data
Quantity whether be more than or equal to pre-determined threshold；
Removing module, for being more than or equal to when the quantity of the available data storage location information of the data
During pre-determined threshold, the disabled data storage location information is deleted；
Preserving module, is less than pre- gating for the quantity when the available data storage location information of the data
In limited time, the disabled data storage location information is preserved.
18. device according to claim 17, it is characterised in that described device also includes：
Request receiving module, for receiving the data storage location information inquiring request from client；
Information return module, for returning to corresponding data storage location information to client, the number of return
Include disabled data storage location information according to storage location information.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610088248.7A CN107092598A (en) | 2016-02-17 | 2016-02-17 | The management method and device of data storage location information |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610088248.7A CN107092598A (en) | 2016-02-17 | 2016-02-17 | The management method and device of data storage location information |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107092598A true CN107092598A (en) | 2017-08-25 |
Family
ID=59645976
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610088248.7A Pending CN107092598A (en) | 2016-02-17 | 2016-02-17 | The management method and device of data storage location information |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107092598A (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109286672A (en) * | 2018-09-30 | 2019-01-29 | 北京金山云网络技术有限公司 | A kind of processing method, device and the server of user's request |
| CN110896406A (en) * | 2018-09-13 | 2020-03-20 | 华为技术有限公司 | Data storage method and device and server |
| CN112148762A (en) * | 2019-06-28 | 2020-12-29 | 西安京迅递供应链科技有限公司 | Statistical method and device for real-time data stream |
| CN112565393A (en) * | 2020-12-01 | 2021-03-26 | 平安科技（深圳）有限公司 | File uploading method, file downloading method, file uploading device, file downloading device, computer equipment and storage medium |
| CN112637327A (en) * | 2020-12-21 | 2021-04-09 | 北京奇艺世纪科技有限公司 | Data processing method, device and system |
| CN113407636A (en) * | 2021-07-09 | 2021-09-17 | 浙江明度智控科技有限公司 | Offline data synchronization method and device for digital factory and server |
| CN115454958A (en) * | 2022-09-15 | 2022-12-09 | 北京百度网讯科技有限公司 | Data processing method, device, equipment, system and medium based on artificial intelligence |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP0750258A2 (en) * | 1995-06-19 | 1996-12-27 | Kabushiki Kaisha Toshiba | Memory state recovering apparatus |
| CN102567490A (en) * | 2011-12-21 | 2012-07-11 | 华为技术有限公司 | Method and apparatus for recovering description information and caching data in database |
| CN102662795A (en) * | 2012-03-20 | 2012-09-12 | 浪潮电子信息产业股份有限公司 | Metadata fault-tolerant recovery method in distributed storage system |
| CN103198122A (en) * | 2013-04-02 | 2013-07-10 | 华为技术有限公司 | Method and device for restarting in-memory database |
| CN104216802A (en) * | 2014-09-25 | 2014-12-17 | 北京金山安全软件有限公司 | Memory database recovery method and device |
| CN105045685A (en) * | 2015-08-04 | 2015-11-11 | 浪潮电子信息产业股份有限公司 | Backup method and apparatus for cached data and recovery method and apparatus for cached data |
-
2016
- 2016-02-17 CN CN201610088248.7A patent/CN107092598A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP0750258A2 (en) * | 1995-06-19 | 1996-12-27 | Kabushiki Kaisha Toshiba | Memory state recovering apparatus |
| CN102567490A (en) * | 2011-12-21 | 2012-07-11 | 华为技术有限公司 | Method and apparatus for recovering description information and caching data in database |
| CN102662795A (en) * | 2012-03-20 | 2012-09-12 | 浪潮电子信息产业股份有限公司 | Metadata fault-tolerant recovery method in distributed storage system |
| CN103198122A (en) * | 2013-04-02 | 2013-07-10 | 华为技术有限公司 | Method and device for restarting in-memory database |
| CN104216802A (en) * | 2014-09-25 | 2014-12-17 | 北京金山安全软件有限公司 | Memory database recovery method and device |
| CN105045685A (en) * | 2015-08-04 | 2015-11-11 | 浪潮电子信息产业股份有限公司 | Backup method and apparatus for cached data and recovery method and apparatus for cached data |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110896406A (en) * | 2018-09-13 | 2020-03-20 | 华为技术有限公司 | Data storage method and device and server |
| US11403227B2 (en) | 2018-09-13 | 2022-08-02 | Huawei Technologies Co., Ltd. | Data storage method and apparatus, and server |
| CN109286672A (en) * | 2018-09-30 | 2019-01-29 | 北京金山云网络技术有限公司 | A kind of processing method, device and the server of user's request |
| CN109286672B (en) * | 2018-09-30 | 2020-11-27 | 北京金山云网络技术有限公司 | User request processing method and device and server |
| CN112148762A (en) * | 2019-06-28 | 2020-12-29 | 西安京迅递供应链科技有限公司 | Statistical method and device for real-time data stream |
| CN112565393A (en) * | 2020-12-01 | 2021-03-26 | 平安科技（深圳）有限公司 | File uploading method, file downloading method, file uploading device, file downloading device, computer equipment and storage medium |
| CN112565393B (en) * | 2020-12-01 | 2024-01-05 | 平安科技（深圳）有限公司 | File uploading method, downloading method, device, computer equipment and storage medium |
| CN112637327A (en) * | 2020-12-21 | 2021-04-09 | 北京奇艺世纪科技有限公司 | Data processing method, device and system |
| CN113407636A (en) * | 2021-07-09 | 2021-09-17 | 浙江明度智控科技有限公司 | Offline data synchronization method and device for digital factory and server |
| CN113407636B (en) * | 2021-07-09 | 2022-06-03 | 明度智云(浙江)科技有限公司 | Offline data synchronization method and device for digital factory and server |
| CN115454958A (en) * | 2022-09-15 | 2022-12-09 | 北京百度网讯科技有限公司 | Data processing method, device, equipment, system and medium based on artificial intelligence |
| CN115454958B (en) * | 2022-09-15 | 2024-03-05 | 北京百度网讯科技有限公司 | Data processing method, device, equipment, system and medium based on artificial intelligence |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107092598A (en) | The management method and device of data storage location information | |
| CN106407040B (en) | A kind of duplicating remote data method and system | |
| US7353335B2 (en) | Storage control method for database recovery in logless mode | |
| CN100440214C (en) | Management of metadata in a storage subsystem, system and computer program | |
| US20140089266A1 (en) | Information processing system | |
| US20150066857A1 (en) | Replication of snapshots and clones | |
| US10496618B2 (en) | Managing data replication in a data grid | |
| US9542279B2 (en) | Shadow paging based log segment directory | |
| CN108701048A (en) | Data load method and device | |
| CN109800185B (en) | Data caching method in data storage system | |
| BR112013032923B1 (en) | APPARATUS, COMPUTER IMPLEMENTED METHOD AND COMPUTER-READABLE MEDIA TO MANAGE REPLICATED VIRTUAL STORAGE IN RECOVERY SITES | |
| CN104516943A (en) | Archival management of database logs | |
| WO2007103141A2 (en) | Method and apparatus for providing virtual machine backup | |
| CN105593829B (en) | Method, system and the medium of file system object are excluded from original image backup | |
| US20140108345A1 (en) | Exchanging locations of an out of synchronization indicator and a change recording indicator via pointers | |
| US11397649B2 (en) | Generating standby cloud versions of a virtual machine | |
| CN107615255A (en) | Utilize the computing device and method of long-time memory | |
| EP3147797B1 (en) | Data management method, node and system for database cluster | |
| CN107851061A (en) | The affairs that hardware aids in remote memory are submitted | |
| EP3089051B1 (en) | Database rollback using wal | |
| US8019953B2 (en) | Method for providing atomicity for host write input/outputs (I/Os) in a continuous data protection (CDP)-enabled volume using intent log | |
| CN108762982B (en) | A kind of database restoring method, apparatus and system | |
| CN111506253A (en) | Distributed storage system and storage method thereof | |
| US10983709B2 (en) | Methods for improving journal performance in storage networks and devices thereof | |
| CN113590596A (en) | Data processing method, system, device, computer program product and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20170825 |