CN109542858A - Data archiving method, device based on distributed memory system, computer equipment - Google Patents
Data archiving method, device based on distributed memory system, computer equipment Download PDFInfo
- Publication number
- CN109542858A CN109542858A CN201811185739.9A CN201811185739A CN109542858A CN 109542858 A CN109542858 A CN 109542858A CN 201811185739 A CN201811185739 A CN 201811185739A CN 109542858 A CN109542858 A CN 109542858A
- Authority
- CN
- China
- Prior art keywords
- file
- memory system
- distributed memory
- store path
- visited
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 33
- 238000012546 transfer Methods 0.000 claims description 52
- 238000004590 computer program Methods 0.000 claims description 12
- 239000002699 waste material Substances 0.000 abstract description 6
- 239000004744 fabric Substances 0.000 description 5
- 238000000151 deposition Methods 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 238000010408 sweeping Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 239000000686 essence Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
This application involves technical field of distributed memory, in particular to a kind of data archiving method based on distributed memory system, device, computer equipment.The method includes: file of the scanning in distributed memory system;Judge whether the file scanned meets the archive condition of data filing strategy;If the file scanned meets archive condition, the store path for the file that writing scan is arrived obtains the first store path;According to the first store path, the directory listing about store path is generated;Directory listing is read, the corresponding file in distributed memory system of the first store path is transferred in archival storage and is stored.Scan the file in distributed memory system, if the file scanned meets the archive condition of data filing strategy, the file for meeting the archive condition of data filing strategy is transferred in archival storage and is stored, aim to solve the problem that the data filing of existing distributed memory system, need the problem of manually participating in, lead to waste of human resource.
Description
Technical field
This application involves technical field of distributed memory, in particular to a kind of data filing based on distributed memory system
Method, apparatus, computer equipment.
Background technique
It with the development of internet technology and applies, a large amount of data can be generated in the application process of internet.Mutual
It needs to store a large amount of data in networking.In traditional memory technology, when the data that user generates are more, it can face
The increasing situation in the space of the storage equipment used, carrying cost become higher and higher as space increases.Existing point
The data filing of cloth storage system, generally requires human intervention, copies filing to by hand from a set of distributed memory system and deposits
Reservoir, then the artificial data deleted after filing, data filing process need artificial participate in.
Apply for content
In view of the shortcomings of the prior art, the application propose a kind of data archiving method based on distributed memory system, device,
Computer equipment, it is intended to which the data filing for solving existing distributed memory system needs manually to participate in, and causes human resources unrestrained
The problem of taking.
The technical solution that the application proposes is:
A kind of data archiving method based on distributed memory system, which comprises
Scan the file in distributed memory system;
Judge whether the file scanned meets the archive condition of data filing strategy;
If the file scanned meets the archive condition of the data filing strategy, the text scanned is recorded
The store path of part obtains the first store path;
According to first store path, the directory listing about store path is generated;
The directory listing is read, the corresponding file in distributed memory system of first store path is transferred to
It is stored in archival storage.
Further, the listed files is read described, first store path is corresponding in distributed storage system
File in system was transferred to after the step of storing in archival storage, comprising:
Receive the request of front-end A PI access file;
Judge whether file to be visited is stored in distributed memory system;
If the file to be visited is previously stored in distributed memory system, allow the request.
Further, it is described receive front-end A PI access file request the step of before, comprising:
Record the transfer record that the file of archival storage is transferred to from distributed memory system;
It is described judge whether file to be visited is stored in the step in distributed memory system after, comprising:
If the file to be visited is not previously stored in distributed memory system, judge that the file to be visited is
No is the file that archival storage is transferred to from distributed memory system;
If the file to be visited is the file for being transferred to archival storage from distributed memory system, described in acquisition
The store path of file to be visited in distributed memory system;
It is recorded according to the transfer, it is corresponding to search store path of the file to be visited in distributed memory system
Store path in archival storage obtains the second store path;
File in second store path is pulled back into the file to be visited in distributed memory system
Store path;
After completing readjustment, allow the request.
Further, described after completing readjustment, after the step of allowing the request, comprising:
After the front-end A PI accesses the end of file, the file adjusted back in distributed memory system is turned again
It moves in archival storage and stores.
Further, it is described receive front-end A PI access file request the step of before, comprising:
Record the transfer record that the file of archival storage is transferred to from distributed memory system;
It is described judge whether file to be visited is stored in the step in distributed memory system after, comprising:
If the file to be visited is not previously stored in distributed memory system, judge that the file to be visited is
No is the file that archival storage is transferred to from distributed memory system;
If file to be visited is the file for being transferred to archival storage from distributed memory system, obtain described wait visit
Store path of the file asked in distributed memory system;
It is recorded according to the transfer, it is corresponding to search store path of the file to be visited in distributed memory system
Store path in archival storage obtains the second store path;
It configures store path of the file to be visited in distributed memory system and jumps to second storage road
Diameter;
After the configuration has been completed, allow the request.
Further, the directory listing is read described, first store path is corresponding in distributed storage system
File in system was transferred in the step of storing in archival storage, comprising:
Read the directory listing;
According to the directory listing, the corresponding file in distributed memory system of first store path is cut one by one
It switches in archival storage;Or
Read the directory listing;
According to the directory listing, the corresponding file in distributed memory system of first store path is answered one by one
It makes in archival storage;
After completing duplication, the corresponding file in distributed memory system of first store path is deleted.
Further, the data filing strategy includes time conditions archiving policies or/and catalogue condition archiving policies, institute
State time conditions archiving policies be at the appointed time before file meet archive condition, the catalogue condition archiving policies are to refer to
The file determined under catalogue meets archive condition.
The application also provides a kind of data filing device based on distributed memory system, and described device includes:
Scan module, for scanning the file in distributed memory system;
Judgment module, for judging whether the file scanned meets the archive condition of data filing strategy;
Logging modle records if meeting the archive condition of the data filing strategy for the file scanned
The store path of the file scanned obtains the first store path;
Generation module, for generating the directory listing about store path according to first store path;
Profiling module, it is for reading the directory listing, first store path is corresponding in distributed memory system
In file be transferred in archival storage and store.
The application also provides a kind of computer equipment, including memory and processor, and the memory is stored with computer
The step of program, the processor realizes method described in any of the above embodiments when executing the computer program.
The application also provides a kind of computer readable storage medium, is stored thereon with computer program, the computer journey
The step of method described in any of the above embodiments is realized when sequence is executed by processor.
According to above-mentioned technical solution, the application is the utility model has the advantages that scan the file in distributed memory system, if scanning
To file meet the archive condition of data filing strategy, the file for meeting the archive condition of data filing strategy is transferred to and is returned
It is stored in shelves memory, it is intended to which the data filing for solving existing distributed memory system needs manually to participate in, manpower is caused to provide
The problem of source wastes.
Detailed description of the invention
Fig. 1 is the flow chart using the data archiving method provided by the embodiments of the present application based on distributed memory system;
Fig. 2 is the functional module using the data filing device provided by the embodiments of the present application based on distributed memory system
Figure;
Fig. 3 is the structural schematic block diagram using computer equipment provided by the embodiments of the present application.
Specific embodiment
It is with reference to the accompanying drawings and embodiments, right in order to which the objects, technical solutions and advantages of the application are more clearly understood
The application is further elaborated.It should be appreciated that specific embodiment described herein is only used to explain the application, and
It is not used in restriction the application.
As shown in Figure 1, the embodiment of the present application proposes a kind of data archiving method based on distributed memory system, the side
Method the following steps are included:
Step S101, file of the scanning in distributed memory system.
File in distributed memory system is scanned, in the present embodiment, timing scan is in distributed storage
File in system is executed according to the time of setting and is scanned to the file in distributed memory system.
In some embodiments, before step S101, comprising:
The filing to the file in distributed memory system is received to instruct.
After receiving the filing instruction to the file in distributed memory system, start scanning in distributed storage system
File in system.That is, can just start to execute scanning in the case where receiving filing instruction.
Step S102, judge whether the file scanned meets the archive condition of data filing strategy.
When scanning the file in distributed memory system, needs to judge the file scanned, judge to scan
To file whether meet the archive condition of data filing strategy.
In the present embodiment, in step s 102, comprising:
According to the file scanned, the file information corresponding with the archive condition of data filing strategy is obtained;
According to the file information, judge whether the file scanned meets the archive condition of data filing strategy.
Due to the available many file informations of the file scanned, in order to improve judging efficiency, acquisition is returned with data
The corresponding the file information of archive condition of shelves strategy, according to the file information, judges the text scanned after obtaining the file information
Whether part meets the archive condition of data filing strategy.
In the present embodiment, data filing strategy includes time conditions archiving policies or/and catalogue condition archiving policies, when
Between condition archiving policies be at the appointed time before file meet archive condition, catalogue condition archiving policies are under specified directory
File meet archive condition.
If step S103, the described file scanned meets the archive condition, the file scanned is recorded
Store path obtains the first store path.
According to judging result, it can obtain whether the file scanned meets the archive condition of data filing strategy, if sweeping
The file retouched does not meet the archive condition of data filing strategy, then does not need to carry out archive operation, if the file symbol scanned
The archive condition for closing data filing strategy, needs to carry out archive operation, then the store path for the file that writing scan is arrived, and obtains the
One store path.
Step S104, according to first store path, the directory listing about store path is generated.
After obtaining the first store path, directory listing is generated according to the first store path, directory listing is about depositing
Store up path.By reading directory listing, the corresponding storage of file for meeting the archive condition of data filing strategy can be obtained
Path.
Step S105, the directory listing is read, first store path is corresponding in distributed memory system
File is transferred in archival storage and stores.
After generating directory listing, directory listing is read, the first store path is obtained, then by the first store path pair
It should be transferred in archival storage and store in the file in distributed memory system.Archival storage is to meet data for storing
The file of the archive condition of archiving policies.
In the present embodiment, in step s105, comprising:
Read the directory listing;
According to the directory listing, the corresponding file in distributed memory system of first store path is cut one by one
It switches in archival storage.
Directory listing is read, the first store path is obtained and is deposited one by one by first according to directory listing according to the sequence of list
The corresponding file in distributed memory system in storage path is sheared, and is pasted in archival storage later, thus release point
The storage space of cloth storage system.
In some embodiments, in step s105, comprising:
Read the directory listing;
According to the directory listing, the corresponding file in distributed memory system of first store path is answered one by one
It makes in archival storage;
After completing duplication, the corresponding file in distributed memory system of first store path is deleted.
Directory listing is read, the first store path is obtained and is deposited one by one by first according to directory listing according to the sequence of list
The corresponding file in distributed memory system in storage path is replicated, and is pasted in archival storage later, completes to replicate
Later, the corresponding file in distributed memory system of the first store path is deleted, thus the storage of release profile formula storage system
Deposit space.
In the present embodiment, after step S105, comprising:
Receive the request of front-end A PI access file;
Judge whether file to be visited is stored in distributed memory system;
If the file to be visited is previously stored in distributed memory system, allow the request.
The corresponding file in distributed memory system of first store path is transferred to after being stored in archival storage,
The transfer record that the file of archival storage is transferred to from distributed memory system is recorded, transfer record includes the staged files, turns
Text move part is before transfer in the store path of distributed memory system, the staged files depositing in archival storage after the transfer
Store up path.After completing to record transfer record, the request of front-end A PI access file is received, it be first to file to be visited
Position is judged, judges whether file to be visited is stored in distributed memory system, if file to be visited is
It is stored in distributed memory system, then allows to request.
The full name of above-mentioned API is Application Programming Interface, and the entitled application program of Chinese is compiled
Journey interface.
In the present embodiment, it is described receive front-end A PI access file request the step of before, comprising:
Record the transfer record that the file of archival storage is transferred to from distributed memory system;
Accordingly, after judging whether file to be visited is stored in the step in distributed memory system, comprising:
If the file to be visited is not previously stored in distributed memory system, judge that the file to be visited is
No is the file that archival storage is transferred to from distributed memory system;
If the file to be visited is the file for being transferred to archival storage from distributed memory system, described in acquisition
The store path of file to be visited in distributed memory system;
It is recorded according to the transfer, it is corresponding to search store path of the file to be visited in distributed memory system
Store path in archival storage obtains the second store path;
File in second store path is pulled back into the file to be visited in distributed memory system
Store path;
After completing readjustment, allow the request.
If file to be visited is not previously stored in distributed memory system, also need to be judged again, judge to
Whether the file of access is the file that archival storage is transferred to from distributed memory system, if file to be visited be not for from
Distributed memory system is transferred to the file of archival storage, then refuses to request.If file to be visited is from distributed storage
System is transferred to the file of archival storage, obtains the store path of file to be visited in distributed memory system, due to
Transfer record exists in the store path of distributed memory system, the staged files before transfer including the staged files, the staged files
In the store path of archival storage after transfer, is recorded according to transfer, search file to be visited in distributed memory system
In the corresponding store path in archival storage of store path, obtain the second store path, obtaining the second store path
Later, the file in the second store path is pulled back into the position before its transfer, that is, pulls back to file to be visited and exists
Store path in distributed memory system allows to request, front-end A PI is from distributed memory system after completing readjustment
Access file.
Described after completing readjustment, after the step of allowing the request, comprising:
After the front-end A PI accesses the end of file, the file adjusted back in distributed memory system is turned again
It moves in archival storage and stores.
After front-end A PI accesses the end of file, for the storage space of release profile formula storage system, it will adjust back
File in distributed memory system, which is again transferred in archival storage, to be stored.
In some embodiments, it is described receive front-end A PI access file request the step of before, comprising:
Record the transfer record that the file of archival storage is transferred to from distributed memory system;
Accordingly, it is described judge whether file to be visited is stored in the step in distributed memory system after, packet
It includes:
If the file to be visited is not previously stored in distributed memory system, judge that the file to be visited is
No is the file that archival storage is transferred to from distributed memory system;
If file to be visited is the file for being transferred to archival storage from distributed memory system, obtain described wait visit
Store path of the file asked in distributed memory system;
It is recorded according to the transfer, it is corresponding to search store path of the file to be visited in distributed memory system
Store path in archival storage obtains the second store path;
It configures store path of the file to be visited in distributed memory system and jumps to second storage road
Diameter;
After the configuration has been completed, allow the request.
If file to be visited is not previously stored in distributed memory system, also need to be judged again, judge to
Whether the file of access is the file that archival storage is transferred to from distributed memory system, if file to be visited be not for from
Distributed memory system is transferred to the file of archival storage, then refuses to request.If file to be visited is from distributed storage
System is transferred to the file of archival storage, obtains the store path of file to be visited in distributed memory system, due to
Transfer record exists in the store path of distributed memory system, the staged files before transfer including the staged files, the staged files
In the store path of archival storage after transfer, is recorded according to transfer, search file to be visited in distributed memory system
In the corresponding store path in archival storage of store path, obtain the second store path, obtaining the second store path
Later, it configures the store path of file to be visited in distributed memory system and jumps to the second store path, it is, will
The store path of file to be visited in distributed memory system jumps in the store path of archival storage, completes to match
After setting, allow to request, front-end A PI accesses file from archival storage.
In conclusion file of the scanning in distributed memory system, if the file scanned meets data filing strategy
Archive condition, the file for meeting the archive condition of data filing strategy is transferred in archival storage and is stored, it is intended to be solved
The data filing of existing distributed memory system, needs the problem of manually participating in, lead to waste of human resource.
As shown in Fig. 2, the embodiment of the present application proposes a kind of data filing device 1 based on distributed memory system, device 1
Including scan module 11, judgment module 12, logging modle 13, generation module 14 and profiling module 15.
Scan module 11, for scanning the file in distributed memory system.
File in distributed memory system is scanned, in the present embodiment, timing scan is in distributed storage
File in system is executed according to the time of setting and is scanned to the file in distributed memory system.
In some embodiments, device 1 includes:
Receiving module, for receiving the instruction of the filing to the file in distributed memory system.
After receiving the filing instruction to the file in distributed memory system, start scanning in distributed storage system
File in system.That is, can just start to execute scanning in the case where receiving filing instruction.
Judgment module 12, for judging whether the file scanned meets the archive condition of data filing strategy.
When scanning the file in distributed memory system, needs to judge the file scanned, judge to scan
To file whether meet the archive condition of data filing strategy.
In the present embodiment, judgment module 12 includes:
First obtains module, for obtaining corresponding with the archive condition of data filing strategy according to the file scanned
The file information;
First judgment module, for judging whether the file scanned meets data filing strategy according to the file information
Archive condition.
Due to the available many file informations of the file scanned, in order to improve judging efficiency, acquisition is returned with data
The corresponding the file information of archive condition of shelves strategy, according to the file information, judges the text scanned after obtaining the file information
Whether part meets the archive condition of data filing strategy.
In the present embodiment, data filing strategy includes time conditions archiving policies or/and catalogue condition archiving policies, when
Between condition archiving policies be at the appointed time before file meet archive condition, catalogue condition archiving policies are under specified directory
File meet archive condition.
Logging modle 13 records described scan if meet the archive condition for the file scanned
The store path of file obtains the first store path.
According to judging result, it can obtain whether the file scanned meets the archive condition of data filing strategy, if sweeping
The file retouched does not meet the archive condition of data filing strategy, then does not need to carry out archive operation, if the file symbol scanned
The archive condition for closing data filing strategy, needs to carry out archive operation, then the store path for the file that writing scan is arrived, and obtains the
One store path.
Generation module 14, for generating the directory listing about store path according to first store path.
After obtaining the first store path, directory listing is generated according to the first store path, directory listing is about depositing
Store up path.By reading directory listing, the corresponding storage of file for meeting the archive condition of data filing strategy can be obtained
Path.
Profiling module 15, it is for reading the directory listing, first store path is corresponding in distributed storage system
File in system, which is transferred in archival storage, to be stored.
After generating directory listing, directory listing is read, the first store path is obtained, then by the first store path pair
It should be transferred in archival storage and store in the file in distributed memory system.Archival storage is to meet data for storing
The file of the archive condition of archiving policies.
In the present embodiment, profiling module 15 includes:
First read module, for reading the directory listing;
First profiling module, for one by one corresponding to first store path in distribution according to the directory listing
Paper cut in storage system is into archival storage.
Directory listing is read, the first store path is obtained and is deposited one by one by first according to directory listing according to the sequence of list
The corresponding file in distributed memory system in storage path is sheared, and is pasted in archival storage later, thus release point
The storage space of cloth storage system.
In some embodiments, profiling module 15 includes:
Second read module, for reading the directory listing;
Second profiling module, for one by one corresponding to first store path in distribution according to the directory listing
File in storage system copies in archival storage;
First removing module, it is corresponding in distributed storage for after completing duplication, deleting first store path
File in system.
Directory listing is read, the first store path is obtained and is deposited one by one by first according to directory listing according to the sequence of list
The corresponding file in distributed memory system in storage path is replicated, and is pasted in archival storage later, completes to replicate
Later, the corresponding file in distributed memory system of the first store path is deleted, thus the storage of release profile formula storage system
Deposit space.
In the present embodiment, device 1 includes:
First receiving module, for receiving the request of front-end A PI access file;
Second judgment module, for judging whether file to be visited is stored in distributed memory system;
First permission module allows institute if being previously stored in distributed memory system for the file to be visited
State request.
The corresponding file in distributed memory system of first store path is transferred to after being stored in archival storage,
The transfer record that the file of archival storage is transferred to from distributed memory system is recorded, transfer record includes the staged files, turns
Text move part is before transfer in the store path of distributed memory system, the staged files depositing in archival storage after the transfer
Store up path.After completing to record transfer record, the request of front-end A PI access file is received, it be first to file to be visited
Position is judged, judges whether file to be visited is stored in distributed memory system, if file to be visited is
It is stored in distributed memory system, then allows to request.
The full name of above-mentioned API is Application Programming Interface, and the entitled application program of Chinese is compiled
Journey interface.
In the present embodiment, device 1 includes:
First logging modle, for recording the transfer note for being transferred to the file of archival storage from distributed memory system
Record;
Third judgment module judges if not being previously stored in distributed memory system for the file to be visited
Whether the file to be visited is the file that archival storage is transferred to from distributed memory system;
Second obtains module, if being to be transferred to archival storage from distributed memory system for the file to be visited
File, then obtain store path of the file to be visited in distributed memory system;
First searching module searches the file to be visited in distributed storage system for recording according to the transfer
The corresponding store path in archival storage of store path in system, obtains the second store path;
First callback module is being divided for the file in second store path to be pulled back to the file to be visited
Store path in cloth storage system;
Second allows module, for allowing the request after completing readjustment.
If file to be visited is not previously stored in distributed memory system, also need to be judged again, judge to
Whether the file of access is the file that archival storage is transferred to from distributed memory system, if file to be visited be not for from
Distributed memory system is transferred to the file of archival storage, then refuses to request.If file to be visited is from distributed storage
System is transferred to the file of archival storage, obtains the store path of file to be visited in distributed memory system, due to
Transfer record exists in the store path of distributed memory system, the staged files before transfer including the staged files, the staged files
In the store path of archival storage after transfer, is recorded according to transfer, search file to be visited in distributed memory system
In the corresponding store path in archival storage of store path, obtain the second store path, obtaining the second store path
Later, the file in the second store path is pulled back into the position before its transfer, that is, pulls back to file to be visited and exists
Store path in distributed memory system allows to request, front-end A PI is from distributed memory system after completing readjustment
Access file.
Device 1 includes:
Third profiling module, for will adjust back in distributed storage system after the front-end A PI accesses the end of file
File in system, which is again transferred in archival storage, to be stored.
After front-end A PI accesses the end of file, for the storage space of release profile formula storage system, it will adjust back
File in distributed memory system, which is again transferred in archival storage, to be stored.
In some embodiments, device 1 includes:
Second logging modle, for recording the transfer note for being transferred to the file of archival storage from distributed memory system
Record;
4th judgment module judges if not being previously stored in distributed memory system for the file to be visited
Whether the file to be visited is the file that archival storage is transferred to from distributed memory system;
Third obtains module, if being the text for being transferred to archival storage from distributed memory system for file to be visited
Part then obtains store path of the file to be visited in distributed memory system;
Second searching module searches the file to be visited in distributed storage system for recording according to the transfer
The corresponding store path in archival storage of store path in system, obtains the second store path;
First configuration module is jumped for configuring store path of the file to be visited in distributed memory system
To second store path;
Third allows module, for after the configuration has been completed, allowing the request.
If file to be visited is not previously stored in distributed memory system, also need to be judged again, judge to
Whether the file of access is the file that archival storage is transferred to from distributed memory system, if file to be visited be not for from
Distributed memory system is transferred to the file of archival storage, then refuses to request.If file to be visited is from distributed storage
System is transferred to the file of archival storage, obtains the store path of file to be visited in distributed memory system, due to
Transfer record exists in the store path of distributed memory system, the staged files before transfer including the staged files, the staged files
In the store path of archival storage after transfer, is recorded according to transfer, search file to be visited in distributed memory system
In the corresponding store path in archival storage of store path, obtain the second store path, obtaining the second store path
Later, it configures the store path of file to be visited in distributed memory system and jumps to the second store path, it is, will
The store path of file to be visited in distributed memory system jumps in the store path of archival storage, completes to match
After setting, allow to request, front-end A PI accesses file from archival storage.
In conclusion file of the scanning in distributed memory system, if the file scanned meets data filing strategy
Archive condition, the file for meeting the archive condition of data filing strategy is transferred in archival storage and is stored, it is intended to be solved
The data filing of existing distributed memory system, needs the problem of manually participating in, lead to waste of human resource.
As shown in figure 3, also providing a kind of computer equipment in the embodiment of the present application, which can be service
Device, internal structure can be as shown in Figure 3.The computer equipment includes processor, the memory, net connected by system bus
Network interface and database.Wherein, the processor of the Computer Design is for providing calculating and control ability.The computer equipment
Memory includes non-volatile memory medium, built-in storage.The non-volatile memory medium is stored with operating system, computer journey
Sequence and database.The internal memory provides environment for the operation of operating system and computer program in non-volatile memory medium.
The database of the computer equipment is for storing the data such as the model of the data archiving method based on distributed memory system.The meter
The network interface for calculating machine equipment is used to communicate with external terminal by network connection.When the computer program is executed by processor
To realize a kind of data archiving method based on distributed memory system.
Above-mentioned processor executes the step of above-mentioned data archiving method based on distributed memory system: scanning is in distribution
File in storage system;Judge whether the file scanned meets the archive condition of data filing strategy;If the scanning is arrived
File meet the archive condition, then record the store path of the file scanned, obtain the first store path;According to
First store path generates the directory listing about store path;The directory listing is read, by first storage road
The corresponding file in distributed memory system of diameter, which is transferred in archival storage, to be stored.
In one embodiment, first store path is corresponded to and is deposited in distribution by the above-mentioned reading listed files
File in storage system was transferred to after the step of storing in archival storage, comprising:
Receive the request of front-end A PI access file;
Judge whether file to be visited is stored in distributed memory system;
If the file to be visited is previously stored in distributed memory system, allow the request.
In one embodiment, before the step of above-mentioned request for receiving front-end A PI access file, comprising:
Record the transfer record that the file of archival storage is transferred to from distributed memory system;
It is described judge whether file to be visited is stored in the step in distributed memory system after, comprising:
If the file to be visited is not previously stored in distributed memory system, judge that the file to be visited is
No is the file that archival storage is transferred to from distributed memory system;
If the file to be visited is the file for being transferred to archival storage from distributed memory system, described in acquisition
The store path of file to be visited in distributed memory system;
It is recorded according to the transfer, it is corresponding to search store path of the file to be visited in distributed memory system
Store path in archival storage obtains the second store path;
File in second store path is pulled back into the file to be visited in distributed memory system
Store path;
After completing readjustment, allow the request.
In one embodiment, above-mentioned after completing readjustment, after the step of allowing the request, comprising:
After the front-end A PI accesses the end of file, the file adjusted back in distributed memory system is turned again
It moves in archival storage and stores.
In one embodiment, before the step of above-mentioned request for receiving front-end A PI access file, comprising:
Record the transfer record that the file of archival storage is transferred to from distributed memory system;
It is described judge whether file to be visited is stored in the step in distributed memory system after, comprising:
If the file to be visited is not previously stored in distributed memory system, judge that the file to be visited is
No is the file that archival storage is transferred to from distributed memory system;
If file to be visited is the file for being transferred to archival storage from distributed memory system, obtain described wait visit
Store path of the file asked in distributed memory system;
It is recorded according to the transfer, it is corresponding to search store path of the file to be visited in distributed memory system
Store path in archival storage obtains the second store path;
It configures store path of the file to be visited in distributed memory system and jumps to second storage road
Diameter;
After the configuration has been completed, allow the request.
In one embodiment, first store path is corresponded to and is deposited in distribution by the above-mentioned reading directory listing
File in storage system was transferred in the step of storing in archival storage, comprising:
Read the directory listing;
According to the directory listing, the corresponding file in distributed memory system of first store path is cut one by one
It switches in archival storage;Or
Read the directory listing;
According to the directory listing, the corresponding file in distributed memory system of first store path is answered one by one
It makes in archival storage;
After completing duplication, the corresponding file in distributed memory system of first store path is deleted.
In one embodiment, above-mentioned data filing strategy includes time conditions archiving policies or/and the filing of catalogue condition
Strategy, the time conditions archiving policies be at the appointed time before file meet archive condition, the catalogue condition filing
Strategy is that the file under specified directory meets archive condition.
The computer equipment of the embodiment of the present application scans the file in distributed memory system, if the file scanned
The file for meeting the archive condition of data filing strategy is transferred to archival storage by the archive condition for meeting data filing strategy
Middle storage, it is intended to which the data filing for solving existing distributed memory system needs manually to participate in, leads to waste of human resource
Problem.
One embodiment of the application also provides a kind of computer readable storage medium, is stored thereon with computer program, calculates
Machine program realizes a kind of data archiving method based on distributed memory system when being executed by processor, specifically: scanning is dividing
File in cloth storage system;Judge whether the file scanned meets the archive condition of data filing strategy;If described sweep
The file retouched meets the archive condition, then records the store path of the file scanned, obtains the first store path;
According to first store path, the directory listing about store path is generated;The directory listing is read, described first is deposited
The corresponding file in distributed memory system in storage path, which is transferred in archival storage, to be stored.
In one embodiment, first store path is corresponded to and is deposited in distribution by the above-mentioned reading listed files
File in storage system was transferred to after the step of storing in archival storage, comprising:
Receive the request of front-end A PI access file;
Judge whether file to be visited is stored in distributed memory system;
If the file to be visited is previously stored in distributed memory system, allow the request.
In one embodiment, before the step of above-mentioned request for receiving front-end A PI access file, comprising:
Record the transfer record that the file of archival storage is transferred to from distributed memory system;
It is described judge whether file to be visited is stored in the step in distributed memory system after, comprising:
If the file to be visited is not previously stored in distributed memory system, judge that the file to be visited is
No is the file that archival storage is transferred to from distributed memory system;
If the file to be visited is the file for being transferred to archival storage from distributed memory system, described in acquisition
The store path of file to be visited in distributed memory system;
It is recorded according to the transfer, it is corresponding to search store path of the file to be visited in distributed memory system
Store path in archival storage obtains the second store path;
File in second store path is pulled back into the file to be visited in distributed memory system
Store path;
After completing readjustment, allow the request.
In one embodiment, above-mentioned after completing readjustment, after the step of allowing the request, comprising:
After the front-end A PI accesses the end of file, the file adjusted back in distributed memory system is turned again
It moves in archival storage and stores.
In one embodiment, before the step of above-mentioned request for receiving front-end A PI access file, comprising:
Record the transfer record that the file of archival storage is transferred to from distributed memory system;
It is described judge whether file to be visited is stored in the step in distributed memory system after, comprising:
If the file to be visited is not previously stored in distributed memory system, judge that the file to be visited is
No is the file that archival storage is transferred to from distributed memory system;
If file to be visited is the file for being transferred to archival storage from distributed memory system, obtain described wait visit
Store path of the file asked in distributed memory system;
It is recorded according to the transfer, it is corresponding to search store path of the file to be visited in distributed memory system
Store path in archival storage obtains the second store path;
It configures store path of the file to be visited in distributed memory system and jumps to second storage road
Diameter;
After the configuration has been completed, allow the request.
In one embodiment, first store path is corresponded to and is deposited in distribution by the above-mentioned reading directory listing
File in storage system was transferred in the step of storing in archival storage, comprising:
Read the directory listing;
According to the directory listing, the corresponding file in distributed memory system of first store path is cut one by one
It switches in archival storage;Or
Read the directory listing;
According to the directory listing, the corresponding file in distributed memory system of first store path is answered one by one
It makes in archival storage;
After completing duplication, the corresponding file in distributed memory system of first store path is deleted.
In one embodiment, above-mentioned data filing strategy includes time conditions archiving policies or/and the filing of catalogue condition
Strategy, the time conditions archiving policies be at the appointed time before file meet archive condition, the catalogue condition filing
Strategy is that the file under specified directory meets archive condition.
The storage medium of the embodiment of the present application scans the file in distributed memory system, if the file symbol scanned
The archive condition for closing data filing strategy, the file for meeting the archive condition of data filing strategy is transferred in archival storage
Storage, it is intended to which the data filing for solving existing distributed memory system needs manually to participate in, leads to asking for waste of human resource
Topic.
Those of ordinary skill in the art will appreciate that realizing all or part of the process in above-described embodiment method, being can be with
Relevant hardware is instructed to complete by computer program, the computer program can be stored in a non-volatile computer
In read/write memory medium, the computer program is when being executed, it may include such as the process of the embodiment of above-mentioned each method.Wherein,
Any reference used in provided herein and embodiment to memory, storage, database or other media,
Including non-volatile and/or volatile memory.Nonvolatile memory may include read-only memory (ROM), programming ROM
(PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM) or flash memory.Volatile memory may include
Random access memory (RAM) or external cache.By way of illustration and not limitation, RAM is available in many forms,
Such as static state RAM (SRAM), dynamic ram (DRAM), synchronous dram (SDRAM), double speed are according to rate SDRAM (SSRSDRAM), enhancing
Type SDRAM (ESDRAM), synchronization link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM
(RDRAM), direct memory bus dynamic ram (DRDRAM) and memory bus dynamic ram (RDRAM) etc..
The foregoing is merely the preferred embodiments of the application, not to limit the application, all essences in the application
Made any modifications, equivalent replacements, and improvements etc. within mind and principle should all include within the scope of protection of this application.
Claims (10)
1. a kind of data archiving method based on distributed memory system, which is characterized in that the described method includes:
Scan the file in distributed memory system;
Judge whether the file scanned meets the archive condition of data filing strategy;
If the file scanned meets the archive condition, the store path of the file scanned is recorded, is obtained
First store path;
According to first store path, the directory listing about store path is generated;
The directory listing is read, the corresponding file in distributed memory system of first store path is transferred to filing
It is stored in memory.
2. the data archiving method according to claim 1 based on distributed memory system, which is characterized in that in the reading
The listed files is taken, the corresponding file in distributed memory system of first store path is transferred to archival storage
After the step of middle storage, comprising:
Receive the request of front-end A PI access file;
Judge whether file to be visited is stored in distributed memory system;
If the file to be visited is previously stored in distributed memory system, allow the request.
3. the data archiving method according to claim 2 based on distributed memory system, which is characterized in that connect described
Before the step of receiving the request of front-end A PI access file, comprising:
Record the transfer record that the file of archival storage is transferred to from distributed memory system;
It is described judge whether file to be visited is stored in the step in distributed memory system after, comprising:
If the file to be visited is not previously stored in distributed memory system, judge the file to be visited whether be
The file of archival storage is transferred to from distributed memory system;
If the file to be visited is the file for being transferred to archival storage from distributed memory system, obtain described wait visit
Store path of the file asked in distributed memory system;
It is recorded according to the transfer, searches store path correspondence of the file to be visited in distributed memory system and returning
Store path in shelves memory, obtains the second store path;
File in second store path is pulled back into storage of the file to be visited in distributed memory system
Path;
After completing readjustment, allow the request.
4. the data archiving method according to claim 3 based on distributed memory system, which is characterized in that it is described
After the step of completing after adjusting back, allowing the request, comprising:
After the front-end A PI accesses the end of file, the file adjusted back in distributed memory system is again transferred to
It is stored in archival storage.
5. the data archiving method according to claim 2 based on distributed memory system, which is characterized in that connect described
Before the step of receiving the request of front-end A PI access file, comprising:
Record the transfer record that the file of archival storage is transferred to from distributed memory system;
It is described judge whether file to be visited is stored in the step in distributed memory system after, comprising:
If the file to be visited is not previously stored in distributed memory system, judge the file to be visited whether be
The file of archival storage is transferred to from distributed memory system;
If file to be visited is the file for being transferred to archival storage from distributed memory system, obtain described to be visited
Store path of the file in distributed memory system;
It is recorded according to the transfer, searches store path correspondence of the file to be visited in distributed memory system and returning
Store path in shelves memory, obtains the second store path;
It configures store path of the file to be visited in distributed memory system and jumps to second store path;
After the configuration has been completed, allow the request.
6. the data archiving method according to claim 1 based on distributed memory system, which is characterized in that in the reading
The directory listing is taken, the corresponding file in distributed memory system of first store path is transferred to archival storage
In the step of middle storage, comprising:
Read the directory listing;
According to the directory listing, the corresponding paper cut in distributed memory system of first store path is arrived one by one
In archival storage;Or
Read the directory listing;
According to the directory listing, the corresponding file in distributed memory system of first store path is copied to one by one
In archival storage;
After completing duplication, the corresponding file in distributed memory system of first store path is deleted.
7. the data archiving method according to claim 1 based on distributed memory system, which is characterized in that the data
Archiving policies include time conditions archiving policies or/and catalogue condition archiving policies, and the time conditions archiving policies are to refer to
File before fixing time meets archive condition, and the catalogue condition archiving policies are that the file under specified directory meets filing item
Part.
8. a kind of data filing device based on distributed memory system, which is characterized in that described device includes:
Scan module, for scanning the file in distributed memory system;
Judgment module, for judging whether the file scanned meets the archive condition of data filing strategy;
Logging modle, if meeting the archive condition of the data filing strategy for the file scanned, described in record
The store path of the file scanned obtains the first store path;
Generation module, for generating the directory listing about store path according to first store path;
Profiling module, it is for reading the directory listing, first store path is corresponding in distributed memory system
File is transferred in archival storage and stores.
9. a kind of computer equipment, including memory and processor, the memory are stored with computer program, feature exists
In the processor realizes method described in any one of claims 1 to 7 when executing computer program the step of.
10. a kind of computer readable storage medium, is stored thereon with computer program, which is characterized in that the computer program
The step of method described in any one of claims 1 to 7 is realized when being executed by processor.
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201811185739.9A CN109542858A (en) | 2018-10-11 | 2018-10-11 | Data archiving method, device based on distributed memory system, computer equipment |
PCT/CN2018/124395 WO2020073517A1 (en) | 2018-10-11 | 2018-12-27 | Data archiving method and apparatus based on distributed storage system, and computer device |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201811185739.9A CN109542858A (en) | 2018-10-11 | 2018-10-11 | Data archiving method, device based on distributed memory system, computer equipment |
Publications (1)
Publication Number | Publication Date |
---|---|
CN109542858A true CN109542858A (en) | 2019-03-29 |
Family
ID=65843648
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201811185739.9A Pending CN109542858A (en) | 2018-10-11 | 2018-10-11 | Data archiving method, device based on distributed memory system, computer equipment |
Country Status (2)
Country | Link |
---|---|
CN (1) | CN109542858A (en) |
WO (1) | WO2020073517A1 (en) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN110457255A (en) * | 2019-07-05 | 2019-11-15 | 中国平安人寿保险股份有限公司 | Method, server and the computer readable storage medium of data filing |
CN111614730A (en) * | 2020-04-28 | 2020-09-01 | 北京金山云网络技术有限公司 | File processing method and device of cloud storage system and electronic equipment |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN103036956A (en) * | 2012-11-30 | 2013-04-10 | 航天恒星科技有限公司 | Filing system and implement method of distributed configured massive data |
CN104199926A (en) * | 2014-09-01 | 2014-12-10 | 北京联创信安科技有限公司 | File archiving implementation method and device and file access method and device |
CN106537380A (en) * | 2014-06-30 | 2017-03-22 | 谷歌公司 | Automated archiving of user generated media files |
CN106648967A (en) * | 2016-10-14 | 2017-05-10 | 曙光信息产业(北京)有限公司 | File scanning method and system |
Family Cites Families (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN103593351A (en) * | 2012-08-15 | 2014-02-19 | 中国银联股份有限公司 | Electronic file filing method and system |
CN103294794B (en) * | 2013-05-23 | 2017-07-28 | 上海爱数信息技术股份有限公司 | A kind of online elite archiving and the system for accessing file |
CN104572751A (en) * | 2013-10-24 | 2015-04-29 | 携程计算机技术(上海)有限公司 | Compression storage method and system for calling center sound recording files |
-
2018
- 2018-10-11 CN CN201811185739.9A patent/CN109542858A/en active Pending
- 2018-12-27 WO PCT/CN2018/124395 patent/WO2020073517A1/en active Application Filing
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN103036956A (en) * | 2012-11-30 | 2013-04-10 | 航天恒星科技有限公司 | Filing system and implement method of distributed configured massive data |
CN106537380A (en) * | 2014-06-30 | 2017-03-22 | 谷歌公司 | Automated archiving of user generated media files |
CN104199926A (en) * | 2014-09-01 | 2014-12-10 | 北京联创信安科技有限公司 | File archiving implementation method and device and file access method and device |
CN106648967A (en) * | 2016-10-14 | 2017-05-10 | 曙光信息产业(北京)有限公司 | File scanning method and system |
Non-Patent Citations (2)
Title |
---|
PP: "金融票据影像数据管理方案, 海量文件轻松管", pages 1 - 5, Retrieved from the Internet <URL:https://server.yesky.com/datacenter/123/36450623.shtml> * |
佚名: "金融票据影像数据管理方案,海量非结构化文件轻松管", pages 1 - 5, Retrieved from the Internet <URL:http://www.ctiforum.com/news/guonei/401900.html> * |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN110457255A (en) * | 2019-07-05 | 2019-11-15 | 中国平安人寿保险股份有限公司 | Method, server and the computer readable storage medium of data filing |
CN110457255B (en) * | 2019-07-05 | 2023-11-21 | 中国平安人寿保险股份有限公司 | Method, server and computer readable storage medium for archiving data |
CN111614730A (en) * | 2020-04-28 | 2020-09-01 | 北京金山云网络技术有限公司 | File processing method and device of cloud storage system and electronic equipment |
CN111614730B (en) * | 2020-04-28 | 2022-07-19 | 北京金山云网络技术有限公司 | File processing method and device of cloud storage system and electronic equipment |
Also Published As
Publication number | Publication date |
---|---|
WO2020073517A1 (en) | 2020-04-16 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN101217571B (en) | Write/read document operation method applied in multi-copy data grid system | |
CN105528368B (en) | A kind of database migration method and device | |
CN104618482B (en) | Access method, server, conventional memory device, the system of cloud data | |
JP5656563B2 (en) | Document management system, document management system control method, and program | |
CN103605726B (en) | A kind of access method of small documents, system and control node and memory node | |
CN108959455A (en) | Single page Web application implementation method, device, computer equipment and storage medium | |
CN104820602A (en) | Method, device and system for publishing software package | |
CN109885299B (en) | Template processing method, device, computer equipment and storage medium in model development | |
CN109542858A (en) | Data archiving method, device based on distributed memory system, computer equipment | |
CN105183389A (en) | Data hierarchical management method and device and electronic equipment | |
CN108563771B (en) | Block chain-based large file management system and method | |
CN103902405B (en) | Quasi-continuity data replication method and device | |
CN112559448A (en) | Data state migration method and device, computer equipment and storage medium | |
CN107679420A (en) | A kind of authority setting method and system based on distributed file system | |
CN105824723A (en) | Method and system for backup of data of public cloud storage account | |
CN109189384A (en) | A kind of multiplexing method and device of code | |
CN114048185B (en) | Method for transparently packaging, storing and accessing massive small files in distributed file system | |
CN105955845A (en) | Data recovery method and device | |
CN105956491B (en) | The data of publicly-owned cloud storage system are carried out with the method and system of more account backups | |
CN108243207B (en) | A kind of date storage method of network cloud disk | |
CN113448946B (en) | Data migration method and device and electronic equipment | |
CN110427343A (en) | A kind of method of deployment file, computer equipment and readable storage medium storing program for executing | |
CN110286904A (en) | Generate method, apparatus, computer equipment and the storage medium of front end skin | |
CN115987759B (en) | Data processing method, device, electronic equipment and storage medium | |
CN101848430A (en) | Device and method for service request authentication, service request authentication system and method of service request authentication system |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination |