CN109542858A - Data archiving method, device based on distributed memory system, computer equipment - Google Patents

Data archiving method, device based on distributed memory system, computer equipment Download PDF

Info

Publication number
CN109542858A
CN109542858A CN201811185739.9A CN201811185739A CN109542858A CN 109542858 A CN109542858 A CN 109542858A CN 201811185739 A CN201811185739 A CN 201811185739A CN 109542858 A CN109542858 A CN 109542858A
Authority
CN
China
Prior art keywords
file
memory system
distributed memory
store path
visited
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201811185739.9A
Other languages
Chinese (zh)
Inventor
管清政
王欣
汪洋
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Ping An Technology Shenzhen Co Ltd
Original Assignee
Ping An Technology Shenzhen Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Ping An Technology Shenzhen Co Ltd filed Critical Ping An Technology Shenzhen Co Ltd
Priority to CN201811185739.9A priority Critical patent/CN109542858A/en
Priority to PCT/CN2018/124395 priority patent/WO2020073517A1/en
Publication of CN109542858A publication Critical patent/CN109542858A/en
Pending legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/13File access structures, e.g. distributed indices
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/18File system types
    • G06F16/182Distributed file systems

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

This application involves technical field of distributed memory, in particular to a kind of data archiving method based on distributed memory system, device, computer equipment.The method includes: file of the scanning in distributed memory system;Judge whether the file scanned meets the archive condition of data filing strategy;If the file scanned meets archive condition, the store path for the file that writing scan is arrived obtains the first store path;According to the first store path, the directory listing about store path is generated;Directory listing is read, the corresponding file in distributed memory system of the first store path is transferred in archival storage and is stored.Scan the file in distributed memory system, if the file scanned meets the archive condition of data filing strategy, the file for meeting the archive condition of data filing strategy is transferred in archival storage and is stored, aim to solve the problem that the data filing of existing distributed memory system, need the problem of manually participating in, lead to waste of human resource.

Description

Data archiving method, device based on distributed memory system, computer equipment
Technical field
This application involves technical field of distributed memory, in particular to a kind of data filing based on distributed memory system Method, apparatus, computer equipment.
Background technique
It with the development of internet technology and applies, a large amount of data can be generated in the application process of internet.Mutual It needs to store a large amount of data in networking.In traditional memory technology, when the data that user generates are more, it can face The increasing situation in the space of the storage equipment used, carrying cost become higher and higher as space increases.Existing point The data filing of cloth storage system, generally requires human intervention, copies filing to by hand from a set of distributed memory system and deposits Reservoir, then the artificial data deleted after filing, data filing process need artificial participate in.
Apply for content
In view of the shortcomings of the prior art, the application propose a kind of data archiving method based on distributed memory system, device, Computer equipment, it is intended to which the data filing for solving existing distributed memory system needs manually to participate in, and causes human resources unrestrained The problem of taking.
The technical solution that the application proposes is:
A kind of data archiving method based on distributed memory system, which comprises
Scan the file in distributed memory system;
Judge whether the file scanned meets the archive condition of data filing strategy;
If the file scanned meets the archive condition of the data filing strategy, the text scanned is recorded The store path of part obtains the first store path;
According to first store path, the directory listing about store path is generated;
The directory listing is read, the corresponding file in distributed memory system of first store path is transferred to It is stored in archival storage.
Further, the listed files is read described, first store path is corresponding in distributed storage system File in system was transferred to after the step of storing in archival storage, comprising:
Receive the request of front-end A PI access file;
Judge whether file to be visited is stored in distributed memory system;
If the file to be visited is previously stored in distributed memory system, allow the request.
Further, it is described receive front-end A PI access file request the step of before, comprising:
Record the transfer record that the file of archival storage is transferred to from distributed memory system;
It is described judge whether file to be visited is stored in the step in distributed memory system after, comprising:
If the file to be visited is not previously stored in distributed memory system, judge that the file to be visited is No is the file that archival storage is transferred to from distributed memory system;
If the file to be visited is the file for being transferred to archival storage from distributed memory system, described in acquisition The store path of file to be visited in distributed memory system;
It is recorded according to the transfer, it is corresponding to search store path of the file to be visited in distributed memory system Store path in archival storage obtains the second store path;
File in second store path is pulled back into the file to be visited in distributed memory system Store path;
After completing readjustment, allow the request.
Further, described after completing readjustment, after the step of allowing the request, comprising:
After the front-end A PI accesses the end of file, the file adjusted back in distributed memory system is turned again It moves in archival storage and stores.
Further, it is described receive front-end A PI access file request the step of before, comprising:
Record the transfer record that the file of archival storage is transferred to from distributed memory system;
It is described judge whether file to be visited is stored in the step in distributed memory system after, comprising:
If the file to be visited is not previously stored in distributed memory system, judge that the file to be visited is No is the file that archival storage is transferred to from distributed memory system;
If file to be visited is the file for being transferred to archival storage from distributed memory system, obtain described wait visit Store path of the file asked in distributed memory system;
It is recorded according to the transfer, it is corresponding to search store path of the file to be visited in distributed memory system Store path in archival storage obtains the second store path;
It configures store path of the file to be visited in distributed memory system and jumps to second storage road Diameter;
After the configuration has been completed, allow the request.
Further, the directory listing is read described, first store path is corresponding in distributed storage system File in system was transferred in the step of storing in archival storage, comprising:
Read the directory listing;
According to the directory listing, the corresponding file in distributed memory system of first store path is cut one by one It switches in archival storage;Or
Read the directory listing;
According to the directory listing, the corresponding file in distributed memory system of first store path is answered one by one It makes in archival storage;
After completing duplication, the corresponding file in distributed memory system of first store path is deleted.
Further, the data filing strategy includes time conditions archiving policies or/and catalogue condition archiving policies, institute State time conditions archiving policies be at the appointed time before file meet archive condition, the catalogue condition archiving policies are to refer to The file determined under catalogue meets archive condition.
The application also provides a kind of data filing device based on distributed memory system, and described device includes:
Scan module, for scanning the file in distributed memory system;
Judgment module, for judging whether the file scanned meets the archive condition of data filing strategy;
Logging modle records if meeting the archive condition of the data filing strategy for the file scanned The store path of the file scanned obtains the first store path;
Generation module, for generating the directory listing about store path according to first store path;
Profiling module, it is for reading the directory listing, first store path is corresponding in distributed memory system In file be transferred in archival storage and store.
The application also provides a kind of computer equipment, including memory and processor, and the memory is stored with computer The step of program, the processor realizes method described in any of the above embodiments when executing the computer program.
The application also provides a kind of computer readable storage medium, is stored thereon with computer program, the computer journey The step of method described in any of the above embodiments is realized when sequence is executed by processor.
According to above-mentioned technical solution, the application is the utility model has the advantages that scan the file in distributed memory system, if scanning To file meet the archive condition of data filing strategy, the file for meeting the archive condition of data filing strategy is transferred to and is returned It is stored in shelves memory, it is intended to which the data filing for solving existing distributed memory system needs manually to participate in, manpower is caused to provide The problem of source wastes.
Detailed description of the invention
Fig. 1 is the flow chart using the data archiving method provided by the embodiments of the present application based on distributed memory system;
Fig. 2 is the functional module using the data filing device provided by the embodiments of the present application based on distributed memory system Figure;
Fig. 3 is the structural schematic block diagram using computer equipment provided by the embodiments of the present application.
Specific embodiment
It is with reference to the accompanying drawings and embodiments, right in order to which the objects, technical solutions and advantages of the application are more clearly understood The application is further elaborated.It should be appreciated that specific embodiment described herein is only used to explain the application, and It is not used in restriction the application.
As shown in Figure 1, the embodiment of the present application proposes a kind of data archiving method based on distributed memory system, the side Method the following steps are included:
Step S101, file of the scanning in distributed memory system.
File in distributed memory system is scanned, in the present embodiment, timing scan is in distributed storage File in system is executed according to the time of setting and is scanned to the file in distributed memory system.
In some embodiments, before step S101, comprising:
The filing to the file in distributed memory system is received to instruct.
After receiving the filing instruction to the file in distributed memory system, start scanning in distributed storage system File in system.That is, can just start to execute scanning in the case where receiving filing instruction.
Step S102, judge whether the file scanned meets the archive condition of data filing strategy.
When scanning the file in distributed memory system, needs to judge the file scanned, judge to scan To file whether meet the archive condition of data filing strategy.
In the present embodiment, in step s 102, comprising:
According to the file scanned, the file information corresponding with the archive condition of data filing strategy is obtained;
According to the file information, judge whether the file scanned meets the archive condition of data filing strategy.
Due to the available many file informations of the file scanned, in order to improve judging efficiency, acquisition is returned with data The corresponding the file information of archive condition of shelves strategy, according to the file information, judges the text scanned after obtaining the file information Whether part meets the archive condition of data filing strategy.
In the present embodiment, data filing strategy includes time conditions archiving policies or/and catalogue condition archiving policies, when Between condition archiving policies be at the appointed time before file meet archive condition, catalogue condition archiving policies are under specified directory File meet archive condition.
If step S103, the described file scanned meets the archive condition, the file scanned is recorded Store path obtains the first store path.
According to judging result, it can obtain whether the file scanned meets the archive condition of data filing strategy, if sweeping The file retouched does not meet the archive condition of data filing strategy, then does not need to carry out archive operation, if the file symbol scanned The archive condition for closing data filing strategy, needs to carry out archive operation, then the store path for the file that writing scan is arrived, and obtains the One store path.
Step S104, according to first store path, the directory listing about store path is generated.
After obtaining the first store path, directory listing is generated according to the first store path, directory listing is about depositing Store up path.By reading directory listing, the corresponding storage of file for meeting the archive condition of data filing strategy can be obtained Path.
Step S105, the directory listing is read, first store path is corresponding in distributed memory system File is transferred in archival storage and stores.
After generating directory listing, directory listing is read, the first store path is obtained, then by the first store path pair It should be transferred in archival storage and store in the file in distributed memory system.Archival storage is to meet data for storing The file of the archive condition of archiving policies.
In the present embodiment, in step s105, comprising:
Read the directory listing;
According to the directory listing, the corresponding file in distributed memory system of first store path is cut one by one It switches in archival storage.
Directory listing is read, the first store path is obtained and is deposited one by one by first according to directory listing according to the sequence of list The corresponding file in distributed memory system in storage path is sheared, and is pasted in archival storage later, thus release point The storage space of cloth storage system.
In some embodiments, in step s105, comprising:
Read the directory listing;
According to the directory listing, the corresponding file in distributed memory system of first store path is answered one by one It makes in archival storage;
After completing duplication, the corresponding file in distributed memory system of first store path is deleted.
Directory listing is read, the first store path is obtained and is deposited one by one by first according to directory listing according to the sequence of list The corresponding file in distributed memory system in storage path is replicated, and is pasted in archival storage later, completes to replicate Later, the corresponding file in distributed memory system of the first store path is deleted, thus the storage of release profile formula storage system Deposit space.
In the present embodiment, after step S105, comprising:
Receive the request of front-end A PI access file;
Judge whether file to be visited is stored in distributed memory system;
If the file to be visited is previously stored in distributed memory system, allow the request.
The corresponding file in distributed memory system of first store path is transferred to after being stored in archival storage, The transfer record that the file of archival storage is transferred to from distributed memory system is recorded, transfer record includes the staged files, turns Text move part is before transfer in the store path of distributed memory system, the staged files depositing in archival storage after the transfer Store up path.After completing to record transfer record, the request of front-end A PI access file is received, it be first to file to be visited Position is judged, judges whether file to be visited is stored in distributed memory system, if file to be visited is It is stored in distributed memory system, then allows to request.
The full name of above-mentioned API is Application Programming Interface, and the entitled application program of Chinese is compiled Journey interface.
In the present embodiment, it is described receive front-end A PI access file request the step of before, comprising:
Record the transfer record that the file of archival storage is transferred to from distributed memory system;
Accordingly, after judging whether file to be visited is stored in the step in distributed memory system, comprising:
If the file to be visited is not previously stored in distributed memory system, judge that the file to be visited is No is the file that archival storage is transferred to from distributed memory system;
If the file to be visited is the file for being transferred to archival storage from distributed memory system, described in acquisition The store path of file to be visited in distributed memory system;
It is recorded according to the transfer, it is corresponding to search store path of the file to be visited in distributed memory system Store path in archival storage obtains the second store path;
File in second store path is pulled back into the file to be visited in distributed memory system Store path;
After completing readjustment, allow the request.
If file to be visited is not previously stored in distributed memory system, also need to be judged again, judge to Whether the file of access is the file that archival storage is transferred to from distributed memory system, if file to be visited be not for from Distributed memory system is transferred to the file of archival storage, then refuses to request.If file to be visited is from distributed storage System is transferred to the file of archival storage, obtains the store path of file to be visited in distributed memory system, due to Transfer record exists in the store path of distributed memory system, the staged files before transfer including the staged files, the staged files In the store path of archival storage after transfer, is recorded according to transfer, search file to be visited in distributed memory system In the corresponding store path in archival storage of store path, obtain the second store path, obtaining the second store path Later, the file in the second store path is pulled back into the position before its transfer, that is, pulls back to file to be visited and exists Store path in distributed memory system allows to request, front-end A PI is from distributed memory system after completing readjustment Access file.
Described after completing readjustment, after the step of allowing the request, comprising:
After the front-end A PI accesses the end of file, the file adjusted back in distributed memory system is turned again It moves in archival storage and stores.
After front-end A PI accesses the end of file, for the storage space of release profile formula storage system, it will adjust back File in distributed memory system, which is again transferred in archival storage, to be stored.
In some embodiments, it is described receive front-end A PI access file request the step of before, comprising:
Record the transfer record that the file of archival storage is transferred to from distributed memory system;
Accordingly, it is described judge whether file to be visited is stored in the step in distributed memory system after, packet It includes:
If the file to be visited is not previously stored in distributed memory system, judge that the file to be visited is No is the file that archival storage is transferred to from distributed memory system;
If file to be visited is the file for being transferred to archival storage from distributed memory system, obtain described wait visit Store path of the file asked in distributed memory system;
It is recorded according to the transfer, it is corresponding to search store path of the file to be visited in distributed memory system Store path in archival storage obtains the second store path;
It configures store path of the file to be visited in distributed memory system and jumps to second storage road Diameter;
After the configuration has been completed, allow the request.
If file to be visited is not previously stored in distributed memory system, also need to be judged again, judge to Whether the file of access is the file that archival storage is transferred to from distributed memory system, if file to be visited be not for from Distributed memory system is transferred to the file of archival storage, then refuses to request.If file to be visited is from distributed storage System is transferred to the file of archival storage, obtains the store path of file to be visited in distributed memory system, due to Transfer record exists in the store path of distributed memory system, the staged files before transfer including the staged files, the staged files In the store path of archival storage after transfer, is recorded according to transfer, search file to be visited in distributed memory system In the corresponding store path in archival storage of store path, obtain the second store path, obtaining the second store path Later, it configures the store path of file to be visited in distributed memory system and jumps to the second store path, it is, will The store path of file to be visited in distributed memory system jumps in the store path of archival storage, completes to match After setting, allow to request, front-end A PI accesses file from archival storage.
In conclusion file of the scanning in distributed memory system, if the file scanned meets data filing strategy Archive condition, the file for meeting the archive condition of data filing strategy is transferred in archival storage and is stored, it is intended to be solved The data filing of existing distributed memory system, needs the problem of manually participating in, lead to waste of human resource.
As shown in Fig. 2, the embodiment of the present application proposes a kind of data filing device 1 based on distributed memory system, device 1 Including scan module 11, judgment module 12, logging modle 13, generation module 14 and profiling module 15.
Scan module 11, for scanning the file in distributed memory system.
File in distributed memory system is scanned, in the present embodiment, timing scan is in distributed storage File in system is executed according to the time of setting and is scanned to the file in distributed memory system.
In some embodiments, device 1 includes:
Receiving module, for receiving the instruction of the filing to the file in distributed memory system.
After receiving the filing instruction to the file in distributed memory system, start scanning in distributed storage system File in system.That is, can just start to execute scanning in the case where receiving filing instruction.
Judgment module 12, for judging whether the file scanned meets the archive condition of data filing strategy.
When scanning the file in distributed memory system, needs to judge the file scanned, judge to scan To file whether meet the archive condition of data filing strategy.
In the present embodiment, judgment module 12 includes:
First obtains module, for obtaining corresponding with the archive condition of data filing strategy according to the file scanned The file information;
First judgment module, for judging whether the file scanned meets data filing strategy according to the file information Archive condition.
Due to the available many file informations of the file scanned, in order to improve judging efficiency, acquisition is returned with data The corresponding the file information of archive condition of shelves strategy, according to the file information, judges the text scanned after obtaining the file information Whether part meets the archive condition of data filing strategy.
In the present embodiment, data filing strategy includes time conditions archiving policies or/and catalogue condition archiving policies, when Between condition archiving policies be at the appointed time before file meet archive condition, catalogue condition archiving policies are under specified directory File meet archive condition.
Logging modle 13 records described scan if meet the archive condition for the file scanned The store path of file obtains the first store path.
According to judging result, it can obtain whether the file scanned meets the archive condition of data filing strategy, if sweeping The file retouched does not meet the archive condition of data filing strategy, then does not need to carry out archive operation, if the file symbol scanned The archive condition for closing data filing strategy, needs to carry out archive operation, then the store path for the file that writing scan is arrived, and obtains the One store path.
Generation module 14, for generating the directory listing about store path according to first store path.
After obtaining the first store path, directory listing is generated according to the first store path, directory listing is about depositing Store up path.By reading directory listing, the corresponding storage of file for meeting the archive condition of data filing strategy can be obtained Path.
Profiling module 15, it is for reading the directory listing, first store path is corresponding in distributed storage system File in system, which is transferred in archival storage, to be stored.
After generating directory listing, directory listing is read, the first store path is obtained, then by the first store path pair It should be transferred in archival storage and store in the file in distributed memory system.Archival storage is to meet data for storing The file of the archive condition of archiving policies.
In the present embodiment, profiling module 15 includes:
First read module, for reading the directory listing;
First profiling module, for one by one corresponding to first store path in distribution according to the directory listing Paper cut in storage system is into archival storage.
Directory listing is read, the first store path is obtained and is deposited one by one by first according to directory listing according to the sequence of list The corresponding file in distributed memory system in storage path is sheared, and is pasted in archival storage later, thus release point The storage space of cloth storage system.
In some embodiments, profiling module 15 includes:
Second read module, for reading the directory listing;
Second profiling module, for one by one corresponding to first store path in distribution according to the directory listing File in storage system copies in archival storage;
First removing module, it is corresponding in distributed storage for after completing duplication, deleting first store path File in system.
Directory listing is read, the first store path is obtained and is deposited one by one by first according to directory listing according to the sequence of list The corresponding file in distributed memory system in storage path is replicated, and is pasted in archival storage later, completes to replicate Later, the corresponding file in distributed memory system of the first store path is deleted, thus the storage of release profile formula storage system Deposit space.
In the present embodiment, device 1 includes:
First receiving module, for receiving the request of front-end A PI access file;
Second judgment module, for judging whether file to be visited is stored in distributed memory system;
First permission module allows institute if being previously stored in distributed memory system for the file to be visited State request.
The corresponding file in distributed memory system of first store path is transferred to after being stored in archival storage, The transfer record that the file of archival storage is transferred to from distributed memory system is recorded, transfer record includes the staged files, turns Text move part is before transfer in the store path of distributed memory system, the staged files depositing in archival storage after the transfer Store up path.After completing to record transfer record, the request of front-end A PI access file is received, it be first to file to be visited Position is judged, judges whether file to be visited is stored in distributed memory system, if file to be visited is It is stored in distributed memory system, then allows to request.
The full name of above-mentioned API is Application Programming Interface, and the entitled application program of Chinese is compiled Journey interface.
In the present embodiment, device 1 includes:
First logging modle, for recording the transfer note for being transferred to the file of archival storage from distributed memory system Record;
Third judgment module judges if not being previously stored in distributed memory system for the file to be visited Whether the file to be visited is the file that archival storage is transferred to from distributed memory system;
Second obtains module, if being to be transferred to archival storage from distributed memory system for the file to be visited File, then obtain store path of the file to be visited in distributed memory system;
First searching module searches the file to be visited in distributed storage system for recording according to the transfer The corresponding store path in archival storage of store path in system, obtains the second store path;
First callback module is being divided for the file in second store path to be pulled back to the file to be visited Store path in cloth storage system;
Second allows module, for allowing the request after completing readjustment.
If file to be visited is not previously stored in distributed memory system, also need to be judged again, judge to Whether the file of access is the file that archival storage is transferred to from distributed memory system, if file to be visited be not for from Distributed memory system is transferred to the file of archival storage, then refuses to request.If file to be visited is from distributed storage System is transferred to the file of archival storage, obtains the store path of file to be visited in distributed memory system, due to Transfer record exists in the store path of distributed memory system, the staged files before transfer including the staged files, the staged files In the store path of archival storage after transfer, is recorded according to transfer, search file to be visited in distributed memory system In the corresponding store path in archival storage of store path, obtain the second store path, obtaining the second store path Later, the file in the second store path is pulled back into the position before its transfer, that is, pulls back to file to be visited and exists Store path in distributed memory system allows to request, front-end A PI is from distributed memory system after completing readjustment Access file.
Device 1 includes:
Third profiling module, for will adjust back in distributed storage system after the front-end A PI accesses the end of file File in system, which is again transferred in archival storage, to be stored.
After front-end A PI accesses the end of file, for the storage space of release profile formula storage system, it will adjust back File in distributed memory system, which is again transferred in archival storage, to be stored.
In some embodiments, device 1 includes:
Second logging modle, for recording the transfer note for being transferred to the file of archival storage from distributed memory system Record;
4th judgment module judges if not being previously stored in distributed memory system for the file to be visited Whether the file to be visited is the file that archival storage is transferred to from distributed memory system;
Third obtains module, if being the text for being transferred to archival storage from distributed memory system for file to be visited Part then obtains store path of the file to be visited in distributed memory system;
Second searching module searches the file to be visited in distributed storage system for recording according to the transfer The corresponding store path in archival storage of store path in system, obtains the second store path;
First configuration module is jumped for configuring store path of the file to be visited in distributed memory system To second store path;
Third allows module, for after the configuration has been completed, allowing the request.
If file to be visited is not previously stored in distributed memory system, also need to be judged again, judge to Whether the file of access is the file that archival storage is transferred to from distributed memory system, if file to be visited be not for from Distributed memory system is transferred to the file of archival storage, then refuses to request.If file to be visited is from distributed storage System is transferred to the file of archival storage, obtains the store path of file to be visited in distributed memory system, due to Transfer record exists in the store path of distributed memory system, the staged files before transfer including the staged files, the staged files In the store path of archival storage after transfer, is recorded according to transfer, search file to be visited in distributed memory system In the corresponding store path in archival storage of store path, obtain the second store path, obtaining the second store path Later, it configures the store path of file to be visited in distributed memory system and jumps to the second store path, it is, will The store path of file to be visited in distributed memory system jumps in the store path of archival storage, completes to match After setting, allow to request, front-end A PI accesses file from archival storage.
In conclusion file of the scanning in distributed memory system, if the file scanned meets data filing strategy Archive condition, the file for meeting the archive condition of data filing strategy is transferred in archival storage and is stored, it is intended to be solved The data filing of existing distributed memory system, needs the problem of manually participating in, lead to waste of human resource.
As shown in figure 3, also providing a kind of computer equipment in the embodiment of the present application, which can be service Device, internal structure can be as shown in Figure 3.The computer equipment includes processor, the memory, net connected by system bus Network interface and database.Wherein, the processor of the Computer Design is for providing calculating and control ability.The computer equipment Memory includes non-volatile memory medium, built-in storage.The non-volatile memory medium is stored with operating system, computer journey Sequence and database.The internal memory provides environment for the operation of operating system and computer program in non-volatile memory medium. The database of the computer equipment is for storing the data such as the model of the data archiving method based on distributed memory system.The meter The network interface for calculating machine equipment is used to communicate with external terminal by network connection.When the computer program is executed by processor To realize a kind of data archiving method based on distributed memory system.
Above-mentioned processor executes the step of above-mentioned data archiving method based on distributed memory system: scanning is in distribution File in storage system;Judge whether the file scanned meets the archive condition of data filing strategy;If the scanning is arrived File meet the archive condition, then record the store path of the file scanned, obtain the first store path;According to First store path generates the directory listing about store path;The directory listing is read, by first storage road The corresponding file in distributed memory system of diameter, which is transferred in archival storage, to be stored.
In one embodiment, first store path is corresponded to and is deposited in distribution by the above-mentioned reading listed files File in storage system was transferred to after the step of storing in archival storage, comprising:
Receive the request of front-end A PI access file;
Judge whether file to be visited is stored in distributed memory system;
If the file to be visited is previously stored in distributed memory system, allow the request.
In one embodiment, before the step of above-mentioned request for receiving front-end A PI access file, comprising:
Record the transfer record that the file of archival storage is transferred to from distributed memory system;
It is described judge whether file to be visited is stored in the step in distributed memory system after, comprising:
If the file to be visited is not previously stored in distributed memory system, judge that the file to be visited is No is the file that archival storage is transferred to from distributed memory system;
If the file to be visited is the file for being transferred to archival storage from distributed memory system, described in acquisition The store path of file to be visited in distributed memory system;
It is recorded according to the transfer, it is corresponding to search store path of the file to be visited in distributed memory system Store path in archival storage obtains the second store path;
File in second store path is pulled back into the file to be visited in distributed memory system Store path;
After completing readjustment, allow the request.
In one embodiment, above-mentioned after completing readjustment, after the step of allowing the request, comprising:
After the front-end A PI accesses the end of file, the file adjusted back in distributed memory system is turned again It moves in archival storage and stores.
In one embodiment, before the step of above-mentioned request for receiving front-end A PI access file, comprising:
Record the transfer record that the file of archival storage is transferred to from distributed memory system;
It is described judge whether file to be visited is stored in the step in distributed memory system after, comprising:
If the file to be visited is not previously stored in distributed memory system, judge that the file to be visited is No is the file that archival storage is transferred to from distributed memory system;
If file to be visited is the file for being transferred to archival storage from distributed memory system, obtain described wait visit Store path of the file asked in distributed memory system;
It is recorded according to the transfer, it is corresponding to search store path of the file to be visited in distributed memory system Store path in archival storage obtains the second store path;
It configures store path of the file to be visited in distributed memory system and jumps to second storage road Diameter;
After the configuration has been completed, allow the request.
In one embodiment, first store path is corresponded to and is deposited in distribution by the above-mentioned reading directory listing File in storage system was transferred in the step of storing in archival storage, comprising:
Read the directory listing;
According to the directory listing, the corresponding file in distributed memory system of first store path is cut one by one It switches in archival storage;Or
Read the directory listing;
According to the directory listing, the corresponding file in distributed memory system of first store path is answered one by one It makes in archival storage;
After completing duplication, the corresponding file in distributed memory system of first store path is deleted.
In one embodiment, above-mentioned data filing strategy includes time conditions archiving policies or/and the filing of catalogue condition Strategy, the time conditions archiving policies be at the appointed time before file meet archive condition, the catalogue condition filing Strategy is that the file under specified directory meets archive condition.
The computer equipment of the embodiment of the present application scans the file in distributed memory system, if the file scanned The file for meeting the archive condition of data filing strategy is transferred to archival storage by the archive condition for meeting data filing strategy Middle storage, it is intended to which the data filing for solving existing distributed memory system needs manually to participate in, leads to waste of human resource Problem.
One embodiment of the application also provides a kind of computer readable storage medium, is stored thereon with computer program, calculates Machine program realizes a kind of data archiving method based on distributed memory system when being executed by processor, specifically: scanning is dividing File in cloth storage system;Judge whether the file scanned meets the archive condition of data filing strategy;If described sweep The file retouched meets the archive condition, then records the store path of the file scanned, obtains the first store path; According to first store path, the directory listing about store path is generated;The directory listing is read, described first is deposited The corresponding file in distributed memory system in storage path, which is transferred in archival storage, to be stored.
In one embodiment, first store path is corresponded to and is deposited in distribution by the above-mentioned reading listed files File in storage system was transferred to after the step of storing in archival storage, comprising:
Receive the request of front-end A PI access file;
Judge whether file to be visited is stored in distributed memory system;
If the file to be visited is previously stored in distributed memory system, allow the request.
In one embodiment, before the step of above-mentioned request for receiving front-end A PI access file, comprising:
Record the transfer record that the file of archival storage is transferred to from distributed memory system;
It is described judge whether file to be visited is stored in the step in distributed memory system after, comprising:
If the file to be visited is not previously stored in distributed memory system, judge that the file to be visited is No is the file that archival storage is transferred to from distributed memory system;
If the file to be visited is the file for being transferred to archival storage from distributed memory system, described in acquisition The store path of file to be visited in distributed memory system;
It is recorded according to the transfer, it is corresponding to search store path of the file to be visited in distributed memory system Store path in archival storage obtains the second store path;
File in second store path is pulled back into the file to be visited in distributed memory system Store path;
After completing readjustment, allow the request.
In one embodiment, above-mentioned after completing readjustment, after the step of allowing the request, comprising:
After the front-end A PI accesses the end of file, the file adjusted back in distributed memory system is turned again It moves in archival storage and stores.
In one embodiment, before the step of above-mentioned request for receiving front-end A PI access file, comprising:
Record the transfer record that the file of archival storage is transferred to from distributed memory system;
It is described judge whether file to be visited is stored in the step in distributed memory system after, comprising:
If the file to be visited is not previously stored in distributed memory system, judge that the file to be visited is No is the file that archival storage is transferred to from distributed memory system;
If file to be visited is the file for being transferred to archival storage from distributed memory system, obtain described wait visit Store path of the file asked in distributed memory system;
It is recorded according to the transfer, it is corresponding to search store path of the file to be visited in distributed memory system Store path in archival storage obtains the second store path;
It configures store path of the file to be visited in distributed memory system and jumps to second storage road Diameter;
After the configuration has been completed, allow the request.
In one embodiment, first store path is corresponded to and is deposited in distribution by the above-mentioned reading directory listing File in storage system was transferred in the step of storing in archival storage, comprising:
Read the directory listing;
According to the directory listing, the corresponding file in distributed memory system of first store path is cut one by one It switches in archival storage;Or
Read the directory listing;
According to the directory listing, the corresponding file in distributed memory system of first store path is answered one by one It makes in archival storage;
After completing duplication, the corresponding file in distributed memory system of first store path is deleted.
In one embodiment, above-mentioned data filing strategy includes time conditions archiving policies or/and the filing of catalogue condition Strategy, the time conditions archiving policies be at the appointed time before file meet archive condition, the catalogue condition filing Strategy is that the file under specified directory meets archive condition.
The storage medium of the embodiment of the present application scans the file in distributed memory system, if the file symbol scanned The archive condition for closing data filing strategy, the file for meeting the archive condition of data filing strategy is transferred in archival storage Storage, it is intended to which the data filing for solving existing distributed memory system needs manually to participate in, leads to asking for waste of human resource Topic.
Those of ordinary skill in the art will appreciate that realizing all or part of the process in above-described embodiment method, being can be with Relevant hardware is instructed to complete by computer program, the computer program can be stored in a non-volatile computer In read/write memory medium, the computer program is when being executed, it may include such as the process of the embodiment of above-mentioned each method.Wherein, Any reference used in provided herein and embodiment to memory, storage, database or other media, Including non-volatile and/or volatile memory.Nonvolatile memory may include read-only memory (ROM), programming ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM) or flash memory.Volatile memory may include Random access memory (RAM) or external cache.By way of illustration and not limitation, RAM is available in many forms, Such as static state RAM (SRAM), dynamic ram (DRAM), synchronous dram (SDRAM), double speed are according to rate SDRAM (SSRSDRAM), enhancing Type SDRAM (ESDRAM), synchronization link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic ram (DRDRAM) and memory bus dynamic ram (RDRAM) etc..
The foregoing is merely the preferred embodiments of the application, not to limit the application, all essences in the application Made any modifications, equivalent replacements, and improvements etc. within mind and principle should all include within the scope of protection of this application.

Claims (10)

1. a kind of data archiving method based on distributed memory system, which is characterized in that the described method includes:
Scan the file in distributed memory system;
Judge whether the file scanned meets the archive condition of data filing strategy;
If the file scanned meets the archive condition, the store path of the file scanned is recorded, is obtained First store path;
According to first store path, the directory listing about store path is generated;
The directory listing is read, the corresponding file in distributed memory system of first store path is transferred to filing It is stored in memory.
2. the data archiving method according to claim 1 based on distributed memory system, which is characterized in that in the reading The listed files is taken, the corresponding file in distributed memory system of first store path is transferred to archival storage After the step of middle storage, comprising:
Receive the request of front-end A PI access file;
Judge whether file to be visited is stored in distributed memory system;
If the file to be visited is previously stored in distributed memory system, allow the request.
3. the data archiving method according to claim 2 based on distributed memory system, which is characterized in that connect described Before the step of receiving the request of front-end A PI access file, comprising:
Record the transfer record that the file of archival storage is transferred to from distributed memory system;
It is described judge whether file to be visited is stored in the step in distributed memory system after, comprising:
If the file to be visited is not previously stored in distributed memory system, judge the file to be visited whether be The file of archival storage is transferred to from distributed memory system;
If the file to be visited is the file for being transferred to archival storage from distributed memory system, obtain described wait visit Store path of the file asked in distributed memory system;
It is recorded according to the transfer, searches store path correspondence of the file to be visited in distributed memory system and returning Store path in shelves memory, obtains the second store path;
File in second store path is pulled back into storage of the file to be visited in distributed memory system Path;
After completing readjustment, allow the request.
4. the data archiving method according to claim 3 based on distributed memory system, which is characterized in that it is described After the step of completing after adjusting back, allowing the request, comprising:
After the front-end A PI accesses the end of file, the file adjusted back in distributed memory system is again transferred to It is stored in archival storage.
5. the data archiving method according to claim 2 based on distributed memory system, which is characterized in that connect described Before the step of receiving the request of front-end A PI access file, comprising:
Record the transfer record that the file of archival storage is transferred to from distributed memory system;
It is described judge whether file to be visited is stored in the step in distributed memory system after, comprising:
If the file to be visited is not previously stored in distributed memory system, judge the file to be visited whether be The file of archival storage is transferred to from distributed memory system;
If file to be visited is the file for being transferred to archival storage from distributed memory system, obtain described to be visited Store path of the file in distributed memory system;
It is recorded according to the transfer, searches store path correspondence of the file to be visited in distributed memory system and returning Store path in shelves memory, obtains the second store path;
It configures store path of the file to be visited in distributed memory system and jumps to second store path;
After the configuration has been completed, allow the request.
6. the data archiving method according to claim 1 based on distributed memory system, which is characterized in that in the reading The directory listing is taken, the corresponding file in distributed memory system of first store path is transferred to archival storage In the step of middle storage, comprising:
Read the directory listing;
According to the directory listing, the corresponding paper cut in distributed memory system of first store path is arrived one by one In archival storage;Or
Read the directory listing;
According to the directory listing, the corresponding file in distributed memory system of first store path is copied to one by one In archival storage;
After completing duplication, the corresponding file in distributed memory system of first store path is deleted.
7. the data archiving method according to claim 1 based on distributed memory system, which is characterized in that the data Archiving policies include time conditions archiving policies or/and catalogue condition archiving policies, and the time conditions archiving policies are to refer to File before fixing time meets archive condition, and the catalogue condition archiving policies are that the file under specified directory meets filing item Part.
8. a kind of data filing device based on distributed memory system, which is characterized in that described device includes:
Scan module, for scanning the file in distributed memory system;
Judgment module, for judging whether the file scanned meets the archive condition of data filing strategy;
Logging modle, if meeting the archive condition of the data filing strategy for the file scanned, described in record The store path of the file scanned obtains the first store path;
Generation module, for generating the directory listing about store path according to first store path;
Profiling module, it is for reading the directory listing, first store path is corresponding in distributed memory system File is transferred in archival storage and stores.
9. a kind of computer equipment, including memory and processor, the memory are stored with computer program, feature exists In the processor realizes method described in any one of claims 1 to 7 when executing computer program the step of.
10. a kind of computer readable storage medium, is stored thereon with computer program, which is characterized in that the computer program The step of method described in any one of claims 1 to 7 is realized when being executed by processor.
CN201811185739.9A 2018-10-11 2018-10-11 Data archiving method, device based on distributed memory system, computer equipment Pending CN109542858A (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN201811185739.9A CN109542858A (en) 2018-10-11 2018-10-11 Data archiving method, device based on distributed memory system, computer equipment
PCT/CN2018/124395 WO2020073517A1 (en) 2018-10-11 2018-12-27 Data archiving method and apparatus based on distributed storage system, and computer device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201811185739.9A CN109542858A (en) 2018-10-11 2018-10-11 Data archiving method, device based on distributed memory system, computer equipment

Publications (1)

Publication Number Publication Date
CN109542858A true CN109542858A (en) 2019-03-29

Family

ID=65843648

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201811185739.9A Pending CN109542858A (en) 2018-10-11 2018-10-11 Data archiving method, device based on distributed memory system, computer equipment

Country Status (2)

Country Link
CN (1) CN109542858A (en)
WO (1) WO2020073517A1 (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110457255A (en) * 2019-07-05 2019-11-15 中国平安人寿保险股份有限公司 Method, server and the computer readable storage medium of data filing
CN111614730A (en) * 2020-04-28 2020-09-01 北京金山云网络技术有限公司 File processing method and device of cloud storage system and electronic equipment

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103036956A (en) * 2012-11-30 2013-04-10 航天恒星科技有限公司 Filing system and implement method of distributed configured massive data
CN104199926A (en) * 2014-09-01 2014-12-10 北京联创信安科技有限公司 File archiving implementation method and device and file access method and device
CN106537380A (en) * 2014-06-30 2017-03-22 谷歌公司 Automated archiving of user generated media files
CN106648967A (en) * 2016-10-14 2017-05-10 曙光信息产业(北京)有限公司 File scanning method and system

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103593351A (en) * 2012-08-15 2014-02-19 中国银联股份有限公司 Electronic file filing method and system
CN103294794B (en) * 2013-05-23 2017-07-28 上海爱数信息技术股份有限公司 A kind of online elite archiving and the system for accessing file
CN104572751A (en) * 2013-10-24 2015-04-29 携程计算机技术(上海)有限公司 Compression storage method and system for calling center sound recording files

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103036956A (en) * 2012-11-30 2013-04-10 航天恒星科技有限公司 Filing system and implement method of distributed configured massive data
CN106537380A (en) * 2014-06-30 2017-03-22 谷歌公司 Automated archiving of user generated media files
CN104199926A (en) * 2014-09-01 2014-12-10 北京联创信安科技有限公司 File archiving implementation method and device and file access method and device
CN106648967A (en) * 2016-10-14 2017-05-10 曙光信息产业(北京)有限公司 File scanning method and system

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
PP: "金融票据影像数据管理方案, 海量文件轻松管", pages 1 - 5, Retrieved from the Internet <URL:https://server.yesky.com/datacenter/123/36450623.shtml> *
佚名: "金融票据影像数据管理方案,海量非结构化文件轻松管", pages 1 - 5, Retrieved from the Internet <URL:http://www.ctiforum.com/news/guonei/401900.html> *

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110457255A (en) * 2019-07-05 2019-11-15 中国平安人寿保险股份有限公司 Method, server and the computer readable storage medium of data filing
CN110457255B (en) * 2019-07-05 2023-11-21 中国平安人寿保险股份有限公司 Method, server and computer readable storage medium for archiving data
CN111614730A (en) * 2020-04-28 2020-09-01 北京金山云网络技术有限公司 File processing method and device of cloud storage system and electronic equipment
CN111614730B (en) * 2020-04-28 2022-07-19 北京金山云网络技术有限公司 File processing method and device of cloud storage system and electronic equipment

Also Published As

Publication number Publication date
WO2020073517A1 (en) 2020-04-16

Similar Documents

Publication Publication Date Title
CN101217571B (en) Write/read document operation method applied in multi-copy data grid system
CN105528368B (en) A kind of database migration method and device
CN104618482B (en) Access method, server, conventional memory device, the system of cloud data
JP5656563B2 (en) Document management system, document management system control method, and program
CN103605726B (en) A kind of access method of small documents, system and control node and memory node
CN108959455A (en) Single page Web application implementation method, device, computer equipment and storage medium
CN104820602A (en) Method, device and system for publishing software package
CN109885299B (en) Template processing method, device, computer equipment and storage medium in model development
CN109542858A (en) Data archiving method, device based on distributed memory system, computer equipment
CN105183389A (en) Data hierarchical management method and device and electronic equipment
CN108563771B (en) Block chain-based large file management system and method
CN103902405B (en) Quasi-continuity data replication method and device
CN112559448A (en) Data state migration method and device, computer equipment and storage medium
CN107679420A (en) A kind of authority setting method and system based on distributed file system
CN105824723A (en) Method and system for backup of data of public cloud storage account
CN109189384A (en) A kind of multiplexing method and device of code
CN114048185B (en) Method for transparently packaging, storing and accessing massive small files in distributed file system
CN105955845A (en) Data recovery method and device
CN105956491B (en) The data of publicly-owned cloud storage system are carried out with the method and system of more account backups
CN108243207B (en) A kind of date storage method of network cloud disk
CN113448946B (en) Data migration method and device and electronic equipment
CN110427343A (en) A kind of method of deployment file, computer equipment and readable storage medium storing program for executing
CN110286904A (en) Generate method, apparatus, computer equipment and the storage medium of front end skin
CN115987759B (en) Data processing method, device, electronic equipment and storage medium
CN101848430A (en) Device and method for service request authentication, service request authentication system and method of service request authentication system

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination