CN110737635B - Data blocking method - Google Patents
Data blocking method Download PDFInfo
- Publication number
- CN110737635B CN110737635B CN201810711503.8A CN201810711503A CN110737635B CN 110737635 B CN110737635 B CN 110737635B CN 201810711503 A CN201810711503 A CN 201810711503A CN 110737635 B CN110737635 B CN 110737635B
- Authority
- CN
- China
- Prior art keywords
- file
- modification
- client
- data
- storing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention discloses a data blocking method, which comprises the following steps of S1-S2: s1, converting a single data file into a data folder; s2, receiving a modification request of a client to request for modifying a target file, wherein a subfolder for storing the target file and a modification lock file for storing feature code information are inserted into the data folder during modification, the inserted subfolder and the modification lock file are deleted after modification is finished, and the feature code information is stored in the modification record file. The invention has the following beneficial effects: the file access mechanism is optimized by modifying the lock file, the response speed is accelerated, and the data transmission time is saved.
Description
Technical Field
The invention relates to the technical field of data blocking, in particular to a data blocking method.
Background
If the file system receives a large user access amount, the file system bears the pressure caused by a large amount of accesses and also needs to perform frequent file reading IO operation, if the accessed file is a large data file, the occupied IO is too long, the access speed is slow, the time consumption is long, and therefore the access bottleneck is easy to appear.
Referring to fig. 1, fig. 1 is a schematic diagram of a data flow of file transmission in the prior art, where a current file process is stored in a server as a whole, and a client is also stored as a whole, for example, file services such as FTP and NFS, a stream operation must be exclusively used for a write operation, and huge resources are occupied for performing read-write processing, which is described in different scenarios:
A. when uploading files, the files are firstly divided into small blocks, and then are merged into the same file stream after being transmitted to a server;
B. when downloading the file, firstly dividing the file into small blocks, transmitting the small blocks to a client, and then merging the small blocks into the same file stream;
C. when the file is modified, the whole file record is deleted, and then a new complete file is uploaded.
This method has the following problems:
disadvantage 1: file service can access a plurality of files, the size of a big data file is not always determined, so that the operation of occupying one file stream for a long time can be caused, and the downloading or uploading speed can be reduced along with the increase of time;
and (2) disadvantage: when the file service accesses or reads the big data files, the whole feature code is verified, and a plurality of big data files are processed at the same time, so that the operation is increased and the CPU (Central processing Unit) is occupied;
disadvantage 3: when the file service processes and downloads the big data file, the file flow is read only forward in a single direction in sequence, the reading efficiency is not high, and the time consumption is long;
disadvantage 4: when processing and storing large data files, the file service needs to occupy the file stream for writing, and writes forward in a single direction in sequence, so that the writing efficiency is not high, and the time consumption is long;
disadvantage 5: when multiple users modify the same file at the same time, the users contend and interfere with each other, download is influenced at the same time, and the file is possibly invalid;
disadvantage 6: the modification history is not easily recorded.
Disclosure of Invention
The invention aims to provide a data partitioning method, which solves the defects in the prior art.
The technical scheme of the invention is realized as follows:
in one aspect, the present invention provides a data partitioning method, including steps S1-S2:
s1, converting a single data file into a data folder, wherein the data folder comprises a summary information file, one or more block files and a modification record file for storing modification records;
s2, receiving a modification request of a client to request modification of a target file, wherein a subfolder for storing the target file and a modification lock file for storing feature code information are inserted into the data folder during modification, the inserted subfolder and the modification lock file are deleted after modification is finished, and the feature code information is stored into the modification record file; the target file is one or more of the one or more block files.
In the data blocking method of the present invention, the step S2 includes steps S21 to S27:
s21, receiving a modification request of a client;
s22, checking whether a modification lock file exists in the data folder or not according to the modification request, if yes, turning to a step S23, and if not, turning to a step S24;
s23, matching the feature code information, if the matching is successful, turning to the step S26, and if the matching is unsuccessful, terminating the operation;
s24, checking whether a modification record exists in the modification record file, if so, turning to the step S23, and if not, turning to the step S25;
s25, inserting a subfolder for storing the target file and a modification lock file for storing feature code information into the data folder;
s26, allowing the client to modify the target file;
and S27, deleting the subfolders and the modification lock files after the modification is finished, and storing the feature code information into the modification record file.
In the data blocking method according to the present invention, in step S23: and the feature code information comprises a modified user name, address information and modified information of the target file, the user name and the address information are matched with information preset by a server, if the matching is successful, the step S26 is switched to, so that the target file is modified according to the modified information, and if the matching is unsuccessful, the operation is terminated.
In the data blocking method of the present invention, the step S25 includes steps S251 to S254:
s251, modifying the modification record file to insert the current modification record;
s252, writing the current modification record into a local file;
s253, generating and inserting a subfolder for storing the target file and a modification lock file for storing feature code information into the data folder;
and S254, granting modification authority to the client, thereby allowing the client to modify the target file.
In the data blocking method of the present invention, the step S1 includes steps S11 to S14:
s11, acquiring the size of the single data file;
s12, configuring the size of one to more block files, and calculating the number of the block files according to the size of the one to more block files and the size of the single data file;
s13, sequentially numbering one or more block files;
s14, converting a single data file into a data folder, wherein the data folder comprises a summary information file, one or more block files and a modification record file for storing modification records; the summary information file comprises the size of the single data file, the size of one to more block files, operation user information, and sequence numbers and file names of the one to more block files.
In the data blocking method of the present invention, the step S2 further includes steps S28 to S29:
s28, receiving an uploading request of the client and allowing the client to upload;
and S29, receiving a downloading request of the client and allowing the client to perform downloading operation.
In the data blocking method of the present invention, the step S28 includes steps S281 to S283:
s281, receiving an uploading request of the client;
s282, establishing a client block file set cache in the client;
and S283, allowing the client to upload, and storing the file in the client block file set cache during uploading.
In the data blocking method of the present invention, the S29 includes steps S291-S293:
s291, receiving a downloading request of the client;
s292, establishing a service block file set cache in the server;
and S293, allowing the client to perform downloading operation through the service blocked file set cache.
In the data blocking method of the present invention, the method further includes step S3:
and S3, judging whether the target file modification of the client is finished or not, and if not, repairing according to a preset strategy.
In the data blocking method of the present invention, in step S3, the preset policy includes:
deleting the modification request; or alternatively
And regularly clearing the overtime target file.
Therefore, the method has the advantages that the file access mechanism is optimized by modifying the lock file, the response speed is increased, and the data transmission time is saved.
Drawings
The invention will be further described with reference to the following drawings and examples, in which:
FIG. 1 is a schematic diagram of data flow of file transmission in the prior art;
FIG. 2 is a data flow diagram of a file transfer according to an embodiment of the present invention;
fig. 3 is a flowchart of a data chunking method according to an embodiment of the present invention;
FIG. 4 is a diagram illustrating a structure of a data folder according to an embodiment of the present invention;
FIG. 5 is a diagram illustrating a structure of a data folder into which a modified lock file and subfolders are inserted according to an embodiment of the present invention;
FIG. 6 is a flowchart of a process for requesting modification of a target file according to an embodiment of the present invention;
fig. 7 is a schematic diagram of uploading a file by a client according to an embodiment of the present invention;
fig. 8 is a schematic diagram of downloading a file by a client according to an embodiment of the present invention.
Detailed Description
In order to more clearly understand the technical features, objects, and effects of the present invention, embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It is to be understood that the following description is only a specific illustration of the embodiments of the present invention and should not be taken as limiting the scope of the invention.
Referring to fig. 2, fig. 2 is a schematic data flow diagram of file transmission according to an embodiment of the present invention, and the overall description of the present invention is as follows:
1. in order to accelerate the uploading and downloading speed, a single same file request is changed into the same file to support multi-concurrent uploading and downloading, and the more data is received, the shorter the file operation time is;
2. in order to avoid monopolizing the operation of writing the same file, the whole file is stored in blocks, each request is written into the corresponding block file at the same time, the whole file is not restored, an information summary file is added independently, and check data are stored;
3. because the files are not stored integrally, a block storage mode is adopted, the downloading is more direct, the downloading is carried out according to blocks, and the influence of the speed of reading the same file is avoided.
Referring to fig. 3, fig. 3 is a flowchart of a data chunking method according to an embodiment of the present invention, where the method includes steps S1-S2:
s1, converting a single data file into a data folder, wherein the data folder comprises a summary information file, one or more block files and a modification record file for storing modification records; the step S1 comprises steps S11-S14:
and S11, acquiring the size of the single data file.
S12, configuring the size of one to a plurality of block files, and calculating the number of the block files according to the size of the one to the plurality of block files and the size of the single data file. The size of the data folder can be configured through the size of a single data file, and the number of the one or more block files can be calculated through the size of the data folder.
And S13, sequentially numbering one or more block files. Referring to fig. 4, one or more of the partitioned files are numbered sequentially 1, 2, 3 \8230, 8230and N.
S14, converting a single data file into a data folder, wherein the data folder comprises a summary information file, one or more block files and a modification record file for storing modification records; the summary information file comprises the size of the single data file, the size of one to more block files, operation user information, and sequence numbers and file names of the one to more block files. Referring to fig. 4, fig. 4 is a schematic structural diagram of a data folder according to an embodiment of the present invention. Converting from a single data file to a folder, comprising: summarizing information files (including record authors, file names, sizes and other information, and responsible for checking and retrieving the whole file), blocking data files (i.e. one or more blocked files) and modification record files (current file modification history); the file data is read directly according to the blocks, the method is suitable for multithread concurrent downloading, one thread occupies one block, the locking operation caused by monopolizing one file stream is reduced, and the reading efficiency is improved.
Thus, the server handles file uploading and downloading as follows:
the method comprises the steps of firstly collecting the size of a whole file, converting the file name into a file folder, determining a default block size (configurable), calculating the number of blocks occupied by the file, numbering the whole block list in sequence, wherein the names of the block files are block numbers, and the single block file is independently stored in the file folder without collection and combination.
When writing, the storage medium records that the modification operation file is written exclusively, the related information of the file being modified, such as an operator, how many blocks of files are modified and the like, is stored, and the modified files are imported into the history record; if the current write already has the write operation file, a failure message of informing the client of the fact that the file has been written is sent back.
And when the breakpoint is continuously transmitted, reading the names of the block files on the hard disk, comparing the block indexes, issuing the block numbers which do not correspond to the block names, and correspondingly re-uploading the block numbers by the client. The breakpoint continuous transmission refers to that when downloading or uploading, a downloading or uploading task (a file or a compressed packet) is artificially divided into a plurality of parts, each part is uploaded or downloaded by adopting a thread, if a network fault is met, the uncompleted downloading part can be continuously uploaded from the uploaded or downloaded part, and the downloading is not required to be uploaded from the beginning, so that the time can be saved for a user, and the speed can be increased.
S2, receiving a modification request of a client to request for modifying a target file, wherein a subfolder for storing the target file and a modification lock file for storing feature code information are inserted into the data folder during modification, the inserted subfolder and the modification lock file are deleted after modification is finished, and the feature code information is stored in the modification record file; the target file is one or more of the one or more block files. Referring to fig. 5, fig. 5 is a schematic structural diagram of a data folder into which a modified lock file and a subfolder are inserted according to an embodiment of the present invention. By adding an exclusive modified lock file (storing the feature code information of a modifier, such as the name of the modifier, IP, mac, file size, blocking information and the like) and a subfolder for storing the blocks of the modified file, after the file blocks are finished, uniformly combining the files into a parent folder, and deleting the lock file and the subfolder; the method has the advantages that multiple threads are written into the block file, a plurality of blocks are operated simultaneously, in addition, the memory blocking of the large file is avoided and the large file is merged into an integral file, the server does not need to operate the integral file, and the integral file is directly provided for downloading according to the block mode.
Specifically, referring to fig. 6, fig. 6 is a flowchart of requesting to modify a target file according to an embodiment of the present invention, where step S2 includes steps S21 to S27:
s21, receiving a modification request of a client;
s22, checking whether a modification lock file exists in the data folder or not according to the modification request, if yes, turning to a step S23, and if not, turning to a step S24;
s23, matching the feature code information, if the matching is successful, turning to the step S26, and if the matching is unsuccessful, terminating the operation; in the step S23: and the feature code information comprises a modified user name, address information and modified information of the target file, the user name and the address information are matched with information preset by a server, if the matching is successful, the step S26 is switched to, so that the target file is modified according to the modified information, and if the matching is unsuccessful, the operation is terminated.
S24, checking whether a modification record exists in the modification record file, if so, turning to the step S23, and if not, turning to the step S25;
s25, inserting a subfolder for storing the target file and a modification lock file for storing feature code information into the data folder; the step S25 includes steps S251-S254:
s251, modifying the modification record file to insert the current modification record;
s252, writing the current modification record into a local file;
s253, generating and inserting a subfolder for storing the target file and a modification lock file for storing the feature code information into the data folder;
and S254, granting modification authority to the client, thereby allowing the client to modify the target file.
S26, allowing the client to modify the target file;
and S27, deleting the subfolders and the modification lock files after the modification is finished, and storing the feature code information into the modification record file.
Therefore, when the client side obtains the modification right, if the cache does not have the modification feature code, the modification record of the file is added, other users are prevented from modifying the same file, a modification lock file is locally established, the whole block data is verified after the modification is completed, the modified file list cache is completely withdrawn, the modification lock file is deleted, and the operation is completed.
Preferably, the step S2 further includes steps S28 to S29:
s28, receiving an uploading request of the client and allowing the client to upload; referring to fig. 7, fig. 7 is a schematic diagram illustrating a client uploading a file according to an embodiment of the present invention; the step S28 includes steps S281-S283:
s281, receiving an uploading request of the client;
s282, establishing a client block file set cache in the client;
and S283, allowing the client to upload, and storing the file in the client block file set cache during uploading.
The following limitations exist when uploading files:
1. the file reading needs to take a long time, particularly for large data files, and high-speed reading can be realized only by reading;
2. reading a file cannot be operated using multiple threads.
The method solves the problem that the speed is poor because the records are divided in the cache and the files are directly put into the cache after being read, and the local reading efficiency is higher than that of network transmission, so that the speed of the network transmission cannot be delayed.
And S29, receiving a downloading request of the client and allowing the client to perform downloading operation. Referring to fig. 8, fig. 8 is a schematic diagram of downloading a file by a client according to an embodiment of the present invention, where the step S29 includes steps S291-S293:
s291, receiving a downloading request of the client;
s292, establishing a service block file set cache in the server;
and S293, allowing the client to perform downloading operation through the service blocked file set cache.
Likewise, there are the following limitations when downloading files:
1. the file writing needs to monopolize the file stream, and if a large data file is downloaded, the time consumption is longer;
2. and writing the mutually contended file streams concurrently.
The solution is to create a block folder in the write folder, access the block file set, and unite them into an integral file after the download is completed.
Preferably, the data blocking method further includes step S3:
and S3, judging whether the target file modification of the client is finished or not, and if not, repairing according to a preset strategy. In step S3, the preset policy includes: deleting the modification request; or periodically clearing the overtime target file. That is, if the user uploads or modifies some files without completing the operation, it may happen that the files are not modified for any reason, such as active or passive, which may result in the generation of junk files, and the following policies are provided for repairing:
A. the delete upload file command is executed by an administrator or a current operator.
B. And clearing the overtime file data by the file service at regular time.
In summary, although the present invention has been described with reference to the preferred embodiments, the above-described preferred embodiments are not intended to limit the present invention, and those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention, therefore, the scope of the present invention shall be determined by the appended claims.
Claims (7)
1. A data blocking method, comprising steps S1-S2:
s1, converting a single data file into a data folder, wherein the data folder comprises a summary information file, one or more block files and a modification record file for storing modification records;
the step S1 comprises steps S11-S14:
s11, acquiring the size of the single data file;
s12, configuring the size of one to a plurality of block files, and calculating the number of the block files according to the size of the one to the plurality of block files and the size of the single data file;
s13, sequentially numbering one or more block files;
s14, converting a single data file into a data folder, wherein the data folder comprises a summary information file, one or more block files and a modification record file for storing modification records; the summary information file comprises the size of the single data file, the size of one to a plurality of block files, operation user information, and the sequence numbers and file names of the one to a plurality of block files;
s2, receiving a modification request of a client to request modification of a target file, wherein a subfolder for storing the target file and a modification lock file for storing feature code information are inserted into the data folder during modification, the inserted subfolder and the modification lock file are deleted after modification is finished, and the feature code information is stored into the modification record file; the target file is one or more of the block files;
the step S2 includes steps S21 to S27:
s21, receiving a modification request of a client;
s22, checking whether a modification lock file exists in the data folder or not according to the modification request, if yes, turning to a step S23, and if not, turning to a step S24;
s23, matching the feature code information, if the matching is successful, turning to the step S26, and if the matching is unsuccessful, terminating the operation;
s24, checking whether a modification record exists in the modification record file, if so, turning to the step S23, and if not, turning to the step S25;
s25, inserting a subfolder for storing the target file and a modification lock file for storing feature code information into the data folder;
s26, allowing the client to modify the target file;
s27, deleting the subfolders and the modification lock files after modification is finished, and storing the feature code information into the modification record file;
in the step S23:
and the feature code information comprises a modified user name, address information and modified information of the target file, the user name and the address information are matched with information preset by the server, if the matching is successful, the step S26 is carried out, so that the target file is modified according to the modified information, and if the matching is unsuccessful, the operation is terminated.
2. The data blocking method according to claim 1, wherein the step S25 comprises steps S251-S254:
s251, modifying the modification record file to insert the current modification record;
s252, writing the current modification record into a local file;
s253, generating and inserting a subfolder for storing the target file and a modification lock file for storing feature code information into the data folder;
and S254, granting modification authority to the client, thereby allowing the client to modify the target file.
3. The data blocking method according to claim 1, wherein said step S2 further comprises steps S28-S29:
s28, receiving an uploading request of the client and allowing the client to upload;
and S29, receiving a downloading request of the client and allowing the client to perform downloading operation.
4. The data blocking method according to claim 3, wherein the step S28 comprises steps S281-S283:
s281, receiving an uploading request of the client;
s282, establishing a client block file set cache in the client;
and S283, allowing the client to upload, and storing the file in the client block file set cache during uploading.
5. The data blocking method according to claim 3, wherein the S29 comprises steps S291-S293:
s291, receiving a downloading request of the client;
s292, establishing a service block file set cache in the server;
and S293, allowing the client to perform downloading operation through the service blocked file set cache.
6. The data blocking method according to claim 1, further comprising step S3:
and S3, judging whether the target file modification of the client is finished, and if not, repairing according to a preset strategy.
7. The data blocking method according to claim 6, wherein in the step S3, the preset policy includes:
deleting the modification request; or alternatively
And regularly clearing the overtime target file.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810711503.8A CN110737635B (en) | 2018-07-02 | 2018-07-02 | Data blocking method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810711503.8A CN110737635B (en) | 2018-07-02 | 2018-07-02 | Data blocking method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110737635A CN110737635A (en) | 2020-01-31 |
| CN110737635B true CN110737635B (en) | 2023-02-10 |
Family
ID=69233335
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810711503.8A Active CN110737635B (en) | 2018-07-02 | 2018-07-02 | Data blocking method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110737635B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111367855B (en) * | 2020-02-08 | 2021-06-18 | 重庆节节高科技发展有限公司 | Method and device for moving out downloaded file, computer equipment and storage medium |
| CN114490693A (en) * | 2022-02-17 | 2022-05-13 | 平安普惠企业管理有限公司 | Data modification method and device, electronic equipment and storage medium |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101969391B (en) * | 2010-10-27 | 2012-08-01 | 北京邮电大学 | Cloud platform supporting fusion network service and operating method thereof |
| CN104572891B (en) * | 2014-12-24 | 2017-12-12 | 北京大学深圳研究生院 | A kind of file updating method for network information separation storage |
| CN105812427B (en) * | 2014-12-31 | 2019-08-13 | 中国电信股份有限公司 | File uploads and method for down loading, device and file server |
| US9684569B2 (en) * | 2015-03-30 | 2017-06-20 | Western Digital Technologies, Inc. | Data deduplication using chunk files |
| CN105007322A (en) * | 2015-07-21 | 2015-10-28 | 陈丹 | File transmission method and system |
| CN105160253B (en) * | 2015-09-29 | 2018-11-09 | 网易（杭州）网络有限公司 | Repair method, apparatus, system and the server of client-side program |
| CN106982132A (en) * | 2016-01-18 | 2017-07-25 | 中兴通讯股份有限公司 | A kind of method of north direction document generation, apparatus and system |
-
2018
- 2018-07-02 CN CN201810711503.8A patent/CN110737635B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN110737635A (en) | 2020-01-31 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP7378870B2 (en) | File system data access method and file system | |
| CN109547566B (en) | Multithreading uploading optimization method based on memory allocation | |
| CN103548003B (en) | Method and system for improving the client-side fingerprint cache of deduplication system backup performance | |
| US7890716B2 (en) | Method of managing time-based differential snapshot | |
| KR20190067158A (en) | METHOD, APPARATUS AND SEARCH METHOD FOR ARCHITECTING BLOCK CHAIN BLOCK DATA | |
| US20080147974A1 (en) | Multi-level caching system | |
| JP4955677B2 (en) | Move data from a file on a storage volume to an alternate location to free up space | |
| EP1902394B1 (en) | Moving data from file on storage volume to alternate location to free space | |
| CN110647497A (en) | HDFS-based high-performance file storage and management system | |
| US20060136779A1 (en) | Object-based storage device with low process load and control method thereof | |
| US7506004B2 (en) | Moving data from file on storage volume to alternate location to free space | |
| US20120284369A1 (en) | System, method and computer program product for managing a remote storage | |
| CN110750497B (en) | Data scheduling system | |
| US7640588B2 (en) | Data processing system and method | |
| CN111475483A (en) | Database migration method and device and computing equipment | |
| CN113032335A (en) | File access method, device, equipment and storage medium | |
| US20080201444A1 (en) | File sharing system and file sharing method | |
| CN110737635B (en) | Data blocking method | |
| CN114610679A (en) | Storage device, data storage method thereof and cloud storage system | |
| US7505986B2 (en) | Moving data from file on storage volume to alternate location to free space | |
| CN107181773A (en) | Data storage and data managing method, the equipment of distributed memory system | |
| US7873681B2 (en) | Moving data from file on storage volume to alternate location to free space | |
| WO2009031158A2 (en) | Method and apparatus for network based data recovery | |
| CN107665224B (en) | Method, system and device for scanning HDFS cold data | |
| CN113835613B (en) | File reading method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |