CN112632021A - Read-write method, system, device and medium of network file system - Google Patents

Read-write method, system, device and medium of network file system Download PDF

Info

Publication number
CN112632021A
CN112632021A CN202011400621.0A CN202011400621A CN112632021A CN 112632021 A CN112632021 A CN 112632021A CN 202011400621 A CN202011400621 A CN 202011400621A CN 112632021 A CN112632021 A CN 112632021A
Authority
CN
China
Prior art keywords
file
transmission
compression
compressed
read
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
CN202011400621.0A
Other languages
Chinese (zh)
Inventor
刘相乐
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Suzhou Inspur Intelligent Technology Co Ltd
Original Assignee
Suzhou Inspur Intelligent Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Suzhou Inspur Intelligent Technology Co Ltd filed Critical Suzhou Inspur Intelligent Technology Co Ltd
Priority to CN202011400621.0A priority Critical patent/CN112632021A/en
Publication of CN112632021A publication Critical patent/CN112632021A/en
Withdrawn legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/18File system types
    • G06F16/182Distributed file systems
    • G06F16/1824Distributed file systems implemented using Network-attached Storage [NAS] architecture
    • G06F16/183Provision of network file services by network file servers, e.g. by using NFS, CIFS
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/17Details of further file system functions
    • G06F16/174Redundancy elimination performed by the file system
    • G06F16/1744Redundancy elimination performed by the file system using compression, e.g. sparse files
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/01Protocols
    • H04L67/06Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L69/00Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
    • H04L69/04Protocols for data compression, e.g. ROHC

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Computer Security & Cryptography (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

The invention discloses a read-write method of a network file system, which comprises the steps of detecting residual resources of a CPU (Central processing Unit), calculating the size and the compression ratio of a transmission file, and writing compressed configuration information into a configuration file; comparing the size and compression ratio of the transmission file with the compression configuration information, and judging whether the transmission condition of the compression file is met; if yes, calculating CPU resources consumed during file compression transmission, and comparing the CPU resources with the residual CPU resources; if the consumed CPU resource is not larger than the CPU residual resource, compressing the transmitted file and executing the read-write operation of the compressed file; if the consumed CPU resource is larger than the residual CPU resource, executing the read-write operation of the common file; if not, executing the read-write operation of the common file; through the mode, the invention can compress the file which occupies a large memory and has a high compression ratio at the same time, and then transmit the file, thereby reducing the network transmission pressure and improving the transmission efficiency.

Description

Read-write method, system, device and medium of network file system
Technical Field
The present invention relates to the field of storage technologies, and in particular, to a method, a system, a device, and a medium for reading and writing a network file system.
Background
The NFS sharing service is a service process for providing file sharing in an NAS cluster storage system, operations of reading and writing large files are often performed in an application scene of the NFS, a client needs to write the files to a storage and also needs to read the large files on the storage to the local, many large files have high compression ratios, the large files are very small after being compressed, in this case, if the large files are directly transmitted without being compressed between the server and the client, a large amount of bandwidth is wasted, and the large amount of bandwidth is wasted under the condition of poor network performance, so that other processes are slow in response, and the efficiency and the use experience of users are affected.
Disclosure of Invention
The technical problem mainly solved by the present invention is to provide a method, a system, a device and a medium for reading and writing a network file system, which can compress a file occupying a relatively large memory and having a relatively high compression ratio first and then transmit the file, thereby reducing network transmission pressure, improving transmission efficiency, improving reading and writing performance of NFS, and improving access experience of NFS sharing.
In order to solve the technical problems, the invention adopts a technical scheme that: a read-write method of a network file system is provided, which comprises the following steps:
detecting residual resources of the CPU, calculating the size and the compression ratio of a transmission file, and writing compression configuration information into a configuration file;
comparing the size and compression ratio of the transmission file with the compression configuration information in the configuration file, and judging whether the transmission file meets the transmission condition of the compression file;
if the transmission condition of the compressed file is met, calculating CPU resources consumed during file compression transmission, and comparing the CPU resources with the CPU residual resources;
if the consumed CPU resource is not larger than the CPU residual resource, compressing the transmitted file and executing the read-write operation of the compressed file;
if the consumed CPU resource is larger than the residual CPU resource, executing the read-write operation of the common file;
and if the compressed file transmission condition is not met, executing the read-write operation of the common file.
Further, the configuration information includes a file size, a file compression ratio and a compression algorithm of compression transmission.
Further, the step of judging whether the transmission condition of the compressed file is met comprises the following steps:
if the compression ratio of the transmission file is larger than the file compression ratio in the configuration information and the size of the transmission file is larger than the size of the file transmitted in the configuration information in a compression mode, the compression condition is met;
if the compression ratio of the transmission file is not more than the file compression ratio in the configuration information and the size of the transmission file is not more than the size of the file compressed and transmitted in the configuration information, the compression condition is not met.
Further, the executing the read-write operation of the compressed file comprises executing a write operation and executing a read operation.
Further, the performing a write operation includes the steps of:
the client compresses the transmission file into a plurality of compressed data packets by using a compression algorithm, and marks the plurality of compressed data packets;
the server receives the compressed data packet, and judges whether the compressed data packet is a compressed data packet of a transmission file or not according to the compressed data packet mark;
if the data packet is a compressed data packet of a transmission file, receiving; if the data packet is not the compressed data packet of the transmission file, refusing to receive;
after the compressed data packets of the transmission files are received, the server decompresses all the compressed data packets by using a compression algorithm and writes the decompressed data packets into a file system.
Further, the performing a read operation includes the steps of:
the server compresses the transmission file by using a compression algorithm, compresses the transmission file into a plurality of compressed data packets, and sets marks on the plurality of compressed data packets;
the client reads the compressed data packet and judges whether the compressed data packet is a compressed data packet of a transmission file or not according to the compressed data packet mark;
if the compressed data packet is the compressed data packet of the transmission file, receiving, and if the compressed data packet is not the compressed data packet of the transmission file, rejecting receiving;
and after reading the compressed data packet, the client decompresses through a compression algorithm.
A read-write system of a network file system comprises a configuration module, a resource detection module, a calculation module, a judgment module and a read-write file module;
the configuration module configures the size of a compressed and transmitted file, the file compression ratio and a compression algorithm;
the resource monitoring module detects the residual used resources of the CPU;
the calculation module calculates the compression ratio of the file and the CPU resource consumed during file compression;
the judging module judges whether the size of a transmission file and the size of a compression file in the configuration module as well as the file compression ratio calculated by the calculating module and the file compression ratio in the configuration module meet the transmission condition of the compression file, if so, the CPU resource consumed in the calculating module is compared with the residual CPU resource;
if the consumed CPU resource is not larger than the CPU residual resource, informing the read-write file module to read and write the compressed file; if not, informing the read-write file module to carry out read-write operation of the common file;
and the file reading and writing module carries out reading and writing operation on the compressed file or reading and writing operation on the common file according to the judgment module.
An electronic device, comprising: a processor, a memory, and a program, wherein the program is stored in the memory and configured to be executed by the processor, the program comprising steps for performing a method of reading from and writing to a network file system as described in any of the above.
A computer-readable storage medium having stored thereon a computer program for executing by a processor the steps of a method for reading from and writing to a network file system as claimed in any one of the preceding claims.
The invention has the beneficial effects that: when the file is read and written, the invention judges whether the file has a higher compression ratio firstly, compresses the file with the higher compression ratio and then transmits the compressed file, so that the time wasted on network transmission is greatly reduced, thereby improving the read and write performance of the file and improving the use experience of NFS sharing service.
Drawings
FIG. 1 is a flow chart of a preferred embodiment of a method for reading from and writing to a network file system according to the present invention;
FIG. 2 is a diagram of a read/write system architecture of a network file system according to the present invention.
Detailed Description
The following detailed description of the preferred embodiments of the present invention, taken in conjunction with the accompanying drawings, will make the advantages and features of the invention easier to understand by those skilled in the art, and thus will clearly and clearly define the scope of the invention.
The embodiment of the invention comprises the following steps:
referring to fig. 1, a method for reading and writing a network file system includes:
writing configuration information related to compression into an NFS configuration file, wherein the configuration information related to compression comprises a minimum file transmitted in a compression mode, a minimum file compression ratio and a compression algorithm;
the minimum file compression transmission means that the size of a file storage occupied space is specified, and compression transmission is carried out only if the size of the transmitted file is larger than the specified size of the file storage occupied space;
the minimum file compression ratio refers to that the compression transmission is considered only when the compression ratio of the file is larger than the value, and when the compression ratio of the file needs to be read and written again, a plurality of positions are selected from the file to be read and written for sampling and calculating the compression ratio;
selecting a plurality of positions, wherein the selected positions are selected randomly; randomly sampling several positions in a file to be transmitted, compressing the several sampling positions through a specified compression algorithm, and calculating a compression ratio;
since the operations of compression and decompression also take some time, take up the performance of cpu, and compression transmission is more valuable only when the compression ratio is higher and the file is larger.
If the file to be transmitted, the minimum file to be transmitted and the minimum compression ratio meet the condition of compressing related configuration information, the size of the space occupied by the transmitted file is larger than that of the minimum file to be transmitted, and the compression ratio of the transmitted file is larger than that of the minimum file, then the compressed transmission is carried out;
if the file to be transmitted, the minimum file to be transmitted and the lowest compression ratio do not meet the condition of compressing related configuration information, the size of the space occupied by the transmitted file is not larger than the size of the space occupied by the compressed and transmitted minimum file, and the compression ratio of the transmitted file is not larger than the minimum file compression ratio, the compressed transmission is not carried out;
the compression algorithm may specify a compression algorithm from the user in the configuration information, such as LZW, LZO, LZMA, LZSS, LZR, LZB, LZH, LZC, LZT, LZMW, LZJ, LZFG, and so forth.
The Cpu residual resource is a residual resource only when the Cpu is idle, and compression transmission is performed when the Cpu resource consumed in transmitting the compressed file is less than the Cpu residual resource, otherwise, other tasks are affected because the compression operation occupies too much Cpu.
Performing file read-write operation, wherein the file read-write operation comprises file read operation and file write operation, and whether the file needs to be compressed or not is judged by compressing related configuration information during the file read-write operation;
if the compression operation is needed, performing the compression operation of writing the file and the compression operation of reading the file;
and if the compression operation is not needed, performing a normal file reading and writing process.
The compression operation of writing the file is the operation of starting the decompression by the server side by the client side, and the compression operation of reading the file is the operation of starting the decompression by the client side by the server side.
The embodiment first performs a description of the operation of writing a file,
the NFS client inquires about the configuration supported by the server, and the NFS server replies according to the configuration in the configuration file;
inquiring whether the content supports compression transmission, minimum compression ratio, minimum file supporting compression and compression algorithm;
calculating a minimum compression ratio;
when the file to be transmitted, the minimum file to be transmitted in a compression mode and the minimum compression ratio meet the condition of compressing related configuration information, performing compression transmission;
if not, performing compression transmission, and performing a reading process of the common file;
if the compressed transmission is carried out, the NFS client carries out the operation of transmitting the compressed file;
calculating CPU resources consumed when the compressed file is transmitted through a calculation tool, comparing the CPU resources with residual CPU resources, and judging whether the compressed file is supported or not;
if the residual CPU resources are larger than the consumed CPU resources, transmitting the compressed file;
and if the residual CPU resources are not greater than the consumed CPU resources, the compressed file is not transmitted, and the compressed file is transmitted after the residual CPU resources are greater than the consumed CPU resources.
The NFS client compresses a file to be transmitted by using a compression algorithm, compresses the transmitted file into a plurality of compressed data packets, and sets a mark of the compressed data packets during transmission; the tag means to identify that the file is a compressed packet of data in order to be the compressed file to which the compressed packet of data belongs.
Since a file may be large, but the network packet required for network transmission may be smaller than the file, in which case the file is transmitted in multiple network packets, it is necessary to determine that multiple network packets are the same file transmitted by the tag.
The NFS server receives the compressed file transmitted by the NFS client, then judges whether the received data packet is the transmitted file compressed data packet or not according to the compressed data packet mark, and receives the compressed file if the received data packet is the transmitted file compressed data packet, or rejects the receiving if the received data packet is not the transmitted file compressed data packet; after the whole compressed file is received, all the received compressed data packets are decompressed through a compression algorithm and written into a file system of a server.
The read file operation is similar to the write file operation, and the read file operation is described below:
the NFS client initiates a file reading request;
the NFS server side compresses the minimum file and the minimum compression ratio transmitted in the related configuration information according to whether the file to be read meets the requirements or not; if yes, performing compression transmission;
if not, performing compression transmission, and performing a reading process of the common file;
calculating CPU resources consumed when the compressed file is transmitted through a calculation tool, comparing the CPU resources with residual CPU resources, and judging whether the compressed file is supported or not;
if the residual CPU resources are larger than the consumed CPU resources, transmitting the compressed file;
and if the residual CPU resources are not greater than the consumed CPU resources, the compressed file is not transmitted, and the compressed file is transmitted after the residual CPU resources are greater than the consumed CPU resources.
The NFS client compresses a file to be transmitted by using a compression algorithm, compresses the transmitted file into a plurality of compressed data packets, and sets a mark of the compressed data packets during transmission; the tag means to identify that the file is a compressed packet of data in order to be the compressed file to which the compressed packet of data belongs.
The NFS server receives the compressed file transmitted by the NFS client, and then according to the compressed data packet mark,
judging whether the received data packet is a transmitted file compression data packet or not, if so, receiving, and if not, rejecting to receive;
after reading the whole compressed file, the NFS client decompresses the compression algorithm of the compression related configuration information inquired when the client is mounted.
An electronic device, comprising: a processor; a memory; and a program, wherein the program is stored in the memory and configured to be executed by the processor, the program comprising steps for performing a read-write method of a network file system as described above.
A computer-readable storage medium, on which a computer program is stored, the computer program being executed by a processor for performing the steps of a method for reading from and writing to a network file system as described above.
Referring to fig. 2, based on the same inventive concept as the read-write method of the network file system in the foregoing embodiment, an embodiment of the present specification further provides a read-write system of a network file system, including a configuration module, a resource detection module, a calculation module, a determination module, and a read-write file module;
the configuration module configures the file value, the file compression ratio and the compression algorithm of compression transmission;
the resource monitoring module detects the residual used resources of the CPU;
the calculation module calculates the compression ratio of the file and the CPU resource consumed when the file is compressed;
the judging module judges whether the transmission condition of the compressed file is met or not according to the size of the transmission file, the size of the compressed file in the configuration module, the file compression ratio calculated by the calculating module and the file compression ratio in the configuration module, and if the transmission condition of the compressed file is met, the CPU resource consumed in the calculating module is compared with the residual CPU resource;
if the consumed CPU resource is not larger than the CPU residual resource, informing the read-write file module to read and write the compressed file; if not, informing the read-write file module to carry out read-write operation of the common file;
and the file reading and writing module carries out reading and writing operation on the compressed file or reading and writing operation on the common file according to the judgment module.
The above description is only an embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes performed by the present specification and drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.

Claims (9)

1. A read-write method of a network file system is characterized by comprising the following steps:
detecting residual resources of the CPU, calculating the size and the compression ratio of a transmission file, and writing compression configuration information into a configuration file;
comparing the size and compression ratio of the transmission file with the compression configuration information in the configuration file, and judging whether the transmission file meets the transmission condition of the compression file;
if the transmission condition of the compressed file is met, calculating CPU resources consumed during file compression transmission, and comparing the CPU resources with the CPU residual resources;
if the consumed CPU resource is not larger than the CPU residual resource, compressing the transmitted file and executing the read-write operation of the compressed file;
if the consumed CPU resource is larger than the residual CPU resource, executing the read-write operation of the common file;
and if the compressed file transmission condition is not met, executing the read-write operation of the common file.
2. A method for reading from and writing to a network file system according to claim 1, wherein: the configuration information comprises the size of the compressed and transmitted file, the file compression ratio and the compression algorithm.
3. A method for reading from and writing to a network file system according to claim 2, characterized in that: the step of judging whether the transmission condition of the compressed file is met comprises the following steps:
if the compression ratio of the transmission file is larger than the file compression ratio in the configuration information and the size of the transmission file is larger than the size of the file transmitted in the configuration information in a compression mode, the compression condition is met;
if the compression ratio of the transmission file is not more than the file compression ratio in the configuration information and the size of the transmission file is not more than the size of the file compressed and transmitted in the configuration information, the compression condition is not met.
4. A method for reading from and writing to a network file system according to claim 2, characterized in that: the executing of the read-write operation of the compressed file comprises executing a write operation and executing a read operation.
5. A method for reading from and writing to a network file system according to claim 4, wherein: the performing a write operation includes the steps of:
the client compresses the transmission file into a plurality of compressed data packets by using a compression algorithm, and marks the plurality of compressed data packets;
the server receives the compressed data packet, and judges whether the compressed data packet is a compressed data packet of a transmission file or not according to the compressed data packet mark;
if the data packet is a compressed data packet of a transmission file, receiving; if the data packet is not the compressed data packet of the transmission file, refusing to receive;
after the compressed data packets of the transmission files are received, the server decompresses all the compressed data packets by using a compression algorithm and writes the decompressed data packets into a file system.
6. A method for reading from and writing to a network file system according to claim 4, wherein: the performing a read operation includes the steps of:
the server compresses the transmission file by using a compression algorithm, compresses the transmission file into a plurality of compressed data packets, and sets marks on the plurality of compressed data packets;
the client reads the compressed data packet and judges whether the compressed data packet is a compressed data packet of a transmission file or not according to the compressed data packet mark;
if the compressed data packet is the compressed data packet of the transmission file, receiving, and if the compressed data packet is not the compressed data packet of the transmission file, rejecting receiving;
and after reading the compressed data packet, the client decompresses through a compression algorithm.
7. A read-write system of a network file system is characterized by comprising a configuration module, a resource detection module, a calculation module, a judgment module and a read-write file module;
the configuration module configures the size of a compressed and transmitted file, the file compression ratio and a compression algorithm;
the resource monitoring module detects the residual used resources of the CPU;
the calculation module calculates the compression ratio of the file and the CPU resource consumed during file compression;
the judging module judges whether the size of a transmission file and the size of a compression file in the configuration module as well as the file compression ratio calculated by the calculating module and the file compression ratio in the configuration module meet the transmission condition of the compression file, if so, the CPU resource consumed in the calculating module is compared with the residual CPU resource;
if the consumed CPU resource is not larger than the CPU residual resource, informing the read-write file module to read and write the compressed file; if not, informing the read-write file module to carry out read-write operation of the common file;
and the file reading and writing module carries out reading and writing operation on the compressed file or reading and writing operation on the common file according to the judgment module.
8. An electronic device comprising a processor, a memory, and a program, wherein the program is stored in the memory and configured to be executed by the processor, the program comprising steps for performing a method of reading from and writing to a network file system as claimed in any one of claims 1 to 6.
9. A computer-readable storage medium, on which a computer program is stored, characterized in that the computer program is executed by a processor for performing the steps of a method for reading from and writing to a network file system as claimed in any one of the preceding claims 1 to 6.
CN202011400621.0A 2020-12-02 2020-12-02 Read-write method, system, device and medium of network file system Withdrawn CN112632021A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011400621.0A CN112632021A (en) 2020-12-02 2020-12-02 Read-write method, system, device and medium of network file system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011400621.0A CN112632021A (en) 2020-12-02 2020-12-02 Read-write method, system, device and medium of network file system

Publications (1)

Publication Number Publication Date
CN112632021A true CN112632021A (en) 2021-04-09

Family

ID=75307867

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011400621.0A Withdrawn CN112632021A (en) 2020-12-02 2020-12-02 Read-write method, system, device and medium of network file system

Country Status (1)

Country Link
CN (1) CN112632021A (en)

Similar Documents

Publication Publication Date Title
KR102435046B1 (en) Lossy compression drive
US20140215170A1 (en) Block Compression in a Key/Value Store
US8539191B2 (en) Estimating space in a compressed volume
US20100050089A1 (en) Web browser system of mobile communication terminal, using proxy server
JP7169098B2 (en) Storage drive, its compression system, and its data compression method
CN112422497B (en) Message transmission method and device and computer equipment
CN112367384B (en) Kafka cluster-based dynamic speed limiting method and device and computer equipment
CN111935261B (en) Response message processing method, device, electronic equipment and storage medium
US8010834B2 (en) Failure information monitoring apparatus and failure information monitoring method
CN111966647A (en) Cloud storage method and device for small files, server and storage medium
CN110324681B (en) Slicing method, device, storage medium and transmission system of multimedia data
CN108599774B (en) Compression method, system, device and computer readable storage medium
CN111857574A (en) Write request data compression method, system, terminal and storage medium
CN109445719B (en) Data storage method and device
CN112632021A (en) Read-write method, system, device and medium of network file system
CN110032432B (en) Example compression method and device and example decompression method and device
CN111414339A (en) File processing method, system, device, equipment and medium
CN115665137A (en) Data processing method and device
CN114519037A (en) Table online previewing method, device and system
CN109947371B (en) Data recording method, device, memory and T-BOX
CN102891732A (en) Data sending method and device and data receiving method and device
CN113590021A (en) Storage system
Huang et al. A Dynamic threshold technique for XML data transmission on networks
CN117666968B (en) Selective data compression method and device for solid-state disk storage system
CN113127012B (en) Software resource construction method based on software engine

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
WW01 Invention patent application withdrawn after publication

Application publication date: 20210409

WW01 Invention patent application withdrawn after publication