WO2022001133A1

WO2022001133A1 - Method and system for improving soft copy read performance, terminal, and storage medium

Info

Publication number: WO2022001133A1
Application number: PCT/CN2021/076951
Authority: WO
Inventors: 苏志恒; 李文鹏
Original assignee: 苏州浪潮智能科技有限公司
Priority date: 2020-06-28
Filing date: 2021-02-19
Publication date: 2022-01-06
Also published as: CN111858665A; CN111858665B

Abstract

A method and system for improving soft copy read performance, a terminal and a storage medium, the method comprising: reading a soft copy file and associating same with source file data within a read range; according to a pre-reading algorithm and a read range, calculating data segments involved in pre-read data; pre-reading corresponding source file data according to a data segment cycle and placing same into a cache; and moving the data in the cache according to a data cache elimination algorithm. By means of self-learning technology of a distributed system pre-reading algorithm and a cache module elimination mechanism, the soft copy file is pre-read, and the read performance and system throughput of the soft copy file are improved.

Description

A method, system, terminal and storage medium for improving soft copy read performance

This application claims the priority of the Chinese patent application submitted to the China Patent Office on June 28, 2020, the application number is 202010598742.4, and the invention title is "A method, system, terminal and storage medium for improving soft copy reading performance", The entire contents of which are incorporated herein by reference.

technical field

The present application belongs to the technical field of distributed systems, and in particular relates to a method, system, terminal and storage medium for improving soft copy read performance.

Background technique

In the era of big data, based on the implemented file soft copy function, when reading a soft copy file, the existing practice is: first, it needs to associate with the source file, and then read the data from the hard disk; when reading the snapshot data of the source file directly from the disk , the data is not in the cache, and the upper-layer application needs to wait for the data to be successfully read from the disk before returning. The time-consuming of reading data directly from the hard disk is 50 times that of the cache reading, which seriously affects the read performance and data transmission efficiency of the storage system. Reading data becomes the bottleneck of read bandwidth and read performance of distributed systems.

SUMMARY OF THE INVENTION

In view of the above deficiencies of the prior art, the present application provides a method, system, terminal and storage medium for improving soft copy read performance, so as to solve the above technical problems.

In a first aspect, the present application provides a method for improving soft copy read performance, including:

Read the soft copy file and associate it with the source file data within the read range;

Calculate the data segments involved in the pre-read data according to the pre-read algorithm and the read range;

Circularly pre-read the corresponding source file data according to the data segment and put it into the cache;

The data in the cache is moved according to the data cache elimination algorithm.

Further, the method also includes:

Create a default cache elimination queue and a corresponding cache elimination queue;

Read the source file data associated with the soft copy file, and determine the source of the read data:

If read directly from the cache, put the read cache data into the default cache elimination queue;

If it is read directly from the disk, during the data reading process, the data read range rules are automatically collected and the cache elimination model is adapted, the read data is moved to the corresponding cache elimination queue according to the matching of the cache elimination model, and the default cache The data of the elimination queue is moved into the corresponding cache elimination queue.

Further, according to the pre-reading algorithm and the read range, the data segments involved in the pre-reading data are obtained, including:

Call the read-ahead algorithm, and calculate the data segment involved in the read-ahead data by passing in the offset of the current read data and the length of the read data according to the read range.

Further, the cyclically pre-reading the corresponding source file data according to the data segment and placing it in the cache includes:

Find the source file corresponding to each pre-read data segment, read the source file data of each data segment asynchronously and load it into the cache, and wait for the next time it is directly read from the cache, put it into the default cache elimination queue for cache update.

In a second aspect, the present application provides a system for improving soft copy read performance, including:

The file reading unit is configured to read the soft copy file and associate it with the source file data within the read range;

The data pre-reading unit is configured to calculate the data segments involved in the pre-reading data according to the pre-reading algorithm and the read range;

a cache writing unit, configured to cyclically pre-read the corresponding source file data according to the data segment and put it into the cache;

The cache elimination unit is configured to move the data in the cache according to the data cache elimination algorithm.

In a third aspect, a terminal is provided, including:

processor, memory, where,

The memory is used to store computer programs,

The processor is used to call and run the computer program from the memory, so that the terminal executes the above-mentioned method of the terminal.

In a fourth aspect, a computer-readable storage medium is provided, and instructions are stored in the computer-readable storage medium, which, when executed on a computer, cause the computer to perform the methods described in the above aspects.

The beneficial effect of the present application is that,

The present application provides a method, system, terminal and storage medium for improving soft copy read performance. Through the distributed system pre-reading algorithm and the self-learning technology of the cache module elimination mechanism, the pre-reading of soft-copy files is realized and the reading of soft-copy files is improved. performance and system throughput.

In addition, the design principle of the present application is reliable, the structure is simple, and has a very broad application prospect.

Description of drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. In other words, other drawings can also be obtained based on these drawings without creative labor.

FIG. 1 is a schematic flowchart of a method according to an embodiment of the present application.

FIG. 2 is a schematic block diagram of a system according to an embodiment of the present application.

FIG. 3 is a schematic structural diagram of a terminal according to an embodiment of the present application.

detailed description

In order to make those skilled in the art better understand the technical solutions in the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described The embodiments are only a part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the scope of protection of the present application.

Key terms appearing in this application are explained below.

FIG. 1 is a schematic flowchart of a method according to an embodiment of the present application. The execution body of FIG. 1 may be a system for improving soft copy read performance.

As shown in Figure 1, the method 100 includes:

Step 110, read the soft copy file and associate it with the source file data within the read range;

Step 120, calculating the data segments involved in the pre-reading data according to the pre-reading algorithm and the read range;

Step 130, cyclically pre-read the corresponding source file data according to the data segment and put it into the cache;

Step 140: Move the data in the cache according to the data cache elimination algorithm.

Optionally, as an embodiment of the present application, the method further includes:

Optionally, as an embodiment of the present application, the data segments involved in the pre-reading data are calculated according to the pre-reading algorithm and the read range, including:

Optionally, as an embodiment of the present application, the cyclically pre-reading the corresponding source file data according to the data segment and placing it in the cache includes:

In order to facilitate the understanding of the present application, the following describes a method for improving soft copy read performance provided by the present application based on the principle of a method for improving soft copy read performance in the present application, combined with the process of performing soft copy reading on a distributed system in the embodiment. method is further described.

Specifically, the method for improving soft copy read performance includes:

First, run the distributed system and client program to create soft copy files;

When the client reads the soft copy file, it first associates the source file snapshot data segment or data segment information corresponding to the read range (OFFSET~OFFSET+SIZE), and then reads the data. If the read data hits the cache, it is directly read from the cache. After the data is returned, the cached data that has been read is put into the default cache elimination queue Q1; if the read data is not in the cache, the data is directly read from the disk, and then returned; during the data read process, the cache module automatically collects the data read range After the cache elimination mode is successfully selected, the data of Q1 is moved into the corresponding cache elimination queue Q2; after the data reading is completed, the cache module pre-reading algorithm is called to pass in the current read data The offset OFFSET and the length of the read data SIZE, calculate the data segments involved in the subsequent pre-read data, and then find the source file corresponding to each pre-read data segment, read the data of each data segment asynchronously and load it into the cache , wait for the next read hit, and put it into the corresponding cache elimination queue for cache update.

The implementation of the client program mainly includes the following steps:

(1) Run distributed systems and client programs to create soft copy files;

(2) Read the soft copy file and wait for the read data to return;

(3) The source file snapshot data segment or data segment information corresponding to the associated read range;

(4) Read the data according to the associated data segment and wait for the return;

(5) After the data reading is completed, according to the range of the current read data and call the cache module pre-reading algorithm to calculate the data segment involved in the pre-reading data;

(6) The source file data segment or snapshot data segment involved in the associated pre-reading data segment;

(7) Circularly issue asynchronous messages to pre-read the data of each data segment;

(8) After the pre-reading message is delivered, the read data is returned to the client.

The cache module handler mainly includes the following steps:

(1) After receiving the client read message, determine whether the data to be read is in the cache;

(2) If so, enter step (3);

(3) Take the data from the cache and put the read cache data into the default cache elimination queue Q1,

After the subsequent cache elimination model is successfully matched, it is moved to the corresponding cache elimination queue;

(4) If not, then enter step (5);

(5) judge whether it is pre-reading, if it is pre-reading, then enter step (6);

(6) Issue an asynchronous read data operation and return it, and put it into the corresponding cache queue after the data is read successfully;

(7) If it is not pre-reading, then enter step (8);

(8) Issue the read data operation and wait for the return;

(9) The self-learning data reads the range data, and matches the corresponding cache elimination model. After the matching is successful, the cache data is moved.

As shown in Figure 2, the system 200 includes:

a file reading unit 210, configured to read the soft copy file and associate it with the source file data within the read range;

The data pre-reading unit 220 is configured to calculate and obtain the data segments involved in the pre-reading data according to the pre-reading algorithm and the reading range;

The cache writing unit 230 is configured to cyclically pre-read the corresponding source file data according to the data segment and put it into the cache;

The cache elimination unit 240 is configured to move the data in the cache according to the data cache elimination algorithm.

FIG. 3 is a schematic structural diagram of a terminal system 300 provided by an embodiment of the present application. The terminal system 300 may be used to execute a method for improving soft copy read performance provided by an embodiment of the present application.

The terminal system 300 may include: a processor 310 , a memory 320 and a communication unit 330 . These components communicate through one or more buses. Those skilled in the art can understand that the structure of the server shown in the figure does not constitute a limitation on this application. It can be either a bus-shaped structure, a star-shaped structure, or a More or fewer components than shown may be included, or some components may be combined, or a different arrangement of components.

Wherein, the memory 320 can be used to store the execution instructions of the processor 310, and the memory 320 can be implemented by any type of volatile or non-volatile storage terminal or their combination, such as static random access memory (SRAM), electrical Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read Only Memory (PROM), Read Only Memory (ROM), Magnetic Memory, Flash Memory, Magnetic Disk or Optical Disk . When the execution instructions in the memory 320 are executed by the processor 310, the terminal 300 is enabled to execute some or all of the steps in the following method embodiments.

The processor 310 is the control center of the storage terminal, using various interfaces and lines to connect various parts of the entire electronic terminal, by running or executing the software programs and/or modules stored in the memory 320, and calling the data stored in the memory, To perform various functions of the electronic terminal and/or process data. The processor may be composed of an integrated circuit (Integrated Circuit, IC for short), for example, may be composed of a single packaged IC, or may be composed of a plurality of packaged ICs connected with the same function or different functions. For example, the processor 310 may only include a central processing unit (Central Processing Unit, CPU for short). In the embodiments of the present application, the CPU may be a single computing core, or may include multiple computing cores.

The communication unit 330 is used for establishing a communication channel, so that the storage terminal can communicate with other terminals. Receive user data sent by other terminals or send user data to other terminals.

The present application also provides a computer storage medium, wherein the computer storage medium can store a program, and when the program is executed, the program can include some or all of the steps in the embodiments provided in the present application. The storage medium may be a magnetic disk, an optical disk, a read-only memory (English: read-only memory, abbreviated as: ROM) or a random access memory (English: random access memory, abbreviated as: RAM) and the like.

Therefore, the present application realizes the pre-reading of soft-copy files through the self-learning technology of the distributed system pre-reading algorithm and the cache module elimination mechanism, and improves the reading performance and system throughput of soft-copy files. The technical effects that can be achieved in this embodiment can be found in The above description will not be repeated here.

Those skilled in the art can clearly understand that the technology in the embodiments of the present application can be implemented by means of software plus a necessary general hardware platform. Based on such understanding, the technical solutions in the embodiments of the present application can be embodied in the form of software products in essence or in the parts that make contributions to the prior art. The computer software products are stored in a storage medium such as a USB flash drive, a mobile Hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program codes, including several instructions to make a computer terminal (It may be a personal computer, a server, or a second terminal, a network terminal, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application.

It is sufficient to refer to each other for the same and similar parts among the various embodiments in this specification. In particular, for the terminal embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and for related parts, please refer to the description in the method embodiment.

In the several embodiments provided in this application, it should be understood that the disclosed systems, systems and methods may be implemented in other manners. For example, the system embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or Can be integrated into another system, or some features can be ignored, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection between systems or units, and may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit.

Although the present application has been described in detail with reference to the accompanying drawings and in conjunction with the preferred embodiments, the present application is not limited thereto. Without departing from the spirit and essence of the present application, those of ordinary skill in the art can make various equivalent modifications or substitutions to the embodiments of the present application, and these modifications or substitutions should all fall within the scope of the present application/any Those skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present application, which should all be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

A method for improving soft copy read performance, comprising:

Read the soft copy file and associate it with the source file data within the read range;

Calculate the data segments involved in the pre-read data according to the pre-read algorithm and the read range;

Circularly pre-read the corresponding source file data according to the data segment and put it into the cache;

The data in the cache is moved according to the data cache elimination algorithm.
The method for improving soft copy read performance according to claim 1, wherein the method further comprises:

Create a default cache elimination queue and a corresponding cache elimination queue;

Read the source file data associated with the soft copy file, and determine the source of the read data:

If read directly from the cache, put the read cache data into the default cache elimination queue;

If it is read directly from the disk, during the data reading process, the data read range rules are automatically collected and the cache elimination model is adapted, the read data is moved to the corresponding cache elimination queue according to the matching of the cache elimination model, and the default cache The data of the elimination queue is moved into the corresponding cache elimination queue.
The method for improving soft copy read performance according to claim 1, wherein the calculating and obtaining the data segments involved in the pre-reading data according to the pre-reading algorithm and the read range comprises:

Call the read-ahead algorithm, and calculate the data segment involved in the read-ahead data by passing in the offset of the current read data and the length of the read data according to the read range.
The method for improving soft copy read performance according to claim 2, wherein the cyclically pre-reading the corresponding source file data according to the data segment and placing it in the cache comprises:

Find the source file corresponding to each pre-read data segment, read the source file data of each data segment asynchronously and load it into the cache, and wait for the next time it is directly read from the cache, put it into the default cache elimination queue for cache update.
A system for improving soft copy read performance, comprising:

The file reading unit is configured to read the soft copy file and associate it with the source file data within the read range;

The data pre-reading unit is configured to calculate the data segments involved in the pre-reading data according to the pre-reading algorithm and the read range;

a cache writing unit, configured to cyclically pre-read the corresponding source file data according to the data segment and put it into the cache;

The cache elimination unit is configured to move the data in the cache according to the data cache elimination algorithm.
A terminal, characterized in that it includes:

processor;

memory for storing instructions for execution of the processor;

wherein the processor is configured to perform the method of any one of claims 1-4.
A computer-readable storage medium storing a computer program, characterized in that, when the program is executed by a processor, the method according to any one of claims 1-4 is implemented.