CN113032335A - File access method, device, equipment and storage medium - Google Patents
File access method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN113032335A CN113032335A CN202110561924.9A CN202110561924A CN113032335A CN 113032335 A CN113032335 A CN 113032335A CN 202110561924 A CN202110561924 A CN 202110561924A CN 113032335 A CN113032335 A CN 113032335A
- Authority
- CN
- China
- Prior art keywords
- file
- server
- specified
- specified file
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0629—Configuration or reconfiguration of storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/505—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the load
Abstract
The application provides a file access method, a file access device, electronic equipment and a storage medium. The specific implementation scheme is as follows: responding to a received file access request of an application program, and calling a program interface to acquire a specified file corresponding to the file access request; sending request information to a server; the request information comprises metadata information of the specified file, and the metadata information comprises storage position information of the specified file in the server; receiving data of a designated file from a server; data specifying the file is returned to the application. The embodiment of the application can improve the reading performance of the small files in the process of artificial intelligence model training, thereby improving the access efficiency of the small files, further improving the service efficiency of the whole storage system, and accelerating the training speed of the artificial intelligence model.
Description
Technical Field
The present application relates to the field of artificial intelligence technologies, and in particular, to a file access method, apparatus, device, and storage medium.
Background
With the development of Artificial Intelligence technology, the number of files required for training an AI (Artificial Intelligence) model is also increasing dramatically. In the data set for AI model training, small files account for a considerable proportion. However, when accessing data of a small file, there are generally problems of strong randomness, small IO granularity, and low disk efficiency. In the process of training the AI model, the small files are read at a low speed, so that the access efficiency of the massive small files is low, the access time is long, and the use efficiency of the whole storage system and the training speed of the AI model are reduced.
Disclosure of Invention
The embodiment of the application provides a file access method, a file access device, an electronic device and a storage medium, which are used for solving the problems in the related art, and the technical scheme is as follows:
in a first aspect, an embodiment of the present application provides a file access method, including:
responding to a received file access request of an application program, and calling a program interface to acquire a specified file corresponding to the file access request;
sending request information to a server; the request information comprises metadata information of the specified file, and the metadata information comprises storage position information of the specified file in the server;
receiving data of a designated file from a server;
data specifying the file is returned to the application.
In one embodiment, the file access request includes an execution task; responding to a received file access request of an application program, calling a program interface to acquire a specified file corresponding to the file access request, wherein the method comprises the following steps:
acquiring an execution task from the file access request;
calling a program interface, acquiring a designated file corresponding to the execution task, and taking the designated file corresponding to the execution task as a designated file corresponding to the file access request; wherein, the assigned file corresponding to the executed task is set in advance through a program interface.
In one embodiment, the method further comprises:
sending a metadata information acquisition request to a server; the metadata information acquisition request is used for acquiring metadata information of all files in the data set mounted by the application program;
and receiving and storing the metadata information returned by the server.
In one embodiment, receiving data from a server specifying a file comprises:
reading all fragment files forming the designated file from a plurality of servers in parallel by adopting multiple threads;
and recombining each fragmented file by adopting a fragmented recombination algorithm.
In one embodiment, the method further comprises:
and storing the data of the designated file into the cache under the condition that the size of the data of the designated file is less than or equal to a preset threshold value.
In one embodiment, sending the request information to the server further comprises:
searching a designated file in a cache;
under the condition that the search is successful, accessing the specified file from the cache;
and in the case of failure of searching, sending request information to the server.
In one embodiment, accessing the designated files from the cache includes pre-reading at least one of the designated files in accordance with a predetermined access sequence in at least one of:
random reading, reading according to the file name sorting sequence, and determining the file name to be read by utilizing the Hash index.
In one embodiment, returning data specifying the file to the application further comprises:
in the case that the current file in the at least one designated file fails to be accessed, accessing the designated file subsequent to the current file in the access sequence;
the data of the specified file following the current file is returned to the application.
In a second aspect, an embodiment of the present application provides a file access method, including:
receiving request information of a specified file from a client;
acquiring metadata information of the specified file from the request information; the metadata information includes storage location information of the specified file in the server;
acquiring data of the designated file according to the storage position information;
and returning the data of the specified file to the client.
In one embodiment, the method further comprises:
and responding to a received metadata information acquisition request from the client, and returning metadata information of all files in the data set mounted by the application program to the client, wherein the application program is the application program requesting to access the specified file at the client.
In a third aspect, an embodiment of the present application provides a file access apparatus, including:
the calling unit is used for responding to the received file access request of the application program and calling a program interface to acquire an appointed file corresponding to the file access request;
a sending unit, configured to send request information to a server; the request information comprises metadata information of the specified file, and the metadata information comprises storage position information of the specified file in the server;
a first receiving unit for receiving data of a specified file from a server;
and the first returning unit is used for returning the data of the specified file to the application program.
In one embodiment, the file access request includes an execution task; the calling unit is used for:
acquiring an execution task from the file access request;
calling a program interface, acquiring a designated file corresponding to the execution task, and taking the designated file corresponding to the execution task as a designated file corresponding to the file access request; wherein, the assigned file corresponding to the executed task is set in advance through a program interface.
In one embodiment, the apparatus further includes a first obtaining unit, configured to:
sending a metadata information acquisition request to a server; the metadata information acquisition request is used for acquiring metadata information of all files in the data set mounted by the application program;
and receiving and storing the metadata information returned by the server.
In one embodiment, the first receiving unit is configured to:
reading all fragment files forming the designated file from a plurality of servers in parallel by adopting multiple threads;
and recombining each fragmented file by adopting a fragmented recombination algorithm.
In one embodiment, the apparatus further includes a storage unit, the storage unit is configured to:
and storing the data of the designated file into the cache under the condition that the size of the data of the designated file is less than or equal to a preset threshold value.
In one embodiment, the transmitting unit includes:
the searching subunit is used for searching the designated file in the cache;
the access subunit is used for accessing the specified file from the cache under the condition of successful searching;
and the sending subunit is used for sending the request information to the server under the condition of failure in searching.
In one embodiment, the access subunit is configured to pre-read the at least one specified file according to a predetermined access sequence in at least one of the following ways:
random reading, reading according to the file name sorting sequence, and determining the file name to be read by utilizing the Hash index.
In one embodiment, the first return unit is further configured to:
in the case that the current file in the at least one designated file fails to be accessed, accessing the designated file subsequent to the current file in the access sequence;
the data of the specified file following the current file is returned to the application.
In a fourth aspect, an embodiment of the present application provides a file access apparatus, including:
a second receiving unit configured to receive request information for a specified file from a client;
a second acquisition unit configured to acquire metadata information of the specified file from the request information; the metadata information includes storage location information of the specified file in the server;
a third acquisition unit configured to acquire data of the specified file based on the storage location information;
and the second returning unit is used for returning the data of the specified file to the client.
In one embodiment, the apparatus further includes a third returning unit, and the third returning unit is configured to:
and responding to a received metadata information acquisition request from the client, and returning metadata information of all files in the data set mounted by the application program to the client, wherein the application program is the application program requesting to access the specified file at the client.
In a fifth aspect, an embodiment of the present application provides an electronic device, where the electronic device includes: a memory and a processor. Wherein the memory and the processor are in communication with each other via an internal connection path, the memory is configured to store instructions, the processor is configured to execute the instructions stored by the memory, and the processor is configured to perform the method of any of the above aspects when the processor executes the instructions stored by the memory.
In a sixth aspect, embodiments of the present application provide a computer-readable storage medium, which stores a computer program, and when the computer program runs on a computer, the method in any one of the above-mentioned aspects is executed.
The advantages or beneficial effects in the above technical solution at least include: the reading performance of the small files can be improved in the process of artificial intelligence model training, so that the access efficiency of the small files is improved, the service efficiency of the whole storage system is improved, and the training speed of the artificial intelligence model is increased.
The foregoing summary is provided for the purpose of description only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features of the present application will be readily apparent by reference to the drawings and following detailed description.
Drawings
In the drawings, like reference numerals refer to the same or similar parts or elements throughout the several views unless otherwise specified. The figures are not necessarily to scale. It is appreciated that these drawings depict only some embodiments in accordance with the disclosure and are therefore not to be considered limiting of its scope.
Fig. 1 is a flowchart of a file access method applied to a client according to an embodiment of the present application;
FIG. 2 is a flow chart of a file access method applied to a client according to another embodiment of the present application;
FIG. 3 is a flow chart of a file access method applied to a client according to another embodiment of the present application;
FIG. 4 is a flow chart of a file access method applied to a client according to another embodiment of the present application;
FIG. 5 is a diagram illustrating an exemplary file system structure of a file access method applied to a client according to another embodiment of the present application;
FIG. 6 is a diagram illustrating an example of a file slice of a file access method according to another embodiment of the present application;
FIG. 7 is an exemplary diagram of cache pools and thread pools for a file access method applied to a client according to another embodiment of the present application;
FIG. 8 is a diagram of a virtual network file storage system architecture for a file access method according to another embodiment of the present application;
FIG. 9 is a flow chart of a file access method applied to a server according to another embodiment of the present application;
FIG. 10 is a schematic structural diagram of a file access device applied to a client according to another embodiment of the present application;
FIG. 11 is a schematic structural diagram of a file access device applied to a client according to another embodiment of the present application;
FIG. 12 is a schematic structural diagram of a file access device applied to a client according to another embodiment of the present application;
fig. 13 is a schematic structural diagram of a sending unit of a file access device applied to a client according to another embodiment of the present application;
fig. 14 is a schematic structural diagram of a file access device according to another embodiment of the present application, applied to a server;
fig. 15 is a schematic structural diagram of a file access device according to another embodiment of the present application, applied to a server;
FIG. 16 is a block diagram of an electronic device used to implement embodiments of the present application.
Detailed Description
In the following, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments may be modified in various different ways, all without departing from the spirit or scope of the present application. Accordingly, the drawings and description are to be regarded as illustrative in nature, and not as restrictive.
Fig. 1 is a flowchart of a file access method applied to a client according to an embodiment of the present application. As shown in fig. 1, the file access method may be applied to a client, and specifically may include:
step S110, responding to the received file access request of the application program, and calling a program interface to obtain a specified file corresponding to the file access request;
step S115, sending request information to a server; the request information comprises metadata information of the specified file, and the metadata information comprises storage position information of the specified file in the server;
step S120, receiving data of a specified file from a server;
step S130, returns the data of the designated file to the application program.
During the process of AI model training, the file can be stored and managed by using a Ceph network file system. Ceph is a distributed file system. When large files are processed, efficient storage and management of the files can be achieved by using Ceph. However, when storing a large amount of small files, Ceph has low storage efficiency.
In the distributed file system, the metadata information records the mapping relationship between the data logic and the physical position, and the information such as the attribute, the access authority and the like of the file. In the related art, when accessing a file, a metadata server needs to be requested to query corresponding metadata information, and then subsequent I/O (Input/Output) operations such as file reading and writing are performed through the obtained metadata information.
In the file access method of the embodiment of the application, all metadata information under all folders which may be accessed by an application program can be obtained in advance, and all metadata information is cached at a client. When the application program initiates a file access request, the file object is read to the backend distributed Ceph network file system by using the metadata information. In the embodiment of the application, the metadata information is used for accessing the files in the Ceph network file system, the metadata server does not need to be requested to inquire the corresponding metadata information when the files are accessed, the performance influence caused by inquiring the metadata server of the Ceph file system can be avoided, the access efficiency of massive small files is improved, the use efficiency of the whole storage system is improved, and the training speed of AI is accelerated.
In an application scenario of AI model training, when an AI training data set is mounted, all metadata information of the data set can be obtained from the Ceph server cluster and cached at a client. The metadata information may include, among other things, information specifying where the file is stored in the server, such as which physical machine of the server cluster the file is stored on. When an upper-layer user for training the AI model needs to access the training data set, a file access request of the application program to the specified file can be sent to the client through the application program trained by the AI model.
In the embodiment of the application, a program interface can be arranged at the client. The upper layer user can set the appointed files which need to be accessed when the application program trains the AI model by using the program interface in advance. For example, a file list of specified files that need to be accessed to train the AI model can be set. As another example, in an application scenario for training an image processing model, the designated files to be accessed may be a plurality of sample picture files used for model training.
In step S110, after receiving the file access request of the application program for the specified file, the client calls the program interface to obtain the specified file corresponding to the file access request. And then the metadata information of the specified file is found in the cache. In step S115, the client transmits request information to the server. The server may be a Ceph server cluster or an object storage server of a distributed file system. The request information carries metadata information of the designated file, so that the server can quickly acquire the data of the designated file according to the storage position information in the metadata information.
And after acquiring the data of the specified file, the server returns the data of the specified file to the client. In step S120, the client receives data specifying a file from the server. In step S130, the client returns data specifying a file to the AI model trained application.
The embodiment of the application can improve the reading performance of the small files in the process of artificial intelligence model training, thereby improving the access efficiency of the small files, further improving the service efficiency of the whole storage system, and accelerating the training speed of the artificial intelligence model.
In one embodiment, the file access request includes an execution task; responding to a received file access request of an application program, calling a program interface to acquire a specified file corresponding to the file access request, wherein the method comprises the following steps:
acquiring an execution task from the file access request;
calling a program interface, acquiring a designated file corresponding to the execution task, and taking the designated file corresponding to the execution task as a designated file corresponding to the file access request; wherein, the assigned file corresponding to the executed task is set in advance through a program interface.
In the embodiment of the application, in the process of executing each execution task, the application program may need to access the designated file corresponding to each execution task. The upper layer user can set the specific file to be accessed for each execution task by using the program interface in advance. In one example, the execution tasks may include a model training task a and a model training task B. Model training task A requires access to training set A1 and training set A2. Model training task B requires access to training set B1 and training set B2. Corresponding designated files may be set at the client for model training task A as training set A1 and training set A2, and corresponding designated files for model training task B as training set B1 and training set B2, using the program interface.
When the application program initiates a file access request to the client, the information for executing the task can be carried in the file access request. The client can obtain the execution task from the file access request, and then call the program interface to obtain the designated file corresponding to the execution task.
Fig. 2 is a flowchart of a file access method applied to a client according to another embodiment of the present application. As shown in fig. 2, in one embodiment, the method further comprises:
in step S140, the data of the designated file is stored in the cache if the size of the data of the designated file is equal to or less than the predetermined threshold.
In such an embodiment, a threshold value of the file data size may be set in advance. And in the case that the size of the data of the specified file is larger than a preset threshold value, reading the data of the specified file from the server and directly returning the data of the specified file to the application program. In this case, the designated file belongs to a large file, and the large file is not cached at the client. In the case where the size of the data of the specified file is equal to or less than a predetermined threshold, it may be determined that the request to access the specified file belongs to scattered small files. In this case, the small file can be read from the server to the small file cache pool of the client according to a pre-configured small file loading mechanism. When the upper layer user requests to read, the file content in the cache pool can be directly returned. The method can reduce direct interaction between the user and the server cluster, and can improve the reading efficiency of the small files.
During the course of AI model training, small files are accessed relatively more frequently than large files. For example, the validation set used for verification belongs to a large file, and is relatively infrequent to access. The small files are cached in the cache pool of the client, so that the access efficiency of the small files can be improved, and the training speed of the artificial intelligence model is further accelerated.
In the embodiment of the application, the upper layer user can set the specified file to be accessed for each execution task by using the program interface in advance. Also, it is also possible to set the access order of the designated files, that is, the access sequence of the predetermined files. The client may store data of the specified file in the cache according to a predetermined access sequence through the program interface. In step S130, the client may also return the data of the specified file to the application program according to a predetermined access sequence.
Fig. 3 is a flowchart of a file access method applied to a client according to another embodiment of the present application. As shown in fig. 3, in an embodiment, step S115 in fig. 1, sending the request information to the server, further includes:
step S111, searching for a designated file in a cache;
step S112, under the condition of successful searching, accessing the specified file from the cache;
in step S113, in the case where the search fails, request information is transmitted to the server.
When a user sends a file access request, the client can judge whether the requested specified file is in the cache of the client. And if the specified file is in the cache of the client, directly accessing the specified file from the cache of the client. And if the specified file is not in the cache of the client, the client uploads the request information to the server. In one example, the server may be a Ceph server cluster. And the client reads the specified file from the rear-end storage system of the Ceph server cluster by sending the request information.
In one example, the client may obtain all metadata information under a folder related to the file access request from the server in advance and cache the metadata information at the client, and then perform associated sorting on the small files according to the obtained metadata information. When a user initiates a file access request, the system may check to see if the requested file is in the cache. If the requested file is in the cache, the requested file can be quickly accessed from the cache. And if the request file is not in the cache, sending request information to the server. In one embodiment, accessing the designated files from the cache includes pre-reading at least one of the designated files in accordance with a predetermined access sequence in at least one of:
random reading, reading according to the file name sorting sequence, and determining the file name to be read by utilizing the Hash index.
In one example, a specified file is accessed from a cache, and the specified file may be pre-read according to a predetermined access sequence. For example, the predetermined access sequence includes accessing file a, file B, file C, and file D in sequence. Then file B, file C, and file D may be pre-read when file a is read in the cache.
Still taking the application scenario of AI model training as an example, in the above random reading manner, the files in the training set can be read randomly. For example, a file capable of being read quickly is selected from the training set and provided to the AI model for training. In another example, the pre-reading may also be performed by using a hash index algorithm, and the file name of the next pre-reading file is calculated according to the file name of the currently read file.
In the embodiment of the application, the file data set can be read into the cache in advance according to the file name sorting by configuring the user directory specified by the pre-cache. The application program can select a proper pre-caching algorithm to pre-read the file in the cache according to the configuration characteristics of the training data set file. According to the embodiment of the application, the access time of a user and a server cluster is reduced at the cost of sacrificing a certain cache through a preset pre-reading cache strategy, and the access efficiency of massive small files is improved.
In the embodiment of the application, in the AI model training process, the access and reading method of the small file is optimized under the condition that the same file needs to be accessed for multiple times and the loading sequence of the file can be predicted in advance for model training to realize pre-caching. On one hand, the purpose of improving the access efficiency is achieved, on the other hand, metadata information in the back-end server cluster does not need to be modified, and extra processing overhead and burden are not added to the back-end server cluster.
In one embodiment, returning data specifying the file to the application further comprises:
in the case that the current file in the at least one designated file fails to be accessed, accessing the designated file subsequent to the current file in the access sequence;
the data of the specified file following the current file is returned to the application.
In one example, specifying the predetermined access sequence for the file includes accessing file a, file B, file C, and file D in that order. In an application scenario of AI model training, the specification file may be data of a model training sample. In this case, the access orders of the designated files can be exchanged, and the training effect of the model is not affected after the access orders are exchanged. For example, if the current file is file a and the current file fails to be accessed, the case of the file a failing to be accessed may be recorded first. And then selecting the next appointed file for access from appointed files after the current file of the access sequence by using a random selection mode or a hash function. For example, file B may be selected for preferential access. And after the file B is successfully accessed and the data of the file B is returned to the application program, the file A is accessed.
In the embodiment of the application, when the current file in at least one designated file fails to be accessed, the access sequence of the designated file can be reversely replaced, and the designated file after the current file in the preset access sequence is accessed. The method can accelerate the access speed of the file, further improve the use efficiency of the whole storage system and accelerate the training speed of the artificial intelligence model.
Fig. 4 is a flowchart of a file access method applied to a client according to another embodiment of the present application. As shown in fig. 4, in one embodiment, the method further includes:
step S101, sending a metadata information acquisition request to a server; the metadata information acquisition request is used for acquiring metadata information of all files in the data set mounted by the application program;
and step S102, receiving and storing the metadata information returned by the server.
Still taking an application scenario of AI model training as an example, when an AI training is performed to mount a data set, all metadata information of the data set is obtained from a Ceph-capable server cluster and cached at a client. In step S101, the client sends a metadata information acquisition request to the server, and acquires metadata information of all files in the application-mounted data set from the server. After receiving the metadata information acquisition request, the server searches metadata distribution information of the Ceph distributed storage corresponding to all files in the data set, and uses the metadata distribution information as metadata information returned to the client. In step S102, the client receives and stores the metadata information returned by the server.
In an application scenario of AI model training, when an AI trains to mount a data set, an application program may configure a reading behavior characteristic of the data set according to the characteristic of the data set when mounting the data set, and mount a data set file that is indicated in parameter configuration and needs to be pre-cached. In one example, metadata information for an entire data set may be cached asynchronously from a root directory. Under the condition that the metadata of the file needing to be accessed is cached in the client, when the application program conducts AI training for file reading, the metadata of the file can be directly obtained from the cache of the client without requesting the metadata of the file from a metadata server.
Fig. 5 is a diagram illustrating an example of a file system structure of a file access method applied to a client according to another embodiment of the present application. The "/" in fig. 5 denotes a root directory; "dense" denotes a directory entry; "File 1-inode" represents an inode for File 1; "File 2-inode" represents an inode for File 2; "File 3-inode" represents an inode for File 3. As shown in FIG. 5, in one example, in a linux operating system, the size of each inode node (index node) may be 128 bytes or 256 bytes. The sum of the number of most dentry (directory entries) and the number of inodes of a single dataset for AI training may not typically exceed 100 million. The cache cost of the entire inode table may not typically exceed 512 MB. By reasonably using the cache with the size of 512MB, the load pressure of a metadata server can be greatly reduced, and the access speed of the file is improved.
In one embodiment, step S120 in fig. 1, receiving data specifying a file from a server, includes:
reading all fragment files forming the designated file from a plurality of servers in parallel by adopting multiple threads;
and recombining each fragmented file by adopting a fragmented recombination algorithm.
Fig. 6 is a diagram illustrating an example of a file fragment of a file access method according to another embodiment of the present application. As shown in FIG. 6, File1 includes sharded files block1, block2, block3, block4, … …, and block n. The sharded file block1 comprises a file metadata block1-meta and a file data block 1-data; the sharded file block2 comprises a file metadata block2-meta and a file data block 2-data; sharded file block3 includes file metadata block3-meta and file data block 3-data. In one example, the client receives data of a designated file from the server, obtains file fragments of the designated file in the Ceph object storage, reads each fragment in parallel by adopting multiple threads, and recombines each fragment in the memory by adopting a fragment recombination algorithm. For example, TCP (Transmission Control Protocol) sliding window algorithm may be used to reassemble each fragment.
Fig. 7 is a diagram illustrating an example of a cache pool and a thread pool of a file access method according to another embodiment of the present application. As shown in fig. 7, if the size of the data of the File1, the File2, and the File3 is equal to or smaller than a predetermined threshold, the above three files belong to small files. The sizes of the data of the File4 and the File5 are greater than a predetermined threshold, the above two files belong to a large File. In the embodiment of the application, for the small file, the small file can be read from a server to be placed in a small file cache pool of a client; and for the large file, not caching in the cache pool of the client. For small files File1, File2 and File3, multiple threads are started from a thread pool respectively, and files are acquired from a distributed network storage cluster in parallel by using multiple threads.
Fig. 8 is a diagram of a virtual network file storage system architecture for a file access method according to another embodiment of the present application. The "Ceph distributed object storage system" in fig. 8 is a remote server storage cluster; "Cephfs Kernel Object" represents a Cephfs Kernel Object; "librados" represents a kernel-mode interface for obtaining training data. As shown in fig. 8, a user obtains a training file of an AI model through a virtual IO read-write interface of a client. The virtual file system read-write driver module can be a kernel-state driver module of the client. The kernel mode driver module is a file read-write driver of the operating system. The kernel mode driving module can comprise a metadata cache pool, a cache strategy management submodule, a multi-file parallel reading submodule and a single-file parallel reading submodule. The single-file parallel reading submodule can be used for reading all the fragment files forming a single file in a multi-thread parallel mode.
In the embodiment of the application, each fragmented file of the designated file may be stored in one server, or may be stored in a plurality of servers of the server cluster respectively. For example, the sharded files of a specific file may be stored in respective corresponding servers. Meanwhile, each fragment file forming the appointed file is read from a plurality of servers in parallel, so that the reading speed of the file can be increased, and the execution efficiency of the system can be improved.
In addition, backup files of respective fragmented files of the designated file may be stored in a plurality of servers other than the respective servers. The file backup can avoid data loss and task blockage caused by the failure of a certain server. The stability and the data safety of the system can be effectively improved through file backup.
In one embodiment, the client may use a standard Unix POSIX (Portable Operating System Interface) Interface to read the file. Software portability at the source code level is available with POSIX. A program written for one POSIX-compatible operating system may be compiled for execution on any other POSIX operating system. In the embodiment of the application, a user only needs to select a proper caching algorithm according to the actual characteristics of the data set when the data set is mounted, and the read-write interface is not changed, so that the use and transplantation cost of upper-layer users is greatly reduced.
The embodiment of the application provides a file access method based on parallel pre-reading, which is realized in a client kernel mode, and can utilize resources of a storage cluster to the maximum extent, reduce storage cost expenditure in AI training and improve the reading speed of small files in the AI model training process. And the interface level is completely compatible with standard Unix Posix system call, and the learning and use cost of a user does not need to be increased.
In summary, the file access method according to the embodiment of the present application has the advantages that:
1) and optimizing the client reading module aiming at the condition that the Cephfs metadata server has lower performance. According to the reading behavior characteristics of the preset data set, the metadata information is cached in full, and the problem that the metadata server of the Cephfs is low in performance is solved.
2) The existing Linux file read-write kernel mainly aims at a traditional HDD (Hard Disk Drive, abbreviation) storage Disk, and does not optimize a network file system. According to the embodiment of the application, the files are acquired from the network storage cluster in parallel by using multiple threads through the preset parallel cache algorithm, and the reading capacity of the client is improved.
3) For a single file, combining with the object storage characteristics of the Ceph bottom layer, different blocks of the file are obtained in parallel by using multiple threads, and the single file reading capability of the client is improved.
Fig. 9 is a flowchart of a file access method applied to a server according to another embodiment of the present application. As shown in fig. 9, the file access method may be applied to a server, and specifically may include:
step S210, receiving request information for a specified file from a client;
step S220, obtaining metadata information of the specified file from the request information; the metadata information includes storage location information of the specified file in the server;
step S230, acquiring data of the designated file according to the storage position information;
step S240, the data of the designated file is returned to the client.
After receiving a file access request of an application program to a specified file, a client sends request information to a server. Wherein, the request information carries metadata information of the specified file. In step S210, the server receives request information for a specified file from the client. In step S220, the server acquires metadata information specifying a file from the request information. In step S230, the server can quickly acquire the data of the specified file according to the storage location information of the specified file in the metadata information. In step S240, after acquiring the data of the designated file, the server returns the data of the designated file to the client.
In the embodiment of the application, after receiving the request information of the specified file from the client, the server can access the file to the back-end file system by using the metadata information. When the files are accessed, the metadata server does not need to be requested to inquire the corresponding metadata information, the performance influence caused by inquiring the metadata server of the distributed file system can be avoided, the access efficiency of massive small files is improved, the use efficiency of the whole storage system is improved, and the training speed of the AI is accelerated.
In one embodiment, the method further comprises:
and responding to a received metadata information acquisition request from the client, and returning metadata information of all files in the data set mounted by the application program to the client, wherein the application program is the application program requesting to access the specified file at the client.
Referring back to fig. 4, in step S101, the client sends a metadata information acquisition request to the server, and acquires metadata information of all files in the dataset on which the application is mounted from the server. After receiving the metadata information acquisition request, the server searches metadata distribution information of the Ceph distributed storage corresponding to all files in the data set, and uses the metadata distribution information as metadata information returned to the client. In step S102, the client receives and stores the metadata information returned by the server. In the subsequent flow, after receiving a file access request of an application program to a specified file, a client can send request information carrying metadata information of the specified file to a server, so that the server can quickly acquire data of the specified file according to storage position information in the metadata information.
The file access method applied to the server in the embodiment of the present application may refer to the corresponding description in the file access method applied to the client, which is not described herein again.
Fig. 10 is a schematic structural diagram of a file access device applied to a client according to another embodiment of the present application. As shown in fig. 10, the apparatus may include:
the calling unit 100 is configured to respond to a received file access request of an application program, and call a program interface to obtain an appointed file corresponding to the file access request;
a sending unit 102, configured to send request information to a server; the request information comprises metadata information of the specified file, and the metadata information comprises storage position information of the specified file in the server;
a first receiving unit 200 for receiving data of a specified file from a server;
the first returning unit 300 is used for returning the data of the specified file to the application program.
In one embodiment, the file access request includes an execution task; the calling unit 100 is configured to:
acquiring an execution task from the file access request;
calling a program interface, acquiring a designated file corresponding to the execution task, and taking the designated file corresponding to the execution task as a designated file corresponding to the file access request; wherein, the assigned file corresponding to the executed task is set in advance through a program interface.
Fig. 11 is a schematic structural diagram of a file access device applied to a client according to another embodiment of the present application. As shown in fig. 11, in an embodiment, the apparatus further includes a first obtaining unit 105, where the first obtaining unit 105 is configured to:
sending a metadata information acquisition request to a server; the metadata information acquisition request is used for acquiring metadata information of all files in the data set mounted by the application program;
and receiving and storing the metadata information returned by the server.
In one embodiment, the first receiving unit 200 is configured to:
reading all fragment files forming the designated file from a plurality of servers in parallel by adopting multiple threads;
and recombining each fragmented file by adopting a fragmented recombination algorithm.
Fig. 12 is a schematic structural diagram of a file access device applied to a client according to another embodiment of the present application. As shown in fig. 12, in an embodiment, the apparatus further includes a storage unit 400, where the storage unit 400 is configured to:
and storing the data of the designated file into the cache under the condition that the size of the data of the designated file is less than or equal to a preset threshold value.
Fig. 13 is a schematic structural diagram of a sending unit of a file access device applied to a client according to another embodiment of the present application. As shown in fig. 13, in one embodiment, the transmitting unit 100 includes:
a searching subunit 110, configured to search for the specified file in the cache;
an accessing subunit 120, configured to, in a case that the search is successful, access the specified file from the cache;
a sending subunit 130, configured to send the request information to the server in a case that the lookup fails.
In one embodiment, the access subunit 120 is configured to pre-read at least one specified file according to a predetermined access sequence in at least one of the following ways:
random reading, reading according to the file name sorting sequence, and determining the file name to be read by utilizing the Hash index.
In one embodiment, the first returning unit 300 is further configured to:
in the case that the current file in the at least one designated file fails to be accessed, accessing the designated file subsequent to the current file in the access sequence;
the data of the specified file following the current file is returned to the application.
Fig. 14 is a schematic structural diagram of a file access device according to another embodiment of the present application, applied to a server. As shown in fig. 14, the apparatus may include:
a second receiving unit 500 for receiving request information for a specified file from a client;
a second obtaining unit 600 for obtaining metadata information of the specified file from the request information; the metadata information includes storage location information of the specified file in the server;
a third acquiring unit 700 for acquiring data of the specified file according to the storage location information;
and a second returning unit 800, configured to return the data of the specified file to the client.
Fig. 15 is a schematic structural diagram of a file access device according to another embodiment of the present application, applied to a server. As shown in fig. 15, in an embodiment, the apparatus further includes a third returning unit 900, where the third returning unit 900 is configured to:
and responding to a received metadata information acquisition request from the client, and returning metadata information of all files in the data set mounted by the application program to the client, wherein the application program is the application program requesting to access the specified file at the client.
The functions of each module, unit, and sub-unit in each device in the embodiments of the present application may refer to the corresponding description in the above method, and are not described herein again.
FIG. 16 is a block diagram of an electronic device used to implement embodiments of the present application. As shown in fig. 16, the electronic apparatus includes: a memory 910 and a processor 920, the memory 910 having stored therein computer programs operable on the processor 920. The processor 920 implements the file access and the file access method in the above-described embodiments when executing the computer program. The number of the memory 910 and the processor 920 may be one or more.
The electronic device further includes:
and a communication interface 930 for communicating with an external device to perform data interactive transmission.
If the memory 910, the processor 920 and the communication interface 930 are implemented independently, the memory 910, the processor 920 and the communication interface 930 may be connected to each other through a bus and perform communication with each other. The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended ISA (EISA) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown in FIG. 16, but this is not intended to represent only one bus or type of bus.
Optionally, in an implementation, if the memory 910, the processor 920 and the communication interface 930 are integrated on a chip, the memory 910, the processor 920 and the communication interface 930 may complete communication with each other through an internal interface.
Embodiments of the present application provide a computer-readable storage medium, which stores a computer program, and when the program is executed by a processor, the computer program implements the method provided in the embodiments of the present application.
The embodiment of the present application further provides a chip, where the chip includes a processor, and is configured to call and execute the instruction stored in the memory from the memory, so that the communication device in which the chip is installed executes the method provided in the embodiment of the present application.
An embodiment of the present application further provides a chip, including: the system comprises an input interface, an output interface, a processor and a memory, wherein the input interface, the output interface, the processor and the memory are connected through an internal connection path, the processor is used for executing codes in the memory, and when the codes are executed, the processor is used for executing the method provided by the embodiment of the application.
It should be understood that the processor may be a Central Processing Unit (CPU), other general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, etc. A general purpose processor may be a microprocessor or any conventional processor or the like. It is noted that the processor may be an advanced reduced instruction set machine (ARM) architecture supported processor.
Further, optionally, the memory may include a read-only memory and a random access memory, and may further include a nonvolatile random access memory. The memory may be either volatile memory or nonvolatile memory, or may include both volatile and nonvolatile memory. The non-volatile memory may include a read-only memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an electrically Erasable EPROM (EEPROM), or a flash memory. Volatile memory can include Random Access Memory (RAM), which acts as external cache memory. By way of example, and not limitation, many forms of RAM are available. For example, Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), double data rate synchronous SDRAM (DDR SDRAM), Enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct memory bus RAM (DR RAM).
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. The procedures or functions according to the present application are generated in whole or in part when the computer program instructions are loaded and executed on a computer. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium.
In the description herein, references to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the application. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, various embodiments or examples and features of different embodiments or examples described in this specification can be combined and combined by one skilled in the art without contradiction.
Furthermore, the terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present application, "a plurality" means two or more unless specifically limited otherwise.
Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps of the process. And the scope of the preferred embodiments of the present application includes other implementations in which functions may be performed out of the order shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved.
The logic and/or steps represented in the flowcharts or otherwise described herein, e.g., an ordered listing of executable instructions that can be considered to implement logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions.
It should be understood that portions of the present application may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. All or part of the steps of the method of the above embodiments may be implemented by hardware that is configured to be instructed to perform the relevant steps by a program, which may be stored in a computer-readable storage medium, and which, when executed, includes one or a combination of the steps of the method embodiments.
In addition, functional units in the embodiments of the present application may be integrated into one processing module, or each unit may exist alone physically, or two or more units are integrated into one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. The integrated module may also be stored in a computer-readable storage medium if it is implemented in the form of a software functional module and sold or used as a separate product. The storage medium may be a read-only memory, a magnetic or optical disk, or the like.
The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive various changes or substitutions within the technical scope of the present application, and these should be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Claims (22)
1. A method for accessing a file, comprising:
responding to a received file access request of an application program, and calling a program interface to acquire an appointed file corresponding to the file access request;
sending request information to a server; the request information comprises metadata information of the specified file, and the metadata information comprises storage position information of the specified file in the server;
receiving data of the specified file from the server;
and returning the data of the specified file to the application program.
2. The method of claim 1, wherein the file access request includes an execution task; responding to a received file access request of an application program, calling a program interface to acquire a specified file corresponding to the file access request, wherein the method comprises the following steps:
acquiring the execution task from the file access request;
calling the program interface, acquiring a designated file corresponding to the execution task, and taking the designated file corresponding to the execution task as a designated file corresponding to the file access request; and the specified file corresponding to the executed task is set in advance through the program interface.
3. The method according to claim 1 or 2, characterized in that the method further comprises:
sending a metadata information acquisition request to a server; the metadata information acquisition request is used for acquiring metadata information of all files in the data set mounted by the application program;
and receiving and storing the metadata information returned by the server.
4. The method of claim 1 or 2, wherein receiving data of the specified file from the server comprises:
reading all fragment files forming the specified file from a plurality of servers in parallel by adopting multiple threads;
and adopting a fragment reassembling algorithm to reassemble the fragment files.
5. The method according to claim 1 or 2, characterized in that the method further comprises:
and storing the data of the specified file into a cache under the condition that the size of the data of the specified file is less than or equal to a preset threshold value.
6. The method of claim 5, wherein sending the request information to the server further comprises:
searching the designated file in the cache;
under the condition that the search is successful, accessing the specified file from the cache;
and sending request information to the server under the condition of failure of searching.
7. The method of claim 6, wherein accessing the designated files from the cache comprises pre-reading at least one of the designated files according to a predetermined access sequence in at least one of:
random reading, reading according to the file name sorting sequence, and determining the file name to be read by utilizing the Hash index.
8. The method of claim 7, wherein returning data of the specified file to the application further comprises:
in the case that the current file in at least one specified file fails to be accessed, accessing a specified file subsequent to the current file in the access sequence;
and returning the data of the specified file after the current file to the application program.
9. A method for accessing a file, comprising:
receiving request information of a specified file from a client;
acquiring metadata information of the specified file from the request information; the metadata information comprises the storage position information of the specified file in the server;
acquiring data of the designated file according to the storage position information;
and returning the data of the specified file to the client.
10. The method of claim 9, further comprising:
and responding to a received metadata information acquisition request from the client, and returning metadata information of all files in the data set mounted by the application program to the client, wherein the application program is the application program requesting to access the specified file at the client.
11. A file access apparatus, comprising:
the calling unit is used for responding to a received file access request of an application program and calling a program interface to acquire an appointed file corresponding to the file access request;
a sending unit, configured to send request information to a server; the request information comprises metadata information of the specified file, and the metadata information comprises storage position information of the specified file in the server;
a first receiving unit configured to receive data of the specified file from the server;
and the first returning unit is used for returning the data of the specified file to the application program.
12. The apparatus of claim 11, wherein the file access request includes an execution task; the calling unit is used for:
acquiring the execution task from the file access request;
calling the program interface, acquiring a designated file corresponding to the execution task, and taking the designated file corresponding to the execution task as a designated file corresponding to the file access request; and the specified file corresponding to the executed task is set in advance through the program interface.
13. The apparatus according to claim 11 or 12, further comprising a first obtaining unit configured to:
sending a metadata information acquisition request to a server; the metadata information acquisition request is used for acquiring metadata information of all files in the data set mounted by the application program;
and receiving and storing the metadata information returned by the server.
14. The apparatus according to claim 11 or 12, wherein the first receiving unit is configured to:
reading all fragment files forming the specified file from a plurality of servers in parallel by adopting multiple threads;
and adopting a fragment reassembling algorithm to reassemble the fragment files.
15. The apparatus according to claim 11 or 12, further comprising a storage unit configured to:
and storing the data of the specified file into a cache under the condition that the size of the data of the specified file is less than or equal to a preset threshold value.
16. The apparatus of claim 15, wherein the sending unit comprises:
the searching subunit is used for searching the designated file in the cache;
the access subunit is used for accessing the specified file from the cache under the condition that the search is successful;
and the sending subunit is used for sending the request information to the server under the condition of failure in searching.
17. The apparatus according to claim 16, wherein the access subunit is configured to pre-read at least one of the specified files according to a predetermined access sequence in at least one of:
random reading, reading according to the file name sorting sequence, and determining the file name to be read by utilizing the Hash index.
18. The apparatus of claim 17, wherein the first return unit is further configured to:
in the case that the current file in at least one specified file fails to be accessed, accessing a specified file subsequent to the current file in the access sequence;
and returning the data of the specified file after the current file to the application program.
19. A file access apparatus, comprising:
a second receiving unit configured to receive request information for a specified file from a client;
a second acquisition unit configured to acquire metadata information of the specified file from the request information; the metadata information comprises the storage position information of the specified file in the server;
a third obtaining unit, configured to obtain data of the specified file according to the storage location information;
and the second returning unit is used for returning the data of the specified file to the client.
20. The apparatus of claim 19, further comprising a third return unit configured to:
and responding to a received metadata information acquisition request from the client, and returning metadata information of all files in the data set mounted by the application program to the client, wherein the application program is the application program requesting to access the specified file at the client.
21. An electronic device comprising a processor and a memory, the memory having stored therein instructions that are loaded and executed by the processor to implement the method of any of claims 1 to 10.
22. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1 to 10.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110561924.9A CN113032335A (en) | 2021-05-24 | 2021-05-24 | File access method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110561924.9A CN113032335A (en) | 2021-05-24 | 2021-05-24 | File access method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113032335A true CN113032335A (en) | 2021-06-25 |
Family
ID=76455518
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110561924.9A Pending CN113032335A (en) | 2021-05-24 | 2021-05-24 | File access method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113032335A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113590915A (en) * | 2021-06-30 | 2021-11-02 | 影石创新科技股份有限公司 | Partitioned data caching method, partitioned data accessing method, partitioned data caching device, partitioned data accessing device, terminal and storage medium |
| CN113868344A (en) * | 2021-09-29 | 2021-12-31 | 全球能源互联网研究院有限公司 | Construction system, method and device for power application, server and storage medium |
| CN115858473A (en) * | 2023-01-29 | 2023-03-28 | 北京阿丘科技有限公司 | Data interaction method and device based on training system and object storage system |
| WO2024001827A1 (en) * | 2022-06-28 | 2024-01-04 | 华为技术有限公司 | Data access method, apparatus, and system |
| WO2024041188A1 (en) * | 2022-08-26 | 2024-02-29 | 华为技术有限公司 | Data management method and corresponding device |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105635196A (en) * | 2014-10-27 | 2016-06-01 | 中国电信股份有限公司 | Method and system of file data obtaining, and application server |
| CN105718484A (en) * | 2014-12-04 | 2016-06-29 | 中兴通讯股份有限公司 | File writing method, file reading method, file deletion method, file query method and client |
| CN106021381A (en) * | 2016-05-11 | 2016-10-12 | 北京搜狐新媒体信息技术有限公司 | Data access/storage method and device for cloud storage service system |
| CN107045530A (en) * | 2017-01-20 | 2017-08-15 | 华中科技大学 | A kind of method that object storage system is embodied as to local file system |
| CN108052622A (en) * | 2017-12-15 | 2018-05-18 | 郑州云海信息技术有限公司 | A kind of storage method based on non-relational database, device and equipment |
| CN109710572A (en) * | 2018-12-29 | 2019-05-03 | 北京赛思信安技术股份有限公司 | A kind of file sharding method based on HBase |
| CN112653639A (en) * | 2020-12-21 | 2021-04-13 | 北京华环电子股份有限公司 | IPv6 message fragment recombination method based on multi-thread interactive processing |
-
2021
- 2021-05-24 CN CN202110561924.9A patent/CN113032335A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105635196A (en) * | 2014-10-27 | 2016-06-01 | 中国电信股份有限公司 | Method and system of file data obtaining, and application server |
| CN105718484A (en) * | 2014-12-04 | 2016-06-29 | 中兴通讯股份有限公司 | File writing method, file reading method, file deletion method, file query method and client |
| CN106021381A (en) * | 2016-05-11 | 2016-10-12 | 北京搜狐新媒体信息技术有限公司 | Data access/storage method and device for cloud storage service system |
| CN107045530A (en) * | 2017-01-20 | 2017-08-15 | 华中科技大学 | A kind of method that object storage system is embodied as to local file system |
| CN108052622A (en) * | 2017-12-15 | 2018-05-18 | 郑州云海信息技术有限公司 | A kind of storage method based on non-relational database, device and equipment |
| CN109710572A (en) * | 2018-12-29 | 2019-05-03 | 北京赛思信安技术股份有限公司 | A kind of file sharding method based on HBase |
| CN112653639A (en) * | 2020-12-21 | 2021-04-13 | 北京华环电子股份有限公司 | IPv6 message fragment recombination method based on multi-thread interactive processing |
Non-Patent Citations (2)
| Title |
|---|
| 王刚: "《顾及要素特征的层次增量矢量网络传输研究》", 31 July 2018 * |
| 贾永红: "《数字图像处理实习教程》", 30 November 2016, 武汉大学出版社 * |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113590915A (en) * | 2021-06-30 | 2021-11-02 | 影石创新科技股份有限公司 | Partitioned data caching method, partitioned data accessing method, partitioned data caching device, partitioned data accessing device, terminal and storage medium |
| CN113868344A (en) * | 2021-09-29 | 2021-12-31 | 全球能源互联网研究院有限公司 | Construction system, method and device for power application, server and storage medium |
| CN113868344B (en) * | 2021-09-29 | 2024-04-16 | 国网智能电网研究院有限公司 | Power application-oriented construction system, method, device, server and storage medium |
| WO2024001827A1 (en) * | 2022-06-28 | 2024-01-04 | 华为技术有限公司 | Data access method, apparatus, and system |
| WO2024041188A1 (en) * | 2022-08-26 | 2024-02-29 | 华为技术有限公司 | Data management method and corresponding device |
| CN115858473A (en) * | 2023-01-29 | 2023-03-28 | 北京阿丘科技有限公司 | Data interaction method and device based on training system and object storage system |
| CN115858473B (en) * | 2023-01-29 | 2023-10-10 | 北京阿丘科技有限公司 | Data interaction method and device based on training system and object storage system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113032335A (en) | File access method, device, equipment and storage medium | |
| CN106506587B (en) | Docker mirror image downloading method based on distributed storage | |
| JP4547263B2 (en) | Apparatus and method for processing data in a network | |
| JP4547264B2 (en) | Apparatus and method for proxy cache | |
| US8463846B2 (en) | File bundling for cache servers of content delivery networks | |
| US9348842B2 (en) | Virtualized data storage system optimizations | |
| US6883068B2 (en) | Methods and apparatus for implementing a chche replacement scheme | |
| US9274956B1 (en) | Intelligent cache eviction at storage gateways | |
| US9268651B1 (en) | Efficient recovery of storage gateway cached volumes | |
| US8078574B1 (en) | Network acceleration device cache supporting multiple historical versions of content | |
| US9559889B1 (en) | Cache population optimization for storage gateways | |
| US20080147974A1 (en) | Multi-level caching system | |
| GB2518158A (en) | Method and system for data access in a storage infrastructure | |
| US20200334292A1 (en) | Key value append | |
| US20130290636A1 (en) | Managing memory | |
| CN109766318B (en) | File reading method and device | |
| CN108540510B (en) | Cloud host creation method and device and cloud service system | |
| CN113835616A (en) | Data management method and system of application and computer equipment | |
| US20200192805A1 (en) | Adaptive Cache Commit Delay for Write Aggregation | |
| CN110554911A (en) | Memory access and allocation method, memory controller and system | |
| US7249219B1 (en) | Method and apparatus to improve buffer cache hit rate | |
| CN114003562B (en) | Directory traversal method, device and equipment and readable storage medium | |
| US8549274B2 (en) | Distributive cache accessing device and method for accelerating to boot remote diskless computers | |
| US8489559B2 (en) | Methods and apparatus for conversion of content | |
| CN111382179A (en) | Data processing method and device and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210625 |