WO2021147360A1 - Method, apparatus and device for processing asynchronous io data, and medium - Google Patents
Method, apparatus and device for processing asynchronous io data, and medium Download PDFInfo
- Publication number
- WO2021147360A1 WO2021147360A1 PCT/CN2020/117021 CN2020117021W WO2021147360A1 WO 2021147360 A1 WO2021147360 A1 WO 2021147360A1 CN 2020117021 W CN2020117021 W CN 2020117021W WO 2021147360 A1 WO2021147360 A1 WO 2021147360A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- asynchronous
- data
- request
- storage request
- tag information
- Prior art date
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
- G06F16/1824—Distributed file systems implemented using Network-attached Storage [NAS] architecture
- G06F16/183—Provision of network file services by network file servers, e.g. by using NFS, CIFS
Definitions
- This application relates to the field of distributed object storage technology, and more specifically, to an asynchronous IO data processing method, device, equipment, and computer-readable storage medium.
- the distributed object storage system is mounted through the NFS (Network File System) export directory to realize the file system access of the upper application and the distributed object storage mechanism of the underlying data.
- NFS Network File System
- the NFS protocol access based on object storage supports two forms of synchronous IO (Iuput/Output, input/output) and asynchronous IO.
- the out-of-sequence IO data fragments in the asynchronous IO data are reordered in the cache to obtain the normal sequence of asynchronous IO data, and make subsequent
- the asynchronous IO data is stored and read according to the reordered IO data fragments.
- the purpose of this application is to provide an asynchronous IO data processing method, device, device, and computer-readable storage medium for improving the processing efficiency of asynchronous IO data.
- An asynchronous IO data processing method including:
- asynchronous IO data storage request Receiving an asynchronous IO data storage request; wherein the asynchronous IO data storage request includes a plurality of asynchronous IO data fragments;
- the tag information Including the object name of the asynchronous IO data storage request, the offset of the asynchronous IO data segment, and the storage location of the asynchronous IO data segment;
- obtaining the tag information of each of the asynchronous IO data fragments includes:
- first tag information of each of the asynchronous IO data fragments is acquired; wherein the first tag information includes the ID (Identity Document, identity of the asynchronous IO data storage request). Identification) information, the offset of the asynchronous IO data segment, and the storage location of the asynchronous IO data segment;
- the method before changing the ID information of the asynchronous IO data storage request in the first tag information to the object name of the asynchronous IO data storage request, the method further includes:
- the metadata information of each asynchronous IO data segment is parsed, and the object name of the asynchronous IO data storage request is obtained from the metadata information.
- receiving an asynchronous IO data storage request includes:
- An asynchronous IO data processing device including:
- the receiving module is configured to receive an asynchronous IO data storage request; wherein the asynchronous IO data storage request includes a plurality of asynchronous IO data fragments;
- the first acquisition module is used to acquire the tag information of each of the asynchronous IO data fragments, and store the tag information in a pre-created data tag storage pool, and store the asynchronous IO data fragments in the data pool ;
- the tag information includes the object name of the asynchronous IO data storage request, the offset of the asynchronous IO data segment, and the storage location of the asynchronous IO data segment;
- the second obtaining module is configured to obtain the object name of the asynchronous IO data reading request when the asynchronous IO data reading request is received;
- the reading module is used to obtain all the tag information corresponding to the object name of the asynchronous IO data read request in the data tag storage pool, and according to the offset and asynchronous of the asynchronous IO data segment in the tag information
- the storage location of the IO data fragment reads the asynchronous IO data corresponding to the asynchronous IO data read request.
- the first acquisition module includes:
- the obtaining unit is configured to obtain the first tag information of each of the asynchronous IO data fragments when the asynchronous IO data storage request is received; wherein the first tag information includes the ID of the asynchronous IO data storage request Information, the offset of the asynchronous IO data segment, and the storage location of the asynchronous IO data segment;
- the changing unit is configured to change the ID information of the asynchronous IO data storage request in the first tag information to the object name of the asynchronous IO data storage request when the asynchronous IO data storage request ends, so as to obtain Tag information of each of the asynchronous IO data fragments.
- it also includes:
- the parsing module is used for parsing the metadata of each asynchronous IO data segment before changing the ID information of the asynchronous IO data storage request in the first tag information to the object name of the asynchronous IO data storage request Data information, obtaining the object name of the asynchronous IO data storage request from the metadata information.
- the receiving module includes:
- the receiving unit is configured to receive the asynchronous IO data storage request under the NFS protocol.
- An asynchronous IO data processing device including:
- Memory used to store computer programs
- the processor is used to implement the steps of the asynchronous IO data processing method as described in any one of the above when executing the computer program.
- This application provides an asynchronous IO data processing method, device, device, and computer-readable storage medium, where the method includes: receiving an asynchronous IO data storage request; wherein the asynchronous IO data storage request includes multiple asynchronous IO data fragments ; Obtain the tag information of each asynchronous IO data segment, and store the tag information in the pre-created data tag storage pool, and store the asynchronous IO data segment in the data pool; where the tag information includes the asynchronous IO data storage request The object name, the offset of the asynchronous IO data fragment, the storage location of the asynchronous IO data fragment; when an asynchronous IO data read request is received, the object name of the asynchronous IO data read request is obtained; and the data tag storage pool is obtained.
- the asynchronous IO data stores all the tag information corresponding to the object name of the request, and reads the asynchronous IO data corresponding to the asynchronous IO data read request according to the offset of the asynchronous IO data fragment in the tag information and
- the above-mentioned solution disclosed in this application obtains the tag information of each asynchronous IO data segment, and stores the tag information in a pre-created data tag storage pool, and when an asynchronous IO data read request is received, the data tag is stored Obtain the tag information corresponding to the object name of the asynchronous IO data read request from the pool, and read the asynchronous IO data according to the offset of the asynchronous IO data fragment in the tag information and the storage location of the asynchronous IO data fragment, so that there is no need to correct or wrong
- the sequential asynchronous IO data fragments are reordered, so that the processing time of asynchronous IO data can be shortened, and the processing efficiency and processing performance of asynchronous IO data can be improved.
- FIG. 1 is a flowchart of an asynchronous IO data processing method provided by an embodiment of the application
- FIG. 2 is a schematic structural diagram of an asynchronous IO data processing device provided by an embodiment of the application.
- FIG. 3 is a schematic structural diagram of an asynchronous IO data processing device provided by an embodiment of the application.
- FIG. 1 shows a flowchart of an asynchronous IO data processing method provided by an embodiment of the present application.
- the asynchronous IO data processing method provided by an embodiment of the present application may include:
- S11 Receive an asynchronous IO data storage request; where the asynchronous IO data storage request includes multiple asynchronous IO data fragments.
- the object gateway when asynchronous IO data needs to be processed, the object gateway receives an asynchronous IO data storage request, where the asynchronous IO data storage request may include multiple asynchronous IO data fragments, and these asynchronous IO data The fragments may not be arranged in the order of the asynchronous IO data, that is, the asynchronous IO data fragments are arranged out of order.
- S12 Obtain the tag information of each asynchronous IO data segment, and store the tag information in the pre-created data tag storage pool, and store the asynchronous IO data segment in the data pool; where the tag information may include asynchronous IO data storage The requested object name, the offset of the asynchronous IO data fragment, and the storage location of the asynchronous IO data fragment.
- asynchronous IO data fragments can be stored in the data pool.
- the data tag storage pool can be pre-created at the bottom layer of the distributed object storage system to store tag information of asynchronous IO data fragments.
- the tag information mentioned here can specifically include the object name of the asynchronous IO data storage request (specifically, the object name of the asynchronous IO data to be stored included in the asynchronous IO data storage request, and it belongs to the same asynchronous IO data.
- the data fragments have the same object name), the offset of the asynchronous IO data fragment, and the storage location of the asynchronous IO data fragment, so that the object name of the asynchronous IO data storage request in the tag information can be used to determine the object to which it belongs.
- the offset of the asynchronous IO data segment in determines the location of the asynchronous IO data segment in the asynchronous IO data, and the storage location of the asynchronous IO data segment determines the location where the asynchronous IO data segment is stored.
- the object name of the asynchronous IO data read request can be obtained (specifically in the asynchronous IO data read request) Contains the object name of the asynchronous IO data to be read), so that the asynchronous IO data can be read through the object name of the asynchronous IO data read request.
- S14 Obtain all tag information corresponding to the object name of the asynchronous IO data read request in the data tag storage pool, and read according to the offset of the asynchronous IO data segment and the storage location of the asynchronous IO data segment in the tag information The asynchronous IO data corresponding to the asynchronous IO data read request.
- all tag information corresponding to the object name of the asynchronous IO data read request can be obtained in the data tag storage pool to ensure that complete asynchronous IO data can be obtained.
- the asynchronous IO data corresponding to the asynchronous IO data read request can be read according to all the acquired tag information. Specifically, the position of each asynchronous IO data segment in the asynchronous IO data can be determined according to the obtained offset of the asynchronous IO data segment in each tag information, and the position of each asynchronous IO data segment in the asynchronous IO data can be determined according to the asynchronous IO data segment in each tag information.
- the storage location of the IO data fragment determines the location where the asynchronous IO data fragment is stored, and then reads the asynchronous IO data fragments sequentially and sequentially in the order of the offset of the asynchronous IO data fragment and the storage location of the asynchronous IO data fragment, thereby Improve the efficiency of reading asynchronous IO data.
- the above solution disclosed in this application obtains the tag information of each asynchronous IO data segment and stores the tag information in a pre-created data tag storage pool, and when an asynchronous IO data read request is received, the data tag is stored Obtain the tag information corresponding to the object name of the asynchronous IO data read request from the pool, and read the asynchronous IO data according to the offset of the asynchronous IO data fragment in the tag information and the storage location of the asynchronous IO data fragment, so that there is no need to correct or wrong
- the sequential asynchronous IO data fragments are reordered, so that the processing time of asynchronous IO data can be shortened, and the processing efficiency and processing performance of asynchronous IO data can be improved.
- An asynchronous IO data processing method provided by an embodiment of the present application, which obtains label information of each asynchronous IO data fragment, may include:
- the first label information of each asynchronous IO data segment is acquired; where the first label information may include the ID information of the asynchronous IO data storage request, the offset of the asynchronous IO data segment, and the asynchronous The storage location of the IO data fragment;
- the ID information of the asynchronous IO data storage request in the first label information is changed to the object name of the asynchronous IO data storage request to obtain the label information of each asynchronous IO data segment.
- the label information of each asynchronous IO data fragment can be obtained in the following ways:
- the corresponding request ID will be assigned to the asynchronous IO data storage request. Therefore, when the asynchronous IO data storage request is received, the first tag information of each asynchronous IO data segment can be obtained , Where the first tag information includes the ID information of the asynchronous IO data storage request (asynchronous IO data fragments that belong to the same asynchronous IO data have the same request ID information), the offset of the asynchronous IO data fragment, and the storage of the asynchronous IO data fragment When the asynchronous IO data is stored, the request ID information will be released.
- the asynchronous IO in the first tag information can be changed
- the ID information of the data storage request is changed to the object name of the asynchronous IO data storage request, so that the label information of each asynchronous IO data fragment can be obtained through the object name.
- An asynchronous IO data processing method provided by an embodiment of the present application may further include: before changing the ID information of the asynchronous IO data storage request in the first tag information to the object name of the asynchronous IO data storage request:
- the object gateway Before changing the ID information of the asynchronous IO data storage request in the first tag information to the object name of the asynchronous IO data storage request, the object gateway can parse out the metadata information of each asynchronous IO data segment, and extract it from the metadata information Obtain the object name of the asynchronous IO data storage request, so as to use the obtained object name to change the request ID information in the first tag information.
- An asynchronous IO data processing method provided by an embodiment of the present application, receiving an asynchronous IO data storage request, may include:
- it can receive asynchronous IO data storage requests under the NFS protocol to realize the distributed object storage system's support for asynchronous IO data under the NFS protocol, while also improving the NFS protocol access efficiency of object storage and the object storage docking NFS interface Storage performance.
- FIG. 2 shows a schematic structural diagram of an asynchronous IO data processing device provided by an embodiment of the present application, which may include:
- the receiving module 21 is configured to receive an asynchronous IO data storage request; wherein, the asynchronous IO data storage request may include multiple asynchronous IO data fragments;
- the first obtaining module 22 is configured to obtain the tag information of each asynchronous IO data segment, and store the tag information in a pre-created data tag storage pool, and store the asynchronous IO data segment in the data pool; wherein, the tag information It can include the object name of the asynchronous IO data storage request, the offset of the asynchronous IO data fragment, and the storage location of the asynchronous IO data fragment;
- the second obtaining module 23 is configured to obtain the object name of the asynchronous IO data reading request when the asynchronous IO data reading request is received;
- the reading module 24 is used to obtain all the tag information corresponding to the object name of the asynchronous IO data read request in the data tag storage pool, and according to the offset of the asynchronous IO data segment and the asynchronous IO data segment in the tag information The storage location of reads the asynchronous IO data corresponding to the asynchronous IO data read request.
- the first obtaining module 22 may include:
- the acquiring unit is used to acquire the first tag information of each asynchronous IO data segment when the asynchronous IO data storage request is received; wherein the first tag information may include the ID information of the asynchronous IO data storage request and the information of the asynchronous IO data segment Offset, storage location of asynchronous IO data fragments;
- the change unit is used to change the ID information of the asynchronous IO data storage request in the first label information to the object name of the asynchronous IO data storage request when the asynchronous IO data storage request ends, so as to obtain the label of each asynchronous IO data segment information.
- the parsing module is used to analyze the metadata information of each asynchronous IO data segment before changing the ID information of the asynchronous IO data storage request in the first tag information to the object name of the asynchronous IO data storage request, from the metadata information Get the object name of the asynchronous IO data storage request.
- the receiving module 21 may include:
- the receiving unit is used to receive an asynchronous IO data storage request under the NFS protocol.
- FIG. 3 shows a schematic structural diagram of an asynchronous IO data processing device provided by an embodiment of the present application, which may include:
- the memory 31 is used to store computer programs
- the processor 32 is configured to execute the computer program stored in the memory 31 and can implement the following steps:
- the asynchronous IO data storage request includes multiple asynchronous IO data fragments; obtain the label information of each asynchronous IO data fragment, and store the label information in the pre-created data label storage pool, and Store asynchronous IO data fragments in the data pool; where the tag information includes the object name of the asynchronous IO data storage request, the offset of the asynchronous IO data fragment, and the storage location of the asynchronous IO data fragment; when asynchronous IO data is received, read When requesting, get the object name of the asynchronous IO data read request; get all the tag information corresponding to the object name of the asynchronous IO data storage request in the data tag storage pool, and according to the offset of the asynchronous IO data fragment in the tag information And the storage location of the asynchronous IO data fragment reads the asynchronous IO data corresponding to the asynchronous IO data read request.
- the embodiment of the present application also provides a computer-readable storage medium.
- the computer-readable storage medium stores a computer program.
- the computer program When the computer program is executed by a processor, the following steps can be implemented:
- the asynchronous IO data storage request includes multiple asynchronous IO data fragments; obtain the label information of each asynchronous IO data fragment, and store the label information in the pre-created data label storage pool, and Store asynchronous IO data fragments in the data pool; where the tag information includes the object name of the asynchronous IO data storage request, the offset of the asynchronous IO data fragment, and the storage location of the asynchronous IO data fragment; when asynchronous IO data is received, read When requesting, get the object name of the asynchronous IO data read request; get all the tag information corresponding to the object name of the asynchronous IO data storage request in the data tag storage pool, and according to the offset of the asynchronous IO data fragment in the tag information And the storage location of the asynchronous IO data fragment reads the asynchronous IO data corresponding to the asynchronous IO data read request.
- the computer-readable storage medium may include: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk, etc., which can store program codes Medium.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
A method, apparatus and device for processing asynchronous IO data, and a medium. The method for processing asynchronous IO data comprises: receiving an asynchronous IO data storage request; acquiring label information of each asynchronous IO data segment, storing the label information in a pre-created data label storage pool, and storing the asynchronous IO data segment in a data pool; when an asynchronous IO data reading request is received, acquiring an object name of the asynchronous IO data reading request; and acquiring, from the data label storage pool, all the label information corresponding to the object name of the asynchronous IO data reading request, and reading, according to an offset of the asynchronous IO data segment in the label information and the storage position of the asynchronous IO data segment, asynchronous IO data corresponding to the asynchronous IO data reading request. Disordered asynchronous IO data segments do not need to be re-sorted, so that the processing efficiency of asynchronous IO data is improved.
Description
本申请要求于2020年01月21日提交中国专利局、申请号为202010071608.9、发明名称为“一种异步IO数据处理方法、装置、设备及介质”的中国专利申请的优先权,其全部内容通过引用结合在本申请中。This application claims the priority of a Chinese patent application filed with the Chinese Patent Office on January 21, 2020, the application number is 202010071608.9, and the invention title is "An asynchronous IO data processing method, device, equipment, and medium". The entire content of the application is approved The reference is incorporated in this application.
本申请涉及分布式对象存储技术领域,更具体地说,涉及一种异步IO数据处理方法、装置、设备及计算机可读存储介质。This application relates to the field of distributed object storage technology, and more specifically, to an asynchronous IO data processing method, device, equipment, and computer-readable storage medium.
在分布式对象存储系统中,通过NFS(Network File System,网络文件系统)导出目录挂载分布式对象存储系统,以实现上层应用的文件系统访问和底层数据的分布式对象存储机制。其中,基于对象存储的NFS协议访问支持同步IO(Iuput/Output,输入/输出)和异步IO这两种形式。In the distributed object storage system, the distributed object storage system is mounted through the NFS (Network File System) export directory to realize the file system access of the upper application and the distributed object storage mechanism of the underlying data. Among them, the NFS protocol access based on object storage supports two forms of synchronous IO (Iuput/Output, input/output) and asynchronous IO.
目前,对于异步IO数据的处理,具体是在接收到异步IO数据之后,将异步IO数据中错序的IO数据片段在缓存中重新进行排序,以得到正常序列的异步IO数据,并使得后续可以按照重新排序的IO数据片段来对异步IO数据进行存储和读取。上述这种对异步IO数据进行处理的方式虽然可以保证异步IO数据能够得到正常处理,但是,由于需要对错序的IO数据片段进行重新排序,因此,则会降低异步IO数据的处理效率。At present, for the processing of asynchronous IO data, after receiving the asynchronous IO data, the out-of-sequence IO data fragments in the asynchronous IO data are reordered in the cache to obtain the normal sequence of asynchronous IO data, and make subsequent The asynchronous IO data is stored and read according to the reordered IO data fragments. Although the above-mentioned method of processing asynchronous IO data can ensure that asynchronous IO data can be processed normally, it will reduce the processing efficiency of asynchronous IO data due to the need to reorder the IO data fragments that are out of order.
综上所述,如何提高异步IO数据的处理效率,是目前本领域技术人员亟待解决的技术问题。In summary, how to improve the processing efficiency of asynchronous IO data is a technical problem urgently to be solved by those skilled in the art.
发明内容Summary of the invention
有鉴于此,本申请的目的是提供一种异步IO数据处理方法、装置、设备及计算机可读存储介质,用于提高异步IO数据的处理效率。In view of this, the purpose of this application is to provide an asynchronous IO data processing method, device, device, and computer-readable storage medium for improving the processing efficiency of asynchronous IO data.
为了实现上述目的,本申请提供如下技术方案:In order to achieve the above objectives, this application provides the following technical solutions:
一种异步IO数据处理方法,包括:An asynchronous IO data processing method, including:
接收异步IO数据存储请求;其中,所述异步IO数据存储请求中包括 多个异步IO数据片段;Receiving an asynchronous IO data storage request; wherein the asynchronous IO data storage request includes a plurality of asynchronous IO data fragments;
获取每个所述异步IO数据片段的标签信息,并将所述标签信息存储在预先创建的数据标签存储池中,且将所述异步IO数据片段存储在数据池中;其中,所述标签信息包括所述异步IO数据存储请求的对象名、所述异步IO数据片段的偏移量、所述异步IO数据片段的存储位置;Obtain the tag information of each of the asynchronous IO data segments, and store the tag information in a pre-created data tag storage pool, and store the asynchronous IO data segments in the data pool; wherein, the tag information Including the object name of the asynchronous IO data storage request, the offset of the asynchronous IO data segment, and the storage location of the asynchronous IO data segment;
当接收到异步IO数据读取请求时,获取所述异步IO数据读取请求的对象名;When an asynchronous IO data read request is received, obtain the object name of the asynchronous IO data read request;
在所述数据标签存储池中获取与所述异步IO数据读取请求的对象名对应的所有的标签信息,并根据述标签信息中异步IO数据片段的偏移量及异步IO数据片段的存储位置读取与所述异步IO数据读取请求对应的异步IO数据。Obtain all tag information corresponding to the object name of the asynchronous IO data read request in the data tag storage pool, and according to the offset of the asynchronous IO data segment and the storage location of the asynchronous IO data segment in the tag information Read the asynchronous IO data corresponding to the asynchronous IO data read request.
优选的,获取每个所述异步IO数据片段的标签信息,包括:Preferably, obtaining the tag information of each of the asynchronous IO data fragments includes:
在接收到所述异步IO数据存储请求时,获取每个所述异步IO数据片段的第一标签信息;其中,所述第一标签信息包括所述异步IO数据存储请求的ID(Identity Document,身份标识)信息、所述异步IO数据片段的偏移量、所述异步IO数据片段的存储位置;When the asynchronous IO data storage request is received, first tag information of each of the asynchronous IO data fragments is acquired; wherein the first tag information includes the ID (Identity Document, identity of the asynchronous IO data storage request). Identification) information, the offset of the asynchronous IO data segment, and the storage location of the asynchronous IO data segment;
当所述异步IO数据存储请求结束时,将所述第一标签信息中的所述异步IO数据存储请求的ID信息更改为所述异步IO数据存储请求的对象名,以获取每个所述异步IO数据片段的标签信息。When the asynchronous IO data storage request ends, change the ID information of the asynchronous IO data storage request in the first tag information to the object name of the asynchronous IO data storage request to obtain each asynchronous IO data storage request. The tag information of the IO data fragment.
优选的,在将所述第一标签信息中的所述异步IO数据存储请求的ID信息更改为所述异步IO数据存储请求的对象名之前,还包括:Preferably, before changing the ID information of the asynchronous IO data storage request in the first tag information to the object name of the asynchronous IO data storage request, the method further includes:
解析每个所述异步IO数据片段的元数据信息,从所述元数据信息中获取所述异步IO数据存储请求的对象名。The metadata information of each asynchronous IO data segment is parsed, and the object name of the asynchronous IO data storage request is obtained from the metadata information.
优选的,接收异步IO数据存储请求,包括:Preferably, receiving an asynchronous IO data storage request includes:
接收NFS协议下的所述异步IO数据存储请求。Receive the asynchronous IO data storage request under the NFS protocol.
一种异步IO数据处理装置,包括:An asynchronous IO data processing device, including:
接收模块,用于接收异步IO数据存储请求;其中,所述异步IO数据存储请求中包括多个异步IO数据片段;The receiving module is configured to receive an asynchronous IO data storage request; wherein the asynchronous IO data storage request includes a plurality of asynchronous IO data fragments;
第一获取模块,用于获取每个所述异步IO数据片段的标签信息,并将 所述标签信息存储在预先创建的数据标签存储池中,且将所述异步IO数据片段存储在数据池中;其中,所述标签信息包括所述异步IO数据存储请求的对象名、所述异步IO数据片段的偏移量、所述异步IO数据片段的存储位置;The first acquisition module is used to acquire the tag information of each of the asynchronous IO data fragments, and store the tag information in a pre-created data tag storage pool, and store the asynchronous IO data fragments in the data pool ; Wherein, the tag information includes the object name of the asynchronous IO data storage request, the offset of the asynchronous IO data segment, and the storage location of the asynchronous IO data segment;
第二获取模块,用于当接收到异步IO数据读取请求时,获取所述异步IO数据读取请求的对象名;The second obtaining module is configured to obtain the object name of the asynchronous IO data reading request when the asynchronous IO data reading request is received;
读取模块,用于在所述数据标签存储池中获取与所述异步IO数据读取请求的对象名对应的所有的标签信息,并根据述标签信息中异步IO数据片段的偏移量及异步IO数据片段的存储位置读取与所述异步IO数据读取请求对应的异步IO数据。The reading module is used to obtain all the tag information corresponding to the object name of the asynchronous IO data read request in the data tag storage pool, and according to the offset and asynchronous of the asynchronous IO data segment in the tag information The storage location of the IO data fragment reads the asynchronous IO data corresponding to the asynchronous IO data read request.
优选的,所述第一获取模块包括:Preferably, the first acquisition module includes:
获取单元,用于在接收到所述异步IO数据存储请求时,获取每个所述异步IO数据片段的第一标签信息;其中,所述第一标签信息包括所述异步IO数据存储请求的ID信息、所述异步IO数据片段的偏移量、所述异步IO数据片段的存储位置;The obtaining unit is configured to obtain the first tag information of each of the asynchronous IO data fragments when the asynchronous IO data storage request is received; wherein the first tag information includes the ID of the asynchronous IO data storage request Information, the offset of the asynchronous IO data segment, and the storage location of the asynchronous IO data segment;
更改单元,用于当所述异步IO数据存储请求结束时,将所述第一标签信息中的所述异步IO数据存储请求的ID信息更改为所述异步IO数据存储请求的对象名,以获取每个所述异步IO数据片段的标签信息。The changing unit is configured to change the ID information of the asynchronous IO data storage request in the first tag information to the object name of the asynchronous IO data storage request when the asynchronous IO data storage request ends, so as to obtain Tag information of each of the asynchronous IO data fragments.
优选的,还包括:Preferably, it also includes:
解析模块,用于在将所述第一标签信息中的所述异步IO数据存储请求的ID信息更改为所述异步IO数据存储请求的对象名之前,解析每个所述异步IO数据片段的元数据信息,从所述元数据信息中获取所述异步IO数据存储请求的对象名。The parsing module is used for parsing the metadata of each asynchronous IO data segment before changing the ID information of the asynchronous IO data storage request in the first tag information to the object name of the asynchronous IO data storage request Data information, obtaining the object name of the asynchronous IO data storage request from the metadata information.
优选的,所述接收模块包括:Preferably, the receiving module includes:
接收单元,用于接收NFS协议下的所述异步IO数据存储请求。The receiving unit is configured to receive the asynchronous IO data storage request under the NFS protocol.
一种异步IO数据处理设备,包括:An asynchronous IO data processing device, including:
存储器,用于存储计算机程序;Memory, used to store computer programs;
处理器,用于执行所述计算机程序时实现如上述任一项所述的异步IO数据处理方法的步骤。The processor is used to implement the steps of the asynchronous IO data processing method as described in any one of the above when executing the computer program.
一种计算机可读存储介质,所述计算机可读存储介质上存储有计算机程序,所述计算机程序被处理器执行时实现如上述任一项所述的异步IO数据处理方法的步骤。A computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the asynchronous IO data processing method as described in any one of the above are implemented.
本申请提供了一种异步IO数据处理方法、装置、设备及计算机可读存储介质,其中,该方法包括:接收异步IO数据存储请求;其中,异步IO数据存储请求中包括多个异步IO数据片段;获取每个异步IO数据片段的标签信息,并将标签信息存储在预先创建的数据标签存储池中,且将异步IO数据片段存储在数据池中;其中,标签信息包括异步IO数据存储请求的对象名、异步IO数据片段的偏移量、异步IO数据片段的存储位置;当接收到异步IO数据读取请求时,获取异步IO数据读取请求的对象名;在数据标签存储池中获取与异步IO数据存储请求的对象名对应的所有的标签信息,并根据标签信息中异步IO数据片段的偏移量及异步IO数据片段的存储位置读取与异步IO数据读取请求对应的异步IO数据。This application provides an asynchronous IO data processing method, device, device, and computer-readable storage medium, where the method includes: receiving an asynchronous IO data storage request; wherein the asynchronous IO data storage request includes multiple asynchronous IO data fragments ; Obtain the tag information of each asynchronous IO data segment, and store the tag information in the pre-created data tag storage pool, and store the asynchronous IO data segment in the data pool; where the tag information includes the asynchronous IO data storage request The object name, the offset of the asynchronous IO data fragment, the storage location of the asynchronous IO data fragment; when an asynchronous IO data read request is received, the object name of the asynchronous IO data read request is obtained; and the data tag storage pool is obtained The asynchronous IO data stores all the tag information corresponding to the object name of the request, and reads the asynchronous IO data corresponding to the asynchronous IO data read request according to the offset of the asynchronous IO data fragment in the tag information and the storage location of the asynchronous IO data fragment .
本申请公开的上述方案,通过获取每个异步IO数据片段的标签信息,并将标签信息存储在预先创建的数据标签存储池中,且在接收到异步IO数据读取请求时,在数据标签存储池中获取与异步IO数据读取请求的对象名对应的标签信息,并根据标签信息中异步IO数据片段的偏移量及异步IO数据片段的存储位置读取异步IO数据,以无需再对错序的异步IO数据片段进行重新排序处理,从而可以缩短异步IO数据的处理时间,提高异步IO数据的处理效率和处理性能。The above-mentioned solution disclosed in this application obtains the tag information of each asynchronous IO data segment, and stores the tag information in a pre-created data tag storage pool, and when an asynchronous IO data read request is received, the data tag is stored Obtain the tag information corresponding to the object name of the asynchronous IO data read request from the pool, and read the asynchronous IO data according to the offset of the asynchronous IO data fragment in the tag information and the storage location of the asynchronous IO data fragment, so that there is no need to correct or wrong The sequential asynchronous IO data fragments are reordered, so that the processing time of asynchronous IO data can be shortened, and the processing efficiency and processing performance of asynchronous IO data can be improved.
为了更清楚地说明本申请实施例或现有技术中的技术方案,下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍,显而易见地,下面描述中的附图仅仅是本申请的一些实施例,对于本领域普通技术人员来讲,在不付出创造性劳动的前提下,还可以根据这些附图获得其他的附图。In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only These are some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work.
图1为本申请实施例提供的一种异步IO数据处理方法的流程图;FIG. 1 is a flowchart of an asynchronous IO data processing method provided by an embodiment of the application;
图2为本申请实施例提供的一种异步IO数据处理装置的结构示意图;2 is a schematic structural diagram of an asynchronous IO data processing device provided by an embodiment of the application;
图3为本申请实施例提供的一种异步IO数据处理设备的结构示意图。FIG. 3 is a schematic structural diagram of an asynchronous IO data processing device provided by an embodiment of the application.
下面将结合本申请实施例中的附图,对本申请实施例中的技术方案进行清楚、完整地描述,显然,所描述的实施例仅仅是本申请一部分实施例,而不是全部的实施例。基于本申请中的实施例,本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例,都属于本申请保护的范围。The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of this application.
参见图1,其示出了本申请实施例提供的一种异步IO数据处理方法的流程图,本申请实施例提供的一种异步IO数据处理方法,可以包括:Referring to FIG. 1, it shows a flowchart of an asynchronous IO data processing method provided by an embodiment of the present application. The asynchronous IO data processing method provided by an embodiment of the present application may include:
S11:接收异步IO数据存储请求;其中,异步IO数据存储请求中包括多个异步IO数据片段。S11: Receive an asynchronous IO data storage request; where the asynchronous IO data storage request includes multiple asynchronous IO data fragments.
在分布式对象存储系统中,当需要对异步IO数据进行处理时,对象网关接收异步IO数据存储请求,其中,该异步IO数据存储请求中可以包括多个异步IO数据片段,且这些异步IO数据片段可能并未按照其在异步IO数据的顺序进行排列,也就是说,这些异步IO数据片段是错序排列的。In a distributed object storage system, when asynchronous IO data needs to be processed, the object gateway receives an asynchronous IO data storage request, where the asynchronous IO data storage request may include multiple asynchronous IO data fragments, and these asynchronous IO data The fragments may not be arranged in the order of the asynchronous IO data, that is, the asynchronous IO data fragments are arranged out of order.
S12:获取每个异步IO数据片段的标签信息,并将标签信息存储在预先创建的数据标签存储池中,且将异步IO数据片段存储在数据池中;其中,标签信息可以包括异步IO数据存储请求的对象名、异步IO数据片段的偏移量、异步IO数据片段的存储位置。S12: Obtain the tag information of each asynchronous IO data segment, and store the tag information in the pre-created data tag storage pool, and store the asynchronous IO data segment in the data pool; where the tag information may include asynchronous IO data storage The requested object name, the offset of the asynchronous IO data fragment, and the storage location of the asynchronous IO data fragment.
在接收到异步IO数据存储请求中包含的多个异步IO数据片段时,获取每个异步IO数据片段的标签信息,并将这些异步IO数据片段的标签信息存储在预先创建的数据标签存储池中,同时可以将异步IO数据片段存储在数据池中。其中,该数据标签存储池可以预先创建在分布式对象存储系统的底层,以用于存储异步IO数据片段的标签信息。When receiving multiple asynchronous IO data fragments included in the asynchronous IO data storage request, obtain the label information of each asynchronous IO data fragment, and store the label information of these asynchronous IO data fragments in the pre-created data label storage pool At the same time, asynchronous IO data fragments can be stored in the data pool. Among them, the data tag storage pool can be pre-created at the bottom layer of the distributed object storage system to store tag information of asynchronous IO data fragments.
另外,这里提及的标签信息具体可以包括异步IO数据存储请求的对象名(具体为异步IO数据存储请求中包含的要进行存储的异步IO数据的对象名,且同属一个异步IO数据的异步IO数据片段具有相同的对象名)、异步IO数据片段的偏移量及异步IO数据片段的存储位置,以便于通过标 签信息中的异步IO数据存储请求的对象名确定其所属的对象,通过标签信息中的异步IO数据片段的偏移量确定该异步IO数据片段在异步IO数据中所属的位置,并通过异步IO数据片段的存储位置确定该异步IO数据片段所存储的位置。In addition, the tag information mentioned here can specifically include the object name of the asynchronous IO data storage request (specifically, the object name of the asynchronous IO data to be stored included in the asynchronous IO data storage request, and it belongs to the same asynchronous IO data. The data fragments have the same object name), the offset of the asynchronous IO data fragment, and the storage location of the asynchronous IO data fragment, so that the object name of the asynchronous IO data storage request in the tag information can be used to determine the object to which it belongs. The offset of the asynchronous IO data segment in determines the location of the asynchronous IO data segment in the asynchronous IO data, and the storage location of the asynchronous IO data segment determines the location where the asynchronous IO data segment is stored.
由上述可知,其并没有对所接收到的异步IO数据片段进行重新排序,然后,再进行存储,因此,可以缩短异步IO数据存储所花费的时间,从而可以提高异步IO数据的存储效率,而且通过创建数据标签存储池来对异步IO数据片段的标签信息进行存储的方式可以保证异步IO数据的完整性和一致性,从而可以提高异步IO数据存储的性能。It can be seen from the above that it does not reorder the received asynchronous IO data fragments and then store them. Therefore, the time spent on asynchronous IO data storage can be shortened, thereby improving the storage efficiency of asynchronous IO data, and By creating a data tag storage pool to store the tag information of the asynchronous IO data fragments, the integrity and consistency of the asynchronous IO data can be guaranteed, and the performance of the asynchronous IO data storage can be improved.
S13:当接收到异步IO数据读取请求时,获取异步IO数据读取请求的对象名。S13: When receiving the asynchronous IO data reading request, obtain the object name of the asynchronous IO data reading request.
在完成对异步IO数据片段标签信息和异步IO数据的存储之后,若接收到异步IO数据读取请求,则可以获取该异步IO数据读取请求的对象名(具体为异步IO数据读取请求中包含的要进行读取的异步IO数据的对象名),以便于通过异步IO数据读取请求的对象名对异步IO数据进行读取。After completing the storage of the asynchronous IO data fragment label information and the asynchronous IO data, if an asynchronous IO data read request is received, the object name of the asynchronous IO data read request can be obtained (specifically in the asynchronous IO data read request) Contains the object name of the asynchronous IO data to be read), so that the asynchronous IO data can be read through the object name of the asynchronous IO data read request.
S14:在数据标签存储池中获取与异步IO数据读取请求的对象名对应的所有的标签信息,并根据述标签信息中异步IO数据片段的偏移量及异步IO数据片段的存储位置读取与异步IO数据读取请求对应的异步IO数据。S14: Obtain all tag information corresponding to the object name of the asynchronous IO data read request in the data tag storage pool, and read according to the offset of the asynchronous IO data segment and the storage location of the asynchronous IO data segment in the tag information The asynchronous IO data corresponding to the asynchronous IO data read request.
在获取到异步IO数据读取请求的对象名时,可以在数据标签存储池中获取与异步IO数据读取请求的对象名对应的所有的标签信息,以保证可以获取到完整的异步IO数据。之后,则可以根据所获取到所有的标签信息来读取与异步IO数据读取请求对应的异步IO数据。具体地,可以根据所获取到的每个标签信息中的异步IO数据片段的偏移量分别确定每个异步IO数据片段在异步IO数据中所处的位置,并根据每个标签信息中的异步IO数据片段的存储位置确定异步IO数据片段所存储的位置,然后,按照异步IO数据片段的偏移量的顺序及异步IO数据片段的存储位置顺序地且依次地读取异步IO数据片段,从而提高异步IO数据的读取效率。When the object name of the asynchronous IO data read request is obtained, all tag information corresponding to the object name of the asynchronous IO data read request can be obtained in the data tag storage pool to ensure that complete asynchronous IO data can be obtained. After that, the asynchronous IO data corresponding to the asynchronous IO data read request can be read according to all the acquired tag information. Specifically, the position of each asynchronous IO data segment in the asynchronous IO data can be determined according to the obtained offset of the asynchronous IO data segment in each tag information, and the position of each asynchronous IO data segment in the asynchronous IO data can be determined according to the asynchronous IO data segment in each tag information. The storage location of the IO data fragment determines the location where the asynchronous IO data fragment is stored, and then reads the asynchronous IO data fragments sequentially and sequentially in the order of the offset of the asynchronous IO data fragment and the storage location of the asynchronous IO data fragment, thereby Improve the efficiency of reading asynchronous IO data.
本申请公开的上述方案,通过获取每个异步IO数据片段的标签信息,并将标签信息存储在预先创建的数据标签存储池中,且在接收到异步IO 数据读取请求时,在数据标签存储池中获取与异步IO数据读取请求的对象名对应的标签信息,并根据标签信息中异步IO数据片段的偏移量及异步IO数据片段的存储位置读取异步IO数据,以无需再对错序的异步IO数据片段进行重新排序处理,从而可以缩短异步IO数据的处理时间,提高异步IO数据的处理效率和处理性能。The above solution disclosed in this application obtains the tag information of each asynchronous IO data segment and stores the tag information in a pre-created data tag storage pool, and when an asynchronous IO data read request is received, the data tag is stored Obtain the tag information corresponding to the object name of the asynchronous IO data read request from the pool, and read the asynchronous IO data according to the offset of the asynchronous IO data fragment in the tag information and the storage location of the asynchronous IO data fragment, so that there is no need to correct or wrong The sequential asynchronous IO data fragments are reordered, so that the processing time of asynchronous IO data can be shortened, and the processing efficiency and processing performance of asynchronous IO data can be improved.
本申请实施例提供的一种异步IO数据处理方法,获取每个异步IO数据片段的标签信息,可以包括:An asynchronous IO data processing method provided by an embodiment of the present application, which obtains label information of each asynchronous IO data fragment, may include:
在接收到异步IO数据存储请求时,获取每个异步IO数据片段的第一标签信息;其中,第一标签信息可以包括异步IO数据存储请求的ID信息、异步IO数据片段的偏移量、异步IO数据片段的存储位置;When an asynchronous IO data storage request is received, the first label information of each asynchronous IO data segment is acquired; where the first label information may include the ID information of the asynchronous IO data storage request, the offset of the asynchronous IO data segment, and the asynchronous The storage location of the IO data fragment;
当异步IO数据存储请求结束时,将第一标签信息中的异步IO数据存储请求的ID信息更改为异步IO数据存储请求的对象名,以获取每个异步IO数据片段的标签信息。When the asynchronous IO data storage request ends, the ID information of the asynchronous IO data storage request in the first label information is changed to the object name of the asynchronous IO data storage request to obtain the label information of each asynchronous IO data segment.
具体可以通过以下方式获取每个异步IO数据片段的标签信息:Specifically, the label information of each asynchronous IO data fragment can be obtained in the following ways:
考虑到在接收到异步IO数据存储请求初期,会为异步IO数据存储请求分配对应的请求ID,因此,在接收到异步IO数据存储请求时,可以获取每个异步IO数据片段的第一标签信息,其中,第一标签信息包括异步IO数据存储请求的ID信息(同属一个异步IO数据的异步IO数据片段具有相同的请求ID信息)、异步IO数据片段的偏移量、异步IO数据片段的存储位置,而当异步IO数据完成存储之后,请求ID信息即会释放,为了便于后续可以准确地读取异步IO数据,则当异步IO数据存储请求结束时,可以将第一标签信息中的异步IO数据存储请求的ID信息更改为异步IO数据存储请求的对象名,以便于通过对象名获取每个异步IO数据片段的标签信息。Considering that in the initial stage of receiving the asynchronous IO data storage request, the corresponding request ID will be assigned to the asynchronous IO data storage request. Therefore, when the asynchronous IO data storage request is received, the first tag information of each asynchronous IO data segment can be obtained , Where the first tag information includes the ID information of the asynchronous IO data storage request (asynchronous IO data fragments that belong to the same asynchronous IO data have the same request ID information), the offset of the asynchronous IO data fragment, and the storage of the asynchronous IO data fragment When the asynchronous IO data is stored, the request ID information will be released. In order to facilitate the subsequent accurate reading of the asynchronous IO data, when the asynchronous IO data storage request ends, the asynchronous IO in the first tag information can be changed The ID information of the data storage request is changed to the object name of the asynchronous IO data storage request, so that the label information of each asynchronous IO data fragment can be obtained through the object name.
本申请实施例提供的一种异步IO数据处理方法,在将第一标签信息中的异步IO数据存储请求的ID信息更改为异步IO数据存储请求的对象名之前,还可以包括:An asynchronous IO data processing method provided by an embodiment of the present application may further include: before changing the ID information of the asynchronous IO data storage request in the first tag information to the object name of the asynchronous IO data storage request:
解析每个异步IO数据片段的元数据信息,从元数据信息中获取异步 IO数据存储请求的对象名。Parse the metadata information of each asynchronous IO data fragment, and obtain the object name of the asynchronous IO data storage request from the metadata information.
在将第一标签信息中的异步IO数据存储请求的ID信息更改为异步IO数据存储请求的对象名之前,对象网关可以解析出每个异步IO数据片段的元数据信息,并从元数据信息中获取异步IO数据存储请求的对象名,以便于利用所获取到的对象名对第一标签信息中的请求ID信息进行更改。Before changing the ID information of the asynchronous IO data storage request in the first tag information to the object name of the asynchronous IO data storage request, the object gateway can parse out the metadata information of each asynchronous IO data segment, and extract it from the metadata information Obtain the object name of the asynchronous IO data storage request, so as to use the obtained object name to change the request ID information in the first tag information.
本申请实施例提供的一种异步IO数据处理方法,接收异步IO数据存储请求,可以包括:An asynchronous IO data processing method provided by an embodiment of the present application, receiving an asynchronous IO data storage request, may include:
接收NFS协议下的异步IO数据存储请求。Receive asynchronous IO data storage requests under the NFS protocol.
具体可以接收NFS协议下的异步IO数据存储请求,以在实现分布式对象存储系统对NFS协议下异步IO数据的支持的同时,还提高了提高对象存储的NFS协议访问效率及对象存储对接NFS接口的存储性能。Specifically, it can receive asynchronous IO data storage requests under the NFS protocol to realize the distributed object storage system's support for asynchronous IO data under the NFS protocol, while also improving the NFS protocol access efficiency of object storage and the object storage docking NFS interface Storage performance.
本申请实施例还提供了一种异步IO数据处理装置,参见图2,其示出了本申请实施例提供的一种异步IO数据处理装置的结构示意图,可以包括:An embodiment of the present application also provides an asynchronous IO data processing device. See FIG. 2, which shows a schematic structural diagram of an asynchronous IO data processing device provided by an embodiment of the present application, which may include:
接收模块21,用于接收异步IO数据存储请求;其中,异步IO数据存储请求中可以包括多个异步IO数据片段;The receiving module 21 is configured to receive an asynchronous IO data storage request; wherein, the asynchronous IO data storage request may include multiple asynchronous IO data fragments;
第一获取模块22,用于获取每个异步IO数据片段的标签信息,并将标签信息存储在预先创建的数据标签存储池中,且将异步IO数据片段存储在数据池中;其中,标签信息可以包括异步IO数据存储请求的对象名、异步IO数据片段的偏移量、异步IO数据片段的存储位置;The first obtaining module 22 is configured to obtain the tag information of each asynchronous IO data segment, and store the tag information in a pre-created data tag storage pool, and store the asynchronous IO data segment in the data pool; wherein, the tag information It can include the object name of the asynchronous IO data storage request, the offset of the asynchronous IO data fragment, and the storage location of the asynchronous IO data fragment;
第二获取模块23,用于当接收到异步IO数据读取请求时,获取异步IO数据读取请求的对象名;The second obtaining module 23 is configured to obtain the object name of the asynchronous IO data reading request when the asynchronous IO data reading request is received;
读取模块24,用于在数据标签存储池中获取与异步IO数据读取请求的对象名对应的所有的标签信息,并根据述标签信息中异步IO数据片段的偏移量及异步IO数据片段的存储位置读取与异步IO数据读取请求对应的异步IO数据。The reading module 24 is used to obtain all the tag information corresponding to the object name of the asynchronous IO data read request in the data tag storage pool, and according to the offset of the asynchronous IO data segment and the asynchronous IO data segment in the tag information The storage location of reads the asynchronous IO data corresponding to the asynchronous IO data read request.
本申请实施例提供的一种异步IO数据处理装置,第一获取模块22可以包括:In an asynchronous IO data processing device provided by an embodiment of the present application, the first obtaining module 22 may include:
获取单元,用于在接收到异步IO数据存储请求时,获取每个异步IO数据片段的第一标签信息;其中,第一标签信息可以包括异步IO数据存储请求的ID信息、异步IO数据片段的偏移量、异步IO数据片段的存储位置;The acquiring unit is used to acquire the first tag information of each asynchronous IO data segment when the asynchronous IO data storage request is received; wherein the first tag information may include the ID information of the asynchronous IO data storage request and the information of the asynchronous IO data segment Offset, storage location of asynchronous IO data fragments;
更改单元,用于当异步IO数据存储请求结束时,将第一标签信息中的异步IO数据存储请求的ID信息更改为异步IO数据存储请求的对象名,以获取每个异步IO数据片段的标签信息。The change unit is used to change the ID information of the asynchronous IO data storage request in the first label information to the object name of the asynchronous IO data storage request when the asynchronous IO data storage request ends, so as to obtain the label of each asynchronous IO data segment information.
本申请实施例提供的一种异步IO数据处理装置,还可以包括:The asynchronous IO data processing device provided by the embodiment of the present application may further include:
解析模块,用于在将第一标签信息中的异步IO数据存储请求的ID信息更改为异步IO数据存储请求的对象名之前,解析每个异步IO数据片段的元数据信息,从元数据信息中获取异步IO数据存储请求的对象名。The parsing module is used to analyze the metadata information of each asynchronous IO data segment before changing the ID information of the asynchronous IO data storage request in the first tag information to the object name of the asynchronous IO data storage request, from the metadata information Get the object name of the asynchronous IO data storage request.
本申请实施例提供的一种异步IO数据处理装置,接收模块21可以包括:In an asynchronous IO data processing apparatus provided by an embodiment of the present application, the receiving module 21 may include:
接收单元,用于接收NFS协议下的异步IO数据存储请求。The receiving unit is used to receive an asynchronous IO data storage request under the NFS protocol.
本申请实施例还提供了一种异步IO数据处理设备,参见图3,其示出了本申请实施例提供的一种异步IO数据处理设备的结构示意图,可以包括:An embodiment of the present application also provides an asynchronous IO data processing device. See FIG. 3, which shows a schematic structural diagram of an asynchronous IO data processing device provided by an embodiment of the present application, which may include:
存储器31,用于存储计算机程序;The memory 31 is used to store computer programs;
处理器32,用于执行存储器31存储的计算机程序时可实现如下步骤:The processor 32 is configured to execute the computer program stored in the memory 31 and can implement the following steps:
接收异步IO数据存储请求;其中,异步IO数据存储请求中包括多个异步IO数据片段;获取每个异步IO数据片段的标签信息,并将标签信息存储在预先创建的数据标签存储池中,且将异步IO数据片段存储在数据池中;其中,标签信息包括异步IO数据存储请求的对象名、异步IO数据片段的偏移量、异步IO数据片段的存储位置;当接收到异步IO数据读取请求时,获取异步IO数据读取请求的对象名;在数据标签存储池中获取与异步IO数据存储请求的对象名对应的所有的标签信息,并根据标签信息中异步IO数据片段的偏移量及异步IO数据片段的存储位置读取与异步IO数据读取请求对应的异步IO数据。Receive an asynchronous IO data storage request; where the asynchronous IO data storage request includes multiple asynchronous IO data fragments; obtain the label information of each asynchronous IO data fragment, and store the label information in the pre-created data label storage pool, and Store asynchronous IO data fragments in the data pool; where the tag information includes the object name of the asynchronous IO data storage request, the offset of the asynchronous IO data fragment, and the storage location of the asynchronous IO data fragment; when asynchronous IO data is received, read When requesting, get the object name of the asynchronous IO data read request; get all the tag information corresponding to the object name of the asynchronous IO data storage request in the data tag storage pool, and according to the offset of the asynchronous IO data fragment in the tag information And the storage location of the asynchronous IO data fragment reads the asynchronous IO data corresponding to the asynchronous IO data read request.
本申请实施例还提供了一种计算机可读存储介质,计算机可读存储介质上存储有计算机程序,计算机程序被处理器执行时可实现如下步骤:The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the following steps can be implemented:
接收异步IO数据存储请求;其中,异步IO数据存储请求中包括多个异步IO数据片段;获取每个异步IO数据片段的标签信息,并将标签信息存储在预先创建的数据标签存储池中,且将异步IO数据片段存储在数据池中;其中,标签信息包括异步IO数据存储请求的对象名、异步IO数据片段的偏移量、异步IO数据片段的存储位置;当接收到异步IO数据读取请求时,获取异步IO数据读取请求的对象名;在数据标签存储池中获取与异步IO数据存储请求的对象名对应的所有的标签信息,并根据标签信息中异步IO数据片段的偏移量及异步IO数据片段的存储位置读取与异步IO数据读取请求对应的异步IO数据。Receive an asynchronous IO data storage request; where the asynchronous IO data storage request includes multiple asynchronous IO data fragments; obtain the label information of each asynchronous IO data fragment, and store the label information in the pre-created data label storage pool, and Store asynchronous IO data fragments in the data pool; where the tag information includes the object name of the asynchronous IO data storage request, the offset of the asynchronous IO data fragment, and the storage location of the asynchronous IO data fragment; when asynchronous IO data is received, read When requesting, get the object name of the asynchronous IO data read request; get all the tag information corresponding to the object name of the asynchronous IO data storage request in the data tag storage pool, and according to the offset of the asynchronous IO data fragment in the tag information And the storage location of the asynchronous IO data fragment reads the asynchronous IO data corresponding to the asynchronous IO data read request.
该计算机可读存储介质可以包括:U盘、移动硬盘、只读存储器(Read-Only Memory,ROM)、随机存取存储器(Random Access Memory,RAM)、磁碟或者光盘等各种可以存储程序代码的介质。The computer-readable storage medium may include: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk, etc., which can store program codes Medium.
本申请实施例提供的一种异步IO数据处理装置、设备及计算机可读存储介质中相关部分的说明可以参见本申请实施例提供的一种异步IO数据处理方法中对应部分的详细说明,在此不再赘述。For descriptions of related parts in an asynchronous IO data processing apparatus, equipment, and computer-readable storage medium provided in the embodiments of the present application, please refer to the detailed description of the corresponding parts in the asynchronous IO data processing method provided in the embodiments of the present application. No longer.
需要说明的是,在本文中,诸如第一和第二等之类的关系术语仅仅用来将一个实体或者操作与另一个实体或操作区分开来,而不一定要求或者暗示这些实体或操作之间存在任何这种实际的关系或者顺序。而且,术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含,从而使得包括一系列要素的过程、方法、物品或者设备所固有的要素。在没有更多限制的情况下,由语句“包括一个……”限定的要素,并不排除在包括所述要素的过程、方法、物品或者设备中还存在另外的相同要素。另外,本申请实施例提供的上述技术方案中与现有技术中对应技术方案实现原理一致的部分并未详细说明,以免过多赘述。It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply one of these entities or operations. There is any such actual relationship or order between. Moreover, the terms "including", "including" or any other variants thereof are intended to cover non-exclusive inclusions, so as to include elements inherent in a process, method, article, or equipment of a series of elements. If there are no more restrictions, the element defined by the sentence "including a..." does not exclude the existence of other same elements in the process, method, article, or equipment that includes the element. In addition, the parts of the foregoing technical solutions provided by the embodiments of the present application that are consistent with the implementation principles of the corresponding technical solutions in the prior art are not described in detail, so as to avoid redundant description.
对所公开的实施例的上述说明,使本领域技术人员能够实现或使用本 申请。对这些实施例的多种修改对本领域技术人员来说将是显而易见的,本文中所定义的一般原理可以在不脱离本申请的精神或范围的情况下,在其它实施例中实现。因此,本申请将不会被限制于本文所示的这些实施例,而是要符合与本文所公开的原理和新颖特点相一致的最宽的范围。The foregoing description of the disclosed embodiments enables those skilled in the art to implement or use this application. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, this application will not be limited to the embodiments shown in this document, but should conform to the widest scope consistent with the principles and novel features disclosed in this document.
Claims (10)
- 一种异步IO数据处理方法,其特征在于,包括:An asynchronous IO data processing method, characterized in that it comprises:接收异步IO数据存储请求;其中,所述异步IO数据存储请求中包括多个异步IO数据片段;Receiving an asynchronous IO data storage request; wherein the asynchronous IO data storage request includes a plurality of asynchronous IO data fragments;获取每个所述异步IO数据片段的标签信息,并将所述标签信息存储在预先创建的数据标签存储池中,且将所述异步IO数据片段存储在数据池中;其中,所述标签信息包括所述异步IO数据存储请求的对象名、所述异步IO数据片段的偏移量、所述异步IO数据片段的存储位置;Obtain the tag information of each of the asynchronous IO data segments, and store the tag information in a pre-created data tag storage pool, and store the asynchronous IO data segments in the data pool; wherein, the tag information Including the object name of the asynchronous IO data storage request, the offset of the asynchronous IO data segment, and the storage location of the asynchronous IO data segment;当接收到异步IO数据读取请求时,获取所述异步IO数据读取请求的对象名;When an asynchronous IO data read request is received, obtain the object name of the asynchronous IO data read request;在所述数据标签存储池中获取与所述异步IO数据读取请求的对象名对应的所有的标签信息,并根据述标签信息中异步IO数据片段的偏移量及异步IO数据片段的存储位置读取与所述异步IO数据读取请求对应的异步IO数据。Obtain all tag information corresponding to the object name of the asynchronous IO data read request in the data tag storage pool, and according to the offset of the asynchronous IO data segment and the storage location of the asynchronous IO data segment in the tag information Read the asynchronous IO data corresponding to the asynchronous IO data read request.
- 根据权利要求1所述的异步IO数据处理方法,其特征在于,获取每个所述异步IO数据片段的标签信息,包括:The asynchronous IO data processing method according to claim 1, wherein acquiring the label information of each of the asynchronous IO data fragments comprises:在接收到所述异步IO数据存储请求时,获取每个所述异步IO数据片段的第一标签信息;其中,所述第一标签信息包括所述异步IO数据存储请求的ID信息、所述异步IO数据片段的偏移量、所述异步IO数据片段的存储位置;When receiving the asynchronous IO data storage request, obtain the first tag information of each of the asynchronous IO data segments; wherein the first tag information includes the ID information of the asynchronous IO data storage request, the asynchronous The offset of the IO data segment, and the storage location of the asynchronous IO data segment;当所述异步IO数据存储请求结束时,将所述第一标签信息中的所述异步IO数据存储请求的ID信息更改为所述异步IO数据存储请求的对象名,以获取每个所述异步IO数据片段的标签信息。When the asynchronous IO data storage request ends, change the ID information of the asynchronous IO data storage request in the first tag information to the object name of the asynchronous IO data storage request to obtain each asynchronous IO data storage request. The tag information of the IO data fragment.
- 根据权利要求2所述的异步IO数据处理方法,其特征在于,在将所述第一标签信息中的所述异步IO数据存储请求的ID信息更改为所述异步IO数据存储请求的对象名之前,还包括:The asynchronous IO data processing method according to claim 2, wherein before changing the ID information of the asynchronous IO data storage request in the first tag information to the object name of the asynchronous IO data storage request ,Also includes:解析每个所述异步IO数据片段的元数据信息,从所述元数据信息中获取所述异步IO数据存储请求的对象名。The metadata information of each asynchronous IO data segment is parsed, and the object name of the asynchronous IO data storage request is obtained from the metadata information.
- 根据权利要求1所述的异步IO数据处理方法,其特征在于,接收 异步IO数据存储请求,包括:The asynchronous IO data processing method according to claim 1, wherein receiving an asynchronous IO data storage request comprises:接收NFS协议下的所述异步IO数据存储请求。Receive the asynchronous IO data storage request under the NFS protocol.
- 一种异步IO数据处理装置,其特征在于,包括:An asynchronous IO data processing device, characterized in that it comprises:接收模块,用于接收异步IO数据存储请求;其中,所述异步IO数据存储请求中包括多个异步IO数据片段;The receiving module is configured to receive an asynchronous IO data storage request; wherein the asynchronous IO data storage request includes a plurality of asynchronous IO data fragments;第一获取模块,用于获取每个所述异步IO数据片段的标签信息,并将所述标签信息存储在预先创建的数据标签存储池中,且将所述异步IO数据片段存储在数据池中;其中,所述标签信息包括所述异步IO数据存储请求的对象名、所述异步IO数据片段的偏移量、所述异步IO数据片段的存储位置;The first acquisition module is used to acquire the tag information of each of the asynchronous IO data fragments, and store the tag information in a pre-created data tag storage pool, and store the asynchronous IO data fragments in the data pool ; Wherein, the tag information includes the object name of the asynchronous IO data storage request, the offset of the asynchronous IO data segment, and the storage location of the asynchronous IO data segment;第二获取模块,用于当接收到异步IO数据读取请求时,获取所述异步IO数据读取请求的对象名;The second obtaining module is configured to obtain the object name of the asynchronous IO data reading request when the asynchronous IO data reading request is received;读取模块,用于在所述数据标签存储池中获取与所述异步IO数据读取请求的对象名对应的所有的标签信息,并根据述标签信息中异步IO数据片段的偏移量及异步IO数据片段的存储位置读取与所述异步IO数据读取请求对应的异步IO数据。The reading module is used to obtain all the tag information corresponding to the object name of the asynchronous IO data read request in the data tag storage pool, and according to the offset and asynchronous of the asynchronous IO data segment in the tag information The storage location of the IO data fragment reads the asynchronous IO data corresponding to the asynchronous IO data read request.
- 根据权利要求5所述的异步IO数据处理装置,其特征在于,所述第一获取模块包括:The asynchronous IO data processing device according to claim 5, wherein the first acquisition module comprises:获取单元,用于在接收到所述异步IO数据存储请求时,获取每个所述异步IO数据片段的第一标签信息;其中,所述第一标签信息包括所述异步IO数据存储请求的ID信息、所述异步IO数据片段的偏移量、所述异步IO数据片段的存储位置;The obtaining unit is configured to obtain the first tag information of each of the asynchronous IO data fragments when the asynchronous IO data storage request is received; wherein the first tag information includes the ID of the asynchronous IO data storage request Information, the offset of the asynchronous IO data segment, and the storage location of the asynchronous IO data segment;更改单元,用于当所述异步IO数据存储请求结束时,将所述第一标签信息中的所述异步IO数据存储请求的ID信息更改为所述异步IO数据存储请求的对象名,以获取每个所述异步IO数据片段的标签信息。The changing unit is configured to change the ID information of the asynchronous IO data storage request in the first tag information to the object name of the asynchronous IO data storage request when the asynchronous IO data storage request ends, so as to obtain Tag information of each of the asynchronous IO data fragments.
- 根据权利要求6所述的异步IO数据处理装置,其特征在于,还包括:The asynchronous IO data processing device according to claim 6, further comprising:解析模块,用于在将所述第一标签信息中的所述异步IO数据存储请求的ID信息更改为所述异步IO数据存储请求的对象名之前,解析每个所述 异步IO数据片段的元数据信息,从所述元数据信息中获取所述异步IO数据存储请求的对象名。The parsing module is used for parsing the metadata of each asynchronous IO data segment before changing the ID information of the asynchronous IO data storage request in the first tag information to the object name of the asynchronous IO data storage request Data information, obtaining the object name of the asynchronous IO data storage request from the metadata information.
- 根据权利要求5所述的异步IO数据处理装置,其特征在于,所述接收模块包括:The asynchronous IO data processing device according to claim 5, wherein the receiving module comprises:接收单元,用于接收NFS协议下的所述异步IO数据存储请求。The receiving unit is configured to receive the asynchronous IO data storage request under the NFS protocol.
- 一种异步IO数据处理设备,其特征在于,包括:An asynchronous IO data processing device, which is characterized in that it comprises:存储器,用于存储计算机程序;Memory, used to store computer programs;处理器,用于执行所述计算机程序时实现如权利要求1至4任一项所述的异步IO数据处理方法的步骤。The processor is configured to implement the steps of the asynchronous IO data processing method according to any one of claims 1 to 4 when the computer program is executed.
- 一种计算机可读存储介质,其特征在于,所述计算机可读存储介质上存储有计算机程序,所述计算机程序被处理器执行时实现如权利要求1至4任一项所述的异步IO数据处理方法的步骤。A computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the asynchronous IO data according to any one of claims 1 to 4 is realized Processing method steps.
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202010071608.9A CN111241036B (en) | 2020-01-21 | 2020-01-21 | Asynchronous IO data processing method, device, equipment and medium |
CN202010071608.9 | 2020-01-21 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2021147360A1 true WO2021147360A1 (en) | 2021-07-29 |
Family
ID=70864992
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2020/117021 WO2021147360A1 (en) | 2020-01-21 | 2020-09-23 | Method, apparatus and device for processing asynchronous io data, and medium |
Country Status (2)
Country | Link |
---|---|
CN (1) | CN111241036B (en) |
WO (1) | WO2021147360A1 (en) |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN111241036B (en) * | 2020-01-21 | 2023-01-10 | 苏州浪潮智能科技有限公司 | Asynchronous IO data processing method, device, equipment and medium |
Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20170318094A1 (en) * | 2016-04-29 | 2017-11-02 | Netapp, Inc. | Self-organizing storage system for asynchronous storage service |
CN108399050A (en) * | 2018-02-02 | 2018-08-14 | 新华三技术有限公司 | A kind of data processing method and device |
CN108763473A (en) * | 2018-05-29 | 2018-11-06 | 郑州云海信息技术有限公司 | A kind of the native object storage method and device of distributed storage |
CN108959500A (en) * | 2018-06-26 | 2018-12-07 | 郑州云海信息技术有限公司 | A kind of object storage method, device, equipment and computer readable storage medium |
US20190020652A1 (en) * | 2017-07-14 | 2019-01-17 | EMC IP Holding Company LLC | System and method for local data ip based network security for preventing data breach attempts in a multi-tenant protection storage deployment |
CN111241036A (en) * | 2020-01-21 | 2020-06-05 | 苏州浪潮智能科技有限公司 | Asynchronous IO data processing method, device, equipment and medium |
Family Cites Families (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN101488104B (en) * | 2009-02-26 | 2011-05-04 | 北京云快线软件服务有限公司 | System and method for implementing high-efficiency security memory |
CN103106233A (en) * | 2012-11-02 | 2013-05-15 | 北京邮电大学 | Asynchronous index and read-write method of massive files applied to search engine |
CN105391649B (en) * | 2015-10-16 | 2018-09-04 | 浪潮(北京)电子信息产业有限公司 | A kind of array dispatching method and device |
CN107277128B (en) * | 2017-06-15 | 2020-09-22 | 苏州浪潮智能科技有限公司 | Method and device for requesting processing order preservation in distributed storage protocol |
CN107516049A (en) * | 2017-07-31 | 2017-12-26 | 广东美的制冷设备有限公司 | Method for reading data, device, safety chip and computer equipment |
CN107797945A (en) * | 2017-10-31 | 2018-03-13 | 郑州云海信息技术有限公司 | A kind of storage system and its date storage method, device, system and equipment |
CN110570075B (en) * | 2019-07-18 | 2022-04-05 | 北京邮电大学 | Power business edge calculation task allocation method and device |
-
2020
- 2020-01-21 CN CN202010071608.9A patent/CN111241036B/en active Active
- 2020-09-23 WO PCT/CN2020/117021 patent/WO2021147360A1/en active Application Filing
Patent Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20170318094A1 (en) * | 2016-04-29 | 2017-11-02 | Netapp, Inc. | Self-organizing storage system for asynchronous storage service |
US20190020652A1 (en) * | 2017-07-14 | 2019-01-17 | EMC IP Holding Company LLC | System and method for local data ip based network security for preventing data breach attempts in a multi-tenant protection storage deployment |
CN108399050A (en) * | 2018-02-02 | 2018-08-14 | 新华三技术有限公司 | A kind of data processing method and device |
CN108763473A (en) * | 2018-05-29 | 2018-11-06 | 郑州云海信息技术有限公司 | A kind of the native object storage method and device of distributed storage |
CN108959500A (en) * | 2018-06-26 | 2018-12-07 | 郑州云海信息技术有限公司 | A kind of object storage method, device, equipment and computer readable storage medium |
CN111241036A (en) * | 2020-01-21 | 2020-06-05 | 苏州浪潮智能科技有限公司 | Asynchronous IO data processing method, device, equipment and medium |
Also Published As
Publication number | Publication date |
---|---|
CN111241036A (en) | 2020-06-05 |
CN111241036B (en) | 2023-01-10 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11086962B2 (en) | Webpage loading method, client and server | |
US9715507B2 (en) | Techniques for reconciling metadata and data in a cloud storage system without service interruption | |
AU2010221620B2 (en) | Content rendering on a computer | |
US6754799B2 (en) | System and method for indexing and retrieving cached objects | |
US20190146681A1 (en) | Data writing and reading method and apparatus, and distributed object storage cluster | |
US9298625B2 (en) | Read and write requests to partially cached files | |
US11687488B2 (en) | Directory deletion method and apparatus, and storage server | |
US9881015B2 (en) | Method and system for previewing file information | |
US8620990B2 (en) | Parsing contents of an e-Form | |
KR19990064246A (en) | A system and method for positioning a page on the World Wide Web and positioning a document from a network computer | |
US10572544B1 (en) | Method and system for document similarity analysis | |
US20190146882A1 (en) | Journal-less recovery for nested crash-consistent storage systems | |
CN103150395B (en) | Directory path analysis method of solid state drive (SSD)-based file system | |
WO2019076102A1 (en) | Data rollback method and system, device, and computer readable storage medium | |
US10489346B2 (en) | Atomic update of B-tree in a persistent memory-based file system | |
US20180349422A1 (en) | Database management system, database server, and database management method | |
WO2021147360A1 (en) | Method, apparatus and device for processing asynchronous io data, and medium | |
US11853284B2 (en) | In-place updates with concurrent reads in a decomposed state | |
WO2015057463A1 (en) | File handlers supporting dynamic data streams | |
CN111290700A (en) | Distributed data reading and writing method and system | |
US20150286672A1 (en) | Information processing method, information processing device, and medium | |
JP4219125B2 (en) | Full-text search device, full-text search method, program, and recording medium | |
US10795875B2 (en) | Data storing method using multi-version based data structure | |
US10706012B2 (en) | File creation | |
WO2021017655A1 (en) | Method, apparatus, and computing device for obtaining inode number, and storage medium |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 20915191 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 20915191 Country of ref document: EP Kind code of ref document: A1 |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 20915191 Country of ref document: EP Kind code of ref document: A1 |