TWI782306B - An access docking device and system, and a method and device applied to the access docking device - Google Patents

Abstract

The present invention discloses an access docking device and system, and a met hod and device applied to the access docking device. The access docking device is set up in a Hadoop computing server and includes a compatibility interface layer for implementing a Hadoop document system interface to achieve an access docking with a Hadoop computing service component; an operational implementation layer for providing a first interface function to the compatibility interface layer to implement the file operation required by the Hadoop computing service component under the document system interface; a storage access layer for providing a second interface function to the operational implementation layer to convert the document operation into an access operation of the target storage in a distributed storage. The access docking device can be used to achieve the effects of decoupling the Hadoop computing service and the storage service and directly accessing the target storage in the distributed storage.

Description

Access docking device, system and method and device for applying the access docking device

The invention belongs to the technical field of distributed storage, and specifically relates to an access docking device, a system, and a method and device for applying the access docking device.

This section is intended to provide a background or context for implementations of the invention that are recited in the claims. The descriptions herein are not admitted to be prior art by inclusion in this section. With the continuous development of big data technology, the decoupling and separation of Hadoop computing services and storage services has gradually become a new development trend due to the following advantages: 1. It can make the technical architecture of storage resources relatively stable and avoid frequent upgrades of computing components Or the impact of expansion; Second, it is convenient to realize the sharing of storage resources. However, in the prior art, there is no solution with good performance and high availability to realize the above-mentioned decoupling and separation of Hadoop computing services and storage services.

Aiming at the problem that it is difficult to realize the decoupling and separation of Hadoop computing services and storage services in the above-mentioned prior art, an access docking device, a system, and a method and device for applying the access docking device are proposed. Using the access docking device, system and application thereof , can solve the above problems.

The present invention provides the following solutions.

In the first aspect, an access docking device is provided, which is deployed on a Hadoop computing server, including: a compatible interface layer, which is used to implement the file system interface of Hadoop in compatibility, so as to realize the access docking with Hadoop computing service components; The interface layer provides the first interface function, so as to realize the file operation required by the Hadoop computing service component under the file system interface; store The access layer converts the file operation into an access operation to the object storage in the distributed storage by providing the second interface function to the operation implementation layer.

In some possible implementation manners, the distributed storage is a Ceph cluster. In some possible implementation manners, the access operation of the object storage is an access operation to the rados cluster in the Ceph cluster.

In some possible implementations, the storage access layer includes: a Crush computing unit, which is used to establish communication with the Mon node of the Ceph cluster to obtain the CrushMap of the Ceph cluster, and calculate the position of the object storage device OSD in the Ceph cluster through the Crush algorithm; The read-write unit is used to establish Socket communication with the object storage device OSD in the Ceph cluster, so as to realize the access operation to the Ceph cluster.

In some possible implementations, the file operations include at least one or more of the following: list files and folders, create folders, delete folders, obtain file status information, rename files, return folders, and open files pointer, write the data stream into the opened file, read the data of the opened file, and realize user authentication.

In some possible implementations, the storage access layer is implemented by a dynamic link library file (Libcephrgw.so) deployed in the Hadoop specified directory, and the second interface function is encapsulated in the dynamic link library file Libcephrgw.so for Access the C++ interface function of the rados cluster in the Ceph cluster.

In some possible implementations, the operation implementation layer is implemented by deploying the second Java package (cephlibrgw.jar) under the Hadoop specified directory, and the second Java package (cephlibrgw.jar) is used to use the dynamic link library file (Libcephrgw. so) The encapsulated C++ interface function is converted into a java interface function, and the first interface function is a java interface function.

In some possible implementation manners, the second Java package (cephlibrgw.jar) utilizes JNI to implement conversion between Java interface functions and C++ interface functions.

In some possible implementation manners, the compatible interface layer is implemented by the first Java package (CephRgwFileSystem.jar) deployed in the specified directory of Hadoop. In some possible implementations, the operations of the file system interface multiplex the HDFS implementation.

In some possible implementation manners, the compatible interface layer is also used for: enabling the yarn component of Hadoop to call the function of the first Java package (CephRgwFileSystem.jar) during operation.

In some possible implementation manners, the access docking device is deployed on each computing server node in the Hadoop computing server cluster.

In some possible implementation manners, the content of the Hadoop configuration file core-site.xml includes the main class information of the access docker.

In some possible implementation manners, the distributed storage utilizes an idle storage interface to provide storage services to other computing platforms other than the Hadoop computing server cluster.

In some possible implementation manners, the distributed storage is a Ceph cluster, and the free storage interface includes a block device storage interface and a file system storage interface.

In a third aspect, a method for applying an access docking device is provided, including: receiving an access request from a Hadoop computing service component; using the access docking device in the first aspect above to convert the access request into an access to object storage in distributed storage operate.

In some possible implementation manners, before receiving the access request of the Hadoop computing service component, it further includes: using the Hadoop configuration file content core-site.xml to obtain the main class information of the access docker.

In a fourth aspect, a device for applying an access docking device is provided, including: a receiving module, configured to receive an access request from a Hadoop computing service component; an access module, configured to convert the access request into Access operations to object storage in distributed storage.

In some possible implementation manners, it also includes: a loading module, configured to use the content core-site.xml of the Hadoop configuration file to obtain the main class information of the access docker.

In a fifth aspect, there is provided an application access docking device, including: one or more multi-core processors; memory for storing one or more programs; when one or more programs are executed by one or more multi-core processors , so that one or more multi-core processors implement the method in the third aspect above.

In a sixth aspect, a computer-readable storage medium is provided. The computer-readable storage medium stores a program, and when the program is executed by a multi-core processor, the multi-core processor executes the method in the third aspect above.

The above-mentioned at least one technical solution adopted in the embodiment of the present application can achieve the following beneficial effects: through the cooperative work of the compatible interface layer, the operation implementation layer and the object access layer in the above-mentioned access docker, it is possible to change any Hadoop storage service and management layer above In the case of interface and software implementation, it supports the heterogeneous decoupling of Hadoop's computing services and storage services, enabling Hadoop's computing service components to directly access heterogeneous distributed storage in the form of object storage access operations, improving performance and availability. Distributed storage is, for example, a Ceph cluster.

It should be understood that the above description is only an overview of the technical solution of the present invention, so as to understand the technical means of the present invention more clearly, so as to be implemented according to the contents of the description. In order to make the above and other objects, features and advantages of the present invention more comprehensible, the specific implementation manners of the present invention are illustrated below.

1:Hadoop computing server

11:Hadoop Computing Service Components

100: Access docker

101: Compatible interface layer

102: Operation implementation layer

103:Storage Access Layer

1031: file reading and writing unit

1032: Crush calculation unit

2: Ceph cluster

300: method

400: device

401: receiving module

402: access module

200: external equipment

500: device

10: Processor

20: memory

21: RAM

22: Cache

23:ROM

24: Program module

30: Display unit

40: I/O interface

50: network adapter

60: bus

600: computer readable storage medium

The advantages and benefits described herein, as well as other advantages and benefits, will be apparent to those of ordinary skill in the art from reading the following detailed description of the exemplary embodiments. The drawings are only for the purpose of illustrating exemplary embodiments and are not to be considered as limiting the invention. Also throughout the drawings, the same reference numerals are used to denote the same parts. In the accompanying drawings: [Fig. 1] is a structural schematic diagram of the access docking device, one embodiment of the access docking device, system, and method and device for applying the access docking device of the present invention.

[ Fig. 2 ] is a schematic diagram of the FileSystem interface of an embodiment of the access docking device, the system and the method and device for using the access docking device of the present invention.

[ Fig. 3 ] is a schematic flowchart of a method for using an access docker in one embodiment of the access docker, system, and method and device for applying the access docker in the present invention.

[ Fig. 4 ] is a structural diagram of a device for applying an access docker in one embodiment of the access docker, system, and method and device for applying the access docker in the present invention.

[ Fig. 5 ] is a structural diagram of a device for applying an access docker in one embodiment of the access docker, the system, and the method and device for applying the access docker in the present invention.

[ FIG. 6 ] is a schematic diagram of a computer-readable storage medium of an embodiment of the access docking device, the system, and the method and device for using the access docking device of the present invention.

In the drawings, the same or corresponding reference numerals denote the same or corresponding parts.

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and It should not be limited by the examples set forth herein. Rather, these embodiments are provided so that the present disclosure can be more thoroughly understood, and the scope of the present disclosure can be fully conveyed to those skilled in the art.

In the present invention, it should be understood that terms such as "comprising" or "having" are intended to indicate the presence of features, numbers, steps, acts, components, parts or combinations thereof disclosed in the specification, and are not intended to exclude one or multiple other features, numbers, steps, acts, parts, parts or combinations thereof.

Before describing the present invention, some technical terms appearing in the present invention are briefly explained.

Hadoop: A distributed system infrastructure developed by the Apache Foundation. Users can develop distributed programs without knowing the underlying details of the distribution. Make full use of the power of the cluster for high-speed computing and storage. Hadoop is currently the most widely used distributed computing platform. It adopts the MapReduce distributed computing model and provides a series of interfaces and frameworks to help users efficiently utilize the computing resources of distributed clusters and improve the parallelism of computing.

Ceph: A unified, distributed file system designed for reliability and scalability, with excellent performance. Object Storage: Object storage, also known as object-based storage, is a general term used to describe methods of addressing and processing discrete units called objects. Like files, objects contain data, but unlike files, objects are never hierarchical within a hierarchy. Every object is in the same level of a flat address space called a storage pool, and one object does not belong to the next level of another object. In addition, it should be noted that, in the case of no conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and examples.

As shown in FIG. 1 , this embodiment provides an access docking device 100 . The access dock 100 is deployed on the Hadoop computing server 1 , and the access dock 100 includes: a compatible interface layer 101 , an operation implementation layer 102 and a storage access layer 103 . Wherein, the compatible interface layer 101 is used for compatiblely implementing the file system interface of Hadoop, thereby realizing the access docking with the Hadoop computing service component 11, and the operation realization layer 102 provides the first interface function to the compatible interface layer 101, thereby under the file system interface Realize the required file operations of the Hadoop computing service components, the storage access layer 103 provides the second interface function to the operation implementation layer 102, thereby converting the file operations into access operations to the object storage in the distributed storage, to realize and distribute Stored access docking.

In some possible implementation manners, the above-mentioned distributed storage may preferably be a Ceph cluster 2 . It can be understood that this embodiment can also be applied to realize the interconnection between Hadoop and other distributed storage devices other than the Ceph cluster. This embodiment uses the Ceph cluster as an example for description, but is not limited thereto. By adopting the CEPH cluster as the distributed storage connected to Hadoop, for Hadoop, it can effectively improve the file read and write performance and improve the file access efficiency. At the same time, the data in Hadoop can be mounted to the user space through CEPH, realizing data storage Diversified management; for the CEPH cluster, the access to the CEPH cluster through the Hadoop platform provides the access interface of the Java programming language for the CEPH cluster, which greatly expands the application scenarios and application scope of CEPH.

件(create)、創建目錄(mkdir)、創建文件流(open)等虛方法，用於Hadoop所需的各種文件操作在Ceph上的實現；其中，配置項fs.cephRgw.impl表示cephRgw的實現類。配置項ceph.auth.id、ceph.conf.file、ceph.auth.access用戶的access key、ceph.auth.secret、ceph.auth.secret、mon host等設置了Ceph集群的參數，配置項fs.AbstractFileSystem.cephRgw.impl表示cephRgw的抽象文件系統的實現類。 In some possible implementation manners, the content of the Hadoop configuration file core-site.xml includes the main class information of the access docker. For example, as shown in Table 1, add the following configuration items to the Hadoop configuration file content core-site.xml:

Create virtual methods such as file (create), create directory (mkdir), and create file stream (open), which are used to implement various file operations required by Hadoop on Ceph; among them, the configuration item fs.cephRgw.impl represents the implementation class of cephRgw . The configuration items ceph.auth.id, ceph.conf.file, ceph.auth.access user's access key, ceph.auth.secret, ceph.auth.secret, mon host, etc. set the parameters of the Ceph cluster, and the configuration item fs. AbstractFileSystem.cephRgw.impl represents the implementation class of cephRgw's abstract file system.

Furthermore, AbstractFileSystem plays a role similar to a virtual file system (VFS) in Hadoop, which is used by Hadoop when the format of the file system is unclear, and realizes the function of creating a file CephRgw Functions include: CephRgw(URIthisUri,Configurationconf)throwsIOException, URISyntaxException ; Indicates that the function in the CephRgwFileSystem class is called during the running process of the Hadoop layer component.

The functions and internal implementation structures of the compatible interface layer 101 , the operation implementation layer 102 and the storage access layer 103 are illustrated below.

(1) Compatible interface layer 101

The compatible interface layer 101 is used to implement the Hadoop file system interface (FileSystem) compatiblely, thereby realizing the access docking with Hadoop computing service components; Service components form an access interface. Specifically, it can be called by Hadoop computing service components to execute various file-related methods or implement file-related operations, realize Hadoop file system interface functions, and shield Hadoop computing service components from calling differences in file IO.

Figure 2 shows the abstract methods contained in the FileSystem interface. The FileSystem interface supports Hadoop computing service components to perform file-related operations on demand. The specific functions include but are not limited to: initializing the file system through configuration files, creating files or folders, obtaining File or folder information, set file or folder permissions, create file read and write data streams, read and write files, rename or delete folders.

In some possible implementation manners, the above-mentioned compatible interface layer 101, that is, the CephRgwFileSystem layer, can be implemented by the first Java package CephRgwFileSystem.jar deployed in the specified directory of Hadoop. For example, the CephRgwFileSystem.jar can be placed under share/Hadoop/common/lib of Hadoop. In addition, the above-mentioned CephRgwFileSystem.jar can also be used to realize the docking of Hadoop's scheduling service components (such as Yarn) for special file storage requirements such as cache storage locations.

In some possible implementation manners, the CephRgwFileSystem layer is also used to: make yarn call the function function in the CephRgwFileSystem class when the Hadoop components are running. For example, the above functions can be realized through the CephRgw function deployed in Hadoop.

For example, the CephRgw functional function can be: CephRgw(URIthisUri, Configuration conf) throws IOException, URISyntaxException; Therefore, when calling the functional functions in the CephRgwFileSystem class, Hadoop components can access the Ceph cluster in a local distributed manner, without rewriting the business code, which simplifies the use of client code.

For example, as shown in Table 2, the functional functions included in the CephRgwFileSystem class include:

It can be seen from the compatibility interface layer 101 above that by introducing CephRgwFileSystem, a new implementation class of FileSystem, compatibility with HDFS access and connection with Hadoop computing service components can be realized.

(2) Operation Realization Layer 102

The operation implementation layer 102 provides the first interface function to the compatible interface layer 101 of the upper layer, thereby realizing the file operation required by the Hadoop computing service component under the FileSystem interface; specifically, the operation implementation layer 102 is also the cephlibrgw layer, which can be deployed in It is realized by the second Java package cephlibrgw.jar in the specified directory of Hadoop. For example, the cephlibrgw.jar can be placed under share/Hadoop/common/lib of Hadoop to implement the above cephlibrgw layer.

In some possible implementations, the above file operations include at least one or more of the following: list files and folders, create folders, delete folders, obtain file status information, rename files, return folders, open The pointer of the file, write the data stream into the opened file, read the data of the opened file, and realize user authentication.

For example, as shown in Table 3, the first interface function provided by the cephlibrgw layer is a Java interface function, which may include:

(3) Storage access layer 103

The storage access layer 103 provides the second interface function to the operation implementation layer 102, thereby transforming the file operation into an access operation to the object storage in the distributed storage. The access operation of the object storage is specifically the access operation of the rados cluster in the Ceph cluster.

In some possible implementations, the storage access layer 103 is a C language layer, which can be implemented by the dynamic link library file Libcephrgw.so deployed in the Hadoop specified directory, such as The libcephrgw.so can be placed under the /usr/lib64/ folder of Hadoop to implement the storage access layer 103 above.

The second interface function provided by the storage access layer 103 to the operation implementation layer 102 may specifically be a C++ interface function encapsulated in Libcephrgw.so for accessing the rados cluster in the Ceph cluster. The C++ interface function provides function interfaces for basic operations such as file creation, file access, file reading, file writing, file update, directory listing, file name query, file status query, and system status query, and repackages the initialization system Handle, obtain operation handle and other functions, the user only needs to apply for the corresponding handle and directly call the operation function to perform the corresponding operation, without manually managing the intermediate variables and parameters inside Ceph.

For example, as shown in Table 4, the second interface function provided by Libcephrgw.so is a C++ interface function, and may include:

In some possible implementations, the above-mentioned operation implementation layer 102 can also call the C++ interface function encapsulated by Libcephrgw.so, and convert it into a java interface function provided to the upper compatible interface layer, that is, the first interface function. Specifically, the operation implementation layer 102 uses JNI to implement conversion between Java interface functions and C++ interface functions. Among them, the above-mentioned JNI provides several calling interfaces to realize the communication between the Java language and the C++ language. It can be understood that the programming language used by Hadoop is Java language, and the language used by Ceph cluster is C++ language. Java language cannot directly operate hardware, so you can call C++ libraries or functions through JNI to operate hardware to avoid repeated development.

In some possible implementations, as shown in Figure 1, the storage access layer 103 specifically includes: a Crush computing unit 1032 and a file reading and writing unit 1031, wherein the Crush computing unit is used to establish communication with the Mon node of the Ceph cluster to obtain the Ceph The CrushMap of the cluster, and calculate the location of the object storage device OSD in the Ceph cluster through the Crush algorithm; the file read and write unit is used to communicate with the Ceph The object storage device OSD in the ph cluster 2 establishes Socket communication to realize the access operation to the Ceph cluster, that is, to realize the connection to the Ceph cluster.

In some possible implementation manners, the access docking device 100 is specifically deployed on each Hadoop computing server node in the Hadoop computing server cluster. In this way, Hadoop's big data computing services can be distributed and directly access Ceph storage without additional gateways, the access path is short, and performance and availability are improved.

Through the cooperative work of the compatible interface layer, operation implementation layer and object access layer in the above-mentioned access docking device, it is possible to support Hadoop computing services and storage services without changing any Hadoop storage services and interfaces and software implementations above the management layer. Structural decoupling enables Hadoop's computing service components to directly access heterogeneous distributed storage in the form of object storage access operations, improving performance and availability.

Based on the above access docking device, an embodiment of the present application further provides an access docking system. It includes: a Hadoop computing server cluster and distributed storage, wherein the above-mentioned access docking device is deployed on each computing server of the Hadoop computing server cluster to connect each computing server to the distributed storage.

In some possible implementation manners, the distributed storage uses an idle storage interface to provide storage services to computing platforms outside the Hadoop computing server cluster. For example, the storage resources of the Ceph cluster can be shared with different applications of big data, virtual machines, and containers at the same time, so as to realize the sharing of storage resources.

In some possible implementation manners, the distributed storage is a Ceph cluster, and the free storage interface includes a block device storage interface and a file system storage interface.

It should be noted that the access docking system in the embodiment of the present application can implement various aspects of the embodiment of the access docking device, and achieve the same effect and function, and will not be repeated here.

Based on the access docking device described above, the embodiment of the present application further provides a method for using the access docking device. Fig. 3 is a schematic flow diagram of a method for application access docking device according to an embodiment of the present application. As shown in Fig. 3, method 300 includes: step S301: receiving the access request of Hadoop computing service component; step S302: using the above-mentioned access docking device , transforming the access request into an access operation to the object storage in the distributed storage.

In some possible implementation manners, the method 300 may further include: acquiring the main class information of the access docker by using the content core-site.xml of the Hadoop configuration file.

Next, the data access process using the put file as an example will describe in detail the method for the above application to access the docking device.

Firstly, it is executed by the compatible interface layer: step S41: fragment the put file.

Step S42: Send the put file to the operation implementation layer in the form of data flow through the create function.

Among them, Hadoop fragments the file according to the io.file.buffer.size configuration item of the core-site.xml file (the default is 4096 bytes); the file output stream of CephRgwOutputStream is constructed through the create function defined in the Filesystem interface, and passed to The lower layer operation is implemented layer by layer; in CephRgwOutputStream, set the buffer size according to the ceph.io.buffer.size configuration item of the core-site.xml file (the default is 4M); Hadoop calls the Write function of CephRgwOutputStream to transfer the file content to cephlibrgw.jar .

Second, performed by the operation implementation layer: Step S43: Realize the connection between the Java interface function and the C++ interface function, and continue to pass the data flow down to the storage access layer.

Wherein, the operation implementation layer transfers the file data flow to the storage access layer by calling the C++ interface function provided by the lower storage access layer.

Again, executed by the storage access layer: S44: obtain Ceph cluster information, and re-shard the data flow. S45: Calculate the OSD position corresponding to each slice. S46: Communicate directly with the OSD and upload files.

Among them, the storage access layer communicates with the ceph mon through the Crush computing unit, obtains the ceph cluster information, and re-segments the file data stream according to the underlying Objects size of the ceph cluster (default is 4M); the storage access layer passes the Crush The computing unit communicates with ceph mon to obtain the Crush Map, and calculates the ip and port number of the main OSD corresponding to each fragment in the Crush computing unit according to the fragmentation information; the storage access layer establishes communication through file read and write operations and OSD The data is transmitted asynchronously. After the transmission is completed, the OSD end will return the message. By applying the above-mentioned access docker, using the compatible interface layer, operation implementation layer and object access layer to work together, it can be used without changing any Hadoop storage services and above the management layer. In the case of interface and software implementation, it supports the heterogeneous decoupling of Hadoop's computing services and storage services, enabling Hadoop's computing service components to directly access heterogeneous distributed storage in the form of object storage access operations, improving performance and availability.

Based on the above access docking device, the present application further provides a device for using the access docking device. Fig. 4 is a schematic structural diagram of an application access docking device according to an embodiment of the present application. As shown in Fig. 4, the device 400 includes: a receiving module 401 for receiving an access request from a Hadoop computing service component; an access module 402 for using The above-mentioned access docker is used to convert the access request into an access operation to the object storage in the distributed storage.

In some embodiments, the apparatus 400 further includes: a loading module, configured to load the access docker using the core-site.xml content of the Hadoop configuration file.

Those skilled in the art can understand that various aspects of the present invention can be implemented as devices, methods, or computer-readable storage media. Therefore, various aspects of the present invention can be embodied in the following forms, that is: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which can be collectively referred to herein as "circuit", "module" or "equipment".

In some possible implementation manners, an apparatus for accessing a docking device by application of the present invention may at least include one or more processors and at least one memory. Wherein, the memory stores a program, and when the program is executed by the processor, the processor is made to perform the steps shown in Figure 3: Step S301: Receive the access request of the Hadoop computing service component; Step S302 uses the above-mentioned access docking device to access The request is converted into an access operation to the object storage in the distributed storage.

An apparatus 500 for accessing a docking device by an application according to this embodiment of the present invention will be described below with reference to FIG. 5 .

The device 500 shown in FIG. 5 is only an example, and should not impose any limitation on the functions and scope of use of this embodiment of the present invention.

As shown in FIG. 5 , the apparatus 500 may be in the form of a general-purpose computing device, including but not limited to: at least one processor 10 , at least one memory 20 , and a bus 60 connecting different device components.

The bus 60 includes a data bus, an address bus and a control bus.

The memory 20 may include a volatile memory, such as a random access memory (RAM) 21 and/or a cache memory 22 , and may further include a read only memory (ROM) 23 .

Memory 20 may also include program modules 24, such program modules 24 including, but not limited to, an operating device, one or more application programs, other program modules, and program data, each or some combination of which may include network realization of the environment.

The apparatus 500 can also communicate with one or more external devices 200 (such as keyboards, pointing devices, Bluetooth devices, etc.), and can also communicate with one or more other devices. Such communication may be performed through an input/output (I/O) interface 40 and displayed on the display unit 30 . Moreover, the device 500 can also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN) and/or a public network such as the Internet) through the network adapter 50 . As shown, network adapter 50 communicates with other modules in device 500 via bus 60 . It should be appreciated that although not shown, other hardware and/or software modules may be used in conjunction with apparatus 500, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID devices, tape drives And data backup storage devices, etc.

Fig. 6 shows a computer-readable storage medium for performing the method described above. In some possible implementations, various aspects of the present invention can also be implemented in the form of a computer-readable storage medium, which includes program code, and when the program code is executed by a processor, the program code is used to use The processor executes the methods described above.

The method described above includes multiple operations and steps shown and not shown in the above figures, which will not be repeated here.

The computer readable storage medium may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. Readable storage media can, for example, Is - but is not limited to - an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor device, device, or device, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: electrical connection with one or more conductors, portable disk, hard disk, random access memory (RAM), read only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

As shown in FIG. 6 , a computer-readable storage medium 600 according to an embodiment of the present invention is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program codes, and can be stored in a terminal device, such as a personal run on the computer. However, the computer-readable storage medium of the present invention is not limited thereto. In this document, the readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution device, device, or device .

Program code for carrying out the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., as well as conventional procedural programming Language - such as "C" or similar programming language. The program code may execute entirely on the user computing device, partly on the user device and partly on the remote computing device, or entirely on the remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it may be connected to an external computing device (for example, using an Internet service Provider via Internet connection).

In addition, although operations of the methods of the present invention are depicted in a particular order in the figures, this does not require or imply that the operations must be performed in that particular order, or that all operations must be performed. The actions shown in the section will achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step for execution, and/or one step may be decomposed into multiple steps for execution.

Although the spirit and principles of the invention have been described with reference to a number of specific embodiments, it should be understood that the invention is not limited to the specific embodiments disclosed, nor does division of aspects imply that features in these aspects cannot be combined to achieve optimal performance. Benefit, this division is only for the convenience of expression. The present invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

1:Hadoop computing server

11:Hadoop Computing Service Components

100: Access docker

101: Compatible interface layer

102: Operation implementation layer

103:Storage Access Layer

1031: file reading and writing unit

1032: Crush calculation unit

2: Ceph cluster

Claims (19)

An access docking device is characterized in that it is deployed on a Hadoop computing server, and includes: a compatible interface layer, which is used to implement compatible file system interfaces of Hadoop, thereby realizing access docking with Hadoop computing service components; The compatible interface layer provides the first interface function, thereby realizing the file operation required by the Hadoop computing service component under the file system interface; the storage access layer provides the second interface function to the operation implementation layer, thereby the described File operations are transformed into access operations on object storage in distributed storage, where the distributed storage is a Ceph cluster, and the storage access layer includes: a Crush computing unit, which is used to establish communication with the Mon node of the Ceph cluster to obtain Ceph The CrushMap of the cluster, and calculate the position of the object storage device OSD in the Ceph cluster by the Crush algorithm, wherein the storage access layer is implemented by the dynamic link library file deployed in the Hadoop specified directory, and the second interface function is the Encapsulated in the dynamic link library file, for accessing the C++ interface function of the object storage in the distributed storage, the operation implementation layer is implemented by the second Java package deployed in the Hadoop specified directory, the second Java The package is used to convert the C++ interface function encapsulated in the dynamic link library file into a java interface function, and the java interface function is the first interface function. The access docking device according to claim 1, wherein the access operation of the object storage is an access operation to the rados cluster in the Ceph cluster. The access docking device as described in claim item 1, wherein the storage access layer also includes: a file read and write unit, which is used to establish Socket communication with the object storage device OSD in the Ceph cluster, so as to realize the access to the Ceph cluster operate. The access docking device according to claim 1, wherein the file operations include at least one or more of the following: list files and folders, create folders, delete folders, Get the status information of the file, rename the file, return the folder, open the pointer of the file, write the data stream into the opened file, read the data of the opened file, and realize user authentication. The access docking device according to claim 1, wherein the second Java package utilizes JNI to implement conversion between the Java interface function and the C++ interface function. The access docking device according to claim 1, wherein the compatible interface layer is implemented by a first Java package deployed in a Hadoop specified directory. The access docking device according to claim 1, wherein the operation of the file system interface reuses the implementation of the Hadoop distributed file system. The access docking device according to claim 6, wherein the compatible interface layer is further configured to: enable the yarn component of Hadoop to call the function in the first Java package during operation. The access docking device according to claim 1, wherein the access docking device is deployed on each computing server node in the Hadoop computing server cluster. The access docking device according to claim 1, wherein the content of the Hadoop configuration file core-site.xml includes the main class information of the access docking device. An access docking system, comprising: a Hadoop computing server cluster and distributed storage, characterized in that the access docking device as described in any one of request items 1-10 is deployed on each computing server node of the Hadoop computing server cluster , for connecting each computing server node to the distributed storage. The access docking system according to claim 11, wherein the distributed storage uses an idle storage interface to provide storage services to computing platforms other than the Hadoop computing server cluster. The access docking system according to claim 12, wherein the distributed storage is a Ceph cluster, and the free storage interface includes a block device storage interface and a file system storage interface. A method for applying an access docking device, comprising: receiving an access request of a Hadoop computing service component; utilizing the access docking device as described in any one of request items 1-10, converting the access request into a distributed The access operation of the object storage in the storage. The method according to claim 14, wherein, before receiving the access request of the Hadoop computing service component, further comprising: obtaining the main class information of the access docker by using the content core-site.xml of the Hadoop configuration file. An application access docking device, characterized in that it includes: a receiving module for receiving the access request of the Hadoop computing service component; an access module for utilizing the access docking device as described in any one of request items 1-10 , converting the access request into an access operation on the object storage in the distributed storage. The device according to claim 16, further comprising: a loading module, configured to obtain the main class information of the access docker by using the Hadoop configuration file content core-site.xml. An application access docking device, characterized in that it includes: one or more multi-core processors; a memory for storing one or more programs; when the one or more programs are processed by the one or more multi-cores When executed by a multi-core processor, the one or more multi-core processors implement the method described in claim 14. A computer-readable storage medium, the computer-readable storage medium stores a program, and when the program is executed by a multi-core processor, the multi-core processor executes the method described in claim 14.

Images