WO2024113898A1

WO2024113898A1 - Metadata reporting method and apparatus, and device and storage medium

Info

Publication number: WO2024113898A1
Application number: PCT/CN2023/108423
Authority: WO
Inventors: 孙业宽; 张在贵
Original assignee: 苏州元脑智能科技有限公司
Priority date: 2022-11-30
Filing date: 2023-07-20
Publication date: 2024-06-06
Also published as: CN115550368B; CN115550368A

Abstract

The present application relates to the technical field of data processing. Provided are a metadata reporting method and apparatus, and a device and a storage medium. The method comprises: determining a corresponding load balanced allocation mode according to a magnitude relationship between the total number of ES services in an ES cluster and the total number of MDSs in a distributed file system; and allocating an ES service to each of the MDSs according to the load balanced allocation mode, and generating configuration information on the basis of the allocation result, such that the MDS performs metadata reporting for the allocated candidate ES service according to the configuration information. By means of balancing, MDSs are connected to different ESs, such that the MDSs realize, by means of load balancing, balance of pressure of metadata reporting according to conditions such as the number of ESs, that is, ES load balancing is realized, thereby exerting reporting performance to the maximum extent.

Description

A metadata reporting method, device, equipment and storage medium

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to the Chinese patent application filed with the China Patent Office on November 30, 2022, with application number 202211518325.X, and application name “A metadata reporting method, device, equipment and storage medium”, all contents of which are incorporated by reference in this application.

Technical Field

The present application relates to the field of data processing technology, and in particular to a metadata reporting method, device, equipment and storage medium.

Background technique

At present, the metadata retrieval function of the distributed file system based on Elasticsearch (ES, an open source distributed, RESTful-style search and data analysis engine) can achieve sub-minute retrieval of tens of billions of files, support users to configure simple and complex retrieval conditions, and efficiently assist users in data management. For example, as shown in Figure 1, when accessing metadata, the file passes through the MDS service (metadata service). All files in the distributed file system are distributed by different MDSs. Each MDS will report the latest metadata to different ES when the metadata of the file it is responsible for is updated. Users configure different retrieval conditions to retrieve a list of files that meet the requirements from the ES cluster. The distributed file system is managed by multiple MDS services, and the ES cluster is also composed of multiple ES services. When reporting metadata, if the MDS reporting pressure is all pressed on the same ES, the ES performance will be reduced.

Summary of the invention

In view of this, the purpose of this application is to provide a metadata reporting method, device, equipment and medium, which can achieve a balanced metadata reporting pressure and maximize the reporting performance of the program. The specific scheme is as follows:

In a first aspect, the present application discloses a metadata reporting method, comprising:

Determine the corresponding load balancing distribution method based on the relationship between the total number of ES services in the ES cluster and the total number of MDS services in the distributed file system;

An ES service is allocated to each MDS service in a load balancing manner, and configuration information is generated based on the allocation result, so that the MDS service reports metadata to an allocated candidate ES service according to the configuration information.

Optionally, according to the relationship between the total number of ES services in the ES cluster and the total number of MDS services in the distributed file system, a corresponding load balancing distribution method is determined, including:

If the total number of ES services is less than the total number of MDS services, the load balancing distribution method is the first target load balancing distribution method.

Optionally, allocate ES services to each MDS service in a load balancing manner, including:

The initial allocation is completed by sorting all MDS services and sorting all ES services, and allocating ES services to MDS services with the same sorting position according to the sorting position of the ES services;

According to the order of ES services, one ES service is selected from the beginning and allocated to the MDS service that has not been allocated an ES service after the initial allocation, so that each MDS service corresponds to at least one ES service.

If the total number of ES services is equal to the total number of MDS services, the load balancing distribution method is the second target load balancing distribution method.

By sorting all the MDS services and sorting all the ES services, the ES services are allocated to the MDS services with the same sorting position according to the sorting position of the ES services, so that there is a one-to-one correspondence between the MDS services and the ES services.

If the total number of ES services is greater than the total number of MDS services, the load balancing distribution method is the third target load balancing distribution method.

The ES services that are not allocated after the initial allocation are reallocated according to the order of the MDS services until all ES services are allocated so that each ES service corresponds to at least one MDS service.

Optionally, before allocating an ES service to each MDS service in a load balancing manner, the following is also included:

The communication address is configured for each ES service in the ES cluster through the distributed file system, so that the MDS service can report data to the corresponding ES service according to the communication address of the ES service.

Optionally, the metadata reporting method also includes:

If the candidate ES service for receiving metadata reports currently corresponding to the target MDS service in the distributed file system fails, the target MDS is switched to the normally operating ES service so that the target MDS reports metadata to the normally operating ES service.

Optionally, before a candidate ES service for receiving metadata reports corresponding to a target MDS service in a distributed file system fails, the method further includes:

The target MDS reports data to a corresponding candidate ES service according to the configuration information, and determines whether the candidate ES service has a fault according to the data reporting result fed back by the candidate ES service.

Optionally, switch the target MDS to a running ES service, including:

Determine whether there is a first target ES service that is operating normally among all candidate ES services corresponding to the target MDS according to the configuration information;

The reporting object switching mode for the target MDS is determined according to the judgment result, and the target MDS is switched to the normally operating ES service according to the reporting object switching mode.

Optionally, determining a reporting object switching method for the target MDS according to the judgment result, and switching the target MDS to a normally operating ES service according to the reporting object switching method, including:

If there is no first target ES service operating normally among all candidate ES services corresponding to the target MDS in the configuration information, all ES services in the ES cluster are polled until a normally operating ES service is found, and the target MDS is switched to the normally operating ES service.

If there is a first target ES service that is operating normally among all candidate ES services corresponding to the target MDS in the configuration information, the target MDS is switched to the first target ES service.

Optionally, the metadata reporting method also includes:

Determine whether the normally running ES service hits the candidate ES service corresponding to the target MDS based on the configuration information;

According to the hit result, it is determined whether to perform the reporting object switchback processing on the normally running ES service currently corresponding to the target MDS.

Optionally, based on the hit result, it is determined whether to perform a report object switchback process on the normally running ES service currently corresponding to the target MDS, including:

If the normally operating ES service does not hit the candidate ES service corresponding to the target MDS, the status of all candidate ES services corresponding to the target MDS will be monitored regularly, and when there is a normally operating second target ES service among all candidate ES services corresponding to the target MDS, the target MDS will be switched back to the second target ES service.

If the normally operating ES service hits the candidate ES service corresponding to the target MDS, the current reporting connection is retained.

In yet another aspect, the present application discloses a metadata reporting system, including an ES cluster and a distributed file system;

The ES service in the ES cluster and the MDS service in the distributed file system are services allocated by a load balancing allocation method; the load balancing allocation method is determined according to the size relationship between the total number of ES services in the ES cluster and the total number of MDS services in the distributed file system;

The MDS service is used to report metadata to a corresponding ES service according to configuration information; the configuration information is configuration information generated based on the allocation result.

In another aspect, the present application discloses a metadata reporting device, comprising:

The load balancing distribution mode determination module is used to determine the corresponding load balancing distribution mode according to the size relationship between the total number of ES services in the ES cluster and the total number of MDS services in the distributed file system;

The allocation module is used to allocate ES services to each MDS service in a load balancing manner, and generate configuration information based on the allocation result so that the MDS service reports data to the corresponding ES service according to the configuration information.

In another aspect, the present application discloses an electronic device, comprising:

Memory, used to store computer programs;

The processor is used to execute a computer program to implement the aforementioned metadata reporting method.

On the other hand, the present application discloses a non-volatile readable storage medium for storing a computer program; wherein the computer program implements the aforementioned metadata reporting method when executed by a processor.

In this application, the corresponding load balancing distribution method is determined according to the size relationship between the total number of ES services in the ES cluster and the total number of MDS services in the distributed file system; ES services are allocated to each MDS service according to the load balancing distribution method, and configuration information is generated based on the distribution result, so that the MDS service reports metadata to an allocated candidate ES service according to the configuration information. It can be seen that by allocating ES services according to the size relationship between the total number of ES services in the ES cluster and the total number of MDS services in the distributed file system, the currently adapted load balancing distribution method is used. Thus, based on the metadata retrieval function of the distributed file system of Elasticsearch, by balancing each MDS connected to different ES, the MDS realizes the balance of metadata reporting pressure according to the number of ES and other conditions through load balancing, that is, ES load balancing is realized, so as to maximize the reporting performance of the program.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

FIG1 is a schematic diagram of the structure of a metadata reporting system in the prior art;

FIG2 is a flowchart of a metadata reporting method provided by the present application;

FIG3 is a schematic diagram of a load balancing distribution method provided by the present application;

FIG4 is a schematic diagram of another load balancing distribution method provided by the present application;

FIG5 is a schematic diagram of another load balancing distribution method provided by the present application;

FIG6 is a flowchart of a specific metadata reporting method provided by the present application;

FIG7 is a schematic diagram of a specific ES service failure switching provided by the present application;

FIG8 is a flowchart of a specific metadata reporting method provided by the present application;

FIG9 is a schematic diagram of a specific ES service switchback provided by the present application;

FIG10 is a schematic diagram of the structure of a metadata reporting device provided by the present application;

FIG11 is a structural diagram of an electronic device provided in this application.

Detailed ways

In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

In the prior art, a distributed file system is managed by multiple MDS services, and an ES cluster is also composed of multiple ES services. If the MDS reporting pressure is all applied to the same ES when reporting metadata, the ES performance will be reduced. In order to overcome the above technical problems, this application proposes a metadata reporting method that can achieve a balanced metadata reporting pressure and maximize the reporting performance.

Some embodiments of the present application disclose a metadata reporting method. Referring to FIG. 2 , the method may include the following steps:

Step S11: Determine a corresponding load balancing distribution method according to the size relationship between the total number of ES services in the ES cluster and the total number of MDS services in the distributed file system.

In some embodiments of the present application, the total number of ES services in the ES cluster and the total number of MDS services in the distributed file system are first counted, and then the corresponding load balancing distribution method is determined based on the size relationship between the total number of ES services and the total number of MDS services, but at least each MDS corresponds to at least one ES. The size relationship between the total number of ES services and the total number of MDS services can be divided into three situations, namely, the total number of ES services is less than the total number of MDS services, the total number of ES services is equal to the total number of MDS services, and the total number of ES services is greater than the total number of MDS services; each situation corresponds to Different load balancing methods.

ES is an open source distributed, RESTful-style search and data analysis engine that encapsulates Lucene. It provides a set of simple and consistent RESTful APIs to help implement storage and retrieval. Multiple ESs construct an ES distributed search cluster. After an ES service fails, the ES cluster can still provide services normally. ES is a distributed search engine and also a distributed database.

A distributed file system refers to a cluster composed of multiple file storage node servers. Files are stored in blocks, with objects as the basic unit. It supports storing a copy of data on multiple nodes. Each node can obtain complete data through inter-node communication. When a node goes down, complete data can be restored according to the configured policy. It has the characteristics of high availability, high performance, and high scalability. Each node provides metadata services, namely MDS, for various metadata access operations to balance business pressure. MDS service, or metadata service, is used to maintain file metadata and process different metadata requests from clients. Multiple MDSs construct a metadata service cluster, and each MDS is responsible for different subtrees of the entire system file tree to form a distributed metadata service cluster.

Step S12: Allocate an ES service to each MDS service in a load balancing manner, and generate configuration information based on the allocation result, so that the MDS service reports metadata to an allocated candidate ES service according to the configuration information.

In some embodiments of the present application, after determining the load balancing allocation method, an ES service is allocated to each MDS service in this manner, and the allocation result is saved to generate configuration information so that the MDS service reports metadata to an allocated candidate ES service according to the configuration information. That is, an MDS service may be allocated multiple ES services, but the MDS service only reports metadata to one candidate ES service among all the allocated candidate ES services at the same time.

In some embodiments of the present application, the corresponding load balancing distribution method is determined according to the size relationship between the total number of ES services in the ES cluster and the total number of MDS services in the distributed file system, which may include: if the total number of ES services is less than the total number of MDS services, the load balancing distribution method is the first target load balancing distribution method. In some embodiments of the present application, allocating ES services to each MDS service according to the load balancing distribution method may include: sorting all MDS services, and sorting all ES services, and allocating ES services to MDS services with the same sorting position according to the sorting position of the ES services to complete the initial allocation; according to the sorting of the ES services, selecting one ES service from the beginning in turn and assigning it to the MDS services that have not been assigned ES services after the initial allocation, so that each MDS service corresponds to at least one ES service.

That is, since the distributed file system and the ES cluster are two systems and are configured separately, there is no dependency between them, that is, the number of ESs and the number of MDSs are different, and the ES number can be assigned by the MDS number, where the MDS number can be uniformly assigned by the MDS cluster and incremented from 0, 0/1/2, etc. For example, as shown in Figure 3, when the number of ESs is less than the number of MDSs, when allocating ESs, the ES services are allocated to the MDS services with the same sorting position according to the sorting position of the ES services to complete the initial allocation, and then, according to the sorting of the ES services, one ES service is selected from the beginning and allocated to the MDS services that have not been allocated ES services after the initial allocation, so that each Each MDS service corresponds to at least one ES service. In this case, one MDS service may correspond to multiple ES services.

In some embodiments of the present application, the corresponding load balancing distribution method is determined according to the size relationship between the total number of ES services in the ES cluster and the total number of MDS services in the distributed file system, which may include: if the total number of ES services is equal to the total number of MDS services, then the load balancing distribution method is the second target load balancing distribution method. In some embodiments of the present application, allocating ES services to each MDS service according to the load balancing distribution method may include: sorting all MDS services, and sorting all ES services, and allocating ES services to MDS services with the same sorting position according to the sorting position of the ES services, so that there is a one-to-one correspondence between MDS services and ES services. For example, as shown in Figure 4, when the number of ES is equal to the number of MDS, a one-to-one correspondence is sufficient.

In some embodiments of the present application, according to the size relationship between the total number of ES services in the ES cluster and the total number of MDS services in the distributed file system, the corresponding load balancing distribution method is determined, which may include: if the total number of ES services is greater than the total number of MDS services, the load balancing distribution method is the third target load balancing distribution method. In some embodiments of the present application, allocating ES services to each MDS service according to the load balancing distribution method may include: sorting all MDS services, and sorting all ES services, and allocating ES services to MDS services with the same sorting position according to the sorting position of the ES services to complete the initial allocation; and re-allocating the ES services that are not allocated after the initial allocation according to the sorting of the MDS services until all ES services are allocated, so that each ES service corresponds to at least one MDS service. For example, as shown in Figure 5, when the number of ES is greater than the number of MDSs, the ESs that are more than the number of MDSs will also be evenly distributed to each MDS, ES0 and ES 3 are candidate ESs for MDS0, ES1 and ES 4 are candidate ESs for MDS1, but initially MDS0 only establishes a connection with ES0. It can be seen that the load balancing of metadata reporting is achieved through the above three allocation methods.

In some embodiments of the present application, before allocating ES services to each MDS service in a load balancing manner, it may also include: configuring a communication address for each ES service in the ES cluster through a distributed file system, so that the MDS service reports data to the corresponding ES service according to the communication address of the ES service. That is, the distributed storage cluster will configure the address of each ES in the ES cluster, and the MDS connects and communicates with the ES through these addresses to report metadata. In addition, some embodiments of the present application do not rely on any service components, will not affect the business system, and are also of reference value in various advanced functions of distributed storage.

As can be seen from the above, in some embodiments of the present application, a corresponding load balancing distribution method is determined based on the size relationship between the total number of ES services in the ES cluster and the total number of MDS services in the distributed file system; ES services are allocated to each MDS service according to the load balancing distribution method, and configuration information is generated based on the allocation result, so that the MDS service reports metadata to an allocated candidate ES service according to the configuration information. It can be seen that by allocating ES services based on the size relationship between the total number of ES services in the ES cluster and the total number of MDS services in the distributed file system using the currently adapted load balancing distribution method, thereby, based on the metadata retrieval function of the distributed file system of Elasticsearch, by balancing each MDS connected to different ES, the MDS achieves the balance of metadata reporting pressure according to conditions such as the number of ES through load balancing, That is, ES load balancing is achieved, thereby maximizing the reporting performance of the program.

Some embodiments of the present application disclose a specific metadata reporting method. Referring to FIG6 , the method may include the following steps:

Step S21: Determine a corresponding load balancing distribution method according to the size relationship between the total number of ES services in the ES cluster and the total number of MDS services in the distributed file system.

Step S22: Allocate an ES service to each MDS service in a load balancing manner, and generate configuration information based on the allocation result, so that the MDS service reports metadata to an allocated candidate ES service according to the configuration information.

Step S23: If the candidate ES service for receiving metadata reporting currently corresponding to the target MDS service in the distributed file system fails, the target MDS is switched to the normally operating ES service so that the target MDS reports metadata to the normally operating ES service.

During the operation of any system, failures and service anomalies are normal. In fact, the normal process only accounts for 20% of the function realization, and the failure and anomaly handling accounts for 80% of the function realization, that is, the 28th rule. Therefore, in the metadata reporting of the distributed file system metadata retrieval function based on Elasticsearch, that is, in the process of MDS uploading data to ES, ES failure anomalies are normal. If not handled, it will affect the metadata reporting business and ultimately affect the user's search results. If the file is not updated in time, the user will retrieve files that do not meet the user's requirements. If the user performs a deletion operation, the consequence is that the data may be deleted by mistake, causing irreparable losses. Therefore, in some embodiments of the present application, if a candidate ES service for receiving metadata reporting currently corresponding to a certain MDS service in the distributed file system fails, the target MDS is switched to the normally operating ES service, that is, when the ES fails, the MDS actively switches to the normal ES service to implement the fault switching function, so that the target MDS reports metadata to the normally operating ES service.

In some embodiments of the present application, before a candidate ES service for receiving metadata reports corresponding to a target MDS service in a distributed file system fails, the process may further include: the target MDS reports data to a corresponding candidate ES service according to configuration information, and determines whether the candidate ES service has a fault according to the data reporting result fed back by the candidate ES service. Specifically, determining whether the ES is normal may be performed by sending an HTTP GET message to the ES and determining based on the response.

In some embodiments of the present application, switching the target MDS to a normally operating ES service may include: judging whether there is a normally operating first target ES service among all candidate ES services corresponding to the target MDS according to the configuration information; determining a reporting object switching method for the target MDS according to the judgment result, and switching the target MDS to a normally operating ES service according to the reporting object switching method. That is, in some embodiments of the present application, when switching services, it is necessary to first judge whether there is a normally operating ES service among all candidate ES services corresponding to the target MDS, and then select a specific service switching object.

In some embodiments of the present application, the reporting object switching mode for the target MDS is determined according to the judgment result, and The reporting object switching method switches the target MDS to a normally operating ES service, which may include: if there is no normally operating first target ES service among all candidate ES services corresponding to the target MDS in the configuration information, poll all ES services in the ES cluster until a normally operating ES service is found, and switch the target MDS to the normally operating ES service.

In some embodiments of the present application, a reporting object switching method for the target MDS is determined based on a judgment result, and the target MDS is switched to a normally operating ES service based on the reporting object switching method, which may include: if there is a first target ES service that is normally operating among all candidate ES services corresponding to the target MDS in the configuration information, the target MDS is switched to the first target ES service.

That is, when an ES fails, its corresponding MDS report will be abnormal. At this time, the ES failure is detected. At this time, the MDS will first try to select a normal ES from the candidate ES. If all the candidate ESs are not normal after selection, it will select from all ESs in turn. If all ESs are not normal, it will select from all ESs again until a normal ES is selected. Metadata reporting is not performed during the selection of ES. As shown in Figure 7, according to the load balancing allocation algorithm, the candidate ESs of each MDS are: MDS0: ES0, ES3; MDS1: ES1, ES4; MDS2: ES2. When ES0 fails, MDS0 will first select a candidate ES, that is, ES3. If ES3 is still not normal, it will try to select one by one from all ESs from ES0 to ES4. If ES0 to ES4 are also not normal, it will traverse ES0 to ES4 again until a normal ES is selected.

It can be seen that when MDS is reporting metadata, the ES service on the node may be abnormal, and the node may also lose power, but ES complies with its own redundancy rules and can still provide services normally when an ES service is hung up or an ES node is down. At this time, MDS should actively select an available ES in the ES cluster to continue metadata reporting, rather than waiting for the ES assigned to this MDS to become normal before continuing to report, thereby achieving business continuity and high availability. In addition, some embodiments of the present application do not rely on any service components, will not affect the business system, and are also of reference value in various advanced functions of distributed storage.

Among them, the specific processes of the above-mentioned steps S21 and S22 can refer to the corresponding contents disclosed in the above-mentioned embodiments, and will not be repeated here.

As can be seen from the above, in some embodiments of the present application, if the candidate ES service for receiving metadata reports currently corresponding to the target MDS service in the distributed file system fails, the target MDS is switched to the normally operating ES service so that the target MDS reports metadata to the normally operating ES service. It can be seen that when an ES fails, it will affect the MDS metadata reporting process. In order to ensure the continuity and timeliness of metadata reporting and prevent users from retrieving files that do not meet user requirements, MDS metadata reporting implements automatic fault switching, achieves global load balancing and high availability of distributed systems, and does not require manual intervention, thereby improving system stability, better assisting user data management, improving product competitiveness, and improving user satisfaction.

Based on the above embodiments, some embodiments of the present application further disclose a specific metadata reporting method. As shown in FIG8 , the method may include the following steps:

Step S31: Determine a corresponding load balancing distribution method according to the size relationship between the total number of ES services in the ES cluster and the total number of MDS services in the distributed file system.

Step S32: Allocate an ES service to each MDS service in a load balancing manner, and generate configuration information based on the allocation result, so that the MDS service reports metadata to an allocated candidate ES service according to the configuration information.

Step S33: If the candidate ES service for receiving metadata reporting currently corresponding to the target MDS service in the distributed file system fails, the target MDS is switched to the normally operating ES service so that the target MDS reports metadata to the normally operating ES service.

Step S34: Determine whether the normally running ES service hits the candidate ES service corresponding to the target MDS according to the configuration information.

Step S35: According to the hit result, it is determined whether to perform a report object switchback process on the normally running ES service currently corresponding to the target MDS.

That is, the above service switching may switch to the candidate ES service corresponding to the target MDS, or it may switch to a normal ES service that is not a candidate. If the service is switched to a candidate ES service, the current state after the switch is also in line with the load balancing. However, if the service is switched to a normal ES service that is not a candidate, the current distribution does not conform to the initial load balancing distribution. Therefore, service switching back is required to achieve load balancing on the basis of normal operation.

In some embodiments of the present application, based on the hit result, it is determined whether to perform report object switching back processing on the normally operating ES service currently corresponding to the target MDS, which may include: if the normally operating ES service does not hit the candidate ES service corresponding to the target MDS, then periodically monitoring the status of all candidate ES services corresponding to the target MDS, and when there is a normally operating second target ES service among all the candidate ES services corresponding to the target MDS, switching the target MDS back to the second target ES service.

In some embodiments of the present application, based on the hit result, it is determined whether to perform a reporting object switchback process on the normally operating ES service currently corresponding to the target MDS, which may include: if the normally operating ES service hits the candidate ES service corresponding to the target MDS, the current reporting connection is retained. That is, if the MDS switches to connect to other ES after an ES failure, and provides reporting metadata services normally, during this period, it will periodically check whether the currently connected ES is a candidate ES. If not, it will periodically send a message to the candidate ES to check whether the candidate ES has returned to normal. If it has returned, it will switch back to the candidate ES. For example, as shown in Figure 9, ES0 fails, and MDS0 switches to ES1. In the process of reporting metadata through ES1, it also periodically checks whether ES0 is normal. If ES0 is normal, it switches back to ES0, thereby achieving fault switching back and load balancing.

It can be seen that the distributed file system metadata retrieval function load balancing high availability implementation method disclosed in some embodiments of the present application is through 1. The distributed file system configures the address of each ES; 2. The MDS evenly distributes each ES to each MDS according to the load balancing algorithm; 3. During the MDS reporting process, if the ES fails, the MDS first tries to select an ES from the candidate ES, and if it is normal, the candidate ES is used; 4. If all candidate ESs are not normal, select from all ESs in turn, and if a normal ES is selected, use this ES to continue metadata reporting. If all ESs are not normal, continue to traverse all ESs until one is selected. Normal ES; 5. If the ES currently used by MDS is not a candidate ES, it will periodically check whether the candidate ES has become normal. If it has become normal, it will switch back to the candidate ES. In the event of an ES failure, the candidate ES is selected first and then all ES are selected to achieve automatic switching of the failure, and continuous availability of the reporting business. The status of the candidate ES is periodically detected to achieve automatic switching back after the failure, thereby achieving dynamic load balancing and high availability functions. Therefore, based on the distributed unstructured storage metadata retrieval function of elasticsearch, MDS independently implements a set of load balancing and high availability functions, which does not rely on any components and is not coupled with distributed system functions. The independent load balancing and high availability functions realize continuous, efficient and high availability of metadata reporting services, ensure the stable operation of the system, better assist users in data management, and efficiently develop data value.

Among them, the specific processes of the above-mentioned steps S31, S32, and S33 can refer to the corresponding contents disclosed in the above-mentioned embodiments, and will not be repeated here.

As can be seen from the above, in some embodiments of the present application, it is determined whether the normally operating ES service hits the candidate ES service corresponding to the target MDS based on the configuration information; based on the hit result, it is determined whether to perform the reporting object switchback processing on the normally operating ES service currently corresponding to the target MDS. The service switchback processing is performed by reporting the object switchback processing, so as to ensure load balancing as much as possible on the basis of normal operation. According to the correspondence between the MDS and ES initially assigned by the load balancing, if the MDS selects an ES that is not assigned according to the load balancing algorithm after the ES fails, it is monitored in real time when the ES assigned by the load balancing becomes normal. If it is normal, the ES is switched back to achieve global dynamic load balancing.

Correspondingly, some embodiments of the present application also disclose a metadata reporting system, including an ES cluster and a distributed file system; wherein the ES service in the ES cluster and the MDS service in the distributed file system are services allocated through a load balancing distribution method; wherein the load balancing distribution method is determined according to the size relationship between the total number of ES services in the ES cluster and the total number of MDS services in the distributed file system; the MDS service is used to report metadata to a corresponding ES service according to configuration information; the configuration information is configuration information generated based on the allocation result.

Accordingly, some embodiments of the present application further disclose a metadata reporting device, as shown in FIG10 , the device includes:

The load balancing distribution mode determination module 11 is used to determine the corresponding load balancing distribution mode according to the size relationship between the total number of ES services in the ES cluster and the total number of MDS services in the distributed file system;

The allocation module 12 is used to allocate an ES service to each MDS service in a load balancing allocation manner, and generate configuration information based on the allocation result, so that the MDS service reports data to the corresponding ES service according to the configuration information.

As can be seen from the above, in some embodiments of the present application, the corresponding load balancing distribution method is determined according to the size relationship between the total number of ES services in the ES cluster and the total number of MDS services in the distributed file system; The method allocates ES services to each MDS service, and generates configuration information based on the allocation result, so that the MDS service reports metadata to an allocated candidate ES service according to the configuration information. It can be seen that by allocating ES services according to the size relationship between the total number of ES services in the ES cluster and the total number of MDS services in the distributed file system, the current adapted load balancing allocation method is used to allocate ES services. Therefore, based on the metadata retrieval function of the distributed file system of Elasticsearch, by balancing each MDS connected to different ES, MDS achieves the balance of metadata reporting pressure according to the number of ES and other conditions through load balancing, that is, ES load balancing is achieved, thereby maximizing the reporting performance of the program.

In some specific embodiments, the load balancing distribution mode determination module 11 may specifically include:

The first load balancing allocation mode determining unit is configured to determine that if the total number of ES services is less than the total number of MDS services, the load balancing allocation mode is the first target load balancing allocation mode.

In some specific embodiments, the allocation module 12 may specifically include:

The primary allocation unit is used to sort all the MDS services and all the ES services, and to allocate the ES services to the MDS services with the same sorting position according to the sorting position of the ES services, so as to complete the primary allocation;

The reallocation unit is used to select an ES service from the beginning in sequence according to the order of the ES services and allocate it to the MDS services that have not been allocated ES services after the initial allocation, so that each MDS service corresponds to at least one ES service.

The second load balancing allocation mode determining unit is configured to determine that if the total number of ES services is equal to the total number of MDS services, the load balancing allocation mode is the second target load balancing allocation mode.

The allocation unit is used to sort all MDS services and all ES services, and allocate ES services to MDS services with the same sorting position according to the sorting position of the ES services, so that there is a one-to-one correspondence between the MDS services and the ES services.

The third load balancing allocation mode determining unit is configured to determine that if the total number of ES services is greater than the total number of MDS services, the load balancing allocation mode is a third target load balancing allocation mode.

The reallocation unit is used to reallocate the ES services that are not allocated after the initial allocation according to the order of the MDS services until all ES services are allocated, so that each ES service corresponds to at least one MDS service.

In some specific embodiments, the metadata reporting device may specifically include:

The address sending unit is used to configure a communication address for each ES service in the ES cluster through a distributed file system, so that the MDS service reports data to the corresponding ES service according to the communication address of the ES service.

The service switching unit is used to switch the target MDS to the normally operating ES service if the candidate ES service currently corresponding to the target MDS service in the distributed file system for receiving metadata reporting fails, so that the target MDS reports metadata to the normally operating ES service.

The fault judgment unit is used for the target MDS to report data to a corresponding candidate ES service according to the configuration information, and to judge whether the candidate ES service has a fault according to the data reporting result fed back by the candidate ES service.

In some specific embodiments, the service switching unit may specifically include:

A candidate service determination unit, configured to determine whether there is a first target ES service that is operating normally among all candidate ES services corresponding to the target MDS according to the configuration information;

The ES service switching unit is used to determine a reporting object switching mode for the target MDS according to the judgment result, and switch the target MDS to a normally operating ES service according to the reporting object switching mode.

In some specific embodiments, the ES service switching unit may specifically include:

The first switching unit is used to poll all ES services in the ES cluster until a normally operating ES service is found, and switch the target MDS to the normally operating ES service if there is no normally operating first target ES service among all candidate ES services corresponding to the target MDS in the configuration information.

The second switching unit is configured to switch the target MDS to the first target ES service if there is a first target ES service that is operating normally among all candidate ES services corresponding to the target MDS in the configuration information.

A hit judgment unit, used to judge whether the normally running ES service hits the candidate ES service corresponding to the target MDS according to the configuration information;

The switchback judgment unit is used to judge whether to perform a report object switchback process on the normally running ES service currently corresponding to the target MDS according to the hit result.

In some specific embodiments, the switchback determination unit may specifically include:

The service switching back unit is used to periodically monitor the status of all candidate ES services corresponding to the target MDS if the normally operating ES service does not hit the candidate ES service corresponding to the target MDS, and when there is a normally operating second target ES service among all the candidate ES services corresponding to the target MDS, switch the target MDS back to the second target ES service.

The service reservation unit is used to reserve the current reporting connection if the normally operating ES service hits the candidate ES service corresponding to the target MDS.

Furthermore, some embodiments of the present application also disclose an electronic device, as shown in FIG11 . The content in the figure cannot be regarded as any limitation on the scope of use of the present application.

FIG11 is a schematic diagram of the structure of an electronic device 20 provided in some embodiments of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input/output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the metadata reporting method disclosed in any of the aforementioned embodiments.

In some embodiments of the present application, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon include an operating system 221, a computer program 222, and data 223 including configuration information, etc. The storage method can be temporary storage or permanent storage.

The operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20, so as to realize the operation and processing of the massive data 223 in the memory 22 by the processor 21, and it can be Windows Server, Netware, Unix, Linux, etc. In addition to including computer programs that can be used to complete the metadata reporting method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include computer programs that can be used to complete other specific tasks.

Furthermore, some embodiments of the present application also disclose a non-volatile readable storage medium, in which computer executable instructions are stored. When the computer executable instructions are loaded and executed by a processor, the metadata reporting method steps disclosed in any of the aforementioned embodiments are implemented.

In this specification, each embodiment is described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple. For the relevant parts, please refer to the method part description. Can.

The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

The above is a detailed introduction to a metadata reporting method, device, equipment and medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for a person skilled in the art, according to the idea of the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

A metadata reporting method, characterized in that it is applied to a metadata retrieval architecture based on ES, comprising:

Determine the corresponding load balancing distribution method based on the relationship between the total number of ES services in the ES cluster and the total number of MDS services in the distributed file system;

An ES service is allocated to each of the MDS services in accordance with the load balancing allocation method, and configuration information is generated based on the allocation result, so that the MDS service reports metadata to an allocated candidate ES service according to the configuration information.
The metadata reporting method according to claim 1 is characterized in that the step of determining the corresponding load balancing distribution mode according to the size relationship between the total number of ES services in the ES cluster and the total number of MDS services in the distributed file system comprises:

If the total number of the ES services is less than the total number of the MDS services, the load balancing allocation method is the first target load balancing allocation method.
The metadata reporting method according to claim 2, characterized in that the allocating ES services to each of the MDS services according to the load balancing distribution method comprises:

By sorting all the MDS services and sorting all the ES services, the ES services are allocated to the MDS services with the same sorting position according to the sorting position of the ES services, so as to complete the initial allocation;

According to the ranking of the ES services, one ES service is selected from the beginning and allocated to the MDS services that have not been allocated an ES service after the initial allocation, so that each of the MDS services corresponds to at least one ES service.
The metadata reporting method according to claim 1 is characterized in that the step of determining the corresponding load balancing distribution mode according to the size relationship between the total number of ES services in the ES cluster and the total number of MDS services in the distributed file system comprises:

If the total number of the ES services is equal to the total number of the MDS services, the load balancing distribution mode is the second target load balancing distribution mode.
The metadata reporting method according to claim 4, characterized in that the allocating ES service to each of the MDS services according to the load balancing distribution method comprises:

By sorting all the MDS services and sorting all the ES services, the ES services are allocated to the MDS services with the same sorting position according to the sorting position of the ES services, so that there is a one-to-one correspondence between the MDS services and the ES services.
The metadata reporting method according to claim 1 is characterized in that the corresponding load is determined according to the size relationship between the total number of ES services in the ES cluster and the total number of MDS services in the distributed file system. Balanced distribution methods include:

If the total number of the ES services is greater than the total number of the MDS services, the load balancing allocation method is the third target load balancing allocation method.
The metadata reporting method according to claim 6, characterized in that the allocating ES services to each of the MDS services according to the load balancing distribution method comprises:

By sorting all the MDS services and sorting all the ES services, the ES services are allocated to the MDS services with the same sorting position according to the sorting position of the ES services, so as to complete the initial allocation;

The ES services that are not allocated after the initial allocation are allocated again according to the order of the MDS services until all the ES services are allocated, so that each of the ES services corresponds to at least one MDS service.
The metadata reporting method according to claim 1, characterized in that before allocating an ES service to each of the MDS services according to the load balancing distribution method, it also includes:

A communication address is configured for each ES service in the ES cluster through the distributed file system, so that the MDS service reports data to the corresponding ES service according to the communication address of the ES service.
The metadata reporting method according to claim 1, further comprising:

If a candidate ES service for receiving metadata reports currently corresponding to the target MDS service in the distributed file system fails, the target MDS is switched to a normally operating ES service so that the target MDS reports metadata to the normally operating ES service.
The metadata reporting method according to claim 9 is characterized in that before the candidate ES service for receiving metadata reporting corresponding to the target MDS service in the distributed file system fails, it also includes:

The target MDS reports data to a corresponding candidate ES service according to the configuration information, and determines whether the candidate ES service has a fault according to the data reporting result fed back by the candidate ES service.
The metadata reporting method according to claim 9, characterized in that the step of switching the target MDS to a normally operating ES service comprises:

Determine, according to the configuration information, whether there is a first target ES service that is operating normally among all the candidate ES services corresponding to the target MDS;

A reporting object switching mode for the target MDS is determined according to the judgment result, and the target MDS is switched to a normally operating ES service according to the reporting object switching mode.
The metadata reporting method according to claim 11, characterized in that the step of determining a reporting object switching mode for the target MDS according to the judgment result, and switching the target MDS to a normally operating ES service according to the reporting object switching mode, comprises:

If there is no first target ES service operating normally among all the candidate ES services corresponding to the target MDS in the configuration information, all ES services in the ES cluster are polled until a normally operating ES service is found, and the target MDS is switched to the normally operating ES service.
The metadata reporting method according to claim 11, characterized in that the step of determining a reporting object switching mode for the target MDS according to the judgment result, and switching the target MDS to a normally operating ES service according to the reporting object switching mode, comprises:

If there is a first target ES service that is operating normally among all the candidate ES services corresponding to the target MDS in the configuration information, the target MDS is switched to the first target ES service.
The metadata reporting method according to claim 9, further comprising:

Determine, according to the configuration information, whether the normally running ES service hits the candidate ES service corresponding to the target MDS;

According to the hit result, it is determined whether to perform a report object switchback process on the normally running ES service currently corresponding to the target MDS.
The metadata reporting method according to claim 14 is characterized in that judging whether to perform a reporting object switchback process on the normally operating ES service currently corresponding to the target MDS according to the hit result comprises:

If the normally operating ES service does not hit the candidate ES service corresponding to the target MDS, the status of all the candidate ES services corresponding to the target MDS is monitored regularly, and when there is a normally operating second target ES service among all the candidate ES services corresponding to the target MDS, the target MDS is switched back to the second target ES service.
The metadata reporting method according to claim 14 is characterized in that judging whether to perform a reporting object switchback process on the normally operating ES service currently corresponding to the target MDS according to the hit result comprises:

If the normally running ES service hits the candidate ES service corresponding to the target MDS, the current reporting connection is retained.
A metadata reporting system, characterized by comprising an ES cluster and a distributed file system;

The ES service in the ES cluster and the MDS service in the distributed file system are services allocated by a load balancing allocation method; wherein the load balancing allocation method is determined according to the size relationship between the total number of ES services in the ES cluster and the total number of MDS services in the distributed file system;

The MDS service is used to report metadata to a corresponding ES service according to configuration information; the configuration information is configuration information generated based on the allocation result.
A metadata reporting device, characterized by comprising:

The load balancing distribution mode determination module is used to determine the corresponding load balancing distribution mode according to the size relationship between the total number of ES services in the ES cluster and the total number of MDS services in the distributed file system;

The allocation module is used to allocate an ES service to each of the MDS services according to the load balancing allocation method, and generate configuration information based on the allocation result, so that the MDS service reports data to the corresponding ES service according to the configuration information.
An electronic device, comprising:

Memory, used to store computer programs;

A processor, configured to execute the computer program to implement the metadata reporting method according to any one of claims 1 to 16.
A non-volatile readable storage medium, characterized in that it is used to store a computer program; wherein when the computer program is executed by a processor, the metadata reporting method as described in any one of claims 1 to 16 is implemented.