CN110569302A - method and device for physical isolation of distributed cluster based on lucene - Google Patents
method and device for physical isolation of distributed cluster based on lucene Download PDFInfo
- Publication number
- CN110569302A CN110569302A CN201910762813.7A CN201910762813A CN110569302A CN 110569302 A CN110569302 A CN 110569302A CN 201910762813 A CN201910762813 A CN 201910762813A CN 110569302 A CN110569302 A CN 110569302A
- Authority
- CN
- China
- Prior art keywords
- cluster
- service
- list
- index
- server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 60
- 238000002955 isolation Methods 0.000 title claims abstract description 58
- 239000012634 fragment Substances 0.000 claims abstract description 101
- 230000008569 process Effects 0.000 claims abstract description 16
- 238000010276 construction Methods 0.000 claims description 6
- 230000004044 response Effects 0.000 abstract description 9
- 238000010586 diagram Methods 0.000 description 6
- 230000008447 perception Effects 0.000 description 6
- 238000012423 maintenance Methods 0.000 description 5
- 238000005192 partition Methods 0.000 description 5
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000002360 preparation method Methods 0.000 description 2
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2272—Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/278—Data partitioning, e.g. horizontal or vertical partitioning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Computing Systems (AREA)
- Computer And Data Communications (AREA)
Abstract
The invention discloses a method and a device for physical isolation of distributed clusters based on lucene, wherein the method comprises an index creating process, and the index creating process comprises the following steps: inquiring a current cluster resource list and a current index relation list stored in a distributed memory according to the index creating request of the service, and acquiring the use range of a server to which the service is pre-distributed in a cluster; acquiring a server list which can be used by the service according to the inquired using range of the server; and distributing the fragments of the index to be created to each server according to the current running state of each server in the server list, wherein the fragments comprise a main fragment and/or a sub-fragment. According to the invention, by pre-allocating the server range in the cluster to the service (namely dividing cluster resources for the service), different services are physically isolated in the cluster, and the response rate of all the services is obviously improved (the response rate of the cluster is in the second level).
Description
Technical Field
the invention relates to the technical field of cluster management, in particular to a method and a device for physical isolation of distributed clusters based on lucene.
background
at present, for a scenario that multiple services need to use a lucene-based distributed cluster (such as an Elasticsearch cluster), there are two general ways. One way is to directly build indexes on the same lucene-based distributed cluster for different services, and in this way, the cluster resources are fully utilized. The other method is to establish a plurality of clusters, and realize that each service corresponds to an independent cluster, so as to avoid the interference caused by different services.
although the first method can fully utilize cluster resources, the services are in a tightly coupled state, which may cause interference between different services, thereby causing some significant performance problems. For example, the data volume of one service a is relatively large, the data volume of another service B is relatively small, and when the service a frequently sends a search request to the cluster, the search request sent by the service B to the cluster may not be responded in time, thereby causing relatively large interference to the service B. That is, the requests of partial service access clusters can not be responded in time delay, and the response delay is in the minute level or even in the hour level, and other performance problems can be caused. More seriously, when a certain service in the cluster is not operated properly, which causes the conditions of insufficient cluster memory or false dead node, etc., all other service requests will be affected.
the second way is to effectively realize the isolation of the service and the resource by the way of isolating the cluster corresponding to each service. However, when the large data volume service and the small data volume service both correspond to different clusters, the clusters become more and more, and the complexity of cluster operation and maintenance is increased.
disclosure of Invention
in order to solve the problems in the prior art, embodiments of the present invention provide a method and an apparatus for physical isolation of a distributed cluster based on lucene, so as to overcome the problems in the prior art that when a plurality of services share a distributed cluster, the services are tightly coupled to cause mutual interference, and when each service establishes an independent distributed cluster, the clusters become more and more, increasing the complexity of cluster operation and maintenance, and the like.
in order to solve one or more technical problems, the invention adopts the technical scheme that:
In one aspect, a method for physical isolation of a lucene-based distributed cluster is provided, where the method includes creating an index flow, where the creating the index flow includes the following steps:
Inquiring a current cluster resource list and a current index relation list stored in a distributed memory according to an index creating request of a service, and acquiring a use range of a server to which the service is pre-distributed in a cluster;
acquiring a server list which can be used by the service according to the inquired using range of the server;
and distributing the fragments of the index to be created to the servers according to the current running state of each server in the server list, wherein the fragments comprise a main fragment and/or a sub-fragment.
Further, the method further includes a process of generating a current cluster resource list and a current index relationship list stored in the distributed memory:
distributing the service range of the servers in the cluster for the service according to the performance index of the service and the running state of the servers in the cluster;
and updating the pre-established cluster resource list and the index relation list according to the use range of the server, generating a current cluster resource list and a current index relation list, and storing the current cluster resource list and the current index relation list in the distributed memory.
Further, the querying a current cluster resource list and a current index relationship list stored in the distributed memory according to the service index creation request includes:
and judging whether the service has a corresponding physical isolation relationship, and if so, inquiring a current cluster resource list and a current index relationship list stored in the distributed memory according to the index creating request of the service.
Further, the method further includes a writing process, and the writing process includes the following steps:
Inquiring a current cluster resource list and a current index relation list stored in the distributed memory according to an index name in a writing request of a service, and assembling server routing information written into a cluster;
constructing an index id of the service according to the routing information, wherein the index id range is in an id range contained in a routing server;
merging the write-in requests of each piece of data of the service and then forwarding the merged write-in requests to the main slice in parallel;
and synchronously writing the data written into the main fragment into the secondary fragment.
further, the method further comprises a search process, wherein the search process comprises the following steps:
Inquiring a current cluster resource list and a current index relation list stored in the distributed memory according to an index name in a search request of a service, and assembling server routing information of a search cluster;
Constructing a destination fragment list according to the server routing information;
Traversing the fragment list, and sending a search request to each fragment in the fragment list in parallel so that each fragment responds to the search request and executes corresponding search.
in another aspect, an apparatus for physical isolation of a lucene-based distributed cluster is provided, the apparatus comprising an index creation module, the index creation module comprising:
The information query unit is used for querying a current cluster resource list and a current index relation list stored in the distributed memory according to the index creating request of the service, and acquiring the service range of a server pre-allocated to the service in the cluster;
The list generating unit is used for acquiring a server list which can be used by the service according to the inquired using range of the server;
and the fragment distribution unit is used for distributing the fragments of the index to be created to the servers according to the current running state of each server in the server list, wherein the fragments comprise a main fragment and/or an auxiliary fragment.
Further, the apparatus further includes a relationship creating module, where the relationship creating module is configured to generate a cluster resource list and an index relationship list that are currently stored in the distributed memory, and the relationship creating module includes:
The resource dividing unit is used for distributing the service range of the servers in the cluster for the service according to the performance index of the service and the running state of the servers in the cluster;
And the list updating unit is used for updating the pre-established cluster resource list and the index relation list according to the use range of the server, generating a current cluster resource list and storing the current index relation list into the distributed memory.
Further, the apparatus further comprises:
and the relationship judgment module is used for judging whether the corresponding physical isolation relationship exists in the service.
further, the apparatus further comprises a data writing module, wherein the data writing module comprises:
The first assembly unit is used for inquiring a current cluster resource list and a current index relation list stored in the distributed memory according to an index name in a writing request of a service, and assembling server routing information written into a cluster;
the index id construction unit is used for constructing an index id of the service according to the routing information, wherein the index id range is in the id range contained in the routing server;
The data writing unit is used for merging the writing requests of each piece of data of the service and then forwarding the merged writing requests to the main slice in parallel;
And the data synchronization unit is used for synchronously writing the data written into the main fragment into the secondary fragment.
Further, the apparatus further comprises a data search module, and the data search module comprises:
The second assembly unit is used for inquiring a current cluster resource list and a current index relation list stored in the distributed memory according to the index name in the search request of the service, and assembling the server routing information of the search cluster;
A list construction unit for constructing a destination shard list according to the server routing information;
And the request sending unit is used for traversing the fragment list and sending a search request to each fragment in the fragment list in parallel so that each fragment responds to the search request and executes corresponding search.
the technical scheme provided by the embodiment of the invention has the following beneficial effects:
1. According to the method and the device for physical isolation of the distributed cluster based on the lucene, provided by the embodiment of the invention, by pre-allocating the server range in the cluster to the service (namely dividing cluster resources for the service), physical isolation of different services in the cluster is realized, and the response rate of all the services is remarkably improved (the response rate of the cluster is in the second level);
2. the method and the device for physical isolation of the distributed cluster based on the lucene provided by the embodiment of the invention manage the fragment distribution by using the physical distributor and the physical perception decision-making machine, realize the fragment balance in the index, and provide the guarantee of high availability and load balance for the physical isolation of the service in the cluster;
3. The method and the device for physical isolation of the distributed cluster based on the lucene provided by the embodiment of the invention unify the cluster management entries and improve the cluster operation and maintenance efficiency.
drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
FIG. 1 is a flow diagram illustrating a create index flow in a method for physical isolation of lucene-based distributed clusters, according to an example embodiment;
FIG. 2 is a flow diagram illustrating a process of generating a current list of cluster resources stored in a distributed memory and a current list of indexing relationships in a method for physical isolation of lucene-based distributed clusters in accordance with an illustrative embodiment;
FIG. 3 is a flow diagram illustrating write flow in a method of physical isolation of a lucene-based distributed cluster, according to an example embodiment;
FIG. 4 is a flow diagram illustrating a search flow in a method of physical isolation of a lucene-based distributed cluster, according to an example embodiment;
Fig. 5 is a block diagram illustrating an apparatus for physical isolation of a lucene-based distributed cluster, according to an example embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
the method and the device for physical isolation of the distributed cluster based on the lucene uniformly manage the service and the server resource range in the distributed cluster based on the lucene, provide access service for all nodes in the cluster, realize physical isolation of different services in one cluster, manage fragment distribution by using a physical distributor and a physical perception decision-making machine, realize fragment balance in indexes, and provide high availability and load balance guarantee for physical isolation of services in the cluster. The method solves the problem that the services are mutually interfered due to tight coupling when the services share one cluster, realizes that the services in one cluster are physically isolated from each other so as to ensure the performance requirement of each service, can avoid the condition of establishing an independent cluster for each service as much as possible, forms a plurality of physical nodes in one cluster for the service by dividing the server resources of a plurality of services on one cluster, gives full play to the utilization rate of the cluster resources, and reduces the complexity of cluster management and operation and maintenance.
it should be noted here that the technical solution provided by the embodiment of the present invention is applicable to a lucene-based distributed cluster, such as an Elasticsearch cluster. The scheme of the present invention is described below by taking an elastic search cluster as an example, but it should be noted here that the scheme provided by the embodiment of the present invention is not limited to be implemented in the elastic search cluster, and other lucene-based distributed clusters are also applicable to the scheme of the present invention.
Fig. 1 is a flowchart illustrating an index creation process in a method for physical isolation of a lucene-based distributed cluster according to an exemplary embodiment, and referring to fig. 1, the index creation process includes the following steps:
s101: and inquiring a current cluster resource list and a current index relation list stored in a distributed memory according to the index creating request of the service, and acquiring the use range of a server to which the service is pre-distributed in a cluster.
Specifically, first, in the embodiment of the present invention, the service is pre-allocated to the use range of the server in the cluster. In specific implementation, a request for applying for a cluster resource may be sent to a cluster (e.g., an elastic search cluster) administrator for each service, after receiving the request, the cluster administrator divides a service range and/or a node range of a server in the cluster according to performance indexes such as data volume of the service and query response, and stores the service range and/or the node range of the server in a cluster resource list (which may be represented by businesscluster range) for subsequent process query, where the node includes but is not limited to a physical machine. Secondly, in the embodiment of the present invention, a corresponding relationship between the service and the index name needs to be agreed in advance, and the corresponding relationship between the service and the index name needs to be stored in an index relationship list (which may be represented by businesslndices mapper) for later use. In the embodiment of the invention, the cluster resource list and the index relation list are maintained and updated in real time or regularly.
after receiving an index creating request of a service, inquiring a current cluster resource list and a current index relation list stored in a distributed memory according to the index creating request of the service, and acquiring a use range of a server pre-allocated to the service in a cluster. During specific implementation, the service name corresponding to the name is found in the index relation list according to the index name in the index creating request, and then the service range of the server corresponding to the service is inquired in the cluster resource list according to the service name, so that preparation is made for fragment distribution.
s102: and acquiring a server list which can be used by the service according to the inquired using range of the server.
Specifically, after the usage scope of the server is queried according to the above steps, all servers that can be used (i.e., pre-allocated) by the current service are obtained from the usage scope of the server, and a server list is generated.
S103: and distributing the fragments of the index to be created to the servers according to the current running state of each server in the server list, wherein the fragments comprise a main fragment and/or a sub-fragment.
specifically, the current operating state (such as load condition) of each server in the server list is comprehensively considered, and the fragments of the index to be created corresponding to the index creation request are distributed to each server, where the fragments include a main fragment and/or a sub-fragment. In a specific implementation, a physical distributor (represented by a physical allocator) and a physical perception decision maker (represented by a physical aware allocation decider) may be additionally arranged, wherein the physical perception decision maker includes a custom decision maker, a load balancing decision maker and the like. And then fixing the server addresses of the main fragment and the copy fragment to be distributed in a corresponding server list through a physical distributor, and finishing final fragment distribution in the circled service addresses through a physical perception decision maker according to the strategies of a user-defined decision maker and a load balancing decision maker. It should be noted that, after the index creation is completed, the newly created index related information needs to be updated into the index relationship list.
As an example, the perceptual decision policy of a physical perceptual decision maker may refer to the following:
1. Server load condition, if the current server load is relatively high, distributing to the server with lower load;
2. the number of fragments, if the number of fragments on the current server is more than that of other servers, the fragments are preferentially distributed to other servers;
3. The method comprises the following steps that (1) disk space is allocated to a server with larger disk space preferentially if the utilization rate of a disk on the server is higher than that of disks of other servers;
4. the primary and secondary sub-slices are prevented from being distributed on the same server;
5. The main fragment and the copy fragment are preferentially distributed on servers crossing the rack and the computer room.
Fig. 2 is a flowchart illustrating a process of generating a current cluster resource list and a current index relationship list stored in a distributed memory in a method for physical isolation of 1 uce-based distributed clusters according to an exemplary embodiment, and as a preferred implementation, referring to fig. 2, in an embodiment of the present invention, the method further includes a process of generating a current cluster resource list and a current index relationship list stored in a distributed memory, and the specific steps are as follows:
s201: and distributing the service range of the servers in the cluster for the service according to the performance index of the service and the running state of the servers in the cluster.
Specifically, the performance index of the service includes indexes such as data volume and query response of the service, and the running state of the server includes load conditions, a CPU, a memory, and the like of the server. And comprehensively considering factors such as the performance index of the service, the running state of the server and the like, and distributing the use range of the server in the cluster for the service. For example, when the data volume of a service is large, the usage range of the server allocated to the service in the cluster is large. The distribution strategy of the use range of the server is not limited too much, the user can set according to the actual requirement, and in the embodiment of the invention, the distribution strategy can be set to support dynamic adjustment.
S202: and updating the pre-established cluster resource list and the index relation list according to the use range of the server, generating a current cluster resource list and a current index relation list, and storing the current cluster resource list and the current index relation list in the distributed memory.
specifically, in the embodiment of the present invention, a cluster resource list and an index relationship list are created in advance, where the cluster resource list stores a correspondence between a service and a cluster server resource, and the index relationship list stores a correspondence between a service and an agreed index name. After the service scope of the server in the cluster is allocated to the service, the information needs to be updated to the cluster resource list and the index relationship list for subsequent use.
in the embodiment of the invention, the updated cluster resource list and the index relation list are persisted to a distributed memory for storage and are provided for cluster access. It should be noted here that storing the physical isolation metadata in the distributed storage may ensure that the physical isolation metadata is still accessible after a cluster fails (e.g., a node goes down), so that high availability of the physical isolation metadata may be achieved.
as a preferred implementation manner, in the embodiment of the present invention, the querying, according to the index creation request of the service, the current cluster resource list and the current index relationship list stored in the distributed memory includes:
And judging whether the service has a corresponding physical isolation relationship, and if so, inquiring a current cluster resource list and a current index relationship list stored in the distributed memory according to the index creating request of the service.
Specifically, after receiving an index creating request of a service, it may be determined whether the service has a corresponding physical isolation relationship (i.e., whether a service has a service range pre-allocated to a server in a cluster), if so, a pre-created cluster resource list and an index relationship list are queried according to the index creating request of the service, and preparation is made for allocating a fragment. When index creation and fragment distribution are directly performed, a user can comprehensively consider factors such as performance indexes of services and operation states of servers in a cluster so as to achieve load balancing.
It should be noted that, in the embodiment of the present invention, a physical partition management module (denoted by physical partition) may be configured to manage the service range and the use range of the cluster server. And when judging whether the corresponding physical isolation relationship exists in the service, the method can be realized by judging whether the physical partition parameter is empty, if the physical partition parameter is empty, the service does not have the corresponding physical isolation relationship, and if the physical partition parameter is not empty, the service has the corresponding physical isolation relationship.
Fig. 3 is a flow chart illustrating a write flow in a method for physical isolation of a lucene-based distributed cluster according to an exemplary embodiment, and referring to fig. 3, as a preferred implementation, in an embodiment of the present invention, the method further includes a write flow, where the write flow includes the following steps:
S301: and inquiring a current cluster resource list and a current index relation list stored in the distributed memory according to the index name in the writing request of the service, and assembling the server routing information written into the cluster.
specifically, a write-in request of a service is received, a current index relation list stored in a distributed memory is queried according to an index name in the write-in request, the service name corresponding to the index name is obtained, then a current cluster resource list stored in the distributed memory is queried according to the service name, a use range of a server pre-allocated to the service in a cluster is obtained, all server information corresponding to the service is queried, and server routing information written into the cluster is assembled.
s302: and constructing an index id of the service according to the routing information, wherein the index id range is in an id range contained in a routing server.
Specifically, the index id is used to identify a document id and exists uniquely. The id is constructed and generated according to the server address list and the index fragment number, and with the id, the server to which the document is written can be definitely known.
S303: and merging the write-in requests of each piece of data of the service and then forwarding the merged write-in requests to the main slice in parallel.
Specifically, the write requests of each piece of data of the service are merged and then forwarded to the corresponding main slice in parallel, and data write operation is performed.
S304: and synchronously writing the data written into the main fragment into the secondary fragment.
fig. 4 is a flowchart illustrating a search flow in a method for physical isolation of a lucene-based distributed cluster according to an exemplary embodiment, and referring to fig. 4, as a preferred implementation, in an embodiment of the present invention, the method further includes a search flow, and the search flow includes the following steps:
s401: and inquiring a current cluster resource list and a current index relation list stored in the distributed memory according to the index name in the search request of the service, and assembling the server routing information of the search cluster.
specifically, a search (or query) request of a service is received, a current index relationship list stored in a distributed memory is queried according to an index name in the search request, the service name corresponding to the index name is obtained, then a current cluster resource list stored in the distributed memory is queried according to the service name, a use range of a server pre-allocated to the service in a cluster is obtained, all server information corresponding to the service is queried, and server routing information of the search cluster is assembled.
s402: and constructing a destination fragment list according to the server routing information.
Specifically, all fragments corresponding to the service are inquired according to the server routing information, and a destination fragment list is constructed.
S403: and traversing the fragment list, and sending query requests to all the fragments in the fragment list in parallel so that all the fragments respond to the query requests and execute corresponding queries.
specifically, the fragment list obtained in the above steps is traversed, a search request is sent to each fragment in the fragment list in parallel, and after receiving the search request, each fragment will respond to the request and execute a corresponding search operation.
The index creation process is illustrated below:
Suppose there are 100 servers (100 nodes) in a certain Eltics research cluster, and it is represented by server00 … server 99. There are 10 business parties that need to use this cluster, and in order to avoid mutual interference between the businesses, the clusters need to be physically isolated. Assuming that the usage of the cluster by 10 service parties is substantially even, 100 servers in the cluster can be divided into 10 shares, each of which contains 10 servers. The average division here is only an example and does not limit the scheme of the present invention. Suppose that the business names start with business _ uniformly, and the names of 10 business parties are business _ A, business _ B.
After dividing the service and cluster server resources, a service and cluster resource list is obtained and represented by BussinessClusterRange. Then BusinessClusterRange has
(business_A，(server00，server01，...，server09))
(business_B，(server10，server11，...，server19))
…
(business_J，(server90，server91，...，server99))
The service has an agreed correspondence with the index name. It is assumed that the index name of the service business _ a is uniformly prefixed to business _ a _ index, the index of business _ B is prefixed to business _ B _ index, and so on.
The corresponding relationship between the index and the service can be constructed and recorded into the index relationship list (represented by businesslndices mapper), and there is businesslndices mapper
(business_A_index_*，business_A)
(business_B_index_*，business_B)
…
(business_J_index_*，business_J)
when the service a needs to create an index with an index name of business _ a _ index _ computer _01 (assuming that the number of master shards is 5 and each master shard has 1 copy shard), the service name of the service a is found in the index relationship list as business _ a, and then a service-owned server list is found in the cluster resource list according to the service name business _ a (server00, server 01.., server 09).
When the index fragments are allocated, a physical allocator (which may be called a physical allocator) is used to determine that the server list for allocating the fragments is (server00, server 01.., server 09).
5 master slices and replica slices are allocated using a physical aware arbiter (which may be called a physical aware decider) equal-slice allocator. And the Elstics research broadcasts the metadata information of the index and the fragment and updates the cluster state, and the index creation is completed.
fig. 5 is a schematic structural diagram illustrating an apparatus for physical isolation of a lucene-based distributed cluster according to an exemplary embodiment, and referring to fig. 5, the apparatus includes an index creating module, which includes:
The information query unit is used for querying a current cluster resource list and a current index relation list stored in the distributed memory according to the index creating request of the service, and acquiring the service range of a server pre-allocated to the service in the cluster;
the list generating unit is used for acquiring a server list which can be used by the service according to the inquired using range of the server;
and the fragment distribution unit is used for distributing the fragments of the index to be created to the servers according to the current running state of each server in the server list, wherein the fragments comprise a main fragment and/or an auxiliary fragment.
As a preferred implementation manner, in an embodiment of the present invention, the apparatus further includes a relationship creating module, where the relationship creating module is configured to generate a current cluster resource list and a current index relationship list stored in the distributed memory, and the relationship creating module includes:
The resource dividing unit is used for distributing the service range of the servers in the cluster for the service according to the performance index of the service and the running state of the servers in the cluster;
and the list updating unit is used for updating the pre-established cluster resource list and the index relation list according to the use range of the server, generating a current cluster resource list and storing the current index relation list into the distributed memory.
as a preferred implementation manner, in an embodiment of the present invention, the apparatus further includes:
and the relationship judgment module is used for judging whether the corresponding physical isolation relationship exists in the service.
As a preferred implementation manner, in an embodiment of the present invention, the apparatus further includes a data writing module, where the data writing module includes:
The first assembly unit is used for inquiring a current cluster resource list and a current index relation list stored in the distributed memory according to an index name in a writing request of a service, and assembling server routing information written into a cluster;
The index id construction unit is used for constructing an index id of the service according to the routing information, wherein the index id range is in the id range contained in the routing server;
the data writing unit is used for merging the writing requests of each piece of data of the service and then forwarding the merged writing requests to the main slice in parallel;
and the data synchronization unit is used for synchronously writing the data written into the main fragment into the secondary fragment.
as a preferred implementation manner, in an embodiment of the present invention, the apparatus further includes a data search module, where the data search module includes:
The second assembly unit is used for inquiring a current cluster resource list and a current index relation list stored in the distributed memory according to the index name in the search request of the service, and assembling the server routing information of the search cluster;
a list construction unit for constructing a destination shard list according to the server routing information;
And the request sending unit is used for traversing the fragment list and sending a search request to each fragment in the fragment list in parallel so that each fragment responds to the search request and executes corresponding search.
in summary, the technical solution provided by the embodiment of the present invention has the following beneficial effects:
1. According to the method and the device for physical isolation of the distributed cluster based on the lucene, provided by the embodiment of the invention, by pre-allocating the server range in the cluster to the service (namely dividing cluster resources for the service), physical isolation of different services in the cluster is realized, and the response rate of all the services is remarkably improved (the response rate of the cluster is in the second level);
2. The method and the device for physical isolation of the distributed cluster based on the lucene provided by the embodiment of the invention manage the fragment distribution by using the physical distributor and the physical perception decision-making machine, realize the fragment balance in the index, and provide the guarantee of high availability and load balance for the physical isolation of the service in the cluster;
3. The method and the device for physical isolation of the distributed cluster based on the lucene provided by the embodiment of the invention unify the cluster management entries and improve the cluster operation and maintenance efficiency.
It should be noted that: in the physical isolation apparatus based on the lucene distributed cluster provided in the foregoing embodiment, when an isolation service is triggered, only the division of each functional module is used for illustration, and in practical applications, the function distribution may be completed by different functional modules according to needs, that is, the internal structure of the apparatus is divided into different functional modules, so as to complete all or part of the functions described above. In addition, the apparatus for physical isolation of a distributed cluster based on lucene provided in the foregoing embodiment and the method embodiment for physical isolation of a distributed cluster based on lucene belong to the same concept, that is, the apparatus is based on the method for physical isolation of a distributed cluster based on lucene, and a specific implementation process thereof is detailed in the method embodiment and is not described herein again.
it will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, where the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.
the above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.
Claims (10)
1. a method for physical isolation of lucene-based distributed clusters, the method comprising creating an index flow, the creating an index flow comprising the steps of:
inquiring a current cluster resource list and a current index relation list stored in a distributed memory according to an index creating request of a service, and acquiring a use range of a server to which the service is pre-distributed in a cluster;
Acquiring a server list which can be used by the service according to the inquired using range of the server;
and distributing the fragments of the index to be created to the servers according to the current running state of each server in the server list, wherein the fragments comprise a main fragment and/or a sub-fragment.
2. The method of physical isolation of a lucene-based distributed cluster as claimed in claim 1, further comprising the process of generating a current list of cluster resources and a current list of indexing relationships stored in the distributed memory:
Distributing the service range of the servers in the cluster for the service according to the performance index of the service and the running state of the servers in the cluster;
and updating the pre-established cluster resource list and the index relation list according to the use range of the server, generating a current cluster resource list and a current index relation list, and storing the current cluster resource list and the current index relation list in the distributed memory.
3. the method of any of claims 1 or 2, wherein querying the current cluster resource list and the current index relationship list stored in the distributed memory according to the index creation request of the service comprises:
and judging whether the service has a corresponding physical isolation relationship, and if so, inquiring a current cluster resource list and a current index relationship list stored in the distributed memory according to the index creating request of the service.
4. the method of physical isolation of a lucene-based distributed cluster according to any of claims 1 or 2, further comprising a write flow, said write flow comprising the steps of:
Inquiring a current cluster resource list and a current index relation list stored in the distributed memory according to an index name in a writing request of a service, and assembling server routing information written into a cluster;
Constructing an index id of the service according to the routing information, wherein the index id range is in an id range contained in a routing server;
Merging the write-in requests of each piece of data of the service and then forwarding the merged write-in requests to the main slice in parallel;
And synchronously writing the data written into the main fragment into the secondary fragment.
5. method of physical isolation of lucene-based distributed clusters according to claim 1 or 2, characterized in that it further comprises a search procedure comprising the steps of:
inquiring a current cluster resource list and a current index relation list stored in the distributed memory according to an index name in a search request of a service, and assembling server routing information of a search cluster;
constructing a destination fragment list according to the server routing information;
traversing the fragment list, and sending a search request to each fragment in the fragment list in parallel so that each fragment responds to the search request and executes corresponding search.
6. an apparatus for physical isolation of a lucene-based distributed cluster, the apparatus comprising an index creation module comprising:
the information query unit is used for querying a current cluster resource list and a current index relation list stored in the distributed memory according to the index creating request of the service, and acquiring the service range of a server pre-allocated to the service in the cluster;
the list generating unit is used for acquiring a server list which can be used by the service according to the inquired using range of the server;
And the fragment distribution unit is used for distributing the fragments of the index to be created to the servers according to the current running state of each server in the server list, wherein the fragments comprise a main fragment and/or an auxiliary fragment.
7. the apparatus for physical isolation of a lucene-based distributed cluster as claimed in claim 6, further comprising a relationship creation module for generating a current cluster resource list and a current index relationship list stored in the distributed memory, the relationship creation module comprising:
the resource dividing unit is used for distributing the service range of the servers in the cluster for the service according to the performance index of the service and the running state of the servers in the cluster;
And the list updating unit is used for updating the pre-established cluster resource list and the index relation list according to the use range of the server, generating a current cluster resource list and storing the current index relation list into the distributed memory.
8. the apparatus for physical isolation of a lucene-based distributed cluster according to claim 6 or 7, further comprising:
and the relationship judgment module is used for judging whether the corresponding physical isolation relationship exists in the service.
9. the apparatus for physical isolation of a lucene-based distributed cluster according to claim 6 or 7, further comprising a data write module comprising:
the first assembly unit is used for inquiring a current cluster resource list and a current index relation list stored in the distributed memory according to an index name in a writing request of a service, and assembling server routing information written into a cluster;
the index id construction unit is used for constructing an index id of the service according to the routing information, wherein the index id range is in the id range contained in the routing server;
The data writing unit is used for merging the writing requests of each piece of data of the service and then forwarding the merged writing requests to the main slice in parallel;
And the data synchronization unit is used for synchronously writing the data written into the main fragment into the secondary fragment.
10. The apparatus for physical isolation of a lucene-based distributed cluster according to claim 6 or 7, further comprising a data search module comprising:
The second assembly unit is used for inquiring a current cluster resource list and a current index relation list stored in the distributed memory according to the index name in the search request of the service, and assembling the server routing information of the search cluster;
A list construction unit for constructing a destination shard list according to the server routing information;
and the request sending unit is used for traversing the fragment list and sending a search request to each fragment in the fragment list in parallel so that each fragment responds to the search request and executes corresponding search.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910762813.7A CN110569302A (en) | 2019-08-16 | 2019-08-16 | method and device for physical isolation of distributed cluster based on lucene |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910762813.7A CN110569302A (en) | 2019-08-16 | 2019-08-16 | method and device for physical isolation of distributed cluster based on lucene |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110569302A true CN110569302A (en) | 2019-12-13 |
Family
ID=68775763
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910762813.7A Pending CN110569302A (en) | 2019-08-16 | 2019-08-16 | method and device for physical isolation of distributed cluster based on lucene |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110569302A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111428114A (en) * | 2020-03-27 | 2020-07-17 | 中国工商银行股份有限公司 | Index creating method and device for Elasticissearch search engine |
| CN111460023A (en) * | 2020-04-29 | 2020-07-28 | 上海东普信息科技有限公司 | Service data processing method, device, equipment and storage medium based on elastic search |
| CN111460024A (en) * | 2020-04-29 | 2020-07-28 | 上海东普信息科技有限公司 | Real-time service system based on Elasticissearch |
| CN112800104A (en) * | 2020-12-08 | 2021-05-14 | 江苏苏宁云计算有限公司 | Method and device for optimizing ES query request link |
| CN113568749A (en) * | 2021-07-28 | 2021-10-29 | 新华智云科技有限公司 | Shard distribution method based on Elasticissearch cluster |
| CN117075823A (en) * | 2023-10-17 | 2023-11-17 | 苏州元脑智能科技有限公司 | Object searching method, system, electronic device and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150229715A1 (en) * | 2014-02-13 | 2015-08-13 | Linkedin Corporation | Cluster management |
| CN107515878A (en) * | 2016-06-16 | 2017-12-26 | 苏宁云商集团股份有限公司 | The management method and device of a kind of data directory |
| CN108197296A (en) * | 2018-01-23 | 2018-06-22 | 马上消费金融股份有限公司 | Data storage method based on Elasticissearch index |
| CN108829507A (en) * | 2018-03-30 | 2018-11-16 | 北京百度网讯科技有限公司 | The resource isolation method, apparatus and server of distributed data base system |
-
2019
- 2019-08-16 CN CN201910762813.7A patent/CN110569302A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150229715A1 (en) * | 2014-02-13 | 2015-08-13 | Linkedin Corporation | Cluster management |
| CN107515878A (en) * | 2016-06-16 | 2017-12-26 | 苏宁云商集团股份有限公司 | The management method and device of a kind of data directory |
| CN108197296A (en) * | 2018-01-23 | 2018-06-22 | 马上消费金融股份有限公司 | Data storage method based on Elasticissearch index |
| CN108829507A (en) * | 2018-03-30 | 2018-11-16 | 北京百度网讯科技有限公司 | The resource isolation method, apparatus and server of distributed data base system |
Non-Patent Citations (1)
| Title |
|---|
| ZPEI0411: "《https://www.jianshu.com/p/297e13045605》", 31 July 2018 * |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111428114A (en) * | 2020-03-27 | 2020-07-17 | 中国工商银行股份有限公司 | Index creating method and device for Elasticissearch search engine |
| CN111428114B (en) * | 2020-03-27 | 2024-04-09 | 中国工商银行股份有限公司 | Index creation method and device of elastic search engine |
| CN111460023A (en) * | 2020-04-29 | 2020-07-28 | 上海东普信息科技有限公司 | Service data processing method, device, equipment and storage medium based on elastic search |
| CN111460024A (en) * | 2020-04-29 | 2020-07-28 | 上海东普信息科技有限公司 | Real-time service system based on Elasticissearch |
| CN111460023B (en) * | 2020-04-29 | 2023-06-09 | 上海东普信息科技有限公司 | Method, device, equipment and storage medium for processing service data based on elastic search |
| CN111460024B (en) * | 2020-04-29 | 2023-06-09 | 上海东普信息科技有限公司 | Real-time service system based on elastic search |
| CN112800104A (en) * | 2020-12-08 | 2021-05-14 | 江苏苏宁云计算有限公司 | Method and device for optimizing ES query request link |
| CN113568749A (en) * | 2021-07-28 | 2021-10-29 | 新华智云科技有限公司 | Shard distribution method based on Elasticissearch cluster |
| CN113568749B (en) * | 2021-07-28 | 2023-09-05 | 新华智云科技有限公司 | Method for distributing shards based on elastic search cluster |
| CN117075823A (en) * | 2023-10-17 | 2023-11-17 | 苏州元脑智能科技有限公司 | Object searching method, system, electronic device and storage medium |
| CN117075823B (en) * | 2023-10-17 | 2024-02-02 | 苏州元脑智能科技有限公司 | Object searching method, system, electronic device and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110569302A (en) | method and device for physical isolation of distributed cluster based on lucene | |
| US10901796B2 (en) | Hash-based partitioning system | |
| EP3334123B1 (en) | Content distribution method and system | |
| US9727590B2 (en) | Data management and indexing across a distributed database | |
| CN110213352B (en) | Method for aggregating dispersed autonomous storage resources with uniform name space | |
| US11726984B2 (en) | Data redistribution method and apparatus, and database cluster | |
| US20150269239A1 (en) | Storage device selection for database partition replicas | |
| JP6086463B2 (en) | Method, device and system for peer-to-peer data replication and method, device and system for master node switching | |
| CN102137014A (en) | Resource management method, system and resource manager | |
| CN113515364B (en) | Data migration method and device, computer equipment and storage medium | |
| KR101714412B1 (en) | Method and apparatus for organizing database system in cloud environment | |
| CN109918021B (en) | Data processing method and device | |
| JP2022505720A (en) | How to Effectively Reduce I / O Consumption in Containerized Relational Databases | |
| US10986065B1 (en) | Cell-based distributed service architecture with dynamic cell assignment | |
| CN112035244A (en) | Deployment of virtual node clusters in a multi-tenant environment | |
| CN112422611A (en) | Virtual bucket storage processing method and system based on distributed object storage | |
| CN111428114B (en) | Index creation method and device of elastic search engine | |
| CN109005071B (en) | Decision deployment method and scheduling equipment | |
| CN113055448B (en) | Metadata management method and device | |
| CN112905703A (en) | User information management method and system | |
| CN115687250A (en) | Storage method, equipment, system and computer storage medium | |
| KR101654969B1 (en) | Method and apparatus for assigning namenode in virtualized cluster environments | |
| CN110928943B (en) | Distributed database and data writing method | |
| JP5278254B2 (en) | Storage system, data storage method and program | |
| CN112748850A (en) | Method, apparatus and computer program product for storage management |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20191213 |
|
| RJ01 | Rejection of invention patent application after publication |