CN115442382A - Method and device for eliminating hot spot data - Google Patents

Method and device for eliminating hot spot data Download PDF

Info

Publication number
CN115442382A
CN115442382A CN202211012805.9A CN202211012805A CN115442382A CN 115442382 A CN115442382 A CN 115442382A CN 202211012805 A CN202211012805 A CN 202211012805A CN 115442382 A CN115442382 A CN 115442382A
Authority
CN
China
Prior art keywords
server
cluster
hotspot
data
hot spot
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202211012805.9A
Other languages
Chinese (zh)
Inventor
李广
郝建明
沈刚
张育新
张琦
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
China Unionpay Co Ltd
Original Assignee
China Unionpay Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by China Unionpay Co Ltd filed Critical China Unionpay Co Ltd
Priority to CN202211012805.9A priority Critical patent/CN115442382A/en
Publication of CN115442382A publication Critical patent/CN115442382A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/104—Peer-to-peer [P2P] networks
    • H04L67/1074—Peer-to-peer [P2P] networks for supporting data block transmission mechanisms
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5083—Techniques for rebalancing the load in a distributed system
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/104—Peer-to-peer [P2P] networks
    • H04L67/1044—Group management mechanisms 

Abstract

A method and a device for eliminating hot spot data are used for detecting and eliminating the hot spot data in a cluster. The method comprises the following steps: acquiring a performance influence factor of each server in a cluster; the performance influence factors comprise access frequency and garbage recovery time; determining a hot spot server in the cluster according to the garbage collection time of each server; the hotspot server is a server in a hotspot data set; and if the performance influence factor of the hotspot server meets a trigger condition, distributing the data in the hotspot server to other servers of the cluster.

Description

Method and device for eliminating hot spot data
Technical Field
The present application relates to the field of big data cluster technologies, and in particular, to a method and an apparatus for eliminating hot spot data.
Background
With the rapid development of the internet and mobile devices, more and more data are generated in daily life and work of people. In order to analyze and process large-scale data, companies and enterprises need to build a big data cluster, and Hadoop is widely applied to the field of analysis and processing of big data as an open-source distributed data processing framework.
The HBase is a distributed database in a Hadoop cluster, has the characteristics of high reliability, high performance, column-oriented performance and scalability, and can provide the capability of quickly and randomly accessing mass data. However, when the HBase is actually accessed, when one or more data tables in the Region Server are frequently accessed, a hot spot phenomenon occurs, and the performance of the Region Server where the hot spot data is located is reduced due to a large number of accesses. For the whole cluster, the uneven data access can cause unbalanced HBase load, and further cause the overall performance of the cluster to be reduced.
Therefore, a solution for detecting and eliminating hot spot data in a cluster is needed.
Disclosure of Invention
The application provides a method and a device for eliminating hot spot data, which are used for detecting and eliminating the hot spot data in a cluster.
In a first aspect, an embodiment of the present application provides a method for eliminating hot spot data, where the method includes: acquiring a performance influence factor of each server in a cluster; the performance influence factors comprise access frequency and garbage recovery time; determining a hot spot server in the cluster according to the garbage collection time of each server; the hotspot server is a server in a hotspot data set; and if the performance influence factor of the hotspot server meets a trigger condition, distributing the data in the hotspot server to other servers of the cluster.
According to the technical scheme, the hot spot server is determined according to the garbage recycling time of each server in the cluster, and when the hot spot server meets the triggering condition, the data in the currently detected hot spot server is distributed to other servers of the cluster by using the high data availability characteristic of the cluster, so that the read-write load of each server in the cluster is kept in dynamic balance, and the high performance of the whole cluster can be continuously kept.
In one possible design, the determining the hotspot server in the cluster according to the garbage collection time of each server includes: and sequencing the garbage recovery time of all the servers in the cluster, and determining the server with the longest garbage recovery time as a hot spot server in the cluster.
In the technical scheme, resources such as a CPU (central processing unit) and the like need to be consumed when the memory is subjected to garbage collection, and the influence of the time consumed by the garbage collection of the server on user experience is large, so that the server with the longest garbage collection time can be determined as a hot spot server in a cluster.
In one possible design, the triggering condition is satisfied if the performance impact factor of the hotspot server satisfies a triggering condition, including if a product of an inverse of the access frequency and the garbage collection time is greater than a preset threshold.
In the above technical solution, when data in the hotspot server is allocated to other servers, the hotspot server and the data in the hotspot server cannot be accessed, and when the determined number of times of access of the hotspot server per minute is particularly large, reallocation of the data on the hotspot server may affect many users, and therefore, the data on the hotspot server needs to be allocated under the condition that the access frequency of the hotspot server is relatively low.
In one possible design, distributing the data in the hotspot server to other servers of the cluster includes: sending an instruction of offline of the hotspot server to a management server in the cluster; and the management server distributes the data in the hotspot server to other servers of the cluster according to a load balancing strategy.
In the above technical solution, after detecting the hotspot server in the data access set, an instruction of offline of the hotspot server is sent to the management server in the cluster, and the data in the server (i.e., the hotspot server) indicating offline is distributed to other servers in the cluster by using the high availability of the data of the cluster itself.
In one possible design, the method further includes: sending an instruction of offline of the hotspot server to a monitoring server in the cluster; and the monitoring server deletes the hotspot server from the registration list, so that the hotspot server does not provide services to the outside.
In the technical scheme, after the hotspot servers with concentrated data access are detected, the command of offline of the hotspot servers is sent to the monitoring server in the cluster to indicate the monitoring server to delete the hotspot servers from the registration list, so that the situation that reading and writing are continuously carried out on the hotspot servers due to the fact that the offline of the hotspot servers is unknown is avoided.
In a possible design, after the management server distributes the data in the hotspot server to other servers of the cluster according to a load balancing policy, the method further includes: the management server brings the hotspot server online; and the monitoring server adds the hotspot server to the registration list, so that the hotspot server provides service for the outside again.
In the technical scheme, after the management server finishes distributing the data in the hotspot server, the hotspot server is put on line again so as to restart the hotspot server; and add the hotspot server to a registration list to inform the hotspot server that the service can continue to be provided to the outside.
In one possible design, the method further includes: detecting a state of the cluster; and if the cluster state is abnormal, suspending the acquisition of the performance impact factors of each server in the cluster until the cluster state is detected to be recovered to normal.
In the technical scheme, when the abnormal state of the cluster is detected, the step of eliminating the hot spot data is suspended, so that the situation that the data in the hot spot server is disordered in the process of distributing the data in the hot spot server to other servers of the cluster due to the abnormal state of the cluster is avoided.
In a second aspect, an embodiment of the present application provides an apparatus for eliminating hot spot data, including:
the acquisition module is used for acquiring the performance influence factor of each server in the cluster; the performance influence factors comprise access frequency and garbage recovery time;
a determining module, configured to determine a hotspot server in the cluster according to the garbage collection time of each server; the hotspot server is a server in a hotspot data set;
and the processing module is used for distributing the data in the hotspot server to other servers of the cluster if the performance influence factor of the hotspot server meets a trigger condition.
In a possible design, the determining module is further configured to sort the garbage collection time of all servers in the cluster, and determine a server with the longest garbage collection time as a hotspot server in the cluster.
In one possible design, the triggering condition is satisfied if the performance impact factor of the hotspot server satisfies a triggering condition, including if a product of an inverse of the access frequency and the garbage collection time is greater than a preset threshold.
In a possible design, the processing module is further configured to send an instruction to the management server in the cluster to drop the hotspot server; and the management server distributes the data in the hotspot server to other servers of the cluster according to a load balancing strategy.
In a possible design, the processing module is further configured to send an instruction that the hotspot server goes offline to a monitoring server in the cluster; and the monitoring server deletes the hotspot server from the registration list, so that the hotspot server does not provide services to the outside.
In a possible design, after the management server distributes the data in the hotspot server to other servers of the cluster according to a load balancing policy, the method further includes: the management server brings the hotspot server online; and the monitoring server adds the hotspot server into the registration list, so that the hotspot server provides service to the outside again.
In one possible design, the apparatus further includes a detection module to detect a status of the cluster; the processing module is further configured to suspend obtaining the performance impact factor of each server in the cluster if the state of the cluster is abnormal until it is detected that the state of the cluster is recovered to normal.
In a third aspect, an embodiment of the present application further provides a computing device, including:
a memory for storing program instructions;
a processor for invoking program instructions stored in said memory for executing a method as described in any one of the possible designs of the first aspect in accordance with the obtained program instructions.
In a fourth aspect, the present application further provides a computer-readable storage medium, in which computer-readable instructions are stored, and when the computer-readable instructions are read and executed by a computer, the method described in any one of the possible designs of the first aspect is implemented.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the description below are only some embodiments of the present application, and it is obvious for those skilled in the art that other drawings may be obtained according to these drawings without inventive labor.
Fig. 1 is a schematic diagram of a system architecture of an HBase provided in an embodiment of the present application;
fig. 2 is a schematic flowchart of a method for eliminating hot spot data according to an embodiment of the present disclosure;
fig. 3 is a garbage collection time line diagram of each server in a cluster before executing the present solution according to the embodiment of the present application;
fig. 4 is a garbage collection time line graph of each server in the cluster after executing the present solution according to the embodiment of the present application;
fig. 5 is a schematic diagram of a specific process for eliminating hot spot data according to an embodiment of the present disclosure;
fig. 6 is a schematic structural diagram of an apparatus for eliminating hot spot data according to an embodiment of the present disclosure;
fig. 7 is a schematic structural diagram of a computing device according to an embodiment of the present application.
Detailed Description
To make the objects, technical solutions and advantages of the present application clearer, the present application will be described in further detail with reference to the accompanying drawings, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present application without making any creative effort belong to the protection scope of the present application.
In the embodiments of the present application, a plurality means two or more. The terms "first," "second," and the like, are used for descriptive purposes only and are not to be construed as indicating or implying relative importance, nor order.
Hadoop as an open-source distributed data processing framework has been widely applied to the field of big data processing and analysis, and becomes an essential option for companies and enterprises to build big data clusters. With the rapid increase of the scale of data to be analyzed and processed, the number of servers in a large data cluster is also rapidly increased, and the application reads and writes the large data cluster more and more frequently. The problem that follows is that the servers in the cluster are frequently accessed and become abnormal or even down, which causes huge losses to companies and enterprises. Therefore, the stability and robustness issues of large data cluster operations become considerable problems.
HBase is short for Hadoop Database, is a distributed column-oriented Database established on a Hadoop file system, has the characteristics of high reliability, high performance, column orientation and scalability, and can provide the capability of quickly and randomly accessing mass data.
Fig. 1 exemplarily shows a schematic diagram of a system architecture of HBase. The HBase is uniformly managed through a large data cluster CDH (distributed computing encapsulation access hadoop) integration package. HBase runs on HDFS, and the HDFS serves as a basic storage facility. As can be seen from the architecture diagram of the HBase, the components in the HBase include a Client, a Master, a Zookeeper cluster and a Region Server cluster.
The Client comprises an interface for accessing the HBase, and the HBase accesses an HBase database through an Application Program Interface (API) provided by the Client to complete writing and reading of data.
The Master manages all the Region servers in the Region Server cluster, monitors the state of each Region Server, distributes Regions to each Region Server, and realizes load balancing and fault transfer of the Region Server cluster.
The Zookeeper cluster consists of a plurality of Zookeeper, and the Zookeeper stores the state information of the Region Server cluster and the addressing entry of each Region in the Region Server. The Master and the Region Server register to the Zookeeper when starting, and the Zookeeper can ensure that only one Master is in the cluster at any time and monitor the Region Server.
The Region Server cluster is composed of a plurality of Region servers, the Region servers are used for maintaining regions distributed to the Region servers by a Master, and the regions are the minimum units of distributed storage and load balancing in HBase.
It should be noted that the system architecture of the HBase shown in fig. 1 is only an example, and this is not specifically limited in this embodiment of the present application.
In the prior art, the data table can be distributed to each Region Server in two ways, one way is to distribute the data table according to an average value upper and lower limit algorithm by a load balancer of an HBase, so as to ensure that the number of regions in each Region Server service is balanced. In the method, when the data table is distributed, factors such as access of hot data, bottom file distribution, physical resources of all nodes and the like are not considered, so that the phenomenon of reading and writing hot spots is easy to generate, and ideal load balance cannot be achieved.
Another way is to scatter a single data table to multiple Region servers by using a command, but the scattering rule in this way is only simple average distribution, for example, 1 data table is evenly distributed to 10 designated Region servers, because hot data and hot Region servers cannot be predicted in advance, after the data table runs for a period of time, because the application is accessed by non-uniform reading and writing, the hot data and hot Region servers will randomly appear in the Hbase cluster, and as the service duration increases, the data distribution in the Region Server cluster is easily degraded gradually, and further the performance of the whole cluster is reduced.
Based on the above description, the present application provides a method for eliminating hot spot data, so as to detect and eliminate hot spot data in a cluster.
Fig. 2 exemplarily illustrates a flowchart of a method for eliminating hot spot data according to an embodiment of the present application, and as shown in fig. 2, the method includes the following steps:
step 201, acquiring a performance influence factor of each Server (Region Server) in the cluster.
The performance impact factors include access frequency and Garbage Collection (GC) time. The access frequency is the number of times of accessing the Region Server per unit time. The garbage collection time is used for recovering and releasing the unneeded memory space, so that the situation that the unneeded memory space exceeds available memory (OOM) due to unlimited memory growth is avoided.
Step 202, determining a hot spot server in the cluster according to the garbage collection time of each server.
In the embodiment of the application, the hotspot Server is a Region Server in a hotspot data set. Resources such as a CPU (central processing unit) and the like need to be consumed when the memory is subjected to garbage collection, and the garbage collection time of the Region Server has a large influence on user experience, so that the hot Region Server in the cluster can be determined according to the garbage collection time of each Region Server.
For example, when determining the hot spot Region servers in the cluster according to the garbage collection time of each Region Server, the garbage collection times of all the Region servers in the cluster may be sorted, and the Region Server with the longest garbage collection time is determined to be the hot spot Region Server in the cluster.
And 203, distributing the data in the hotspot server to other servers of the cluster if the performance influence factor of the hotspot server meets the trigger condition.
In one example, when determining whether the hotspot Region Server performance impact factor satisfies the trigger condition, the product of the reciprocal of the access frequency and the garbage collection time may be greater than a preset threshold as the trigger condition. If the product of the reciprocal of the access frequency and the garbage recovery time is larger than a preset threshold, the triggering condition is met, and the data in the hot spot Region Server is distributed to other Region servers of the cluster; if the product of the reciprocal of the access frequency and the garbage collection time is not greater than the preset threshold, the trigger condition is not satisfied, and the steps 501 to 503 are repeatedly executed.
It should be noted that the preset threshold may be determined according to the range of the actual garbage collection time of the Region Server in the cluster and the range of the access times per minute of the Region Server. For example, the actual garbage collection time of the Region Server in a cluster ranges from about 40s to 180s, and the range of the access times of the Region Server per minute ranges from about 10 times to about 300 times, so that the preset threshold may be set to 1, that is, the garbage collection time/access frequency of the hot-spot Region Server > 1 meets the trigger condition, where the unit of the garbage collection time is second.
In the technical scheme, when the hot spot Region Server is offline and the data in the hot spot Region Server is distributed to other servers, the hot spot Region Server and the data in the hot spot Region Server cannot be accessed, and when the determined number of times of access of the hot spot Region Server per minute is large, the data on the hot spot Region Server is redistributed, so that more users are affected, and therefore, the data on the hot spot Region Server needs to be distributed under the condition that the access frequency of the hot spot Region Server is relatively low. For example, the current garbage collection time of the hot spot Region Server is 180s, and when the range of the access times per minute of the Region Server is less than 180 times, the influence of the lower line of the hot spot Region Server on the user is considered to be small, and the data in the hot spot Region Server can be distributed to other servers. When the range of the access times of the Region Server per minute is more than or equal to 180 times, the influence of the lower line of the hot spot Region Server on the user is considered to be large, and the data in the hot spot Region Server is not processed at the moment.
It is understood that the above-mentioned trigger condition may also be that the product of the access frequency and the inverse of the garbage collection time is smaller than a preset threshold.
In yet another example, when determining whether the hotspot Region Server performance impact factor satisfies the trigger condition, the access frequency may be smaller than a preset threshold as the trigger condition. If the access frequency is less than a preset threshold value, the influence of the lower line of the hot spot Region Server on a user is considered to be small, the triggering condition is met, and the data in the hot spot Region Server is distributed to other Region servers of the cluster; if the access frequency is not less than the preset threshold, the influence of the lower line of the hot spot Region Server on the user is considered to be large, the trigger condition is not met, and the steps 501 to 503 are repeatedly executed. For example, the actual garbage collection time range of the Region Server in the cluster is about 40s-180s, and the range of the access frequency of the Region Server per minute is about 10 times-300 times, then the preset threshold value may be set to 180, that is, the access frequency of the hot-spot Region Server is less than 180 times, that is, the trigger condition is satisfied.
And after judging that the triggering condition is met, sending an offline instruction of the hotspot Region Server to a Master and a Zookeeper in the cluster through an API (application programming interface). After receiving the offline instruction of the hot spot Region Server, the Master distributes the data in the hot spot Region Server to other Region servers of the cluster according to the strategy of load balancing. After receiving the instruction of offline of the hotspot Region Server, the Zookeeper deletes the hotspot Region Server from the registration list, so that the hotspot Region Server does not provide service to the outside.
It is understood that, during actual execution, the Master allocates the data in the hot spot Region Server to other Region servers of the cluster, and instead of allocating the data actually stored in the hot spot Region Server, the Master restores and allocates copies of the hot spot Region Server to other Region servers of the cluster.
After the Master distributes the data in the hotspot Region Server to other Region servers of the cluster, the Master brings the hotspot Region Server on line, and the Zookeeper adds the hotspot Server to a registration list, so that the hotspot Region Server provides services for the outside again.
Through the operation, the data on the Region Server with the most concentrated hot spot data currently detected can be dispersed to other Region servers of the cluster. By repeatedly executing the operation, the garbage recovery time of each Region Server in the cluster approaches to be consistent, namely, the difference between the longest garbage recovery time consumption and the shortest garbage recovery time consumption is smaller, and further the Region Server of the whole cluster realizes real-time load balancing. For example, fig. 3 is a line graph of garbage collection time of each server in the cluster before executing the present solution; fig. 4 is a line diagram of garbage collection time of each server in the cluster after the scheme is executed. The garbage collection time of the uppermost Region Server in fig. 3 is obviously longer than that of other Region servers in the cluster, and the data distribution in the Region Server cluster is easily degraded along with the increase of the service life, so that the performance of the whole cluster is reduced. After detecting the hot spot Region Server for multiple times and scattering the data in the hot spot Region Server to other Region servers in the cluster, as shown in fig. 4, the garbage collection time of each Region Server in the cluster is nearly consistent.
In a possible implementation manner, before the performance impact factor of each server in the cluster is acquired, the state of the cluster is detected, and if the state of the cluster is abnormal, the acquisition of the performance impact factor of each server in the cluster is suspended until the state of the cluster is detected to be recovered to be normal. The situation that data are disordered in the process of distributing the data in the hotspot server to other servers of the cluster due to abnormal cluster state is avoided.
The method for eliminating the hot data determines the Region Server with the longest garbage recovery time as the hot Region Server according to the garbage recovery time consumption of each Region Server in the cluster, and distributes the currently detected data in the hot Region Server to other Region servers of the cluster by using the high data availability characteristic of the cluster when the hot Region Server meets the triggering condition, so that the read-write load of each Region Server in the cluster keeps dynamic balance, and the high performance of the whole Hbase cluster can be continuously kept.
For better explaining the embodiment of the present application, fig. 5 exemplarily shows a schematic diagram of a specific process for eliminating hot spot data provided by the embodiment of the present application, and the process includes the following steps.
And step 501, acquiring the state information of the cluster.
And acquiring the state information of the cluster through the API.
And 502, judging whether the cluster state is normal or not.
Judging whether the cluster normally operates according to the acquired state information of the cluster; if the cluster state is normal, go to step 503; if the cluster status is abnormal, step 508 is executed.
Step 503, determining the hot spot Region Server in the cluster according to the garbage recovery time of each Region Server in the cluster.
Specifically, the garbage collection time of all Region servers in the cluster may be sorted, and the Region Server with the longest garbage collection time is determined as the hot Region Server in the cluster.
And step 504, judging whether the garbage collection time/access frequency is more than 1.
Judging whether the product of the reciprocal of the access frequency of the hot spot Region Server and the garbage recovery time is more than 1; if so, go to step 505; otherwise, step 501 is performed.
The access frequency is the number of times of accessing the hot spot Region Server every minute, and the unit of garbage recovery time is second.
And 505, sending an instruction of the hot spot Region Server offline to Master and Zookeeper in the cluster.
And step 506, the Master distributes the data in the hot spot Region Server to other Region servers of the cluster according to a load balancing strategy.
The hotspot Region Server is deleted from the registration list by the Zookeeper, so that the hotspot Region Server does not provide services to the outside.
And step 507, the Master brings the hotspot Region Server on line.
And the Zookeeper adds the hot spot Region Server into the registration list, and the hot spot Region Server provides services for the outside again.
After step 507 is executed, step 501 is repeatedly executed.
Step 508, tentative 5 minutes.
After a pause of 5 minutes, step 501 is repeated.
Based on the same technical concept, fig. 6 exemplarily illustrates an apparatus for eliminating hot spot data provided by an embodiment of the present application. As shown in fig. 6, the apparatus 600 includes:
an obtaining module 601, configured to obtain a performance impact factor of each server in a cluster; the performance influence factors comprise access frequency and garbage recovery time;
a determining module 602, configured to determine a hotspot server in the cluster according to the garbage collection time of each server; the hotspot server is a server in a hotspot data set;
a processing module 603, configured to allocate the data in the hotspot server to other servers of the cluster if the performance impact factor of the hotspot server meets a trigger condition.
In a possible design, the determining module 602 is further configured to sort the garbage collection time of all servers in the cluster, and determine that a server with the longest garbage collection time is a hotspot server in the cluster.
In one possible design, the triggering condition is satisfied if the performance impact factor of the hotspot server satisfies a triggering condition, including if a product of an inverse of the access frequency and the garbage collection time is greater than a preset threshold.
In a possible design, the processing module 603 is further configured to send an instruction to the management server in the cluster to drop the hotspot server; and the management server distributes the data in the hotspot server to other servers of the cluster according to a load balancing strategy.
In a possible design, the processing module 603 is further configured to send an instruction to the monitoring server in the cluster to drop the hotspot server; and the monitoring server deletes the hotspot server from the registration list, so that the hotspot server does not provide services to the outside.
In a possible design, after the management server distributes the data in the hotspot server to other servers of the cluster according to a load balancing policy, the method further includes: the management server brings the hotspot server online; and the monitoring server adds the hotspot server to the registration list, so that the hotspot server provides service for the outside again.
In one possible design, the apparatus further includes a detection module 604 for detecting a status of the cluster; the processing module 603 is further configured to suspend obtaining the performance impact factor of each server in the cluster if the state of the cluster is abnormal, until it is detected that the state of the cluster is recovered to normal.
Based on the same technical concept, the embodiment of the present application provides a computing device, as shown in fig. 7, including at least one processor 701 and a memory 702 connected to the at least one processor, where a specific connection medium between the processor 701 and the memory 702 is not limited in this embodiment, and the processor 701 and the memory 702 are connected through a bus in fig. 7 as an example. The bus may be divided into an address bus, a data bus, a control bus, etc.
In this embodiment, the memory 702 stores instructions executable by the at least one processor 701, and the at least one processor 701 may execute the method for removing hot spot data by executing the instructions stored in the memory 702.
The processor 701 is a control center of the computing device, and may connect various parts of the computing device by using various interfaces and lines, and perform resource setting by executing or executing instructions stored in the memory 702 and calling data stored in the memory 702.
Alternatively, the processor 701 may include one or more processing units, and the processor 701 may integrate an application processor, which mainly handles an operating system, a user interface, application programs, and the like, and a modem processor, which mainly handles wireless communication. It will be appreciated that the modem processor described above may not be integrated into the processor 701. In some embodiments, processor 701 and memory 702 may be implemented on the same chip, or in some embodiments they may be implemented separately on separate chips.
The processor 701 may be a general-purpose processor, such as a Central Processing Unit (CPU), a digital signal processor, an Application Specific Integrated Circuit (ASIC), a field programmable gate array or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or the like, that may implement or perform the methods, steps, and logic blocks disclosed in the embodiments of the present Application. The general purpose processor may be a microprocessor or any conventional processor or the like. The steps of a method disclosed in connection with the embodiments of the present application may be directly implemented by a hardware processor, or may be implemented by a combination of hardware and software modules in a processor.
Memory 702, which is a non-volatile computer-readable storage medium, may be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. The Memory 702 may include at least one type of storage medium, and may include, for example, a flash Memory, a hard disk, a multimedia card, a card-type Memory, a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a Programmable Read Only Memory (PROM), a Read Only Memory (ROM), a charge Erasable Programmable Read Only Memory (EEPROM), a magnetic Memory, a magnetic disk, an optical disk, and so on. The memory 702 is any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited to such. The memory 702 in the embodiments of the present application may also be circuitry or any other device capable of performing a storage function to store program instructions and/or data.
Based on the same technical concept, embodiments of the present application further provide a computer-readable storage medium, where a computer-executable program is stored, and the computer-executable program is configured to enable a computer to execute the method for eliminating hot spot data listed in any of the above manners.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
While the preferred embodiments of the present application have been described, additional variations and modifications of these embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including preferred embodiments and all alterations and modifications as fall within the scope of the application.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present application without departing from the spirit and scope of the application. Thus, if such modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is intended to include such modifications and variations as well.

Claims (10)

1. A method of eliminating hotspot data, the method comprising:
acquiring a performance influence factor of each server in a cluster; the performance influence factors comprise access frequency and garbage recovery time;
determining a hot spot server in the cluster according to the garbage collection time of each server; the hotspot server is a server in a hotspot data set;
and if the performance influence factor of the hotspot server meets a trigger condition, distributing the data in the hotspot server to other servers of the cluster.
2. The method of claim 1, wherein the determining the hotspot server in the cluster according to the garbage collection time of each server comprises:
sorting the garbage collection time of all the servers in the cluster, and determining the server with the longest garbage collection time as a hot spot server in the cluster.
3. The method of claim 1, wherein the determining if the performance impact factor of the hotspot server satisfies a trigger condition comprises
And if the product of the reciprocal of the access frequency and the garbage recycling time is greater than a preset threshold value, a triggering condition is met.
4. The method of claim 1, wherein distributing the data in the hotspot server to other servers of the cluster comprises:
sending an instruction of offline of the hotspot server to a management server in the cluster;
and the management server distributes the data in the hotspot server to other servers of the cluster according to a load balancing strategy.
5. The method of claim 4, further comprising:
sending an instruction of offline of the hotspot server to a monitoring server in the cluster;
and the monitoring server deletes the hotspot server from the registration list, so that the hotspot server does not provide services to the outside.
6. The method of claim 5, wherein after the management server distributes the data in the hotspot server to other servers of the cluster according to a load balancing policy, the method further comprises:
the management server brings the hotspot server online;
and the monitoring server adds the hotspot server into the registration list, so that the hotspot server provides service to the outside again.
7. The method according to any one of claims 1 to 6, further comprising:
detecting a state of the cluster;
and if the cluster state is abnormal, suspending the acquisition of the performance impact factors of each server in the cluster until the cluster state is detected to be recovered to normal.
8. An apparatus for eliminating hot spot data, comprising:
the acquisition module is used for acquiring the performance influence factor of each server in the cluster; the performance influence factors comprise access frequency and garbage recovery time;
a determining module, configured to determine a hotspot server in the cluster according to the garbage collection time of each server; the hotspot server is a server in a hotspot data set;
and the processing module is used for distributing the data in the hotspot server to other servers of the cluster if the performance influence factor of the hotspot server meets a trigger condition.
9. A computing device, comprising:
a memory for storing program instructions;
a processor for calling program instructions stored in said memory and for executing the method of any one of claims 1 to 7 in accordance with the obtained program instructions.
10. A computer readable storage medium comprising computer readable instructions which, when read and executed by a computer, cause the method of any one of claims 1 to 7 to be carried out.
CN202211012805.9A 2022-08-23 2022-08-23 Method and device for eliminating hot spot data Pending CN115442382A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202211012805.9A CN115442382A (en) 2022-08-23 2022-08-23 Method and device for eliminating hot spot data

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202211012805.9A CN115442382A (en) 2022-08-23 2022-08-23 Method and device for eliminating hot spot data

Publications (1)

Publication Number Publication Date
CN115442382A true CN115442382A (en) 2022-12-06

Family

ID=84245400

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202211012805.9A Pending CN115442382A (en) 2022-08-23 2022-08-23 Method and device for eliminating hot spot data

Country Status (1)

Country Link
CN (1) CN115442382A (en)

Similar Documents

Publication Publication Date Title
CN107360206B (en) Block chain consensus method, equipment and system
CN109597685B (en) Task allocation method, device and server
US8726290B2 (en) System and/or method for balancing allocation of data among reduce processes by reallocation
CN112162865B (en) Scheduling method and device of server and server
CN109783237B (en) Resource allocation method and device
KR101600129B1 (en) Application efficiency engine
CN106326002B (en) Resource scheduling method, device and equipment
CN108549583B (en) Big data processing method and device, server and readable storage medium
CN107968810B (en) Resource scheduling method, device and system for server cluster
CN106874100B (en) Computing resource allocation method and device
US20170017511A1 (en) Method for memory management in virtual machines, and corresponding system and computer program product
US11010190B2 (en) Methods, mediums, and systems for provisioning application services
CN106713388B (en) Burst service processing method and device
EP3238055A1 (en) Allocating cloud computing resources in a cloud computing environment
US20180359338A1 (en) Data driven bin packing implementation for data centers with variable node capabilities
CN104580194A (en) Virtual resource management method and device oriented to video applications
CN109359060B (en) Data extraction method, device, computing equipment and computer storage medium
CN112346871A (en) Request processing method and micro-service system
CN112650575A (en) Resource scheduling method and device and cloud service system
CN110321364B (en) Transaction data query method, device and terminal of credit card management system
CN107408073A (en) Using memory compression expense is submitted to reduce memory
CN109788013B (en) Method, device and equipment for distributing operation resources in distributed system
Tsai et al. Learning-based memory allocation optimization for delay-sensitive big data processing
CN113741991A (en) Method and device for improving SSD service quality, computer equipment and storage medium
CN111338579B (en) Read-write cache optimization method, system, terminal and storage medium based on storage pool

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination