CN114385081A - Disk protection method for kafka cluster and related equipment - Google Patents
Disk protection method for kafka cluster and related equipment Download PDFInfo
- Publication number
- CN114385081A CN114385081A CN202111611075.XA CN202111611075A CN114385081A CN 114385081 A CN114385081 A CN 114385081A CN 202111611075 A CN202111611075 A CN 202111611075A CN 114385081 A CN114385081 A CN 114385081A
- Authority
- CN
- China
- Prior art keywords
- segment
- partition
- file
- segment file
- cleanable
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 74
- 238000005192 partition Methods 0.000 claims abstract description 215
- 238000003860 storage Methods 0.000 claims abstract description 113
- 238000009826 distribution Methods 0.000 claims abstract description 45
- 238000004140 cleaning Methods 0.000 claims description 77
- 238000012163 sequencing technique Methods 0.000 claims description 7
- 238000012544 monitoring process Methods 0.000 claims description 4
- 238000004364 calculation method Methods 0.000 claims description 3
- 238000010586 diagram Methods 0.000 description 7
- 238000012217 deletion Methods 0.000 description 6
- 230000037430 deletion Effects 0.000 description 6
- 230000014759 maintenance of location Effects 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 238000012545 processing Methods 0.000 description 4
- 230000004044 response Effects 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 239000003795 chemical substances by application Substances 0.000 description 3
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000001514 detection method Methods 0.000 description 3
- 230000007246 mechanism Effects 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 238000007689 inspection Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- ZLIBICFPKPWGIZ-UHFFFAOYSA-N pyrimethanil Chemical compound CC1=CC(C)=NC(NC=2C=CC=CC=2)=N1 ZLIBICFPKPWGIZ-UHFFFAOYSA-N 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0643—Management of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0644—Management of space entities, e.g. partitions, extents, pools
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
- G06F3/0652—Erasing, e.g. deleting, data cleaning, moving of data to a wastebasket
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the application provides a disk protection method and related equipment for a kafka cluster, wherein in the disk protection method, the disk capacities of all storage servers in the kafka cluster are polled; when the disk capacity of any storage server reaches a protection threshold value, acquiring partition distribution information, inquiring a data directory of the storage server for each storage server, and finding out a first preset number of segment files according to a descending order of occupied space; for each segment file, obtaining the theme and the partition of the segment file, and judging whether the partition of the segment file is a main partition according to the distribution information; when the partition to which the segment file belongs is a main partition, the segment file under the partition to which the segment file belongs is cleaned according to a preset proportion, so that the available capacity of a disk is ensured, the problem of overload of a disk space is avoided when the kafka system generates burst flow, and the high availability of the kafka service is ensured.
Description
[ technical field ] A method for producing a semiconductor device
The embodiment of the application relates to the technical field of computers, in particular to a disk protection method and related equipment for a kafka cluster.
[ background of the invention ]
Kafka is an open source stream processing platform developed by the Apache software foundation, written in Scala and Java. Kafka is a high-throughput publish-subscribe messaging system that can handle all action flow data in a consumer-sized web site.
The architecture of Kafka includes the following components:
topic (Topic) is a particular type of message stream. The message is a payload (Playload) of bytes, and the topic is a classification name or a seed (feed) name of the message.
A producer is any object that can publish a message to a topic.
A Broker (Broker) or kafla cluster is a set of servers that hold published messages.
The consumer may subscribe to one or more topics and pull data from the Broker (Broker) to consume the published messages.
The Kafka existing deletion strategy can be divided into: the method comprises a time-based deletion strategy and a file size-based deletion strategy, wherein the two strategies cannot cope with the condition that burst traffic occurs in a short time, so that the disk space is full, and the kafka service is unavailable.
[ summary of the invention ]
The embodiment of the application provides a disk protection method and related equipment for a kafka cluster, so that the available capacity of a disk is ensured, the disk space cannot be overloaded when a kafka system has burst traffic, and the high availability of kafka service is ensured.
In a first aspect, an embodiment of the present application provides a disk protection method for a kafka cluster, including: polling the disk capacity of all storage servers in the kafka cluster; when the disk capacity of any storage server reaches a protection threshold value, obtaining partition distribution information, wherein the partition distribution information comprises themes set on each storage server and main partitions subordinate to each theme; for each storage server, inquiring a data directory of the storage server, and finding out a first preset number of segment files according to the sequence of the occupied space from large to small; for each segment file, obtaining the theme and the partition of the segment file, and judging whether the partition of the segment file is a main partition according to the distribution information; and when the partition to which the segment file belongs is a main partition, cleaning the segment file under the partition to which the segment file belongs according to a preset proportion.
In the disk protection method, the use condition of the disk space of the storage server is monitored, and when the size of the occupied space of the disk exceeds a preset threshold value, the disk space is cleaned, so that the available capacity of the disk is ensured, the disk space cannot be overloaded when the kafka system has burst flow, and the high availability of the kafka service is ensured. The method ensures that the disk space is not occupied by burst traffic under the scene that the disk space is limited, and realizes a mechanism of kafka cluster service self-protection, thereby improving the availability of the kafka cluster.
In one possible implementation manner, the clearing, according to a preset proportion, a segment file under a partition to which the segment file belongs includes: searching for a cleanable segment file under a partition to which the segment file belongs, wherein the cleanable segment file is a segment file without a snapshot file; and selecting the segment files to be cleaned according to a preset proportion from the cleanable segment files, and cleaning the segment files to be cleaned.
In one possible implementation manner, the selecting, according to a preset ratio, a segment file to be cleaned from the cleanable segment files includes: sequencing the cleanable segment files according to a time sequence; and selecting the segment files with a preset proportion in the sequence from far to near as the file units to be cleaned.
In one possible implementation manner, the clearing the segment file to be cleared includes: for each segment file to be cleaned, acquiring the file name of the segment file to be cleaned, and calculating the initial offset of data according to the file name; and cleaning the segment files to be cleaned according to the starting offset in the partitions to which the segment files belong.
In one possible implementation manner, the method further includes: if finding the cleanable segment file in the partition to which the segment file belongs fails, adjusting the segment file parameters to adjust the size of the segment file; searching the cleanable segment file under the partition to which the segment file belongs again, wherein the cleanable segment file is a segment file without a snapshot file; and selecting the segment files to be cleaned according to a preset proportion from the cleanable segment files, and cleaning the segment files to be cleaned.
In a second aspect, an embodiment of the present application provides a disk protection device for a kafka cluster, where the disk protection device is arranged in a terminal device, and the disk protection device includes: the monitoring module is used for polling the disk capacities of all the storage servers in the kafka cluster; the system comprises an acquisition module, a storage module and a management module, wherein the acquisition module is used for acquiring partition distribution information when the disk capacity of any storage server reaches a protection threshold, and the partition distribution information comprises themes set on each storage server and main partitions subordinate to each theme; the query module is used for querying a data directory of each storage server and finding out a first preset number of segment files according to the sequence of occupied space from large to small; the judging module is used for acquiring the theme and the partition of each segment file and judging whether the partition of each segment file is a main partition according to the distribution information; and the first cleaning module is used for cleaning the segment files under the partition to which the segment files belong according to a preset proportion when the partition to which the segment files belong is a main partition.
In one possible implementation manner, the first cleaning module includes: the searching module is used for searching the cleanable segment file under the partition to which the segment file belongs, wherein the cleanable segment file is a segment file without a snapshot file; and the second cleaning module is used for selecting the segment files to be cleaned in the cleanable segment files according to a preset proportion and cleaning the segment files to be cleaned.
In one possible implementation manner, the second cleaning module includes: the sorting module is used for sorting the cleanable segment files according to a time sequence; and the selection module is used for selecting the segment files with the preset proportion in the sequence from far to near as the file units to be cleaned.
In one possible implementation manner, the second cleaning module includes: the calculation module is used for acquiring the file name of each segment file to be cleaned and calculating the initial offset of data according to the file name; and the third cleaning module is used for cleaning the segment file to be cleaned in the partition to which the segment file belongs according to the starting offset.
In one possible implementation manner, the apparatus further includes: the adjusting module is used for adjusting segment file parameters to adjust the size of the segment file if finding the cleanable segment file in the partition to which the segment file belongs fails; the second searching module is used for searching the cleanable segment file under the partition to which the segment file belongs again, wherein the cleanable segment file is a segment file without a snapshot file; and the fourth cleaning module is used for selecting the segment files to be cleaned in the cleanable segment files according to a preset proportion and cleaning the segment files to be cleaned.
In a third aspect, an embodiment of the present application provides a disk protection method for a kafka cluster, where the method is applied in a console of the kafka cluster, the kafka cluster further includes a plurality of storage servers and a coordination server, and the method includes: polling the disk capacity of all storage servers in the kafka cluster; when the disk capacity of any storage server reaches a protection threshold value, obtaining partition distribution information stored on the coordination server, wherein the partition distribution information comprises themes set on each storage server and main partitions subordinate to each theme; sending a cleaning instruction to each storage server to enable each storage server to clean files, wherein the cleaning instruction comprises the partition distribution information
In a fourth aspect, an embodiment of the present application provides a disk protection method for a kafka cluster, where the method is applied in a storage server of the kafka cluster, and the kafka cluster further includes a console, where the method includes: receiving a cleaning instruction sent by the console, wherein the cleaning instruction comprises the partition distribution information, and the partition distribution information comprises themes set on each storage server and main partitions subordinate to each theme; searching a first preset number of segment files according to the sequence of occupied space from large to small in the inquired data directory; for each segment file, obtaining the theme and the partition of the segment file, and judging whether the partition of the segment file is a main partition according to the distribution information; and when the partition to which the segment file belongs is a main partition, cleaning the segment file under the partition to which the segment file belongs according to a preset proportion.
In one possible implementation manner, the clearing, according to a preset proportion, a segment file under a partition to which the segment file belongs includes: searching for a cleanable segment file under a partition to which the segment file belongs, wherein the cleanable segment file is a segment file without a snapshot file; and selecting the segment files to be cleaned according to a preset proportion from the cleanable segment files, and cleaning the segment files to be cleaned.
In one possible implementation manner, the selecting, according to a preset ratio, a segment file to be cleaned from the cleanable segment files includes: sequencing the cleanable segment files according to a time sequence; and selecting the segment files with a preset proportion in the sequence from far to near as the file units to be cleaned.
In one possible implementation manner, the clearing the segment file to be cleared includes: for each segment file to be cleaned, acquiring the file name of the segment file to be cleaned, and calculating the initial offset of data according to the file name; and cleaning the segment files to be cleaned according to the starting offset in the partitions to which the segment files belong.
In one possible implementation manner, the method further includes: if finding the cleanable segment file in the partition to which the segment file belongs fails, adjusting the segment file parameters to adjust the size of the segment file; searching the cleanable segment file under the partition to which the segment file belongs again, wherein the cleanable segment file is a segment file without a snapshot file; and selecting the segment files to be cleaned according to a preset proportion from the cleanable segment files, and cleaning the segment files to be cleaned.
In a fifth aspect, an embodiment of the present application provides a terminal device, including: at least one processor; and at least one memory communicatively coupled to the processor, wherein: the memory stores program instructions executable by the processor, the processor calling the program instructions to be able to perform the method provided by the first aspect.
In a sixth aspect, embodiments of the present application provide a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the method provided in the first aspect.
It should be understood that the second, fifth, and sixth aspects of the embodiments of the present application are consistent with the technical solution of the first aspect of the embodiments of the present application, and beneficial effects obtained by various aspects and corresponding possible implementations are similar and will not be described again.
[ description of the drawings ]
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present specification, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a flowchart of a disk protection method for a kafka cluster according to a first embodiment of the present application;
fig. 2 is a flowchart of another disk protection method for a kafka cluster according to the first embodiment of the present application;
fig. 3 is a flowchart of a disk protection method for a kafka cluster according to a first embodiment of the present application;
fig. 4 is a schematic structural diagram of a disk protection device for a kafka cluster according to a second embodiment of the present application;
fig. 5 is a schematic structural diagram of another disk protection apparatus for a kafka cluster according to a second embodiment of the present specification;
FIG. 6 is a schematic structural diagram of a kafka cluster provided in the present specification;
fig. 7 is a flowchart of a disk protection method for a kafka cluster according to a third embodiment of the present specification;
fig. 8 is a flowchart of a disk protection method for a kafka cluster according to a fourth embodiment of the present disclosure.
[ detailed description ] embodiments
For better understanding of the technical solutions in the present specification, the following detailed description of the embodiments of the present application is provided with reference to the accompanying drawings.
It should be understood that the described embodiments are only a few embodiments of the present specification, and not all embodiments. All other embodiments obtained by a person skilled in the art based on the embodiments in the present specification without any inventive step are within the scope of the present specification.
The terminology used in the embodiments of the present application is for the purpose of describing particular embodiments only and is not intended to be limiting of the specification. As used in the examples of this application and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.
In the prior art, kafka provides two deletion strategies: a time-based deletion policy and a file size-based deletion policy, both of which may be set by the following system parameters:
log. retention. hours — set data retention time;
log.retention. bytes — set the maximum size of the file;
however, when burst traffic occurs in both of these two strategies within a short time, there is a risk that the disk space is occupied, for example, assuming that the disk space is 100G, the data retention time is set to 24 hours, but the data volume exceeds 100G within a burst 24 hour, which may result in the disk space being full. In addition, for the policy based on the file size, the policy is actually periodically queried, that is, kafka queries the saved files periodically to find whether any file exceeds the maximum limit, and if so, deletes the file. When a large amount of burst traffic occurs during two queries, the file cannot be deleted by the strategy, which results in the full disk space.
When the disk space is full, the kafka service is down, and the service is unavailable.
Based on the above problems, embodiments of the present application provide a disk protection method for a kafka cluster, which monitors a usage situation of a disk space of a storage server, and when a size of an occupied space of a disk exceeds a preset threshold, cleans the occupied space of the disk, and ensures an available capacity of the disk, so that when a burst traffic occurs in a kafka system, an overload problem does not occur in the disk space, and high availability of a kafka service is ensured. The method ensures that the disk space is not occupied by burst traffic under the scene that the disk space is limited, and realizes a mechanism of kafka cluster service self-protection, thereby improving the availability of the kafka cluster.
Fig. 1 is a flowchart of a disk protection method for a kafka cluster according to an embodiment of the present application, and as shown in fig. 1, the disk protection method may include:
and step 101, polling the disk capacities of all storage servers in the kafka cluster.
And sending out an inquiry at regular time, sequentially inquiring the disk capacity of each storage server, or simultaneously sending out inquiries to all the storage servers and inquiring the disk use conditions of the storage servers.
When the inquiry result shows that the disk capacity of any storage server reaches a preset protection threshold, acquiring the distribution information of Kafka, wherein the distribution information records the theme set on each agent (Broker) for storing messages in Kafka and the partition under the theme, and for convenience of description, the agent (Broker) is called the storage server in the embodiment.
In Kafka, a Topic (Topic) may set one or more partitions (partitation). The partitions are stored on agents (Broker), in order to deal with the situation that data partitions are lost due to sudden factors such as server downtime and the like, Kafka may set multiple copies (replias) for each partition, randomly select one copy (replia) as a leader partition, which is called a master partition in this embodiment, and use the other copies (replias) as follower partitions, which is called slave partitions in this embodiment. When sending data, a producer directly sends the data to a main partition (leader partition), and then removes the synchronous data of the main partition (leader partition) from a partition (follower partition), and when consuming the data, a consumer consumes the data from the main partition (leader partition).
That is, each storage server has one or more partitions with one or more topics stored thereon, and the stored partitions may be master partitions or slave partitions. In the present application, data in which a set theme, a partition under the set theme, and master-slave attributes of the partition are recorded on each storage server is referred to as distribution information.
Step 103, for each storage server, querying a data directory of the storage server, and finding out segment files of a first preset number according to a descending order of occupied space.
For each storage server, scanning the file stored on the storage server, and deleting the file, specifically:
the data directory of the storage server is queried, the data directory refers to metadata of all data stored on the storage server, and can be simply understood as a directory of saved data.
By querying the data directory, the storage condition on the storage server can be known: which files are saved, the size of the space occupied by each file, the saving path of each file, etc.
The Partition (Partition) is divided into a plurality of segments (Segment files) for storage, and the segments (Segment files) are the smallest storage unit of kafka. Segment file is composed of two parts, namely an index file (index file) and a data file (data file), and the data file stores the message issued by the producer.
And finding out a first preset number of segment files according to the data directory and the size of the occupied space. In an example, all the segment files are sorted in the descending order of occupied space, and a first preset number of segment files are selected from the initial position of the queue.
And 104, acquiring the theme and the partition of each segment file, and judging whether the partition of each segment file is a main partition according to the distribution information.
And 105, when the partition to which the segment file belongs is a main partition, cleaning the segment file under the partition to which the segment file belongs according to a preset proportion.
For the segment files found in one storage server, for each segment file, the following method is performed:
and obtaining the theme and the partition to which the section of file belongs, and judging whether the partition to which the section of file belongs is a main partition according to the distribution information.
When the partition to which the segment file belongs is a main partition, cleaning all the segment files under the partition according to a preset proportion.
For example, suppose the current state in the kafka cluster is themed: firsttropic and secondtropic, firsttropic has two subordinates, e.g. 01 and 02, secondtropic has two subordinates, e.g. 01 and 02, assuming that the distribution information is: the storage server A is provided with a primary partition of a 01 partition of firsttpic, a secondary partition of a 02 partition of firsttpic, a primary partition of a 02 partition of secondpic, a secondary partition of a 01 partition of secondpic, and the storage server B is provided with a primary partition of a 02 partition of firsttpic, a primary partition of a 01 secondpic-01 partition of firsttpic, and a secondary partition of a 01 partition of secondpic-02.
And performing file cleaning on the storage server A and the storage server B, inquiring a data directory of the storage server for the storage server A, and finding out a first preset number of segment files, such as 10 segment files, from the files stored in the storage server A according to the sequence that the occupied space of the segment files is reduced from large to small.
For each of the 10 segment files found the following method is performed:
for example, the storage path of the segment file can be directly searched through a data directory, and since the 01 partition, such as firsttpic, in Kafka, which stores partitions by adding the topic names and partition numbers, may be named as firsttpic-01 for storage. Therefore, for a segment file such as 00001, the storage path such as file/firstttopic-01 is queried, and the upper-level path thereof is the partition to which the segment file belongs and the subject to which the partition belongs, namely, firstttopic-01.
After the topic and the partition to which the segment file belongs are obtained through query, whether the partition is a main partition or not is judged according to the distribution information, for example, if the distribution information shows that a 01 partition with firstttop on the storage server A is the main partition and the segment file belongs to the 01 partition with firstttop, the partition to which the segment file belongs is proved to be the main partition.
When the judgment result shows that the partition is the main partition, all files under the partition are cleared according to a preset proportion, for example, 20% of all files under the partition are cleared. The cleaning proportion is set to ensure that the historical data can be kept to the maximum extent while the disk is not fully occupied.
In the disk protection method for the kafka cluster, the use condition of the disk space of the storage server is monitored, and when the size of the occupied space of the disk exceeds a preset threshold, the disk space is cleaned, so that the available capacity of the disk is ensured, the disk space cannot be overloaded when burst traffic occurs in the kafka system, and high availability of kafka service is ensured. The method ensures that the disk space is not occupied by burst traffic under the scene that the disk space is limited, and realizes a mechanism of kafka cluster service self-protection, thereby improving the availability of the kafka cluster.
Fig. 2 is a flowchart of a disk protection method for a kafka cluster according to another embodiment of the present application, as shown in fig. 2, in the embodiment shown in fig. 1 of the present application, step 105 further includes:
Finding out all cleanable segment files under the partition to which the segment files belong, wherein the cleanable segment files do not contain snapshot files.
It is described above that a Segment file (Segment) in kafka is the smallest unit of storage, wherein the Segment file (Segment) being written includes four files with the same file name and different suffixes (index, log, timeseindex, snapshot), while a history Segment has two files with the same name (index, log) without snapshot. Therefore, it is not cleanable when a snapshot file (· snapshot) is included in the Segment file (Segment).
After the cleanable files in the partition are selected, files with a preset ratio of 20% are selected as files to be cleaned from all the cleanable files, and the selected files to be cleaned are cleaned.
Further, the step 202 further includes:
and step 203, sequencing the cleanable segment files according to a time sequence.
And after the cleanable files under the partitions are selected, sequencing all the cleanable files according to the time sequence.
And 204, selecting the segment files with a preset proportion in the sequence from far to near as the file units to be cleaned.
Sorting all cleanable files according to time, selecting segment files with a preset proportion according to a sequence from far to near, taking the selected segment files as files to be cleaned, preferentially deleting historical segment files, and reserving new segment files.
Further, the step 202 further includes:
For a segment file to be cleaned, the cleaning method may be:
acquiring the file name of the segment file, wherein Kafka specifies that the segment file naming rule in one partition is as follows: the first segment file starts from 0, and the file name of the second and subsequent segment files is the offset (offset) of the segment file stored in the previous segment file, that is, the file name indicates the logical storage location of a segment file in the partition. After the file name of the file is obtained, the file name is used as the starting offset of the partition.
After the initial offset is obtained, the file name of the next segment file of the segment file is obtained, or the file name of the last message stored in the segment file is obtained, the end offset position of the segment file is obtained, the segment file is deleted from the initial offset to the end offset position in the partition, and the segment file is deleted from the partition.
Fig. 3 is a flowchart of a disk protection method for a kafka cluster according to still another embodiment of the present application, and as shown in fig. 3, in the embodiment shown in fig. 2 of the present application, the method further includes:
step 301, if finding the cleanable segment file in the partition to which the segment file belongs fails, adjusting the segment file parameters to adjust the size of the segment file.
When the number of partitions in the Kafka cluster is large and the data is uniform, there may be a situation where the space occupied by each topic is close and there is no cleanable file, that is, there is a snapshot file (. snapshot file) in all Segment files (segments).
This problem can be solved by adjusting the size of the kafka Segment file (Segment), for example, adjusting the log.segment.bytes parameter reduces the file Segment size a little, which ensures that there are cleanable segments, and by reducing the Segment space size, the number of segments can be increased, the history segments can be increased, and the cleanable data can be increased. For example, the original Segment file 1G can be adjusted to 500M, and the Segment number is 2 times that of the previous Segment file, so that the cleanable file number is increased by 1 time.
Step 302, re-searching for a cleanable segment file under a partition to which the segment file belongs, where the cleanable segment file is a segment file without a snapshot file.
After the segment size of the segment file is adjusted, all cleanable segment files under the partition to which the segment file belongs are found out again, and the cleanable segment files do not contain the snapshot file.
After the cleanable files in the partition are selected, files with a preset ratio of 20% are selected as files to be cleaned from all the cleanable files, and the selected files to be cleaned are cleaned.
The foregoing description has been directed to specific embodiments of this disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.
Fig. 4 is a schematic structural diagram of a disk protection device for a kafka cluster according to an embodiment of the present invention, where the disk protection device is disposed in a terminal device, and as shown in fig. 4, the disk protection device may include: the system comprises a monitoring module 41, an obtaining module 42, a query module 43, a judgment module 44 and a first cleaning module 45;
the monitoring module 41 is configured to poll disk capacities of all storage servers in the kafka cluster;
an obtaining module 42, configured to obtain partition distribution information when the disk capacity of any storage server reaches a protection threshold, where the partition distribution information includes a theme set on each storage server and a main partition to which each theme belongs;
the query module 43 is configured to query the data directory of each storage server, and find out segment files of a first preset number in a descending order of occupied space;
the judging module 44 is configured to, for each segment file, obtain a theme and a partition to which the segment file belongs, and judge whether the partition to which the segment file belongs is a main partition according to the distribution information;
and a first cleaning module 45, configured to clean the segment files under the partition to which the segment files belong according to a preset ratio when the partition to which the segment files belong is a main partition.
The disk protection device for the kafka cluster provided by the embodiment shown in fig. 4 may be used to execute the technical solution of the method embodiment shown in fig. 1 in this specification, and further reference may be made to the relevant description in the method embodiment for implementation principles and technical effects.
Fig. 5 is a schematic structural diagram of a disk protection device for a kafka cluster according to another embodiment of the present disclosure, in this embodiment, a first cleaning module 45 includes a search module 51 and a second cleaning module 52,
the searching module 51 is configured to search a cleanable segment file in a partition to which the segment file belongs, where the cleanable segment file is a segment file for which a snapshot file does not exist;
and the second cleaning module 52 is configured to select a segment file to be cleaned from the cleanable segment files according to a preset ratio, and clean the segment file to be cleaned.
Further, the second cleaning module 52 includes: a sorting module 53 and a selection module 54,
the sorting module 53 is configured to sort the cleanable segment files according to a time sequence;
and the selecting module 54 is used for selecting the segment files with the preset proportion in the sorting according to the sequence from far to near as the file units to be cleaned.
Further, the second cleaning module further includes:
the calculation module 55 is configured to, for each segment file to be cleaned, obtain a file name of the segment file to be cleaned, and calculate a data start offset according to the file name;
a third cleaning module 55, configured to clean the segment file to be cleaned according to the starting offset in the partition to which the segment file belongs.
Further, the apparatus further comprises:
an adjusting module 57, configured to adjust a segment file parameter if finding a cleanable segment file in a partition to which the segment file belongs fails, so as to adjust the size of the segment file;
a second searching module 58, configured to re-search a cleanable segment file in the partition to which the segment file belongs, where the cleanable segment file is a segment file where no snapshot file exists;
and a fourth cleaning module 59, configured to select a segment file to be cleaned from the cleanable segment files according to a preset ratio, and clean the segment file to be cleaned.
The disk protection device for the kafka cluster provided by the embodiment shown in fig. 5 may be used to execute the technical solutions of the method embodiments shown in fig. 1 to fig. 3 of the present application, and the implementation principle and the technical effects of the technical solutions may further refer to the related descriptions in the method embodiments.
Fig. 6 is a schematic diagram of a kafka cluster 600 provided in one embodiment of the present specification, where the kafka cluster 600 includes a console 610, a storage server 620, a storage server 630, a storage server 640, a coordination server 650,
the storage server 620, the storage server 630 and the storage server 640 are identical in structure, and the storage server is used for storing the server of the message issued in the kafka cluster and is used for the consumption device to pull the stored message. The storage server may include Node (disk detection service) and disk. The disk is a storage medium specifically used for storing messages, and the Node is used for detecting the disk.
The console 610 is a management console of the Node, and is used to control the Node (disk inspection service) on each storage server.
The coordination server 650 is a server installed with a distributed application coordination service such as Zookeeper, nacos, eureka, etc., for managing the storage servers. Kafka is a master-slave architecture, where the master node is a selected server, called controller, and in this embodiment, called coordinating server, and the other servers except the master node are slave nodes. Distributed application coordination services such as Zookeeper, nacos, eureka and the like run on the coordination server, the distributed application coordination services are used for setting themes and partitions under the themes in Kafka, and the controller needs to cooperate with the distributed application coordination services to manage the whole Kafka cluster.
Fig. 7 is a flowchart of a disk protection method for the kafka cluster according to an embodiment of the present application, where the method is applied to the console 610 in the kafka cluster shown in fig. 6, and the method includes:
in step 701, disk capacities of all storage servers in the kafka cluster are polled.
Step 702, when the disk capacity of any storage server reaches a protection threshold, obtaining partition distribution information stored on the coordination server, where the partition distribution information includes a topic set on each storage server and a main partition subordinate to each topic.
Fig. 8 is a flowchart of a disk protection method for a kafka cluster according to an embodiment of the present application, where the method is applied to a storage server of the kafka cluster, and the kafka cluster further includes a console, where the method includes:
Optionally, in step 804, cleaning the segment file under the partition to which the segment file belongs according to a preset proportion includes:
searching for a cleanable segment file under a partition to which the segment file belongs, wherein the cleanable segment file is a segment file without a snapshot file;
and selecting the segment files to be cleaned according to a preset proportion from the cleanable segment files, and cleaning the segment files to be cleaned.
Further, the selecting the segment files to be cleaned according to a preset ratio in the cleanable segment files includes:
sequencing the cleanable segment files according to a time sequence;
and selecting the segment files with a preset proportion in the sequence from far to near as the file units to be cleaned.
Further, the cleaning the segment file to be cleaned includes:
for each segment file to be cleaned, acquiring the file name of the segment file to be cleaned, and calculating the initial offset of data according to the file name;
and cleaning the segment files to be cleaned according to the starting offset in the partitions to which the segment files belong.
Further, the method further comprises:
if finding the cleanable segment file in the partition to which the segment file belongs fails, adjusting the segment file parameters to adjust the size of the segment file;
searching the cleanable segment file under the partition to which the segment file belongs again, wherein the cleanable segment file is a segment file without a snapshot file;
and selecting the segment files to be cleaned according to a preset proportion from the cleanable segment files, and cleaning the segment files to be cleaned.
The embodiment of the present application provides a schematic structural diagram of a terminal device, where the terminal device may include at least one processor; and at least one memory communicatively coupled to the processor, wherein: the memory stores program instructions executable by the processor, and the processor calls the program instructions to execute the disk protection method for the kafka cluster provided by the embodiments shown in fig. 1 to fig. 3 in the present specification.
The terminal device may be an intelligent electronic device such as a smart phone, a tablet computer, or a notebook computer, and the form of the terminal device is not limited in this embodiment.
The embodiment of the present application provides a non-transitory computer-readable storage medium, which stores computer instructions, where the computer instructions cause the computer to execute the disk protection method for the kafka cluster provided in the embodiment shown in fig. 1 to fig. 3 in this specification.
The non-transitory computer readable storage medium described above may take any combination of one or more computer readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a Read Only Memory (ROM), an Erasable Programmable Read Only Memory (EPROM) or flash memory, an optical fiber, a portable compact disc read only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, Radio Frequency (RF), etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for aspects of the present description may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
The foregoing description has been directed to specific embodiments of this disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.
In the description of embodiments of the invention, reference to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the specification. In this specification, the schematic representations of the terms used above are not necessarily intended to refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, various embodiments or examples and features of different embodiments or examples described in this specification can be combined and combined by one skilled in the art without contradiction.
Furthermore, the terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present specification, "a plurality" means at least two, e.g., two, three, etc., unless explicitly defined otherwise.
Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing steps of a custom logic function or process, and alternate implementations are included within the scope of the preferred embodiment of the present description in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the embodiments of the present description.
The word "if" as used herein may be interpreted as "at … …" or "when … …" or "in response to a determination" or "in response to a detection", depending on the context. Similarly, the phrases "if determined" or "if detected (a stated condition or event)" may be interpreted as "when determined" or "in response to a determination" or "when detected (a stated condition or event)" or "in response to a detection (a stated condition or event)", depending on the context.
It should be noted that the terminal referred to in the embodiments of the present application may include, but is not limited to, a Personal Computer (PC), a Personal Digital Assistant (PDA), a wireless handheld device, a tablet computer (tablet computer), a mobile phone, an MP3 player, an MP4 player, and the like.
In the several embodiments provided in this specification, it should be understood that the disclosed system, apparatus, and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and there may be other divisions in actual implementation, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
In addition, functional units in the embodiments of the present description may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, or in a form of hardware plus a software functional unit.
The integrated unit implemented in the form of a software functional unit may be stored in a computer readable storage medium. The software functional unit is stored in a storage medium and includes several instructions to enable a computer device (which may be a personal computer, a server, or a network device) or a processor (processor) to execute some steps of the methods described in the embodiments of the present disclosure. And the aforementioned storage medium includes: various media capable of storing program codes, such as a U disk, a removable hard disk, a Read Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
The above description is only a preferred embodiment of the present disclosure, and should not be taken as limiting the present disclosure, and any modifications, equivalents, improvements, etc. made within the spirit and principle of the present disclosure should be included in the scope of the present disclosure.
Claims (18)
1. A disk protection method for a kafka cluster is characterized by comprising the following steps:
polling the disk capacity of all storage servers in the kafka cluster;
when the disk capacity of any storage server reaches a protection threshold value, obtaining partition distribution information, wherein the partition distribution information comprises themes set on each storage server and main partitions subordinate to each theme;
for each storage server, inquiring a data directory of the storage server, and finding out a first preset number of segment files according to the sequence of the occupied space from large to small;
for each segment file, obtaining the theme and the partition of the segment file, and judging whether the partition of the segment file is a main partition according to the distribution information;
and when the partition to which the segment file belongs is a main partition, cleaning the segment file under the partition to which the segment file belongs according to a preset proportion.
2. The method according to claim 1, wherein the cleaning up the segment files under the partition to which the segment files belong according to a preset ratio comprises:
searching for a cleanable segment file under a partition to which the segment file belongs, wherein the cleanable segment file is a segment file without a snapshot file;
and selecting the segment files to be cleaned according to a preset proportion from the cleanable segment files, and cleaning the segment files to be cleaned.
3. The method of claim 2, wherein selecting the segment files to be cleaned according to a preset ratio from the cleanable segment files comprises:
sequencing the cleanable segment files according to a time sequence;
and selecting the segment files with a preset proportion in the sequence from far to near as the file units to be cleaned.
4. The method of claim 2, wherein the cleaning the segment file to be cleaned comprises:
for each segment file to be cleaned, acquiring the file name of the segment file to be cleaned, and calculating the initial offset of data according to the file name;
and cleaning the segment files to be cleaned according to the starting offset in the partitions to which the segment files belong.
5. The method of claim 2, further comprising:
if finding the cleanable segment file in the partition to which the segment file belongs fails, adjusting the segment file parameters to adjust the size of the segment file;
searching the cleanable segment file under the partition to which the segment file belongs again, wherein the cleanable segment file is a segment file without a snapshot file;
and selecting the segment files to be cleaned according to a preset proportion from the cleanable segment files, and cleaning the segment files to be cleaned.
6. A disk protection device for a kafka cluster, which is arranged in a terminal device, is characterized in that the disk protection device comprises:
the monitoring module is used for polling the disk capacities of all the storage servers in the kafka cluster;
the system comprises an acquisition module, a storage module and a management module, wherein the acquisition module is used for acquiring partition distribution information when the disk capacity of any storage server reaches a protection threshold, and the partition distribution information comprises themes set on each storage server and main partitions subordinate to each theme;
the query module is used for querying a data directory of each storage server and finding out a first preset number of segment files according to the sequence of occupied space from large to small;
the judging module is used for acquiring the theme and the partition of each segment file and judging whether the partition of each segment file is a main partition according to the distribution information;
and the first cleaning module is used for cleaning the segment files under the partition to which the segment files belong according to a preset proportion when the partition to which the segment files belong is a main partition.
7. The apparatus of claim 6, wherein the first cleaning module comprises:
the searching module is used for searching the cleanable segment file under the partition to which the segment file belongs, wherein the cleanable segment file is a segment file without a snapshot file;
and the second cleaning module is used for selecting the segment files to be cleaned in the cleanable segment files according to a preset proportion and cleaning the segment files to be cleaned.
8. The apparatus of claim 7, wherein the second cleaning module comprises:
the sorting module is used for sorting the cleanable segment files according to a time sequence;
and the selection module is used for selecting the segment files with the preset proportion in the sequence from far to near as the file units to be cleaned.
9. The apparatus of claim 7, wherein the second cleaning module comprises:
the calculation module is used for acquiring the file name of each segment file to be cleaned and calculating the initial offset of data according to the file name;
and the third cleaning module is used for cleaning the segment file to be cleaned in the partition to which the segment file belongs according to the starting offset.
10. The apparatus of claim 7, further comprising:
the adjusting module is used for adjusting segment file parameters to adjust the size of the segment file if finding the cleanable segment file in the partition to which the segment file belongs fails;
the second searching module is used for searching the cleanable segment file under the partition to which the segment file belongs again, wherein the cleanable segment file is a segment file without a snapshot file;
and the fourth cleaning module is used for selecting the segment files to be cleaned in the cleanable segment files according to a preset proportion and cleaning the segment files to be cleaned.
11. A disk protection method for a kafka cluster, wherein the method is applied to a console of the kafka cluster, the kafka cluster further includes a plurality of storage servers and a coordinating server, and the method includes:
polling the disk capacity of all storage servers in the kafka cluster;
when the disk capacity of any storage server reaches a protection threshold value, obtaining partition distribution information stored on the coordination server, wherein the partition distribution information comprises themes set on each storage server and main partitions subordinate to each theme;
and sending a cleaning instruction to each storage server to enable each storage server to clean the file, wherein the cleaning instruction comprises the partition distribution information.
12. A disk protection method for a kafka cluster, wherein the method is applied to a storage server of the kafka cluster, and the kafka cluster further comprises a console, and the method comprises:
receiving a cleaning instruction sent by the console, wherein the cleaning instruction comprises the partition distribution information, and the partition distribution information comprises themes set on each storage server and main partitions subordinate to each theme;
searching a first preset number of segment files according to the sequence of occupied space from large to small in the inquired data directory;
for each segment file, obtaining the theme and the partition of the segment file, and judging whether the partition of the segment file is a main partition according to the distribution information;
and when the partition to which the segment file belongs is a main partition, cleaning the segment file under the partition to which the segment file belongs according to a preset proportion.
13. The method of claim 12, wherein the cleaning up the segment files under the partition to which the segment files belong according to a preset ratio comprises:
searching for a cleanable segment file under a partition to which the segment file belongs, wherein the cleanable segment file is a segment file without a snapshot file;
and selecting the segment files to be cleaned according to a preset proportion from the cleanable segment files, and cleaning the segment files to be cleaned.
14. The method of claim 13, wherein selecting the segment files to be cleaned according to a preset ratio from the cleanable segment files comprises:
sequencing the cleanable segment files according to a time sequence;
and selecting the segment files with a preset proportion in the sequence from far to near as the file units to be cleaned.
15. The method of claim 13, wherein the cleaning the segment file to be cleaned comprises:
for each segment file to be cleaned, acquiring the file name of the segment file to be cleaned, and calculating the initial offset of data according to the file name;
and cleaning the segment files to be cleaned according to the starting offset in the partitions to which the segment files belong.
16. The method of claim 13, further comprising:
if finding the cleanable segment file in the partition to which the segment file belongs fails, adjusting the segment file parameters to adjust the size of the segment file;
searching the cleanable segment file under the partition to which the segment file belongs again, wherein the cleanable segment file is a segment file without a snapshot file;
and selecting the segment files to be cleaned according to a preset proportion from the cleanable segment files, and cleaning the segment files to be cleaned.
17. A terminal device, comprising:
at least one processor; and
at least one memory communicatively coupled to the processor, wherein:
the memory stores program instructions executable by the processor, the processor invoking the program instructions to perform the method of any of claims 1 to 5.
18. A non-transitory computer readable storage medium storing computer instructions that cause the computer to perform the method of any of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111611075.XA CN114385081B (en) | 2021-12-27 | 2021-12-27 | Disk protection method for kafka cluster and related equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111611075.XA CN114385081B (en) | 2021-12-27 | 2021-12-27 | Disk protection method for kafka cluster and related equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114385081A true CN114385081A (en) | 2022-04-22 |
| CN114385081B CN114385081B (en) | 2023-03-24 |
Family
ID=81198719
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111611075.XA Active CN114385081B (en) | 2021-12-27 | 2021-12-27 | Disk protection method for kafka cluster and related equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114385081B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110648178A (en) * | 2019-09-24 | 2020-01-03 | 四川长虹电器股份有限公司 | Method for increasing kafka consumption capacity |
| CN111125260A (en) * | 2020-01-21 | 2020-05-08 | 重庆文理学院 | Data synchronization method and system based on SQL Server |
| US20200387309A1 (en) * | 2019-06-07 | 2020-12-10 | Research & Business Foundation Sungkyunkwan University | Segment cleaning method using non-volatile random-access memory and memory management apparatus thereof |
| CN112269661A (en) * | 2020-10-30 | 2021-01-26 | 上海哔哩哔哩科技有限公司 | Partition migration method and device based on Kafka cluster |
| CN112272217A (en) * | 2020-10-16 | 2021-01-26 | 苏州浪潮智能科技有限公司 | Kafka cluster load balancing method, system, equipment and medium |
| CN113055430A (en) * | 2019-12-27 | 2021-06-29 | 华为技术有限公司 | Data synchronization method and related equipment |
| CN113297309A (en) * | 2021-05-31 | 2021-08-24 | 平安证券股份有限公司 | Stream data writing method, device, equipment and storage medium |
-
2021
- 2021-12-27 CN CN202111611075.XA patent/CN114385081B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20200387309A1 (en) * | 2019-06-07 | 2020-12-10 | Research & Business Foundation Sungkyunkwan University | Segment cleaning method using non-volatile random-access memory and memory management apparatus thereof |
| CN110648178A (en) * | 2019-09-24 | 2020-01-03 | 四川长虹电器股份有限公司 | Method for increasing kafka consumption capacity |
| CN113055430A (en) * | 2019-12-27 | 2021-06-29 | 华为技术有限公司 | Data synchronization method and related equipment |
| CN111125260A (en) * | 2020-01-21 | 2020-05-08 | 重庆文理学院 | Data synchronization method and system based on SQL Server |
| CN112272217A (en) * | 2020-10-16 | 2021-01-26 | 苏州浪潮智能科技有限公司 | Kafka cluster load balancing method, system, equipment and medium |
| CN112269661A (en) * | 2020-10-30 | 2021-01-26 | 上海哔哩哔哩科技有限公司 | Partition migration method and device based on Kafka cluster |
| CN113297309A (en) * | 2021-05-31 | 2021-08-24 | 平安证券股份有限公司 | Stream data writing method, device, equipment and storage medium |
Non-Patent Citations (3)
| Title |
|---|
| DANIEL R.TORRES等: "《An open source framework based on Kafka-ML for Distributed DNN inference over the Cloud-to-Things continuum》", 《JOURNAL OF SYSTEMS ARCHITECTURE》 * |
| 汪涛: "Kafka中Broker节点磁盘问题的故障处理方法", 《现代信息科技》 * |
| 高子妍等: "面向云服务的分布式消息系统负载均衡策略", 《计算机科学》 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114385081B (en) | 2023-03-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP5585062B2 (en) | Information processing apparatus, information processing method, data management server, and data synchronization system | |
| CN111200657B (en) | Method for managing resource state information and resource downloading system | |
| CN104537076A (en) | File reading and writing method and device | |
| CN112445626B (en) | Data processing method and device based on message middleware | |
| US11102289B2 (en) | Method for managing resource state information and system for downloading resource | |
| CN107181821A (en) | A kind of information push method and device based on SSE specifications | |
| CN111562889B (en) | Data processing method, device, system and storage medium | |
| CN110413845B (en) | Resource storage method and device based on Internet of things operating system | |
| CN108427619B (en) | Log management method and device, computing equipment and storage medium | |
| CN110543512B (en) | Information synchronization method, device and system | |
| CN111797091A (en) | Method and device for querying data in database, electronic equipment and storage medium | |
| CN112231296B (en) | Distributed log processing method, device, system, equipment and medium | |
| CN112306369A (en) | Data processing method, device, server and storage medium | |
| CN110795041A (en) | Quota method, quota device, server and storage medium of distributed file system | |
| CN101483668A (en) | Network storage and access method, device and system for hot spot data | |
| CN114385081B (en) | Disk protection method for kafka cluster and related equipment | |
| CN111209159A (en) | Information processing method, device, equipment and storage medium | |
| CN112579633A (en) | Data retrieval method, device, equipment and storage medium | |
| CN113779412B (en) | Message touch method, node and system based on blockchain network | |
| CN111881086B (en) | Big data storage method, query method, electronic device and storage medium | |
| CN112148513B (en) | Method, device and storage medium for dumping single-point service data | |
| CN112905541B (en) | Mirror image warehouse garbage cleaning method and device | |
| CN115878121A (en) | Terminal code increment compiling method, system, device, server and storage medium | |
| CN104869171A (en) | Information processing method and smart home equipment | |
| US20220382747A1 (en) | Source-adapted data retrieval for multi-tenant system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |