CN115509769A - Kafka deployment method and device in private cloud and electronic equipment - Google Patents
Kafka deployment method and device in private cloud and electronic equipment Download PDFInfo
- Publication number
- CN115509769A CN115509769A CN202211156387.0A CN202211156387A CN115509769A CN 115509769 A CN115509769 A CN 115509769A CN 202211156387 A CN202211156387 A CN 202211156387A CN 115509769 A CN115509769 A CN 115509769A
- Authority
- CN
- China
- Prior art keywords
- data
- kafka
- memory
- message
- production
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 40
- 238000004519 manufacturing process Methods 0.000 claims abstract description 43
- 238000012544 monitoring process Methods 0.000 claims description 5
- 230000006870 function Effects 0.000 description 3
- 238000009825 accumulation Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 239000007795 chemical reaction product Substances 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/448—Execution paradigms, e.g. implementations of programming paradigms
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention discloses a Kafka deployment method, a Kafka deployment device and electronic equipment in a private cloud, wherein the method is used for reproducing first data failed in production of a production message through a preset callback interface, and the preset callback interface is connected with the Kafka; and then when the number of times of re-production exceeds a preset number threshold and the production of the message fails, storing the corresponding first data into a second memory, re-producing the message according to the first data in the second memory when a preset time period arrives, and simultaneously storing the message with successful production into Kafka, so that the data is not lost on the premise of reducing the cost and resources of a privatized Internet platform, and the use experience of a user is effectively improved.
Description
Technical Field
The invention belongs to the technical field of computers, and particularly relates to a Kafka deployment method and device in private cloud and electronic equipment.
Background
In an industrial enterprise, in order to facilitate management of devices in the enterprise, the devices are generally connected to an industrial internet platform, and two schemes are generally used for connecting the devices to the industrial internet platform, one scheme is to directly connect the devices to a mature public cloud platform, the other scheme is to build a set of privatized industrial internet platform, namely a private cloud, for the devices, the public cloud has huge data volume facing millions and millions of device volumes, so that each service, middleware and database can adopt multi-node and clustered deployment to realize high availability, and stable operation of the service and integrity of data are guaranteed, wherein most importantly, after the devices are connected to the platform, the platform can produce uplink messages of the devices into Kafka middleware, the Kafka is a high-throughput distributed publishing and subscribing message system, and can process all streaming data actions of the devices in the platform, abnormal situations such as downtime of the Kafka may be caused in the message production process, and data loss is caused, so the public cloud platform builds a group to guarantee high availability of the Kafka Kafka group.
In the private industrial internet platform, the access amount and the data amount of the equipment are low, but an enterprise needs to ensure that data are not lost, and needs to additionally consume more hardware resources to build a Kafka cluster environment, so that operation and maintenance work is increased, the cost is high, and the single-machine version Kafka deployment cannot ensure that data are not lost, so that the result of incomplete data is caused.
Therefore, how to ensure that data is not lost on the premise of reducing the cost and resources of the privatized internet platform is a technical problem to be solved by technical personnel in the field.
Disclosure of Invention
The invention aims to solve the technical problem that data is lost under the condition of reducing the cost and resources of a privatized Internet platform in the prior art.
In order to achieve the above technical object, in one aspect, the present invention provides a Kafka deployment method in a private cloud, including:
reproducing the first data failed in producing the message through a preset callback interface, wherein the preset callback interface is connected with Kafka;
when the number of times of reproduction exceeds a preset number threshold and the production of the message fails, storing the corresponding first data into a second memory, reproducing the message according to the first data in the second memory when a preset time period is reached, and storing the message of successful production into Kafka.
Preferably, the first data that the production message fails is reproduced through a preset callback interface, specifically:
saving the first data to a first memory;
retrieve the first data from the first memory and regenerate the message.
Preferably, the storing the first data in a first memory specifically includes:
generating a queue in the first memory according to a preset time interval;
and sequentially storing the first data in the same time interval in corresponding queues according to a time sequence, wherein each queue is provided with a first key and a second key, the first key is used for monitoring whether the first memory is expired or not, if so, messages are sequentially reproduced according to the first data in the corresponding queue, and the second key is used for storing metadata of the corresponding queue.
Preferably, the first memory is in particular a Redis.
Preferably, if the message is successfully produced according to the first data within a preset threshold of times, the first data is deleted from the first memory after a preset time.
In another aspect, the present invention further provides a Kafka deployment apparatus in a private cloud, where the apparatus includes:
the callback module is used for reproducing the first data failed in message production through a preset callback interface, and the preset callback interface is connected with the Kafka;
and the second production module is used for storing the corresponding first data into the second memory when the number of times of re-production exceeds a preset number threshold and the production of the message fails, re-producing the message according to the first data in the second memory when a preset time period arrives, and storing the message of successful production into Kafka.
Preferably, the callback module is specifically configured to:
saving the first data to a first memory;
retrieve the first data from the first memory and regenerate the message.
Preferably, the callback module is specifically configured to:
generating a queue in the first memory according to a preset time interval,
and sequentially storing the first data in the same time interval in corresponding queues according to a time sequence, wherein each queue is provided with a first key and a second key, the first key is used for monitoring whether the first memory is expired, if so, messages are sequentially reproduced according to the first data in the corresponding queue, and the second key is used for storing metadata of the corresponding queue.
Preferably, the first memory is in particular a Redis.
In another aspect, the present invention provides an electronic device, including:
a processor;
a third memory for storing the processor-executable instructions;
the processor is configured for performing the method as described above.
Compared with the prior art, the method, the device and the electronic equipment for deploying Kafka in private cloud provided by the invention have the advantages that the first data failed in message production is produced again through the preset callback interface, and the preset callback interface is connected with Kafka; and then when the number of times of reproduction exceeds a preset number threshold and the message production fails, storing the corresponding first data into a second memory, reproducing the message according to the first data in the second memory when a preset time period is reached, and storing the message of successful production into Kafka, so that the data is not lost on the premise of reducing the cost and resources of a privatized Internet platform, and the use experience of a user is effectively improved.
Drawings
In order to more clearly illustrate the embodiments of the present specification or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the description below are only some embodiments described in the present specification, and it is also possible for those skilled in the art to obtain other drawings based on the drawings without creative efforts.
Fig. 1 is a schematic flowchart illustrating a Kafka deployment method in a private cloud according to an embodiment of the present specification;
fig. 2 is a schematic structural diagram of a Kafka deployment apparatus in a private cloud according to an embodiment of the present specification.
Detailed Description
In order to make those skilled in the art better understand the technical solutions in the present specification, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
Although the present description provides the method operation steps or device structures shown in the following embodiments or drawings, more or less operation steps or module units after partial merger may be included in the method or device based on conventional or non-creative labor, and in the steps or structures in which the necessary cause-and-effect relationship does not exist logically, the execution sequence of the steps or the module structure of the device is not limited to the execution sequence or module structure shown in the embodiments or drawings of the present description. The described method or module structure can be executed in sequence or in parallel according to the embodiments or the method or module structure shown in the drawings (for example, in the environment of parallel processors or multi-thread processing, or even in the environment of distributed processing and server cluster) when the method or module structure is applied to a device, a server or an end product in practice.
The Kafka deployment method in the private cloud provided in the embodiment of the present description may be applied to terminal devices such as a client and a server, and as shown in fig. 1, the method specifically includes the following steps:
and S101, reproducing the first data failed in the production of the message through a preset callback interface, wherein the preset callback interface is connected with Kafka.
Specifically, the preset Callback interface may be a custom Kafka producercalback class, which implements the kalback class of the Kafka itself, and is used for processing the Kafka production message failure, and the Kafka producercalback is used as a Callback method, and the entry parameters are retry times and message metadata. If abnormity occurs in the callback method, the production message fails, which represents that Kafka has a fault, and the message is produced again through the preset callback interface.
In order to avoid data loss, in the embodiment of the present application, the first data that fails in producing the message is produced again through the preset callback interface, specifically:
saving the first data to a first memory;
retrieve the first data from the first memory and reproduce the message.
In order to generate a message more timely and accurately, in this embodiment of the application, the storing the first data in the first memory specifically includes:
generating a queue in the first memory according to a preset time interval;
and sequentially storing the first data in the same time interval in corresponding queues according to a time sequence, wherein each queue is provided with a first key and a second key, the first key is used for monitoring whether the first memory is expired or not, if so, messages are sequentially reproduced according to the first data in the corresponding queue, and the second key is used for storing metadata of the corresponding queue.
Specifically, the first memory may be Redis, which is a high-performance Key-value storage system and is more efficient, and during a downtime or a failure of Kafka, there may be first data entering the callback method every second, each first data performs setting a Key, that is, a Key, which may cause a bloat, so that queues may be generated according to a preset time interval, and the first data located in the same time interval are sequentially stored in the corresponding queues according to a time sequence, the format of the first Key may be shadow: 202141700.
In order to avoid data accumulation in the first memory, in the embodiment of the present application, if a message is successfully generated according to the first data within a preset time threshold, the first data is deleted from the first memory after a preset time.
And S102, when the number of times of re-production exceeds a preset number threshold and the production of the message fails, storing the corresponding first data into a second memory, re-producing the message according to the first data in the second memory when a preset time period is reached, and simultaneously storing the message of successful production into Kafka.
Specifically, when the number of times of re-production in the callback process exceeds a preset number threshold, it indicates that the message has been failed to be produced, and Kafka is still not recovered, in order to prevent data accumulation in the first memory, that is, the Redis, the corresponding first data is stored in the second memory, and when a preset time period arrives, the message is re-produced according to the first data in the second memory, that is, a timing task is started in the second memory to read the first data in the second memory at a fixed time, and the message is produced again until the production is successful and stored in Kafka.
After the processing, the data can be ensured not to be lost on the premise of reducing the cost and resources of the privatized Internet platform, and the integrity of the data is ensured.
Based on the foregoing method for deploying Kafka in a private cloud, one or more embodiments of the present disclosure further provide a terminal for deploying Kafka in a private cloud, where the terminal may include a device, software, a module, a plug-in, a server, a client, and the like that use the method described in the embodiments of the present disclosure and incorporate a necessary device for implementing hardware, and based on the same innovative concept, a system in one or more embodiments provided in the embodiments of the present disclosure is as described in the following embodiments. Although the system described in the embodiments below is preferably implemented in software, hardware, a combination of hardware and software is also possible and contemplated.
Specifically, fig. 2 is a schematic block structure diagram of an embodiment of a Kafka deployment apparatus in a private cloud provided in this specification, and as shown in fig. 2, the data security access apparatus provided in this specification includes:
the callback module 201 is used for reproducing the first data failed in message production through a preset callback interface, and the preset callback interface is connected with the Kafka;
and the second production module 202 is configured to, when the number of times of re-production exceeds a preset number threshold and the production of the message fails, store the corresponding first data in the second memory, and re-produce the message according to the first data in the second memory when a preset time period arrives, and at the same time store the message that the production succeeded in Kafka.
It should be noted that the description of the system according to the corresponding method embodiment may also include other embodiments, and specific implementation manners may refer to the description of the corresponding method embodiment, which is not described in detail herein.
An embodiment of the present application further provides an electronic device, including:
a processor;
a third memory for storing the processor-executable instructions;
the processor is configured to perform the method as provided in the above embodiments.
According to the electronic device provided by the embodiment of the application, the executable instruction of the processor is stored through the third storage, when the processor executes the executable instruction, the first data failed in message production can be reproduced through the preset callback interface, and the preset callback interface is connected with Kafka; and then when the number of times of re-production exceeds a preset number threshold and the production of the message fails, storing the corresponding first data into a second memory, re-producing the message according to the first data in the second memory when a preset time period arrives, and simultaneously storing the message of successful production into Kafka, so that the data is not lost on the premise of reducing the cost and resources of the privatized Internet platform.
It is understood that the same or similar parts in the above embodiments may be mutually referred to, and the same or similar contents in other embodiments may be referred to for the contents which are not described in detail in some embodiments.
Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps of the process, and the scope of the preferred embodiments of the present application includes other implementations in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the present application.
It should be understood that portions of the present application may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one or combination of the following technologies, which are well known in the art, may be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application specific integrated circuit having an appropriate combinational logic gate circuit, a Programmable Gate Array (PGA), a Field Programmable Gate Array (FPGA), or the like.
The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for system embodiments, since they are substantially similar to method embodiments, the description is relatively simple, and the relevant points can be referred to only part of the description of the method embodiments. In the description of the specification, reference to the description of "one embodiment," "some embodiments," "an example," "a specific example," or "some examples" or the like means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the specification. In this specification, the schematic representations of the terms used above are not necessarily intended to refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Moreover, various embodiments or examples and features of different embodiments or examples described in this specification can be combined and combined by one skilled in the art without contradiction
It will be appreciated by those of ordinary skill in the art that the embodiments described herein are intended to assist the reader in understanding the principles of the invention and are to be construed as being without limitation to such specifically recited embodiments and examples. Those skilled in the art can make various other specific changes and combinations based on the teachings of the present invention without departing from the spirit of the invention, and these changes and combinations are within the scope of the invention.
Claims (10)
1. A Kafka deployment method in a private cloud, the method comprising:
reproducing the first data failed in the message production through a preset callback interface, wherein the preset callback interface is connected with the Kafka;
when the number of times of reproduction exceeds a preset number threshold and the production of the message fails, storing the corresponding first data into a second memory, reproducing the message according to the first data in the second memory when a preset time period is reached, and storing the message of successful production into Kafka.
2. The Kafka deployment method in the private cloud of claim 1, wherein the first data that the production message failed is regenerated through a preset callback interface, specifically:
saving the first data to a first memory;
retrieve the first data from the first memory and regenerate the message.
3. The private cloud Kafka deployment method of claim 2, wherein the saving the first data to a first memory specifically comprises:
generating a queue in the first memory according to a preset time interval;
the method comprises the steps that first data located in the same time interval are sequentially stored in corresponding queues according to a time sequence, each queue is provided with a first key and a second key, the first keys are used for monitoring whether the first data are overdue or not through the first storage, if yes, messages are sequentially reproduced according to the first data in the corresponding queues, and the second keys are used for storing metadata of the corresponding queues.
4. The private cloud Kafka deployment method of claim 2, wherein the first storage is specifically Redis.
5. The private cloud Kafka deployment method of claim 2, wherein if the message is successfully generated according to the first data within a preset threshold number of times, deleting the first data from the first memory after a preset time.
6. A Kafka deployment apparatus in a private cloud, the apparatus comprising:
the callback module is used for reproducing the first data failed in message production through a preset callback interface, and the preset callback interface is connected with the Kafka;
and the second production module is used for storing the corresponding first data into the second memory when the number of times of re-production exceeds a preset number threshold and the production of the message fails, re-producing the message according to the first data in the second memory when a preset time period arrives, and storing the message of successful production into Kafka.
7. The Kafka deployment device in a private cloud of claim 6, wherein the callback module is specifically configured to:
saving the first data to a first memory;
retrieve the first data from the first memory and regenerate the message.
8. The Kafka deployment device in a private cloud of claim 7, wherein the callback module is specifically configured to:
generating a queue in the first memory according to a preset time interval,
and sequentially storing the first data in the same time interval in corresponding queues according to a time sequence, wherein each queue is provided with a first key and a second key, the first key is used for monitoring whether the first memory is expired, if so, messages are sequentially reproduced according to the first data in the corresponding queue, and the second key is used for storing metadata of the corresponding queue.
9. The Kafka deployment device in a private cloud of claim 7, wherein the first storage is specifically Redis.
10. An electronic device, comprising:
a processor;
a third memory for storing the processor-executable instructions;
the processor is configured to perform the method of any one of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211156387.0A CN115509769A (en) | 2022-09-22 | 2022-09-22 | Kafka deployment method and device in private cloud and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211156387.0A CN115509769A (en) | 2022-09-22 | 2022-09-22 | Kafka deployment method and device in private cloud and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115509769A true CN115509769A (en) | 2022-12-23 |
Family
ID=84505702
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211156387.0A Pending CN115509769A (en) | 2022-09-22 | 2022-09-22 | Kafka deployment method and device in private cloud and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115509769A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116032671A (en) * | 2023-03-30 | 2023-04-28 | 杭州华卓信息科技有限公司 | Communication method and network system based on hybrid cloud |
-
2022
- 2022-09-22 CN CN202211156387.0A patent/CN115509769A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116032671A (en) * | 2023-03-30 | 2023-04-28 | 杭州华卓信息科技有限公司 | Communication method and network system based on hybrid cloud |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106933843B (en) | Database heartbeat detection method and device | |
| CN112527879B (en) | Kafka-based real-time data extraction method and related equipment | |
| CN108089915B (en) | Method and system for business control processing based on message queue | |
| CN110650164B (en) | File uploading method and device, terminal and computer storage medium | |
| CN107346270B (en) | Method and system for real-time computation based radix estimation | |
| CN115509769A (en) | Kafka deployment method and device in private cloud and electronic equipment | |
| CN111010318A (en) | Method and system for discovering loss of connection of terminal equipment of Internet of things and equipment shadow server | |
| CN112395124A (en) | Robot abnormity control method and device in cluster environment | |
| US8719622B2 (en) | Recording and preventing crash in an appliance | |
| CN110620798A (en) | Control method, system, equipment and storage medium for FTP connection | |
| CN116701352A (en) | Database data migration method and system | |
| JP2015230674A (en) | Management object device, management device, and network management system | |
| US20200285520A1 (en) | Information processor, information processing system, and method of processing information | |
| CN105550065A (en) | Database server communication management method and device | |
| CN113098978B (en) | Data transmission method, device and medium | |
| CN116166470A (en) | Redis cluster clone replication method and device, medium and equipment | |
| CN115174596A (en) | Equipment remote copying method, device and medium | |
| CN110557453B (en) | Data distribution method, device, equipment and storage medium | |
| CN112488462A (en) | Unified pushing method, device and medium for workflow data | |
| CN111970329A (en) | Method, system, equipment and medium for deploying cluster service | |
| CN112423162A (en) | Configuration synchronization method and system under distributed architecture | |
| CN112463514A (en) | Monitoring method and device for distributed cache cluster | |
| CN115599521A (en) | Event backlog processing method, device, equipment and medium | |
| CN111143280B (en) | Data scheduling method, system, device and storage medium | |
| CN114780634A (en) | Real-time database synchronization method and device and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |