CN112702415B - Method for converting Kafka long connection consumption into service - Google Patents
Method for converting Kafka long connection consumption into service Download PDFInfo
- Publication number
- CN112702415B CN112702415B CN202011517727.9A CN202011517727A CN112702415B CN 112702415 B CN112702415 B CN 112702415B CN 202011517727 A CN202011517727 A CN 202011517727A CN 112702415 B CN112702415 B CN 112702415B
- Authority
- CN
- China
- Prior art keywords
- kafka
- service
- consumption
- consumer
- long connection
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/14—Session management
- H04L67/141—Setup of application sessions
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/133—Protocols for remote procedure calls [RPC]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/14—Session management
- H04L67/142—Managing session states for stateless protocols; Signalling session states; State transitions; Keeping-state mechanisms
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/51—Discovery or management thereof, e.g. service location protocol [SLP] or web services
Abstract
The invention relates to a method for converting Kafka long connection consumption into service, which comprises the following steps: the service issuing party issues the Kafka consumption service; the Kafka consumption Service is opened by a Web Service interface; the service requester calls the service request and transmits a topic name to be consumed and an ID of a consumption group; kafka's service engine creates Kafka consumers; the Kafka consumer acquires a long connection data packet from the Kafka cluster in a long connection mode; the service engine of Kafka obtains the short connection data packet from the Kafka consumer; and providing the short connection data packet to a Web Service interface. According to the invention, when the Kafka consumption process adopts a long connection mode, long connection consumption is converted into short connection Service, so that the Kafka consumption mode is changed from a long connection mode to a short connection mode, the connection modes of production and consumption are unified, and the production and consumption behaviors in a Kafka cluster are opened into Web Service.
Description
Technical Field
The invention relates to the technical field of Kafka consumption, in particular to a method for converting Kafka long connection consumption into service. The Service refers to a Web Service.
Background
Kafka (open source streaming platform developed by Apache software foundation, written by Scala and Java) is a high throughput distributed publish-subscribe messaging system that can handle all the action stream data of a consumer in a web site. As shown in fig. 1, the Kafka message system is composed of three parts, namely, a Producer (Producer, Producer for short), a Broker Server (Broker Server) and a Consumer (Consumer for short), wherein the Producer and the Consumer are written by a developer and connected to the Broker Server through an API for data operation.
Under the conventional condition, when a software project needs to use Kafka products, development problems are often faced when the software project is in interface with Kafka clusters, the jar package of Kafka needs to be integrated in the software project, various commonly used Kafka parameters are configured, and Kafka production and consumption need to develop a client, so that the cost is increased for the software project, and certain development time pressure is brought. Namely: when a software project needs to use the Kafka message system, the Kafka client needs to be newly developed for the software project, and each time the Kafka client is developed, a large amount of test cost and maintenance cost are required.
In general, in order to improve the work efficiency and reduce the enterprise cost, a software enterprise may want to directly open the functions of the Kafka product (particularly, the Kafka consumption process) to each software project through a form of Service (referred to as Web Service), so as to avoid the cost pressure and development time pressure caused by the development of the Kafka client.
However, the following application scenarios have a problem that the connection method conflicts, which may result in that the Kafka consumption process cannot be opened to the Web Service:
the Kafka production process adopts a short connection mode, accords with a Web Service short connection response mode, and is suitable for being converted into Service;
the Kafka consumption process adopts a long connection mode, namely, the data is extracted from the Kafka cluster by keeping the connection state all the time, and the Kafka consumption process is not suitable for converting into the Service because the Web Service adopts a short connection response mode.
There is therefore a need to further optimize the Kafka consumption process.
The information disclosed in this background section is only for enhancement of understanding of the general background of the invention and should not be taken as an acknowledgement or any form of suggestion that this information forms the prior art already known to a person skilled in the art.
Disclosure of Invention
Aiming at the defects in the prior art, the invention aims to provide a method for converting Kafka long connection consumption into Service, wherein when a long connection mode is adopted in the Kafka consumption process, the long connection consumption is converted into short connection Service, so that the Kafka consumption mode is changed from the long connection mode into the short connection mode, the connection modes of production and consumption are unified, and the production and consumption behaviors in a Kafka cluster are opened into Web Service.
In order to achieve the above purposes, the technical scheme adopted by the invention is as follows:
a method for converting Kafka long connection consumption to service, comprising the steps of:
the service issuing party issues the service, wherein the service issuing refers to Kafka consumption service issuing;
after the Service is released, the Kafka consumption Service is opened by a Web Service interface and is accessed by taking a Web Service protocol as a standardized protocol;
a service requester calls a service request, and transmits a topic name to be consumed and an ID of a consumption group, wherein the service request call refers to Kafka consumption service request call;
the Kafka service engine receives the topic name and the ID of the consumption group, then creates a Kafka consumer and stores the Kafka consumer in a cache;
The Kafka consumer acquires a long connection data packet from the Kafka cluster in a long connection mode, wherein the long connection data packet comprises a plurality of messages;
the Service engine of Kafka acquires a short connection data packet from a Kafka consumer by adopting a short connection mode which is adaptive to a response mode that Web Service only consumes once;
the Service engine of Kafka provides short connection packets to the Web Service interface.
On the basis of the technical scheme, the specific mode of service publishing by the service publishing party is as follows: selecting a certain data application function in the application program, accessing the function to the large service bus platform in the form of service, and publishing the function to the outside in a public way, so that a service requester can access or apply for use on line.
On the basis of the technical scheme, before service release, the following service basic information is set:
selecting JMS protocol, and further selecting Kafka product,
the processing mode is set to be a mode of a consumer,
setting the cluster address of the corresponding Kafka and the account password of the Kafka,
setting basic parameters of Kafka consumers, wherein the basic parameters at least comprise: the number of failed retransmissions of Kafka, the maximum number of strips consumed by Kafka,
and encapsulating the setting of the service basic information into service parameters, and simultaneously issuing the service parameters when service issuing is carried out.
On the basis of the technical scheme, when the Service request is called, at least the following two parameters are transmitted through a Web Service protocol:
a Name parameter, corresponding to the topic Name of Kafka;
the Groupid parameter, which corresponds to the ID of the consumption group of Kafka.
On the basis of the technical scheme, the Web Service interface receives the parameters transmitted by the Web Service protocol, converts the transmitted parameters into the topic name to be consumed and the ID of the consumption group, and transmits the topic name and the ID to the Service engine of Kafka.
On the basis of the technical scheme, the Kafka consumer is created, and the specific steps are as follows:
and creating a Kafka consumer according to the basic parameters of the Kafka consumer in the service parameters encapsulated when the service is released.
On the basis of the technical scheme, the Kafka Service engine adopts a short connection mode adaptive to a response mode that Web Service only consumes once, and obtains a short connection data packet from a Kafka consumer, and the method specifically comprises the following steps:
analyzing the maximum number of Kafka consumption from the encapsulated service parameters, assigning the analysis result as the preset number of each consumption to a max.polar.records parameter in the Kafka consumer, wherein the parameter specifies the maximum consumption message number of each time;
Setting a mode of automatically submitting the consumption progress offset by Kafka, so that the Kafka can automatically manage the offset, and the accuracy of the offset under high concurrency is guaranteed;
setting a lock of the Kafka consumer, ensuring that the Kafka consumer is not used simultaneously, and ensuring the visibility of the Kafka consumer in the memory;
the service engine of Kafka takes a predetermined number of messages from the long connection data packet to form a short connection data packet.
On the basis of the technical scheme, a service engine of the Kafka sets a strategy, so that the Kafka consumer can keep alive in the memory within a preset timeout period, and the efficiency of subsequent consumption is ensured.
On the basis of the technical scheme, the Kafka service engine adopts a mode of controlling the number of service threads, and the integrity of data under the condition of high concurrency is ensured.
On the basis of the technical scheme, after the service request calling of the service requester is processed, the Kafka cluster logs the service request calling condition for follow-up problem troubleshooting and statistics.
The method for converting Kafka long connection consumption into service has the following beneficial effects:
when the Kafka consumption process adopts a long connection mode, long connection consumption is converted into short connection Service, so that the Kafka consumption mode is changed from the long connection mode to the short connection mode, the connection modes of production and consumption are unified, and the production and consumption behaviors in a Kafka cluster are opened into Web Service.
The method for converting Kafka long connection consumption into Service opens the production and consumption behaviors in the Kafka cluster into Web Service, and can better provide an international standardized access mode. The Kafka product production and consumption actions are converted into a Web Service standardized protocol mode, the Kafka consumer availability maintenance is realized through internal processing, the consumption mode of single short connection is ensured, and meanwhile, the Service condition can be monitored through a monitoring platform by managing Service information through a management platform.
Web Service is a platform for building interoperable distributed applications, and generally, an application calling Web Service is called a client, and an application providing Web Service is called a server.
The interoperable distributed application is a platform-independent, low-coupling, self-contained, programmable-based Web application (meaning an application accessible through the Web) that Web Service may use open XML (a subset of standard universal markup language) standards to describe, publish, discover, coordinate, and configure the interoperable distributed application.
Drawings
The invention has the following drawings:
The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:
fig. 1 is a schematic diagram of the architecture of the Kafka messaging system.
FIG. 2 is a flow diagram of a method of converting Kafka long connection consumption to service in accordance with the present invention.
Fig. 3 is a flowchart of setting service basic information.
Fig. 4 shows an embodiment of Kafka service consumption process.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings. The detailed description, while indicating exemplary embodiments of the invention, is given by way of illustration only, in which various details of embodiments of the invention are included to assist understanding. Accordingly, it will be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
As shown in fig. 2, the method for converting Kafka long connection consumption into service according to the present invention includes the following steps:
the service issuing party issues the service, wherein the service issuing refers to Kafka consumption service issuing;
as an alternative embodiment, the specific way for the service publisher to publish the service is as follows: selecting a certain data application function in an application program, accessing the function to a large service bus platform in a service form, and publishing the function to the outside in a public way, so that a service requester can access or apply for use on line;
As shown in fig. 3, before service delivery, the following service basic information is set:
selecting JMS protocol, and further selecting Kafka product,
the processing mode is set to be a mode of a consumer,
setting the cluster address of the corresponding Kafka and the account password of the Kafka,
setting basic parameters of Kafka consumers, wherein the basic parameters at least comprise: the number of failed retransmissions of Kafka, the maximum number of strips consumed by Kafka,
encapsulating the setting of the service basic information into service parameters, and simultaneously publishing the service parameters when service publishing is carried out;
the service publisher protects the Kafka information in a mode of encapsulating service parameters, provides reasonable calling parameters and ensures the flexibility of the service requester;
after the Service is released, the Kafka consumption Service is opened by a Web Service interface, is accessed by taking a Web Service protocol as a standardized protocol, and ensures the universality of the Service by adopting the standardized protocol;
a service requester calls a service request, and transmits a topic name to be consumed and an ID of a consumption group, wherein the service request call refers to Kafka consumption service request call;
as an alternative embodiment, at the time of Service request call, at least the following two parameters are introduced through the Web Service protocol:
A Name parameter, corresponding to the topic Name of Kafka;
a Groupid parameter, corresponding to the ID of the consumption group of Kafka;
the Web Service interface receives parameters transmitted by a Web Service protocol, converts the transmitted parameters into a topic name to be consumed and an ID of a consumption group, and transmits the topic name and the ID to a Service engine of Kafka;
a service engine (simply called a service engine) of Kafka receives the topic name and the ID of a consumption group, then a Kafka consumer is created and stored in a cache; the service engine of Kafka is the core of service processing and the node of service calling; if the Kafka consumer is not used within a period of time, the cache can be automatically cleared, so that the occupation of the memory is avoided;
the Kafka consumer is created by the following specific steps:
creating a Kafka consumer according to the basic parameters of the Kafka consumer in the service parameters encapsulated when the service is released; as previously mentioned, the basic parameters include at least: number of failed retransmissions of Kafka, maximum number of consumption of Kafka;
the Kafka consumer acquires a long connection data packet from the Kafka cluster in a long connection mode, wherein the long connection data packet comprises a plurality of messages;
the long connection data packet is cached in the Kafka consumer;
the Service engine of Kafka acquires a short connection data packet from a Kafka consumer by adopting a short connection mode adapted to a response mode that Web Service only consumes once;
The Service engine of Kafka provides short connection packets to the Web Service interface.
Through the mode, after the Kafka Service engine converts the data packet from the long connection form into the short connection, the short connection is returned to the Web Service interface (the Web Service interface is returned to the requester), and the Web Service interface acquires the short connection data packet, so that the Kafka long connection consumption is converted into the Service. The requester can interact data with the Web Service interface only through a Web Service protocol, so that the cost pressure and the development time pressure caused by the development of the Kafka client are avoided.
On the basis of the technical scheme, the Kafka Service engine adopts a short connection mode adaptive to a response mode that Web Service only consumes once, and obtains a short connection data packet from a Kafka consumer, and the method specifically comprises the following steps:
analyzing the maximum number of Kafka consumption from the encapsulated service parameters, assigning the analysis result as the preset number of each consumption to a max.polar.records parameter in the Kafka consumer, wherein the parameter specifies the maximum consumption message number of each time;
setting a mode of automatically submitting consumption progress offset by Kafka, so that the Kafka can automatically manage the offset, and the accuracy of the offset under high concurrency is guaranteed; in order to ensure the integrity of Kafka consumption data, offset of each consumption is recorded in Kafka so as to perform next query;
Setting a lock of the Kafka consumer, ensuring that the Kafka consumer is not used simultaneously, and ensuring the visibility of the Kafka consumer in the memory; the addition of the lock of the Kafka consumer aims at the following problems of the Kafka consumer in the case of high concurrency: when the first thread is in use, the second thread can not consume any more;
the service engine of Kafka takes a predetermined number of messages (predetermined number per consumption) from the long connection packet to form a short connection packet.
When the requester requests once, the service engine takes out a predetermined number of messages from the long connection data packet to form a short connection data packet until the messages in the long connection data packet are consumed, and the Kafka consumer acquires the long connection data packet from the Kafka cluster by adopting the long connection mode again.
On the basis of the technical scheme, a service engine of the Kafka sets a strategy, so that the Kafka consumer can keep alive in the memory within a preset timeout period, and the efficiency of subsequent consumption is ensured.
On the basis of the technical scheme, the Kafka service engine adopts a mode of controlling the number of service threads, and the integrity of data under the condition of high concurrency is ensured.
On the basis of the technical scheme, after the service request calling of the service requester is processed, the Kafka cluster logs the service request calling condition for follow-up problem troubleshooting and statistics.
As an alternative embodiment, the Kafka cluster manages the service through the management platform, and monitors the service condition through the monitoring platform.
The invention adopts the scheme to complete the Kafka consumption process, and has the beneficial effects that:
the original long connection mode is changed into single consumption short connection through certain configuration and processing of Kafka consumers and the flow, and the problem of conflict of the original long and short connections is solved.
The Kafka Service is opened into a more universal Web Service, the problem of the opening capability of Kafka products of various versions is solved, the software development experience is improved, and the working efficiency is improved.
The invention can greatly reduce the workload of repeated redundancy in the software development process, greatly improve the efficiency, support the service form to call Kafka related service, meet the trend of the information development era, reduce the coupling degree of software system development and provide a more perfect solution for data push.
The following are specific examples.
As shown in fig. 4, in the present embodiment, the method for converting Kafka long connection consumption into service, and the request process of the service,
the requestor in the figure refers to the service requestor,
the Service engine in the figure refers to a Service engine of Kafka, and the Service engine provides a Web Service interface for a requester, namely, Kafka consumption Service opened by the Web Service interface,
the Web Service call between the requestor and the Service engine in the figure employs a short connection,
long connections are used between Kafka consumer groups and Kafka clusters in the figure,
the invention discloses a method for converting Kafka long connection consumption into Service, which is used for opening a Web Service interface in a Service engine for a requester to call a Service request, wherein the long connection between a Kafka consumer group and a Kafka cluster is formed.
This realizes the conversion of Kafka consumption of long connection into Web Service.
In addition to the foregoing, fig. 4 illustrates an embodiment in which an adapter is provided between the service engine and the requestor for content routing and protocol conversion,
adapters are provided between the service engine and the Kafka consumer group, also for content routing and protocol conversion,
the arrangement of the two adapters is implemented by the prior art, and the invention is not described in detail.
The specific steps of the service requester for invoking the service request shown in fig. 4 are as follows:
the Service issuing party issues the Kafka consumption Service, the Kafka consumption Service is issued, the data in the Kafka cluster are provided in a Service mode, namely the Kafka consumption Service is opened by a Web Service interface, and a Service document of a Service bus is provided for the Service requesting party to use;
as described above, before service distribution, the following service basic information is set:
selecting a JMS protocol and further selecting Kafka products therein, the Kafka products corresponding to Kafka clusters illustrated in the figure,
the processing mode is set to be the mode of the consumer,
setting the cluster address of the corresponding Kafka and the account password of the Kafka,
setting basic parameters of Kafka consumers, wherein the basic parameters at least comprise: the number of failed retransmissions of Kafka, the maximum number of strips consumed by Kafka,
encapsulating the setting of the service basic information into service parameters, and simultaneously publishing the service parameters when service publishing is carried out;
after the Service is released, the Kafka consumption Service is opened in a Service engine by a Web Service interface and is accessed by taking a Web Service protocol as a standardized protocol;
the service requester calls a service request, transmits a topic name to be consumed and an ID of a consumption group, and obtains the topic name of the data to perform single extraction on the data in the Kafka cluster, wherein the single extraction is that a single consumption message return mode is adopted between the Kafka consumer group and the Kafka cluster, and the single extraction action ensures the synchronism of data processing;
Meanwhile, a user can know the specific data pushing condition by checking the service calling log and the message storage condition, and can check the daily average, the monthly average and the total Kafka data pushing condition through a statistical report;
when a Service request is called, a requester transmits at least the following two parameters to a Web Service interface in a Service engine through a Web Service protocol:
a Name parameter, corresponding to the topic Name of Kafka;
a Groupid parameter, corresponding to the ID of the consumption group of Kafka;
the Web Service interface receives parameters transmitted by a Web Service protocol, converts the transmitted parameters into a topic name to be consumed and an ID of a consumption group, and transmits the topic name and the ID to a Service engine of Kafka;
the Kafka consumer group shown in fig. 4, includes Kafka consumers created by the steps of:
the service engine receives the topic name and the ID of the consumption group, then creates a Kafka consumer according to the basic parameters of the Kafka consumer in the service parameters encapsulated when the service is released, and stores the Kafka consumer in the cache; as previously mentioned, the basic parameters include at least: number of failed retransmissions of Kafka, maximum number of consumption of Kafka;
the Kafka consumer acquires a long connection data packet from the Kafka cluster in a long connection mode, wherein the long connection data packet comprises a plurality of messages;
Long connection packets are cached at Kafka consumers;
the Service engine of Kafka acquires a short connection data packet from a Kafka consumer by adopting a short connection mode which is adaptive to a response mode that Web Service only consumes once;
the Service engine of Kafka provides short connection packets to the Web Service interface.
The Service engine converts the Kafka consumer adopting the long connection mode into a short connection mode adaptive to a response mode which only consumes once by the Web Service by adopting the following steps:
analyzing the Kafka consumption maximum number from the encapsulated service parameters, taking the analysis result as the preset number of each consumption, and assigning the preset number to a max.pole.records parameter in the Kafka consumer;
setting a mode of automatically submitting consumption progress offset by Kafka, so that the Kafka can automatically manage the offset, and the accuracy of the offset under high concurrency is guaranteed; in order to ensure the integrity of Kafka consumption data in Kafka, offset of each consumption needs to be recorded so as to perform the next query;
setting a lock of the Kafka consumer, ensuring that the Kafka consumer is not used simultaneously, and ensuring the visibility of the Kafka consumer in the memory; the addition of the lock of the Kafka consumer aims at the following problems of the Kafka consumer in the case of high concurrency: when the first thread is in use, the second thread can not consume any more;
The service engine of Kafka takes a predetermined number (predetermined number per consumption) of messages from the long connection data packet to form a short connection data packet; namely: the Kafka service engine obtains a long connection data packet sent by a Kafka consumer adopting a long connection mode, and converts the long connection data packet into a plurality of short connection data packets which accord with the parameter setting of max.
As described above, in the embodiment shown in fig. 4, the method of establishing a long connection with the Kafka cluster is converted into the method of establishing a short-connection Web Service interface, so that the problem of openability of Kafka products of various versions is solved, and development cost and maintenance cost are reduced. And the working efficiency is improved, and meanwhile, a perfect solution for data pushing is provided.
In summary, the processing method of the embodiment implements enterprise application integration of Kafka services in the government field through the design and implementation of the invention, and provides a solution that Kafka of various versions issues Web Service, thereby beneficially exploring and practicing how to improve the business and data values of application systems in the government field and improve the business actual combat capability of the application systems.
Those not described in detail in this specification are well within the skill of the art.
The above description is only a preferred embodiment of the present invention, and the scope of the present invention is not limited to the above embodiment, but equivalent modifications or changes made by those skilled in the art according to the present disclosure should be included in the scope of the present invention as set forth in the appended claims.
Claims (6)
1. A method for converting Kafka long connection consumption into service, comprising the steps of:
before service release, the following service basic information is set:
selecting JMS protocol, and further selecting Kafka product,
the processing mode is set to be a mode of a consumer,
setting the cluster address of the corresponding Kafka and the account password of the Kafka,
setting basic parameters of Kafka consumers, wherein the basic parameters at least comprise: the number of failed retransmissions of Kafka, the maximum number of strips consumed by Kafka,
encapsulating the setting of the service basic information into service parameters, and simultaneously publishing the service parameters when service publishing is carried out;
the service issuing party issues the service, wherein the service issuing refers to Kafka consumption service issuing;
after the Service is released, the Kafka consumption Service is opened by a Web Service interface and is accessed by taking a Web Service protocol as a standardized protocol;
A service requester calls a service request, and transmits a topic name to be consumed and an ID of a consumption group, wherein the service request call refers to Kafka consumption service request call;
at the time of Service request call, at least the following two parameters are introduced through the Web Service protocol:
a Name parameter, corresponding to the topic Name of Kafka;
a Groupid parameter, corresponding to the ID of the consumption group of Kafka;
the Kafka service engine receives the topic name and the ID of the consumption group, then creates a Kafka consumer according to the basic parameters of the Kafka consumer in the service parameters encapsulated when the service is issued, and stores the Kafka consumer in a cache;
the Kafka consumer acquires a long connection data packet from the Kafka cluster in a long connection mode, wherein the long connection data packet comprises a plurality of messages;
the Kafka Service engine adopts a short connection mode adaptive to a response mode that Web Service only consumes once, and obtains a short connection data packet from a Kafka consumer, and the method specifically comprises the following steps:
analyzing the maximum number of Kafka consumption from the encapsulated service parameters, assigning the analysis result as the preset number of each consumption to a max.polar.records parameter in the Kafka consumer, wherein the parameter specifies the maximum consumption message number of each time;
Setting a mode of automatically submitting consumption progress offset by Kafka, so that the Kafka can automatically manage the offset, and the accuracy of the offset under high concurrency is guaranteed;
setting a lock of the Kafka consumer, ensuring that the Kafka consumer is not used simultaneously, and ensuring the visibility of the Kafka consumer in the memory;
the Kafka service engine takes out a preset number of messages from the long connection data packet to form a short connection data packet;
the Service engine of Kafka provides short connection packets to the Web Service interface.
2. The method for converting Kafka long connection consumption into service of claim 1, wherein the service publisher issues the service by: selecting a certain data application function in the application program, accessing the function to the large service bus platform in the form of service, and publishing the function to the outside in a public way, so that a service requester can access or apply for use on line.
3. The method of converting Kafka long connection consumption to a Service of claim 1, wherein the Web Service interface receives parameters incoming via the Web Service protocol, converts the incoming parameters to a topic name to be consumed and an ID of the consuming group, and forwards to the Kafka Service engine.
4. The method for converting Kafka long connection consumption into service of claim 1, wherein the Kafka service engine sets a policy so that the Kafka consumer remains alive in the memory for a predetermined timeout period to ensure efficiency of subsequent consumption.
5. The method for converting Kafka long connection consumption into service of claim 1, wherein the Kafka service engine guarantees data integrity in high concurrency by controlling the number of service threads.
6. The method for converting Kafka long connection consumption into service of claim 1, wherein after the service request call made by the service requester is processed, the Kafka cluster logs the service request call for subsequent troubleshooting and statistics.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011517727.9A CN112702415B (en) | 2020-12-21 | 2020-12-21 | Method for converting Kafka long connection consumption into service |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011517727.9A CN112702415B (en) | 2020-12-21 | 2020-12-21 | Method for converting Kafka long connection consumption into service |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112702415A CN112702415A (en) | 2021-04-23 |
| CN112702415B true CN112702415B (en) | 2022-06-28 |
Family
ID=75509193
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011517727.9A Active CN112702415B (en) | 2020-12-21 | 2020-12-21 | Method for converting Kafka long connection consumption into service |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112702415B (en) |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106131205B (en) * | 2016-07-29 | 2019-07-26 | 北京赛佰特科技有限公司 | Big data information processing system and method for Internet of Things |
| CN106874585A (en) * | 2017-01-26 | 2017-06-20 | 北京恒泰实达科技股份有限公司 | A kind of Power Plant Simulation method for realizing and other effects high-performance B/S issues |
| CN109684370A (en) * | 2018-09-07 | 2019-04-26 | 平安普惠企业管理有限公司 | Daily record data processing method, system, equipment and storage medium |
| CN111049846A (en) * | 2019-12-20 | 2020-04-21 | 北京明略软件系统有限公司 | Data processing method and device, electronic equipment and computer readable storage medium |
| CN111884883A (en) * | 2020-07-29 | 2020-11-03 | 北京宏达隆和科技有限公司 | Quick auditing processing method for service interface |
-
2020
- 2020-12-21 CN CN202011517727.9A patent/CN112702415B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN112702415A (en) | 2021-04-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109327509B (en) | Low-coupling distributed streaming computing system of master/slave architecture | |
| US8468541B2 (en) | Event driven sendfile | |
| WO2023077952A1 (en) | Data processing method and system, related device, storage medium and product | |
| CN106021315B (en) | Log management method and system for application program | |
| EP0794490A2 (en) | Dynamic execution unit management for high performance server system | |
| EP1233590A1 (en) | Content provider for a computer system | |
| WO2022105736A1 (en) | Data processing method and apparatus, device, computer storage medium, and program | |
| CN111309374A (en) | Micro-service system and service calling method in micro-service system | |
| US10303529B2 (en) | Protocol for communication of data structures | |
| CN113703997A (en) | Bidirectional asynchronous communication middleware system integrating multiple message agents and implementation method | |
| CA2358131A1 (en) | Dynamic class loading | |
| WO2022214012A1 (en) | System and method for implementing multi-language translation of application program, device and medium | |
| WO2022156542A1 (en) | Data access method and system, and storage medium | |
| CN114221995B (en) | Service calling method and device and electronic equipment | |
| CN112702415B (en) | Method for converting Kafka long connection consumption into service | |
| CN110266787B (en) | Hybrid cloud management system and method and computer equipment | |
| US10402307B2 (en) | System and method for providing runtime tracing for a web-based client accessing a transactional middleware platform using an extension interface | |
| CN107196800B (en) | Overtime hierarchical control method based on configuration | |
| CN111724262B (en) | Subsequent package query system of application server and working method thereof | |
| CN115086311A (en) | Management system of enterprise cross-system service based on cloud service bus | |
| CN111294383B (en) | Internet of things service management system | |
| CN113687958A (en) | Data processing method, system, computer device and storage medium | |
| CN112333106A (en) | High-concurrency scene offline psychological consultation doctor resource allocation method | |
| CN115914330B (en) | Heterogeneous inter-application communication method based on NIO asynchronous thread model | |
| CN117076508B (en) | Method for supporting batch data processing by stream data processing system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |