CN117149461A

CN117149461A - Java-based distributed middleware system and implementation method

Info

Publication number: CN117149461A
Application number: CN202311114469.3A
Authority: CN
Inventors: 李水兵
Original assignee: Beiyin Financial Technology Co ltd
Current assignee: Beiyin Financial Technology Co ltd
Priority date: 2023-08-31
Filing date: 2023-08-31
Publication date: 2023-12-01

Abstract

The invention provides a Java-based distributed middleware system and an implementation method thereof, wherein the implementation method comprises the following steps: the design based on the micro-service architecture is adopted, the system is split into a plurality of services, and each service operates in an independent process and interacts through network communication; establishing a Netty-based high-performance communication framework; establishing a Zookeeper-based distributed coordination framework; establishing a distributed cache based on Redis; establishing a Spring Cloud-based micro-service framework; ELK-based logging and monitoring; JWT based authentication and authorization. The security of the middleware is enhanced, and encryption, authentication, authorization and other mechanisms are adopted to ensure the safe transmission and processing of sensitive information.

Description

Java-based distributed middleware system and implementation method

Technical Field

The invention relates to the field of micro-service architecture design, in particular to a Java-based distributed middleware system and an implementation method.

Background

The middleware example is realized by adopting a lightweight container, supports plug-in expansion, and dynamically adds or deletes plug-ins according to the requirement, thereby improving the expandability and flexibility of the system.

The middleware instance supports a variety of protocols including HTTP, TCP, UDP, etc., with appropriate protocols being selected according to the needs of the application.

The middleware instance supports load balancing, and is configured according to different load balancing strategies, so that the performance and reliability of the system are improved.

The middleware instance supports fault transfer, and when a certain node fails, the middleware instance automatically switches to a standby node, so that the availability of the system is ensured.

The application instance is implemented by using a standard java web application, and is developed by using various java web frameworks, such as Spring, struts.

The application instance and the middleware instance communicate through the RESTful API, so that decoupling of the application and the middleware is achieved.

The middleware examples are coordinated through distributed coordination services such as ZooKeeper and the like, so that dynamic discovery and load balancing of the nodes are realized.

The middleware instance and the application instance are respectively deployed on different servers, so that the expandability and the reliability of the system are improved.

The prior middleware has the technical defects that:

complexity: middleware often requires configuration and management, which complicates their deployment and use and furthermore, middleware may need to be integrated with many applications and systems, which increases its complexity.

Performance problems: middleware may degrade system performance because it needs to handle tasks such as data transfer and conversion. Furthermore, middleware may introduce additional delays and network overhead.

Single point of failure: middleware may become a single point of failure for the system and if middleware fails, the entire system may be affected.

Safety problem: the middleware may be the source of security vulnerabilities because it acts as an entry for an attacker to invade the system.

Dependency problem: many applications and systems rely on middleware, which can cause the applications and systems to fail to function properly if the middleware fails or needs to be upgraded.

Cost problem: middleware typically requires purchase of licenses and maintenance fees, which can increase the cost of the system. Furthermore, middleware may require specialized skills and knowledge to configure and manage, which may require additional training and expense.

Disclosure of Invention

In view of the above, the present invention has been made to provide a Java-based distributed middleware system and implementation method that overcomes or at least partially solves the above-mentioned problems.

According to one aspect of the present invention, there is provided a Java-based distributed middleware system and an implementation method, the implementation method including:

the design based on the micro-service architecture is adopted, the system is split into a plurality of services, and each service operates in an independent process and interacts through network communication;

establishing a Netty-based high-performance communication framework;

establishing a Zookeeper-based distributed coordination framework;

establishing a distributed cache based on Redis;

establishing a Spring Cloud-based micro-service framework;

ELK-based logging and monitoring;

JWT based authentication and authorization.

Optionally, the system is split into a plurality of services by adopting a design based on a micro-service architecture, each service operates in an independent process, and the interaction through network communication specifically includes: each service has a corresponding database and cache, and is independently extended and upgraded.

Optionally, the establishing a Netty-based high-performance communication framework specifically includes:

netty is a high-performance network communication framework that helps achieve fast, reliable network communication;

java middleware uses the Netty framework to realize the message passing and processing functions among nodes. Optionally, the building the Zookeeper-based distributed coordination framework specifically includes:

the Zookeeper is a distributed coordination framework and helps to realize coordination and management in a distributed system;

java middleware uses a Zookeeper framework to realize dynamic load balancing and fault transfer functions so as to ensure the usability and stability of the system.

Optionally, the establishing the distributed cache based on Redis specifically includes:

redis is a high-performance distributed cache, and helps to realize rapid reading, writing and inquiring of data;

java middleware patent solution uses Redis as a distributed cache.

Optionally, the establishing the Spring Cloud-based micro service framework specifically includes:

spring Cloud is a popular micro-service framework and helps to realize functions of service registration, discovery and load balancing under the micro-service framework;

the Java middleware uses a Spring Cloud framework to realize various functions under the micro-service architecture.

Optionally, the ELK-based logging and monitoring specifically includes:

ELK is a popular log and monitoring framework, which helps to realize the recording and analysis of real-time log and the real-time monitoring of the system;

the Java middleware patent solution uses an ELK framework to implement the log and monitoring functions of the system to facilitate management and debugging of the system.

Optionally, the JWT-based authentication and authorization specifically includes:

the JWT protocol is a lightweight authentication and authorization protocol, which helps to realize the security authentication and authority control of the system;

java middleware uses the JWT protocol to achieve authentication and authorization of users.

The foregoing description is only an overview of the present invention, and is intended to be implemented in accordance with the teachings of the present invention in order that the same may be more clearly understood and to make the same and other objects, features and advantages of the present invention more readily apparent.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present invention, and that other drawings are also obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a diagram of a Netty mechanism communication provided by an embodiment of the present invention;

fig. 2 is a schematic diagram of a Zookeeper according to an embodiment of the present invention;

FIG. 3 is an IO multiplexing diagram of redis according to an embodiment of the present invention;

FIG. 4 is a component operation diagram of a spring closed micro-service provided by an embodiment of the present invention.

Detailed Description

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure is embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

The terms "comprising" and "having" and any variations thereof in the description embodiments of the invention and in the claims and drawings are intended to cover a non-exclusive inclusion, such as a series of steps or elements.

The technical scheme of the invention is further described in detail below with reference to the accompanying drawings and the examples.

A Java-based distributed middleware system and an implementation method thereof comprise the following steps:

1) The Java middleware patent scheme adopts a design based on a micro-service architecture, the system is split into a plurality of services, and each service operates in an independent process and interacts through network communication. Each service has its own database and cache, and is independently extended and upgraded to improve the availability and scalability of the system.

2) Netty-based high performance communication framework: netty is a high performance network communication framework that helps achieve fast, reliable network communications. The Java middleware patent proposal uses the Netty framework to realize the message passing and processing functions among nodes so as to improve the performance and throughput of the system.

3) Zookeeper-based distributed coordination framework: zookeeper is a distributed coordination framework that helps achieve coordination and management in a distributed system. The Java middleware patent scheme uses a Zookeeper framework to realize dynamic load balancing and fault transfer functions so as to ensure the usability and stability of the system.

4) Redis-based distributed caching: redis is a high-performance distributed cache, and helps to realize rapid reading, writing and inquiring of data. The Java middleware patent scheme uses Redis as a distributed cache to improve the performance and throughput of the system.

5) Spring Cloud based micro-service framework: spring Cloud is a popular micro-service framework and helps to achieve functions such as service registration, discovery and load balancing under the micro-service framework. The Java middleware patent scheme uses a Spring Cloud framework to realize various functions under a micro-service architecture so as to improve the availability and expandability of the system. As shown in fig. 1.

6) Hystrix-based fault tolerant framework: hystrix is a fault-tolerant framework which helps to realize fault tolerance, fusing and other functions. The Java middleware patent scheme uses the Hystrix framework to realize fault tolerance and fusing functions so as to improve the usability and stability of the system.

7) ELK-based logging and monitoring: ELK is a popular logging and monitoring framework that helps to achieve real-time logging and analysis, as well as real-time monitoring of the system. The Java middleware patent scheme uses an ELK framework to realize the log and monitoring functions of the system so as to facilitate the management and debugging of the system.

8) JWT-based authentication and authorization: JWT is a lightweight authentication and authorization protocol that helps achieve secure authentication and entitlement control of the system. The Java middleware patent proposal uses the JWT protocol to realize the authentication and authorization of the user so as to ensure the security of the system.

2. Netty is a high-performance network communication framework for building various types of middleware. Has the following characteristics:

1) Asynchronous event driven: netty handles network requests and responses by way of event notification based on asynchronous event-driven design, without using a traditional blocking I/O model. This approach avoids thread resource waste and performance degradation due to large numbers of blocked I/O operations.

2) High performance: since Netty employs the underlying implementation of NIO, it utilizes an operating system level multiplexing mechanism to achieve efficient network communications while avoiding the problem of requiring separate threads for each connection. This approach significantly reduces the overhead of thread creation and context switching, thereby improving the throughput and concurrency capabilities of the system.

3) Scalability: the Netty design allows the user to extend the functionality of the framework by adding more processors, codecs and protocol implementations. This approach helps developers quickly build web applications with different protocols and formats.

4) Safety: netty provides support for Secure Socket Layer (SSL) and Transport Layer Security (TLS) protocols, helping developers to secure network communications.

5) Ease of use: the Netty API is simple and easy to use in design, provides rich documents and examples, and enables developers to quickly get on hand to construct high-performance network application programs.

Summarizing: netty is used as a bottom communication framework of message queue middleware, and high-concurrency message transmission and processing are supported through asynchronous event driving of Netty and high-performance network communication capability; as a bottom communication component in the RPC framework, supporting high concurrency RPC calls and responses; as a bottom communication framework of the distributed cache middleware, high concurrency cache requests and responses are supported. As shown in fig. 2, a Zookeeper architecture.

3. The Zookeeper is applied to middleware. The Zookeeper is an open-source distributed coordination framework and is mainly used for coordinating the work of distributed processes and services in a distributed system. It provides a data storage service with strong consistency and a distributed lock service that helps developers to implement the following functions in a distributed system:

1) Distributed lock: a Zookeeper is used for realizing distributed lock so as to coordinate concurrent access among a plurality of processes or services and avoid race conditions.

2) Distributed coordination: a Zookeeper is used to implement distributed coordination, such as election algorithms, distributed computing, etc., to coordinate the co-operation between multiple processes or services.

3) Configuration management: distributed configuration management is implemented using Zookeeper to manage configuration information in a distributed system, such as database connection information, caching policies, and the like.

4) Naming service: a Zookeeper is used to implement a distributed naming service to manage service name and address information in a distributed system.

As shown in fig. 3, the IO multiplexing diagram of redis.

4. Middleware distributed cache based on Redis; redis acts as a high-performance distributed cache system, serving as a distributed cache in middleware. The following are some application scenarios where Redis is a middleware distributed cache:

1) Session management: in Web applications, redis is used as session storage to implement distributed session management. By storing the session data in the Redis, the session sharing across servers is realized, and the expandability and reliability of the application program are improved.

2) Data caching: redis is used as a data cache to speed up data access and improve application performance. By storing the commonly used data in the Redis, the access times of the database are reduced, the load of the database is reduced, and the response speed of the application program is improved.

3) Message queues: redis is used as message queue middleware to implement distributed messaging. By using the Pub/Sub mechanism of Redis, an efficient publish/subscribe mode is realized, and multiple subscribers are supported to subscribe to the same topic at the same time, so that high-concurrency message transmission is realized.

4) Distributed lock: redis is used as a distributed lock middleware to implement the functions of a distributed lock. By using the atomic operation of Redis, an efficient lock mechanism is realized, and the phenomenon that a plurality of processes or threads access the same resource simultaneously to cause inconsistent data and race conditions is avoided;

5) In general, redis acts as a high-performance, reliable distributed caching system that helps developers build various types of middleware applications, such as session management, data caching, message queuing, and distributed locks. As shown in FIG. 4, the IO multiplexing diagram of redis can be operated efficiently as a distributed cache in middleware.

5. The Spring Cloud's micro-service framework provides rich tools and components that help developers build message middleware applications quickly. The following are some steps for building message middleware using Spring Cloud:

1) Message queues: a message queue middleware application is created using Spring Cloud Stream. Spring Cloud Stream provides a unified programming model for building message-driven microservice applications, supporting a variety of message middleware such as Kafka, rabbitMQ, activeMQ, etc.

2) And (3) cache management: a Spring Cache is used to create a Cache management middleware application. Spring Cache provides a unified Cache abstraction layer to easily integrate various Cache implementations, such as Redis, ehcache, guava, etc.

3) Service discovery and administration: spring Cloud Netflix is used to create service discovery and administration middleware applications. Spring CloudNetflix provides a series of components, e.g., eureka, ribbon, feign, zuul, etc., for implementing service discovery, load balancing, service invocation, and API gateway functions.

4) Writing a message producer: using the comments and APIs provided by Spring Boot and Spring Cloud Stream, a message producer program is written and messages are sent to the message middleware.

5) Composition message consumer: message consumer programs are written using the annotations and APIs provided by Spring Boot and Spring Cloud Stream, receiving and processing messages from message middleware.

6) Configuration message middleware: relevant parameters such as connection address, authentication information, message queue name, etc. are configured according to different message middleware.

7) Testing and deployment: the message middleware functions and performances are tested in the local environment, and deployment and tuning are performed according to the needs. As shown in the figure, the components of the spring closed micro-service run the graph.

6. ELK-based logging and monitoring. ELK is a set of open source log management and monitoring tools, including Elasticsearch, logstash and Kibana components. The following are some application scenarios for the logging and monitoring of ELKs:

1) And (3) log management: ELKs are used to collect, store, and analyze various types of log data, such as application logs, system logs, web logs, and the like. By collecting and filtering log data using logstack, storing and indexing log data using elastsearch, and visualizing and analyzing using Kibana, efficient log management and analysis is achieved.

2) And (3) log monitoring: ELKs are used to monitor log data in a system in real time, and to detect and analyze anomalies and faults in the system. By using Logstar to collect and filter log data, using elastic search to perform real-time query and search, using Kibana to perform visualization and analysis, timely discovering and solving problems in the system, and improving the reliability and usability of the system.

3) Application program monitoring: ELK is used to monitor and analyze the performance and behavior of applications. Log data, index data and event data of the application program are collected and filtered through the Logstar, the log data, the index data and the event data are stored and indexed through the elastic search, visualization and analysis are conducted through the Kibana, the running condition, the performance bottleneck and the abnormal condition of the application program are known, and developers are helped to optimize and improve the performance and the reliability of the application program.

4) In general, ELK has high flexibility and expandability as a set of open-source log management and monitoring tools for various types of log management and monitoring application scenarios, such as log management, log monitoring, and application monitoring. The developer selects proper components and tools according to actual requirements to construct an efficient and reliable log and monitoring system.

7. JWT based authentication and authorization; JWT (JSON Web Token) is a lightweight authentication and authorization protocol based on a JSON format data representation. JWT consists of three parts: header, payload, and signature. The header contains information such as algorithm and token type, the payload contains data such as user information and authorization information, and the signature is used for verifying the validity of the token. The process of authentication and authorization by middleware using JWT is as follows:

1) User login: the user inputs the user name and the password, the system performs identity verification according to the user name and the password, and a JWT token is generated and contains data such as user information, authorization information and the like.

Token transmission: the system transmits the JWT token to the client, which stores the token locally, e.g., localStorage or sessionStorage.

2) Request verification: when the client sends a request to the system, the JWT token is transmitted to the system as a request header or request parameter, and the system verifies the legitimacy of the token by decrypting the token and verifying the signature.

3) Authorization verification: the system judges whether the user has operation authority according to the user information and the authorization information contained in the token.

The advantages of using JWT for authentication and authorization are as follows:

4) Maintenance of session state is not required: because the JWT token is stateless, the system does not need to maintain session state, and the scenes such as distributed system and multi-server load balancing are easily realized.

The expandability is good: the JWT token contains data in any format, and the custom load field is expanded according to the actual demand.

5) The safety is high: the JWT token uses a signature to verify the legitimacy of the token, preventing the token from being tampered with or counterfeited.

6) In general, authentication and authorization using JWT is a secure, simple, efficient way of authentication and authorization, applicable to various types of Web applications and API services. The developer uses the existing JWT library or framework to easily realize the functions of JWT generation, decryption, verification and the like.

The beneficial effects are that: performance and scalability: the performance and scalability of middleware is a very important key point. In the development of Java middleware, consideration needs to be given to how to maximize performance and scalability, and to ensure stability and reliability of the system.

Safety and stability: the security and stability of middleware is also a very important key point. In the development of Java middleware, consideration is required to maximize security and stability, and to ensure usability and reliability of the system.

Configurability and customizable: the configurability and customizable nature of middleware is also a very important key point. In development of Java middleware, consideration is required on how to maximize configurability and customizable, and ensure flexibility and scalability of the system.

Modularity and reusability: the modularity and reusability of middleware is also a very important key point. In development of Java middleware, consideration is required as to how to maximize modularization and reusability and to ensure maintainability and scalability of a system.

The foregoing detailed description of the invention has been presented for purposes of illustration and description, and it should be understood that the invention is not limited to the particular embodiments disclosed, but is intended to cover all modifications, equivalents, alternatives, and improvements within the spirit and principles of the invention.

Claims

1. The Java-based distributed middleware system and the realization method are characterized in that the realization method comprises the following steps:

establishing a Netty-based high-performance communication framework;

establishing a Zookeeper-based distributed coordination framework;

establishing a distributed cache based on Redis;

establishing a Spring Cloud-based micro-service framework;

ELK-based logging and monitoring;

JWT based authentication and authorization.

2. The Java-based distributed middleware system and implementation method according to claim 1, wherein the design based on a micro-service architecture is adopted to split the system into a plurality of services, each service running in an independent process, and the interaction through network communication specifically includes: each service has a corresponding database and cache, and is independently extended and upgraded.

3. The Java-based distributed middleware system and implementation method according to claim 1, wherein the establishing a Netty-based high-performance communication framework specifically includes:

java middleware uses the Netty framework to realize the message passing and processing functions among nodes.

4. The Java-based distributed middleware system and implementation method according to claim 1, wherein the establishing a Zookeeper-based distributed coordination framework specifically includes:

the Zookeeper is a distributed coordination framework, and can help to realize coordination and management in a distributed system;

5. The Java-based distributed middleware system and implementation method according to claim 1, wherein the establishing a Redis-based distributed cache specifically includes:

java middleware patent solution uses Redis as a distributed cache.

6. The Java-based distributed middleware system and implementation method according to claim 1, wherein the building a Spring Cloud-based micro-service framework specifically includes:

7. The Java-based distributed middleware system and implementation method according to claim 1, wherein the ELK-based logging and monitoring specifically includes:

8. The Java-based distributed middleware system and implementation method according to claim 1, wherein the JWT-based authentication and authorization specifically includes: