CN117834346A - Protocol conversion gateway, protocol conversion method, equipment and storage medium - Google Patents
Protocol conversion gateway, protocol conversion method, equipment and storage medium Download PDFInfo
- Publication number
- CN117834346A CN117834346A CN202311866517.4A CN202311866517A CN117834346A CN 117834346 A CN117834346 A CN 117834346A CN 202311866517 A CN202311866517 A CN 202311866517A CN 117834346 A CN117834346 A CN 117834346A
- Authority
- CN
- China
- Prior art keywords
- request
- protocol conversion
- target
- service system
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000006243 chemical reaction Methods 0.000 title claims abstract description 218
- 238000000034 method Methods 0.000 title claims abstract description 51
- 238000003860 storage Methods 0.000 title claims abstract description 12
- 238000012545 processing Methods 0.000 claims description 34
- 238000004458 analytical method Methods 0.000 claims description 17
- 238000004590 computer program Methods 0.000 claims description 15
- 238000012795 verification Methods 0.000 claims description 8
- 238000004891 communication Methods 0.000 abstract description 25
- 238000007726 management method Methods 0.000 description 15
- 230000008569 process Effects 0.000 description 15
- 238000010586 diagram Methods 0.000 description 12
- 230000004048 modification Effects 0.000 description 6
- 238000012986 modification Methods 0.000 description 6
- 230000006870 function Effects 0.000 description 5
- 230000007246 mechanism Effects 0.000 description 5
- 230000009471 action Effects 0.000 description 3
- 238000004422 calculation algorithm Methods 0.000 description 2
- 238000007405 data analysis Methods 0.000 description 2
- 230000018109 developmental process Effects 0.000 description 2
- 230000011218 segmentation Effects 0.000 description 2
- 238000012356 Product development Methods 0.000 description 1
- 230000003044 adaptive effect Effects 0.000 description 1
- 238000007792 addition Methods 0.000 description 1
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013501 data transformation Methods 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000008571 general function Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000003032 molecular docking Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000010076 replication Effects 0.000 description 1
Landscapes
- Computer And Data Communications (AREA)
- Communication Control (AREA)
Abstract
The application relates to the technical field of data communication and discloses a protocol conversion gateway, a protocol conversion method, equipment and a storage medium. The protocol conversion gateway includes: the system comprises a management platform, a configuration center, a reverse proxy module and a protocol conversion module; the management platform is configured to generate protocol conversion configuration between the third-party service system and the target service system and send the protocol conversion configuration to the configuration center; a configuration center configured to store protocol conversion configurations of at least one third party service system; the reverse proxy module is configured to receive an original request sent by the third-party service system and route and distribute the original request to the protocol conversion module; the protocol conversion module is configured to perform protocol conversion on the original request based on the Spark framework according to the protocol conversion configuration corresponding to the original request, obtain a target request, and send the target request to the target service system. The protocol conversion gateway provided by the application can simultaneously support large-batch data communication among a plurality of service platforms.
Description
Technical Field
The present invention relates to the field of data communications technologies, and in particular, to a protocol conversion gateway, a protocol conversion method, a device, and a storage medium.
Background
Along with the development of the Internet, the problem of diversification of communication protocols in the network is more and more prominent, the performance, the construction period, the cost and the system stability of an automation system are seriously affected, and the method has great significance in solving the problems of conversion and communication standardization of the communication protocols of the automation system. Different internet manufacturers respectively establish proprietary system platforms, and when the service of a third party network platform needs to be connected or accessed in the product development process, protocol conversion between the system platforms is involved. The protocol conversion gateway realizes network interconnection at a network layer, and the data of one system platform is converted into the communication protocol of the other system platform, so that the system platforms of two different protocols can communicate. With the entry of the internet into a big data era, the data communication between systems has higher and higher requirements on the efficiency of a protocol conversion gateway, and how to efficiently complete protocol conversion between a plurality of system platforms so as to cope with flexible and changeable data communication requirements is a problem to be solved at present.
Disclosure of Invention
In view of the foregoing, the present application aims to propose a protocol conversion gateway, a protocol conversion method, a device and a storage medium, so as to improve the data communication efficiency between heterogeneous network systems.
In order to achieve the above purpose, the technical scheme of the application is as follows:
an embodiment of the present application provides a protocol conversion gateway in a first aspect, where the protocol conversion gateway includes a management platform, a configuration center, a reverse proxy module, and a protocol conversion module;
the management platform is configured to generate protocol conversion configuration between the third party service system and the target service system and send the protocol conversion configuration to the configuration center;
the configuration center is configured to store protocol conversion configuration of at least one third party service system;
the reverse proxy module is configured to receive an original request sent by a third-party service system and distribute the original request to the protocol conversion module in a routing way;
the protocol conversion module is configured to perform protocol conversion on the original request based on Spark framework according to the protocol conversion configuration corresponding to the original request to obtain a target request, and send the target request to a target service system.
Optionally, the management platform is specifically configured to perform the following steps:
Acquiring protocol request information of an application, a service or a plug-in each third party service system; the protocol request information includes: request address, request protocol, request mode and request data format;
determining a target service system to be accessed by the third party service system and acquiring target protocol information of the target service system, wherein the method comprises the following steps: target address, target protocol, target request mode and target request data format;
and generating protocol conversion configuration between the third party service system and the target service system according to the protocol request information and the target protocol information.
Optionally, the reverse proxy module is specifically configured to perform the following steps:
receiving an original request sent by a third-party service system;
analyzing the original request to obtain an analysis result, including: request address, request header information, and service request data;
storing the analysis result;
and acquiring protocol conversion configuration corresponding to the original request from the configuration center, and transmitting the service request data and the protocol conversion configuration to the protocol conversion module.
Optionally, the protocol conversion module is specifically configured to perform the following steps:
Acquiring service request data of the original request;
according to the protocol conversion configuration corresponding to the original request, performing protocol conversion on the service request data by using a Spark framework to generate a result data set;
converting the result data set into target format data according to the request format information of the target service system;
according to the target protocol information in the protocol conversion configuration, the target format data is assembled into a target request;
and sending the target request to the target service system.
Optionally, the protocol conversion module is specifically configured to perform the following steps:
judging the data type of the service request data; the data type comprises real-time data and batch data;
if the service request data comprises batch data, processing the batch data by using Spark Job to obtain a batch data set;
if the service request data comprises real-time data, processing the real-time data by using Spark Streaming to generate a small data set;
based on the bulk data set and the small data set, a result data set is generated.
Optionally, the protocol conversion gateway further comprises a query module;
The protocol conversion module is further configured to store the result dataset in a database;
the query module is configured to query and modify the result data set in the database through Spark SQL.
Optionally, the protocol conversion gateway further comprises an authentication module;
the authentication module is configured to read a request address of the original request and perform identity verification on the request address;
if the identity verification is passed, further performing authority check on the request header information of the original request;
and the reverse proxy module is further configured to send the service request data and the protocol conversion configuration to the protocol conversion module if the authority check passes.
According to a second aspect of embodiments of the present application, there is provided a protocol conversion method applied to implementing the protocol conversion gateway provided in the first aspect of embodiments of the present application, where the method includes:
setting protocol conversion configuration between a third party service system and a target service system; the protocol conversion configuration comprises a request address, a request protocol, a request mode and a request data format of the third party service system, and a target address, a target protocol, a target request mode and a target request data format of the target service system;
When an original request sent by a third-party service platform is received, analyzing the original request to obtain an analysis result; the analysis result comprises: request address, request header information, and service request data;
according to the protocol conversion configuration corresponding to the original request, carrying out protocol conversion on the original request based on a Spark framework to obtain a target request;
and sending the target request to a target service system.
According to a third aspect of embodiments of the present application, there is provided a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the method according to the second aspect of embodiments of the present application.
According to a fourth aspect of embodiments of the present application, there is provided an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps in the method according to the second aspect of embodiments of the present application when the computer program is executed.
By adopting the protocol conversion gateway provided by the application, the protocol conversion configuration between the third-party service system and the target service system is stored in advance through the management platform, when an original request sent by the third-party service platform is received, the original request is forwarded to the protocol conversion module through the reverse proxy module, the protocol conversion module automatically converts the data of each original request based on the Spark framework, a target request meeting the protocol requirements of the target service system is generated, and the target request is sent to the target service system for processing. The protocol conversion gateway performs intelligent protocol conversion processing among a plurality of service systems based on a Spark framework, gives request data and a calculation program to Spark processing by utilizing the strong distributed computing capability characteristic of the Spark framework, and automatically performs data segmentation, algorithm replication, distributed execution and result merging on requests sent among different service platforms through Spark to finally obtain a protocol conversion result. The protocol conversion gateway provided by the application can simultaneously support large-batch data communication among a plurality of service platforms.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the description of the embodiments of the present application will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a schematic architecture diagram of a protocol conversion gateway according to an embodiment of the present application;
FIG. 2 is a flow chart of a protocol conversion method according to an embodiment of the present application;
FIG. 3 is a schematic diagram illustrating an architecture of a protocol conversion gateway according to an embodiment of the present application;
FIG. 4 is a flowchart of the protocol conversion module according to one embodiment of the present application;
FIG. 5 is a schematic flow chart of a third party business system accessing platform services according to an embodiment of the present application;
fig. 6 is a schematic diagram of an application scenario of a protocol conversion gateway in the present application;
fig. 7 is a flowchart of the operation of the authentication module in an embodiment of the present application.
Detailed Description
The following description of the embodiments of the present application will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are some, but not all, of the embodiments of the present application. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments herein without making any inventive effort, are intended to be within the scope of the present application.
It should be appreciated that reference throughout this specification to "one embodiment" or "an embodiment" means that a particular feature, structure or characteristic described in connection with the embodiment is included in at least one embodiment of the present application. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
In various embodiments of the present application, it should be understood that the sequence numbers of the following processes do not mean the order of execution, and the order of execution of the processes should be determined by the functions and internal logic thereof, and should not constitute any limitation on the implementation process of the embodiments of the present application.
Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The implementations described in the following exemplary examples are not representative of all implementations consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present application as detailed in the accompanying claims.
It should be noted that, in the case of no conflict, the embodiments and features in the embodiments may be combined with each other.
Fig. 6 is a schematic application scenario diagram of a protocol conversion gateway in the present application. As shown in fig. 6, the protocol conversion gateway may be deployed on the cloud or locally, where the protocol conversion module converts a service request of the third party service system from one communication protocol to another communication protocol (a communication protocol of the target service system), and the protocol conversion module uplink converts the third party protocol data into the target system format data and invokes a downstream service machine interface of the target system to forward; and downlink data returned by the target service system (namely the platform service system) is received in a downlink mode, converted into third-party protocol data and forwarded to the third-party protocol system. The downstream service machine may be a platform docking SDK, and in practical application, the downstream machine of the platform providing the relevant service may be set according to the requirement to support the general function of the target system gateway, which is not limited in this application.
The request supported by the third party service system may be an HTTP request or an HTTPs request, and the communication protocol supported by the target service system may be a protocol supported by a service inside the platform, including: gRPC, thread, web Service, HTTP, etc. It should be noted that, in this embodiment, when the target service system returns data to the third party service system after executing the service processing according to the target request, the data sent by the target service system is still converted into data meeting the communication protocol requirement of the third party service system by the protocol conversion gateway, and is returned to the third party service system. For example, the communication protocol of the third party service system is HTTP, the communication protocol of the target service system is gRPC, the protocol conversion gateway converts between the HTTP protocol and the gRPC protocol, and the platform internal service interface of the gRPC is opened to the outside in the form of HTTP interface, so that the user of the third party service system does not need to develop a new interface standard in order to adapt to the protocol of the target service platform.
The present application will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.
Fig. 1 is a schematic architecture diagram of a protocol conversion gateway 100 according to an embodiment of the present application. As shown in fig. 1, the protocol conversion gateway includes a management platform 101, a configuration center 102, a reverse proxy module 103, and a protocol conversion module 104;
the management platform 101 is configured to generate a protocol conversion configuration between a third party service system and a target service system, and send the configuration to the configuration center 102;
the configuration center 102 is configured to store protocol conversion configuration of at least one third party service system;
the reverse proxy module 103 is configured to receive an original request sent by a third party service system and route and distribute the original request to the protocol conversion module 104;
the protocol conversion module 104 is configured to perform protocol conversion on the original request based on Spark framework according to the protocol conversion configuration corresponding to the original request, obtain a target request, and send the target request to a target service system.
Spark is a memory-based fast, general-purpose, scalable big data analysis and computation engine that can handle large-scale offline data analysis and processing tasks. In this embodiment, an operator accesses a protocol of the third party service platform through the management platform, sets a protocol conversion configuration between the third party service system and the target service system, and stores the protocol conversion configuration in the configuration center, so that the reverse proxy module is convenient to read. The reverse proxy module receives and analyzes the original request sent by the third-party service system, and forwards the original request to the protocol conversion module for processing according to the corresponding protocol conversion configuration.
In this embodiment, the protocol conversion module performs protocol conversion on the original request based on the Spark framework to obtain a target request meeting the communication protocol requirement of the target service system, and then sends the target request to the target service system for processing. The method and the device utilize the characteristic that the Spark framework can perform distributed computation of a large amount of data, process requests from different third party service systems simultaneously, intelligently execute data segmentation, perform algorithm copying and distributed execution on a computing program (data processing program), and intelligently combine results to finally obtain protocol conversion results, thereby greatly improving the efficiency of data communication among system platforms conforming to different protocols. In addition, in this embodiment, the protocol conversion gateway performs flexible access and management protocol conversion configuration on the third party service platform through the management platform and the configuration center, and for application scenarios such as newly adding a third party service system and replacing a data communication protocol, the protocol conversion gateway does not need to redevelop an adaptive interface standard, and can better adapt to flexible and changeable network service scenarios, and is particularly suitable for service scenarios involving big data.
As an embodiment of the present application, the management platform 101 is specifically configured to perform the following steps:
Acquiring protocol request information of an application, a service or a plug-in each third party service system; the protocol request information includes: request address, request protocol, request mode and request data format;
determining a target service system to be accessed by the third party service system and acquiring target protocol information of the target service system, wherein the method comprises the following steps: target address, target protocol, target request mode and target request data format;
and generating protocol conversion configuration between the third party service system and the target service system according to the protocol request information and the target protocol information.
In this embodiment, the protocol request information of the third party service system to be accessed is provided to the protocol conversion gateway in advance, and the protocol conversion configuration is generated through the management platform and stored in the configuration center for later use in the protocol conversion. The management platform displays a configurable interface to an operator, and the operator performs protocol conversion related settings. Specifically, when the third party service system is newly accessed, the protocol request information of the third party service system and the target protocol information of the target service system need to be configured. For example, a third party service system a with a communication protocol of HTTPS needs to access two platform services using the gRPC and thread protocols in the target service system B through the Web end respectively; or, a plurality of third party service systems or access terminals (such as APP, SDK, web terminal, server, etc.) with different protocols access to the platform service using the gRPC protocol in the target service system C through different paths (HTTP/HTTPS), and a request conversion mode needs to be set for each third party service system. The third party service system needs to provide information such as request address, request protocol (i.e. access protocol, such as HTTP), request mode and request data format. The operator also needs to configure an accessed target service system for the third-party service system, and the target service system needs to provide target protocol information, including information such as a target address, a target protocol (such as gRPC protocol), a target request mode, a target request data format and the like, so as to correspond to the protocol request information of the third-party service system, and construct a connection channel of the two systems. And the protocol conversion gateway performs protocol conversion and data forwarding between the two systems according to the configured protocol conversion configuration.
It should be noted that, before setting the protocol conversion configuration, first, a third party service system to be accessed and a target service system to be accessed need to be registered in the service registration center. And then, according to the service requirement, configuring a conversion mode of the third-party service protocol and the platform service protocol in the management platform, and storing the conversion mode into a configuration center.
Similarly, when the target service system returns a request to the third party service system after finishing service processing, the protocol conversion gateway needs to convert the request into a third party service system request and forward the request to the third party service system. And when the third party service system receives the request, service processing is carried out according to the request information, and service interaction of the two service systems is completed.
As an embodiment of the present application, the reverse proxy module 103 is specifically configured to perform the following steps:
receiving an original request sent by a third-party service system;
analyzing the original request to obtain an analysis result, including: request address, request header information, and service request data;
storing the analysis result;
and acquiring protocol conversion configuration corresponding to the original request from the configuration center, and transmitting the service request data and the protocol conversion configuration to the protocol conversion module.
In this embodiment, when the protocol conversion gateway receives an original request sent by the third party service system, the original request is parsed by the reverse proxy module, and specifically, a request address, request header information and service request data (data requiring service processing are requested) are parsed from the original request. After the analysis is finished, the reverse proxy module stores the analysis result so as to be read and used by a subsequent protocol conversion module. The reverse proxy module reads the pre-stored protocol conversion configuration in the configuration center according to the request address and the request header information of the original request, and sends the service request data of the original request to the protocol conversion module, and the protocol conversion module processes the service request data.
As an implementation mode of the application, the protocol conversion gateway further comprises an authentication module;
the authentication module is configured to read a request address of the original request and perform identity verification on the request address;
if the identity verification is passed, further performing authority check on the request header information of the original request;
and the reverse proxy module is further configured to send the service request data and the protocol conversion configuration to the protocol conversion module if the authority check passes.
In one embodiment, the protocol conversion gateway further comprises an authentication module, and when the protocol conversion gateway receives the request of the third party service system, the request is analyzed and authentication is performed according to the configuration information when the request is accessed. After passing the authentication, the protocol conversion is performed according to the protocol conversion configuration, and then the request is forwarded to the platform service system through a downstream service machine (for example, a client of the platform service system, an SDK (software development kit) and the like) of the target service system.
In this embodiment, the reverse proxy module analyzes the original request, and the authentication module authenticates the original request. And after the authentication is passed, the reverse proxy module forwards the service request data to the protocol conversion module for processing. In this embodiment, when generating the protocol conversion configuration between the third party service system and the target service system, the operator also needs to set a corresponding access right for the third party service system to improve the security of services and data in the target service system, and authenticate the request before forwarding the request to the target service system, so as to ensure the security of data access or operation to the target service system. Specifically, when the third party service system is accessed, identity registration is performed for the third party service system, and corresponding access rights are set. For example, permission for allowing data to be queried and read is set for an accessed third party service system, and new data addition, modification and deletion are forbidden.
Fig. 7 is a flowchart of the operation of the authentication module in an embodiment of the present application. As shown in fig. 7, when the original request is authenticated, the authentication module reads the request address (including information such as a host name and a port number) of the original request, searches corresponding information from the configuration center to perform identity authentication, and checks whether the identity information of the requester is registered and stored in the configuration center. If the requester is not registered, authentication is failed, and error information is returned to the request (interface) address of the third party system.
If the authentication module confirms that the requesting party is the registered legal identity, the request header information of the original request is further read, authority check is carried out on the request header information, whether the content of the original request accords with the access authority corresponding to the identity is determined, if the content does not accord with the corresponding access authority, the authority check is not passed, the authentication fails, and error information is returned to the request (interface) address of the third party system.
Fig. 3 is a schematic architecture diagram of a protocol conversion gateway according to an embodiment of the present application. As shown in fig. 3, the protocol conversion flow between service systems in the present embodiment is as follows:
(1) An operator configures a protocol request of a third party service system supporting access in a protocol conversion gateway through a management platform. The third party service system accesses the target service system in a APP, SDK, WEB or Server mode, provides protocol request information, and comprises: request address, request protocol, request mode, request data format, etc. And generating protocol conversion configuration by combining target protocol information of a target service system accessed by the third-party service system, and storing the protocol conversion configuration in a configuration center. The target protocol information includes: destination address, destination protocol, destination request mode, and destination request data format. In addition, the operator can register the identity for the third party service system and allocate the corresponding access right.
(2) When the protocol conversion gateway receives a third-party protocol request, the request is analyzed through the reverse proxy module, the request is authenticated uniformly through the authentication module, the authentication module comprises identity authentication and authority check, and after the authentication is passed, service request data obtained by analyzing the request is sent to the protocol conversion module. In the process, the authentication module reads the protocol conversion configuration stored in the configuration center, authenticates the request according to the protocol conversion configuration, and the reverse proxy module reads the protocol conversion configuration from the configuration center and forwards the protocol conversion configuration and service request data to the protocol conversion module.
(3) The protocol conversion module performs processing such as conversion and cleaning on service request data based on Spark framework according to protocol conversion configuration, then preprocesses the processed data according to a target request data format in the protocol conversion configuration, generates a target request, acquires a downstream service machine (for example, a request client of a target service system), and then sends the target request to the target service system for service processing. In this embodiment, protocols that can be supported by the services of the target Service platform include, but are not limited to, gRPC, thread, web Service, HTTP, and in practical application, the protocols of the third party Service system and the protocols of the target Service system may be configured according to practical situations.
As an embodiment of the present application, the protocol conversion module 104 is specifically configured to perform the following steps:
acquiring service request data of the original request;
according to the protocol conversion configuration corresponding to the original request, performing protocol conversion on the service request data by using a Spark framework to generate a result data set;
converting the result data set into target format data according to the request format information of the target service system;
according to the target protocol information in the protocol conversion configuration, the target format data is assembled into a target request;
and sending the target request to the target service system.
In this embodiment, the protocol conversion module performs protocol conversion processing on service request data based on Spark framework to generate a result data set, then converts the result data set into data conforming to the format of the target service system according to the request format information of the target service system, and assembles the target format data according to the target protocol information of the target service system in the protocol conversion configuration, so as to finally generate a target request conforming to the communication protocol of the target service system.
Optionally, the protocol conversion module 104 is specifically configured to perform the following steps:
Acquiring at least one downstream service machine of a target service system through a service discovery mechanism;
the target request is sent to all downstream service machines through load balancing.
In this embodiment, after generating the target request, the protocol conversion module obtains at least one downstream service machine (for example, a request client of the target service platform) of the target service system through a service discovery mechanism, and sends the converted target request to the target service system through load balancing. When a large number of requests are processed, load balancing is adopted to distribute the requests, so that the reliability of data communication is improved, and access failure or data loss caused by overload of a single machine is avoided.
Fig. 5 is a schematic flow chart of accessing a platform service by a third party business system according to an embodiment of the present application. As shown in fig. 5, in this embodiment, the reverse proxy module analyzes the original request to obtain the Service name to be accessed, and the calling protocol Service uses the Service discovery mechanism to obtain the Service IP address and port information corresponding to the Service name, analyzes and obtains the Service request data, and forwards the Service request data to the protocol conversion module. The protocol conversion module performs distributed computation (i.e., protocol conversion processing) on the service request data based on the Spark framework.
In this embodiment, the connection pool is used to cache connection instances of the third party service system and the target service system. After the protocol conversion is completed, a connection instance of the target service system is acquired through the connection pool, and a protocol object (target request) with the completed protocol conversion is sent to the target service system.
After the protocol conversion is finished, the protocol conversion module calls a protocol Client to take out a target request from a connection pool and send the target request to a target service system, the protocol Client finishes data reading and forwarding through a new instance (process), before the new instance, the protocol Client judges whether an available instance exists in the connection pool, if the available instance does not exist, the new instance is built, the target request in the connection pool is obtained, then a downstream machine (for example, a request Client of the target service system) of the target service system is obtained through a service discovery mechanism, and platform service of the target service system is called through the downstream machine. And the target service system analyzes the target request and executes service processing.
As an embodiment of the present application, the protocol conversion module 104 is specifically configured to perform the following steps:
judging the data type of the service request data; the data type comprises real-time data and batch data;
If the service request data comprises batch data, processing the batch data by using Spark Job to obtain a batch data set;
if the service request data comprises real-time data, processing the real-time data by using Spark Streaming to generate a small data set;
based on the bulk data set and the small data set, a result data set is generated.
In one embodiment, the Spark framework is used for data protocol conversion, and the data protocol conversion can be processed for real-time data and batch data respectively. Specifically, spark Streaming can be used to process real-time data and Spark Job can be used to process non-real-time bulk data.
Fig. 4 is a flowchart of the protocol conversion module according to an embodiment of the present application. As shown in fig. 4, the protocol conversion module converts the service request data and generates a target request as follows:
(1) And acquiring service request data obtained by analyzing the reverse proxy module, and integrating the data. If a plurality of requests are processed simultaneously, integrating service request data obtained by analyzing all the requests, and uniformly carrying out protocol conversion;
(2) And respectively carrying out protocol conversion according to the type of the integrated service request data. Specifically, judging whether the service request data is real-time data or batch data, and calculating the real-time data in real time by Spark Streaming to obtain a small data set; and carrying out batch calculation on the non-real-time batch data by adopting Spark Job to obtain a batch data set. For example, when the request is to add or delete a small amount of data, the request can be divided into real-time data for processing, and when the request is to push tens of thousands of pieces of form data stored in a database, the request is divided into non-real-time batch data for subsequent processing through Spark Job;
(3) Integrating the small data set obtained by Spark Streaming processing and the batch data set obtained by Spark Job processing to generate a result data set;
(4) A target request is generated based on the result dataset. Firstly, the request data format of a target service system is acquired, a result data set is converted into target format data which can be identified by the target service system, and then the target format data is assembled to generate a target request according to other configuration information (for example, the IP address of the target service system) in the target protocol information.
In this embodiment, data integration is performed on a plurality of request data based on the Spark framework, and a distributed computing manner is adopted to process real-time data and batch data simultaneously, so that the transmission efficiency of communication data between systems can be improved, the access rate of a user accessing a service in a target service system in a third party service system can be improved, and the user experience can be improved.
As an implementation mode of the application, the protocol conversion gateway further comprises a query module;
the protocol conversion module 104 is further configured to store the result dataset in a database;
the query module is configured to query and modify the result data set in the database through Spark SQL.
In one embodiment, as shown in fig. 4, the protocol conversion gateway further includes a query module that performs structural operations on the result data set stored in the database, including queries, modifications, and the like. Spark SQL provides a declarative language for processing structured data, and it provides a DataFrame API and an SQL API that can be used to perform SQL queries, data transformations, data aggregations, and the like. In this embodiment, by combining Spark Job and Spark SQL, structured query and management of a large amount of service request data can be implemented, and complex processing and analysis of batch data can be supported.
Based on the same inventive concept, an embodiment of the present application provides a protocol conversion method. Referring to fig. 2, fig. 2 is a flowchart of a protocol conversion method according to an embodiment of the present application. As shown in fig. 2, the method includes:
s21: setting protocol conversion configuration between a third party service system and a target service system; the protocol conversion configuration comprises a request address, a request protocol, a request mode and a request data format of the third party service system, and a target address, a target protocol, a target request mode and a target request data format of the target service system;
S22: when an original request sent by a third-party service platform is received, analyzing the original request to obtain an analysis result; the analysis result comprises: request address, request header information, and service request data;
s23: according to the protocol conversion configuration corresponding to the original request, carrying out protocol conversion on the original request based on a Spark framework to obtain a target request;
s24: and sending the target request to a target service system.
As one embodiment of the present application, setting a protocol conversion configuration between a third party service system and a target service system includes:
acquiring protocol request information of an application, a service or a plug-in each third party service system; the protocol request information includes: request address, request protocol, request mode and request data format;
determining a target service system to be accessed by the third party service system and acquiring target protocol information of the target service system, wherein the method comprises the following steps: target address, target protocol, target request mode and target request data format;
and generating protocol conversion configuration between the third party service system and the target service system according to the protocol request information and the target protocol information.
As an embodiment of the present application, according to a protocol conversion configuration corresponding to the original request, performing protocol conversion on the original request based on Spark framework to obtain a target request, including:
acquiring service request data of the original request;
according to the protocol conversion configuration corresponding to the original request, performing protocol conversion on the service request data by using a Spark framework to generate a result data set;
converting the result data set into target format data according to the request format information of the target service system;
and according to the target protocol information in the protocol conversion configuration, assembling the target format data into a target request.
As an embodiment of the present application, the performing protocol conversion on the service request data using a Spark framework, to generate a result data set includes:
judging the data type of the service request data; the data type comprises real-time data and batch data;
if the service request data comprises batch data, processing the batch data by using Spark Job to obtain a batch data set;
if the service request data comprises real-time data, processing the real-time data by using Spark Streaming to generate a small data set;
Based on the bulk data set and the small data set, a result data set is generated.
As an embodiment of the present application, the protocol conversion method further includes:
storing the result dataset in a database;
and querying and modifying the result data set in the database through Spark SQL.
As an embodiment of the present application, after parsing the original request and obtaining the parsing result, the method further includes:
carrying out identity verification on the request address in the analysis result;
if the identity verification is passed, further performing authority check on the request header information in the analysis result;
and if the authority check is passed, carrying out protocol conversion on the original request.
As one embodiment of the present application, sending the target request to a target service system includes:
acquiring at least one downstream service machine of a target service system through a service discovery mechanism;
the target request is sent to all downstream service machines through load balancing.
Based on the same inventive concept, an embodiment of the present application provides a readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the protocol conversion method according to any of the embodiments of the present application.
Based on the same inventive concept, an embodiment of the present application provides an electronic device. The electronic device includes: a memory, a processor and a computer program stored on the memory and executable on the processor, which when executed implements the steps in the protocol conversion method described in any of the embodiments of the present application.
The specific manner in which the various modules perform the operations in the apparatus of the above embodiments have been described in detail in connection with the embodiments of the method, and will not be described in detail herein.
The foregoing description of the preferred embodiments of the present application is not intended to be limiting, but rather is intended to cover any and all modifications, equivalents, alternatives, and improvements within the spirit and principles of the present application.
For the purposes of simplicity of explanation, the methodologies are shown as a series of acts, but one of ordinary skill in the art will recognize that the subject application is not limited by the order of acts described, as some acts may, in accordance with the subject application, occur in other orders or concurrently. Further, those skilled in the art will also appreciate that the embodiments described in the specification are all preferred embodiments and that the acts and components referred to are not necessarily required for the present application.
It will be apparent to those skilled in the art that embodiments of the present application may be provided as a method, apparatus, or computer program product. Accordingly, the present embodiments may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, embodiments of the present application may take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.
Embodiments of the present application are described with reference to flowchart illustrations and/or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
While the preferred embodiments of the present embodiments have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. It is therefore intended that the following claims be interpreted as including the preferred embodiments and all such alterations and modifications as fall within the scope of the embodiments of the present application.
Finally, it is further noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or terminal. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or terminal device comprising the element.
The protocol conversion gateway, the protocol conversion method, the device and the storage medium provided by the application are described in detail, and specific examples are applied to the description of the principle and the implementation of the application, and the description of the above examples is only used for helping to understand the method and the core idea of the application; meanwhile, as those skilled in the art will have modifications in the specific embodiments and application scope in accordance with the ideas of the present application, the present description should not be construed as limiting the present application in view of the above.
Claims (10)
1. The protocol conversion gateway is characterized by comprising a management platform, a configuration center, a reverse proxy module and a protocol conversion module;
the management platform is configured to generate protocol conversion configuration between the third party service system and the target service system and send the protocol conversion configuration to the configuration center;
the configuration center is configured to store protocol conversion configuration of at least one third party service system;
the reverse proxy module is configured to receive an original request sent by a third-party service system and distribute the original request to the protocol conversion module in a routing way;
the protocol conversion module is configured to perform protocol conversion on the original request based on Spark framework according to the protocol conversion configuration corresponding to the original request to obtain a target request, and send the target request to a target service system.
2. The protocol conversion gateway according to claim 1, wherein the management platform is specifically configured to perform the steps of:
acquiring protocol request information of an application, a service or a plug-in each third party service system; the protocol request information includes: request address, request protocol, request mode and request data format;
Determining a target service system to be accessed by the third party service system and acquiring target protocol information of the target service system, wherein the method comprises the following steps: target address, target protocol, target request mode and target request data format;
and generating protocol conversion configuration between the third party service system and the target service system according to the protocol request information and the target protocol information.
3. The protocol conversion gateway according to claim 1, wherein the reverse proxy module is specifically configured to perform the steps of:
receiving an original request sent by a third-party service system;
analyzing the original request to obtain an analysis result, including: request address, request header information, and service request data;
storing the analysis result;
and acquiring protocol conversion configuration corresponding to the original request from the configuration center, and transmitting the service request data and the protocol conversion configuration to the protocol conversion module.
4. A protocol conversion gateway according to claim 3, characterized in that the protocol conversion module is specifically configured to perform the following steps:
acquiring service request data of the original request;
According to the protocol conversion configuration corresponding to the original request, performing protocol conversion on the service request data by using a Spark framework to generate a result data set;
converting the result data set into target format data according to the request format information of the target service system;
according to the target protocol information in the protocol conversion configuration, the target format data is assembled into a target request;
and sending the target request to the target service system.
5. The protocol conversion gateway according to claim 4, wherein the protocol conversion module is specifically configured to perform the steps of:
judging the data type of the service request data; the data type comprises real-time data and batch data;
if the service request data comprises batch data, processing the batch data by using Spark Job to obtain a batch data set;
if the service request data comprises real-time data, processing the real-time data by using Spark Streaming to generate a small data set;
based on the bulk data set and the small data set, a result data set is generated.
6. The protocol conversion gateway according to claim 4, further comprising a query module;
The protocol conversion module is further configured to store the result dataset in a database;
the query module is configured to query and modify the result data set in the database through Spark SQL.
7. A protocol conversion gateway according to claim 3, further comprising an authentication module;
the authentication module is configured to read a request address of the original request and perform identity verification on the request address;
if the identity verification is passed, further performing authority check on the request header information of the original request;
and the reverse proxy module is further configured to send the service request data and the protocol conversion configuration to the protocol conversion module if the authority check passes.
8. A protocol conversion method, applied to the protocol conversion gateway of any one of claims 1 to 7, comprising:
setting protocol conversion configuration between a third party service system and a target service system; the protocol conversion configuration comprises a request address, a request protocol, a request mode and a request data format of the third party service system, and a target address, a target protocol, a target request mode and a target request data format of the target service system;
When an original request sent by a third-party service platform is received, analyzing the original request to obtain an analysis result; the analysis result comprises: request address, request header information, and service request data;
according to the protocol conversion configuration corresponding to the original request, carrying out protocol conversion on the original request based on a Spark framework to obtain a target request;
and sending the target request to a target service system.
9. A computer readable storage medium, on which a computer program is stored which, when being executed by a processor, implements the steps of the method as claimed in claim 8.
10. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor performs the steps in the method of claim 8 when the computer program is executed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311866517.4A CN117834346A (en) | 2023-12-29 | 2023-12-29 | Protocol conversion gateway, protocol conversion method, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311866517.4A CN117834346A (en) | 2023-12-29 | 2023-12-29 | Protocol conversion gateway, protocol conversion method, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117834346A true CN117834346A (en) | 2024-04-05 |
Family
ID=90522593
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311866517.4A Pending CN117834346A (en) | 2023-12-29 | 2023-12-29 | Protocol conversion gateway, protocol conversion method, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117834346A (en) |
-
2023
- 2023-12-29 CN CN202311866517.4A patent/CN117834346A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP6912469B2 (en) | Access request conversion method and device | |
| CN112333227B (en) | Internet of things platform of mixed type protocol based on rule engine | |
| US11457082B2 (en) | Service processing method and apparatus of service-based architecture | |
| CN109510846B (en) | API calling system, method and device, electronic equipment and storage medium | |
| US9251211B2 (en) | Generation of a query plan for accessing a database | |
| CN112613010A (en) | Authentication service method, device, server and authentication service system | |
| CN114025021B (en) | Communication method, system, medium and electronic equipment crossing Kubernetes cluster | |
| US20120240184A1 (en) | System and method for on the fly protocol conversion in obtaining policy enforcement information | |
| CN101729491B (en) | Method, device and system for enhancing application reliability of script-based business | |
| CN113573303B (en) | Method and device for determining edge application server | |
| US9742835B2 (en) | System and method for backend control of frontend user interfaces | |
| CN114205191B (en) | API gateway system and operation method | |
| CN107241254B (en) | Network connection device, network system and networking method | |
| CN101808051A (en) | Application integration gateway and control method thereof | |
| CN116802628A (en) | Data management and query method and device, service grid system and computing equipment | |
| CN114650281B (en) | File downloading method, device, equipment and storage medium based on complex network | |
| CN117834346A (en) | Protocol conversion gateway, protocol conversion method, equipment and storage medium | |
| CN116700705A (en) | Service system docking method and system based on configuration | |
| CN113676483B (en) | Multi-block chain access management method and system and capacity opening device thereof | |
| CN115202800A (en) | Edge cloud service data processing method and device, computer equipment and storage medium | |
| CN111988269A (en) | Policy management system providing authorization information via distributed data stores | |
| US20230412466A1 (en) | Managing cloud-native virtual network functions | |
| TW201828093A (en) | Visit request conversion method and device that identifies a target service type of a visit request and breaks down the visit request to a data structure corresponding to the target service type to be supplied to a corresponding server | |
| CN112788341B (en) | Video information processing method, multimedia information processing method, device and electronic equipment | |
| CN115473881A (en) | Gateway service method, device and system based on cross-protocol communication |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |