CN116048991A - Construction method and device of multi-source data service REST API interface - Google Patents

Abstract

The invention discloses a method and a device for constructing a multi-source data service REST API interface. The method comprises the following steps: 1) Performing verification test on a REST API interface defined by a user, and if the verification test passes, performing configuration and online test on the interface; 2) Receiving configuration of an interface by a user; 3) Detecting whether the data source configured by the interface is registered or not, and if not, registering the data source; 4) Detecting whether the interface relates to a custom parameter conversion function, and if so, prompting a user to register; 5) According to the interface parameter information and the related data, query SQL statement information, and convert the query request into SQL query statements; 6) Accessing a corresponding data source through an interface to acquire a query result; 7) Packaging the query result whether a custom result conversion function is involved or not, and prompting a user to register if the custom result conversion function is involved and not registered; 8) And putting the REST API interface after the test on line.

Description

Construction method and device of multi-source data service REST API interface

Technical Field

The invention belongs to the technical field of network security, and particularly relates to an automatic construction method and device for a multi-source heterogeneous big data service REST API interface.

Background

When large-scale systems such as a network security situation awareness platform and the like are built, a layered system architecture is generally adopted, an independent data center is abstracted, a data center is built, data assets are deposited, data services are provided for a plurality of business systems on the upper layer through one large data center, and building intensification, data centralization and service standardization are achieved. Under the architecture, the upper business system has a large number of data service demands in the form of REST APIs, and the data platform layer is required to provide various, customizable, flexible and changeable data service REST API interfaces with quick response.

The core ideas of the existing common solutions can be summarized as follows:

(1) The data platform and the service system communicate offline, and the contents such as interface names, parameter information, return information and the like of the REST API are reserved;

(2) The data platform customizes the development interface based on a Web Service development framework such as spring, springMVC, SSH and the like according to the agreed interface description information;

(3) After the custom interface developed by the data platform passes the debugging, the interface is directly exposed to an upper-layer service system or managed through an interface management gateway.

The existing solution needs to customize each data service REST API interface as required, has long interface development and update period, low efficiency, and can not quickly respond to the requirements of multiple customized data services of multiple business systems. In addition, existing solutions are typically based on specific relational structured databases (MySQL, oracle, etc.), and cannot be used

Therefore, the invention mainly designs an automatic construction technology for the multi-source heterogeneous big data service REST API interface, supports the automatic construction of the data service REST API interface and shortens the time for generating and updating the interface.

Disclosure of Invention

Aiming at the problems existing in the prior art, the invention aims to provide an automatic construction method and device for a multi-source heterogeneous big data service REST API interface.

The technical scheme of the invention is as follows:

a multi-source data service REST API interface construction method includes the steps:

1) The interface service module performs verification test on the availability and the correctness of the returned result of the REST API interface defined by the user, and if the verification test passes, the REST API interface is configured and tested on line through a visual interface;

2) An interface management module receives the configuration of the REST API interface by a user; the configuration information comprises interface parameter information and associated data query SQL statement information; the interface parameter information comprises called API names, API paths, API parameters and data sources;

3) The interface management module detects whether a data source configured by the REST API interface is registered or not, if not, the data source registration is carried out through the visual interface, when registering, a data source name, a data source type, a data source access address, a user name, a password and a JDBC driving JAR packet are transmitted to the data source JDBC management sub-module, a JDBC data source connector is registered, and a corresponding dependent packet xxx-JDBC-connection. JAR is uploaded;

4) The interface management module detects whether the REST API interface relates to a custom parameter conversion function, if so, whether the custom parameter conversion function is registered or not is detected, and if not, a user is prompted to register the custom parameter conversion function;

5) The interface management module queries SQL statement information according to the REST API interface parameter information and the associated data, converts a query request received by the REST API interface into an SQL query statement, and sends the SQL query statement to the data query module;

6) The data query module accesses a corresponding data source through a JDBC data source connector corresponding to the REST API interface, acquires a query result from the data source according to the SQL query statement and sends the query result to the data encapsulation module;

7) The data packaging module detects whether the query result is packaged or not and judges whether the self-defined result conversion function is registered or not if the self-defined result conversion function is related to the self-defined result conversion function, and prompts a user to register the self-defined result conversion function if the self-defined result conversion function is not registered;

8) And the interface service module is used for loading the REST API interface after completing configuration and testing.

Further, the interface service module includes a REST API interface import and export sub-module, configured to export and convert a REST API interface in a multi-source data service REST API interface automation construction device into a JSON file, and then import the JSON file into another multi-source data service REST API interface automation construction device.

Further, the interface service module comprises a REST API interface authentication sub-module and a REST API interface service sub-module; the REST API interface authentication submodule is used for providing access account authentication and authorization functions, authorizing an access account before the REST API interface externally provides service, and ensuring that only the authorized account can have authority to access the appointed data service REST API interface; the REST API interface service sub-module is configured to support access to the REST API interface by the user account with authority.

Further, the interface service module further comprises a REST API interface monitoring sub-module, configured to provide monitoring and statistics for the number of access times and the returned result data size of the REST API interface.

Further, the interface service module further comprises a REST API interface on-line sub-module and a REST API interface off-line sub-module; the REST API interface online submodule is used for providing a visual interface to support online operation of the REST API interface; and the REST API interface offline sub-module is used for offline putting the REST API interface which does not provide service to the outside.

Further, the interface management module comprises a REST API interface result conversion function definition sub-module, which is used for supporting the custom registration extension to utilize the custom result conversion function written in the JAVA language, and generating the custom conversion function by inheriting the Transform () method of the resultltTransformPlugin class; when registering the custom transfer function, compiling the resultlform plug class in the custom transfer function, packaging the resultlform plug class into a JAR package, and registering the name, the JAR package and the class name of the custom result transfer function.

Further, the method for obtaining the query result by the data query module comprises the following steps:

11 A JDBC connection pool of a resident memory is established;

12 A JDBC connection pool knowledge base is established and maintained, and the JDBC connection pool knowledge base maintains a JDBC connection pool handle associated with each data source capable of providing data service to the outside and is used for automatically selecting JDBC connection matched with a target data source through knowledge comparison;

13 Analyzing the received SQL query statement to obtain a queried data source list;

14 Based on the data source list, carrying out knowledge matching query on the JDBC connection pool knowledge base, selecting a JDBC connection pool conforming to a target data source and issuing SQL sentences;

15 Monitoring the execution state of the SQL sentence, and collecting the SQL execution result to obtain the query result.

Further, the result packaging module packages the query result into Response information in the JSON format of the REST API according to the agreed parameter information of the REST API; when packaging is carried out, each return record corresponds to one element number in the JSON array; the different fields of each record map to agreed Response KEY information.

Further, in the online application stage, the data packaging module detects whether the query result is packaged or not and directly packages the query result if the query result does not relate to the custom result conversion function; otherwise, the query result is packaged by using the registered self-defined result conversion function.

The multi-source data service REST API interface construction device is characterized by comprising an interface service module, an interface management module, a data query module and a data encapsulation module;

the interface service module is used for carrying out verification test on the availability and the correctness of the returned result of the REST API interface defined by the user, and if the verification test passes, the REST API interface is configured and tested on line through the visual interface; and the REST API interface after configuration and testing is put on line;

the interface management module is used for receiving the configuration of the REST API interface by a user; the configuration information comprises interface parameter information and associated data query SQL statement information; the interface parameter information comprises called API names, API paths, API parameters and data sources;

the interface management module is used for detecting whether a data source configured for the REST API interface is registered or not, if not, the data source registration is carried out through the visual interface, when registering, the data source name, the data source type, the data source access address, the user name, the password and the JDBC driving JAR are packaged to the data source JDBC management sub-module, a JDBC data source connector is registered, and a corresponding dependency package xxx-JDBC-connection. JAR is uploaded; detecting whether the REST API interface relates to a custom parameter conversion function, if so, detecting whether the custom parameter conversion function is registered, and if not, prompting a user to register the custom parameter conversion function; according to the REST API parameter information and the related data query SQL statement information, the query request received by the REST API is converted into an SQL query statement, and the SQL query statement is sent to a data query module;

the data query module is used for accessing a corresponding data source through a JDBC data source connector corresponding to the REST API interface, acquiring a query result from the data source according to the SQL query statement and sending the query result to the data encapsulation module;

the data packaging module is used for detecting whether the query result is packaged and relates to a self-defined result conversion function, if so, judging whether the self-defined result conversion function is registered, and if not, prompting a user to register the self-defined result conversion function.

The invention has the following advantages:

1. the system can be used after being opened, the dependence items are few, and the system can be operated only by java and node environments.

2. Multiple engine data source support, data sources providing services including MYSQL, postgreSQL, hive, elasticsearch, clinkHouse, etc., support dynamic expansion of the data source engine.

3. The mass heterogeneous data source is supported, the service data source format comprises, but is not limited to, CSV, JSON, syslog, full text and the like, the service data source size can reach trillion levels, and the data size is supported to be horizontally expanded.

4. And the SQL statement acquired data is supported to directly generate a data service REST API interface, and heterogeneous data differences of multiple engine data sources are shielded.

5. And supporting a custom API parameter conversion function plug-in and a query result conversion function plug-in, and dynamically loading the plug-in to take effect immediately by uploading corresponding plug-in files.

6. The standard-based JDBC interface is realized, has good expansibility and supports service data source expansion.

7. The SQL grammar is expanded, and the support of the SQL grammar on the data service REST API interface parameter information and the custom function is enriched; and designing and realizing an SQL statement automatic analysis and conversion framework based on parameters, supporting the query condition and result return information of dynamically receiving parameter information and converting the parameter information into the SQL statement.

Drawings

Fig. 1 is a frame diagram of the device of the present invention.

FIG. 2 is a flow chart of the method of the present invention

Detailed Description

The invention will now be described in further detail with reference to the accompanying drawings, which are given by way of illustration only and are not intended to limit the scope of the invention.

Fig. 1 is a frame diagram of an automatic data service interface construction device according to the present invention, which mainly includes 5 parts of interface service, interface management, result response, data query, and data source. The function design of each module is as follows:

1. the interface service module provides visual data service REST API interface import and export, REST API interface online and offline, REST API interface authentication, REST API interface test, REST API interface service, REST API interface monitoring and other functions.

The REST API interface import and export submodule supports import and export of a predefined REST API interface through a JSON format template; partial or all data service REST API interfaces in the automatic construction device supporting the multi-source data service REST API interface are exported into a JSON file according to a specified template, and the JSON file is imported into the automatic construction device supporting the multi-source data service REST API interface, so that the requirements of one-time construction and cross-device rapid migration of the data service REST API interface are met;

the REST API interface online submodule provides a visual interface to support online operation of the REST API interface formed by modes of manually defining the introduction of the REST API interface and the like; the REST API interface offline sub-module can offline the REST API interface which does not provide service to the outside any more; the REST API interface which is not on line or off line does not provide service to the outside;

the REST API interface testing submodule provides support for verification testing on the usability, the correctness of returned results and the like of the REST API interface, and the predefined REST API interface is subjected to online operation after the verification testing is passed, so that the REST API interface is ensured to meet the requirements;

the REST API interface authentication submodule provides the functions of access account authentication, authorization and the like, and authorizes the user account before the REST API interface provides service to the outside, so that only the authorized account can access the appointed data service REST API interface with permission, and the data security is ensured;

the REST API interface service submodule supports the user account with the authority to access the constructed data service REST API interface;

and 6. The REST API interface monitoring module is used for monitoring and counting the access times of the REST API interface, the scale of returned result data and the like.

2. The interface management module provides functions of data source JDBC management, REST API parameter conversion function definition, REST API result conversion function definition, REST API basic information definition, REST API SQL definition, REST API converter and the like.

1. The data source JDBC management sub-module provides data source registration and management functions, is internally provided with MYSQL, postgreSQL,

Hive, elasticsearch, clinkHouse, and the like, supports registering a new data storage source JDBC through a visual interface and uploading a dependent package, wherein registration information comprises a data source name, a data source type, a data source access address, a user name, a password and a JDBC driver JAR package.

The REST API parameter conversion function definition submodule supports custom functions written by JAVA language for extension, the custom functions can be obtained by inheriting a Transform () method of a paramtrans form plug in class, when registering, the classes are compiled and packed into JAR packages to be uploaded, and a plurality of custom functions can be packed in one same JAR package. When registering, the definition parameter is selected to be converted into a custom function name, a JAR package and a class name.

The REST API interface result conversion function definition submodule supports the custom result conversion function which is written by using JAVA language and is extended, the custom function only needs to inherit a Transform () method of a resultTransformPlugin class, when registering, the class is compiled and packed into a JAR package to be uploaded, and a plurality of custom functions can be packed in the same JAR package. When registering, the definition result is selected to be converted into the custom function name, JAR package and class name.

The REST API interface basic information definition mainly takes charge of the REST API interface definition of the visual mode, and specifically comprises the following steps:

information such as API name, API path, API parameters, belonging packets, data source, etc.

The REST API interface SQL definition submodule supports SQL sentences corresponding to the self-defined REST API interface data return result screening logic, wherein the SQL sentences comprise SELECT clauses, FROM clauses, WHERE clauses and GROUPBY clauses;

and 6, the REST API interface conversion sub-module analyzes the actual parameters of the REST API request issued by the interface access module, and translates the actual REST API request into a standardized SQL statement which can be sunk to the data query module by associating and utilizing basic definition information, SQL information, a custom parameter conversion function and a custom result conversion function of the REST API interface.

3. The data query module provides a JDBC Connector adapter and maintains a JDBC query connection knowledge base of each data source; the JDBC Connector adapter may select a matching JDBC connection for a standardized SQL statement that the interface management module sinks. The method comprises the following steps:

1. the JDBC connection pool of the resident memory is built, the time consumption for building the JDBC connection is reduced, and the efficiency is improved;

2. establishing and maintaining a JDBC connection pool knowledge base which maintains a JDBC connection pool handle associated with each data source capable of providing data service externally, and automatically selecting JDBC connection matched with a target data source through knowledge comparison;

3. receiving and analyzing a standardized SQL query statement to obtain a data source list corresponding to the SQL query;

4. based on the knowledge base of the JDBC connection pool, carrying out knowledge matching inquiry, selecting the JDBC connection pool conforming to the target data source and issuing SQL sentences;

5. and monitoring the SQL execution state, and collecting the SQL execution result and returning the SQL execution result to the result response module in a result set mode.

4. And the result packaging module packages the JDBC query result returned by the data query module into Response information in the JSON format of the REST API according to the agreed parameter information of the REST API. When packaging is carried out, each return record corresponds to one element number in the JSON array; the different fields of each record map to agreed Response KEY information. The result response module provides functions such as query result encapsulation, query result filtering, query result statistics and the like. Wherein, the liquid crystal display device comprises a liquid crystal display device,

1. the query result packaging submodule converts the returned result set statement into the data of the response of the REST API interface as a JSON format, packages and constructs the msg module at the same time, and returns the execution state information of the REST API interface;

2. the query result filtering sub-module constructs filtering conditions (such as empty string filtering) by utilizing a result conversion function, and screens data conforming to the filtering conditions;

3. the query result statistics submodule uses the result conversion function to count the data record number of the returned result.

5. The data source module is responsible for storing and managing the data source to be queried, receiving the query request submitted by the data query module, completing the actual query operation, and returning the result to the data query module through the standardized JDBC interface after the query is finished.

Fig. 2 is a flow chart of REST API interface construction of the automatic data service interface construction device according to the present invention, and the specific construction steps are as follows:

1. judging whether a data source corresponding to the RST API interface for providing the service is registered or not, if not, registering the data source through a visual interface, inputting information such as a data source name, a data source type, a data source access address, a user name, a password and the like to a data source JDBC management submodule during registration, registering a JDBC data source connector, and uploading corresponding xxx-JDBC-connection.

2. Judging whether the REST API interface of the service is related to the custom parameter conversion function, if so, judging whether the related custom parameter conversion function is registered, and if not, finishing development, test and registration of the custom parameter conversion function in an off-line mode;

rest API interface basic information definition: configuring information such as names, paths, parameters, belonging groups, data sources and the like of the REST API interface through a visual interface;

rest API interface SQL definition: configuring REST API interface associated data query SQL statement information through a visual interface;

5. the automatic data service interface construction device automatically converts the information into query sentences according to the REST API interface parameter information and the SQL information, and sinks to the corresponding data source query module to complete the query and return the result;

6. and (5) packaging a query result: if the query result encapsulation does not involve the custom result conversion function, the data service interface automatic construction device automatically completes the query result encapsulation; otherwise, judging whether the self-defined result conversion function is registered or not, if not, completing development, testing and registration of the self-defined result conversion function in an off-line manner;

7. data services REST API interface testing, online and authorization.

In order to realize conversion and mapping of data service REST API interface parameters and SQL sentences, the invention extends based on standard SQL grammar, so that parameter information configured when REST API interface definition is supported in the SQL sentences. The conditional statement corresponding to keywords such as SELECT, WHERE, GROUP BY and the like in the SQL statement is supported to be configured based on parameters.

Although specific embodiments of the invention have been disclosed for illustrative purposes, it will be appreciated by those skilled in the art that the invention may be implemented with the help of a variety of examples: various alternatives, variations and modifications are possible without departing from the spirit and scope of the invention and the appended claims. Therefore, it is intended that the invention not be limited to the particular embodiment disclosed as the best mode contemplated for carrying out this invention, but that the invention will have the scope indicated by the scope of the appended claims.

Claims (10)

1. A multi-source data service REST API interface construction method includes the steps:

1) The interface service module performs verification test on the availability and the correctness of the returned result of the REST API interface defined by the user, and if the verification test passes, the REST API interface is configured and tested on line;

2) An interface management module receives the configuration of the REST API interface by a user; the configuration information comprises interface parameter information and associated data query SQL statement information; the interface parameter information comprises called API names, API paths, API parameters and data sources;

3) The interface management module detects whether a data source configured by the REST API interface is registered or not, if not, the data source registration is carried out through the visual interface, when registering, a data source name, a data source type, a data source access address, a user name, a password and a JDBC driving JAR packet are transmitted to the data source JDBC management sub-module, a JDBC data source connector is registered, and a corresponding dependent packet xxx-JDBC-connection. JAR is uploaded;

4) The interface management module detects whether the REST API interface relates to a custom parameter conversion function, if so, whether the custom parameter conversion function is registered or not is detected, and if not, a user is prompted to register the custom parameter conversion function;

5) The interface management module queries SQL statement information according to the REST API interface parameter information and the associated data, converts a query request received by the REST API interface into an SQL query statement, and sends the SQL query statement to the data query module;

6) The data query module accesses a corresponding data source through a JDBC data source connector corresponding to the REST API interface, acquires a query result from the data source according to the SQL query statement and sends the query result to the data encapsulation module;

7) The data packaging module detects whether the query result is packaged or not and judges whether the self-defined result conversion function is registered or not if the self-defined result conversion function is related to the self-defined result conversion function, and prompts a user to register the self-defined result conversion function if the self-defined result conversion function is not registered;

8) And the interface service module is used for loading the REST API interface after completing configuration and testing.

2. The method of claim 1, wherein the interface service module comprises a REST API interface import export sub-module for exporting and converting REST API interfaces in a multi-source data service REST API interface automation construction device to JSON files, which are then imported into another multi-source data service REST API interface automation construction device.

3. The method of claim 1, wherein the interface service module comprises a REST API interface authentication sub-module and a REST API interface service sub-module; the REST API interface authentication submodule is used for providing access account authentication and authorization functions, authorizing an access account before the REST API interface externally provides service, and ensuring that only the authorized account can have authority to access the appointed data service REST API interface; the REST API interface service sub-module is configured to support access to the REST API interface by the user account with authority.

4. The method of claim 1, wherein the interface service module further comprises a REST API interface monitoring sub-module for providing monitoring and statistics of the number of accesses to the REST API interface and the size of the returned result data.

5. The method of claim 1, wherein the interface service module further comprises a REST API interface on-line sub-module, a REST API interface off-line sub-module; the REST API interface online submodule is used for providing a visual interface to support online operation of the REST API interface; and the REST API interface offline sub-module is used for offline putting the REST API interface which does not provide service to the outside.

6. The method according to claim 1, wherein the interface management module includes a REST API interface result conversion function definition sub-module for supporting a custom registration extension to generate a custom conversion function by inheriting a Transform () method of a resultlform plug class using a custom result conversion function written in JAVA language; when registering the custom transfer function, compiling the resultlform plug class in the custom transfer function, packaging the resultlform plug class into a JAR package, and registering the name, the JAR package and the class name of the custom result transfer function.

7. The method of claim 1, wherein the method for obtaining the query result by the data query module is as follows:

11 A JDBC connection pool of a resident memory is established;

12 A JDBC connection pool knowledge base is established and maintained, and the JDBC connection pool knowledge base maintains a JDBC connection pool handle associated with each data source capable of providing data service to the outside and is used for automatically selecting JDBC connection matched with a target data source through knowledge comparison;

13 Analyzing the received SQL query statement to obtain a queried data source list;

14 Based on the data source list, carrying out knowledge matching query on the JDBC connection pool knowledge base, selecting a JDBC connection pool conforming to a target data source and issuing SQL sentences;

15 Monitoring the execution state of the SQL sentence, and collecting the SQL execution result to obtain the query result.

8. The method according to claim 1 or 7, wherein the result packaging module packages the query result into Response information in JSON format of REST API according to the agreed REST API parameter information; when packaging is carried out, each return record corresponds to one element number in the JSON array; the different fields of each record map to agreed Response KEY information.

9. The method of claim 1, wherein the data encapsulation module detects whether the encapsulation of the query result involves a custom result conversion function at the online application stage, and if not, encapsulates the query result directly; otherwise, the query result is packaged by using the registered self-defined result conversion function.

10. The multi-source data service REST API interface construction device is characterized by comprising an interface service module, an interface management module, a data query module and a data encapsulation module;

the interface service module is used for carrying out verification test on the availability and the correctness of the returned result of the REST API interface defined by the user, and if the verification test passes, the REST API interface is configured and tested on line through the visual interface; and the REST API interface after configuration and testing is put on line;

the interface management module is used for receiving the configuration of the REST API interface by a user; the configuration information comprises interface parameter information and associated data query SQL statement information; the interface parameter information comprises called API names, API paths, API parameters and data sources;

the interface management module is used for detecting whether a data source configured for the REST API interface is registered or not, if not, the data source registration is carried out through the visual interface, when registering, the data source name, the data source type, the data source access address, the user name, the password and the JDBC driving JAR are packaged to the data source JDBC management sub-module, a JDBC data source connector is registered, and a corresponding dependency package xxx-JDBC-connection. JAR is uploaded; detecting whether the REST API interface relates to a custom parameter conversion function, if so, detecting whether the custom parameter conversion function is registered, and if not, prompting a user to register the custom parameter conversion function; according to the REST API parameter information and the related data query SQL statement information, the query request received by the REST API is converted into an SQL query statement, and the SQL query statement is sent to a data query module;

the data query module is used for accessing a corresponding data source through a JDBC data source connector corresponding to the REST API interface, acquiring a query result from the data source according to the SQL query statement and sending the query result to the data encapsulation module;

the data packaging module is used for detecting whether the query result is packaged and relates to a self-defined result conversion function, if so, judging whether the self-defined result conversion function is registered, and if not, prompting a user to register the self-defined result conversion function.

Images