Disclosure of Invention
In view of the above, the present invention aims to overcome the deficiencies of the prior art and provide a pre data source hot loading method based on MQ.
The invention provides a presto data source hot loading method based on MQ, comprising the following steps:
step S1: constructing a redis client, and initializing presto service through a redis queue;
step S2: acquiring data source configuration information from a redis client by monitoring a redis queue, and identifying a corresponding operation type according to the acquired data source configuration information;
step S3: calling a corresponding method of the ConnectorManager class file according to the identified operation type to operate the data source;
step S4: the presto service is deployed, and the operated data sources are verified.
As a preferred aspect of the present invention, step S1 of the pre data source hot loading method based on MQ of the present invention comprises:
step S11: creating a configuration file in a presto service, writing redis configuration information in the created configuration file, adding maven dependence of a redis client in a presto-main module, writing a redis client class, reading the redis configuration information of the written configuration file, and constructing a redis client;
step S12: creating a redis queue, storing the total data source configuration information in the service system in a redis client through the created redis queue, sending the changed data source name in the service system to presto service, and updating the changed data source connection information to the redis client;
step S13: creating a catalogMap in prestoc service, storing the data source loaded by the current prestoc service, initializing prestoc service, acquiring the full data source configuration information from a redis client, and serializing the acquired full data source configuration information;
step S14: and sequentially traversing and reading each data source according to the serialized data source configuration information, loading the read data sources into a presto-server, and simultaneously storing the read data sources in the created catalogMap.
As a preferred embodiment of the present invention, in step S11 of the pre data source hot loading method based on MQ, a configuration file is created inside the pre service, and redis configuration information is written in the created configuration file, which includes: creating a source.properties file under the pre-internal etc/directory, and writing the configuration information of redis in the created source.properties file in a key-value mode.
As a preferred aspect of the present invention, step S3 of the pre data source hot loading method based on MQ of the present invention comprises: and constructing an attribute object of the data source according to the identified operation type, and calling a corresponding method of the connectionmanager class file to operate the data source.
As a preferred aspect of the present invention, step S3 of the pre data source hot loading method based on MQ of the present invention comprises: when the identified operation type is newly added, calling a connector factor file to create a connector object according to connection information, a data source name and data source type information in the data source configuration information, generating a catalyst object according to the created connector object, registering the generated catalyst object to a catalyst manager through the catalyst manager file to finish the addition of the data source, advertising the whole through the Announcer file, and updating a connector id set.
As a preferred aspect of the present invention, step S3 of the pre data source hot loading method based on MQ of the present invention comprises: when the identified operation type is deletion, removing the registered catalogName data source name through a CatalogManager manager, removing metadata, authority control data, index data, schema mode configuration, table configuration and column configuration according to a connector id, advertising the global through an Announcer file, and updating the connector id set.
As a preferred aspect of the present invention, step S3 of the pre data source hot loading method based on MQ of the present invention comprises: when the identified operation type is modification, deleting the data source, and then adding the data source according to the modification requirement.
As a preferred aspect of the present invention, step S3 of the pre data source hot loading method based on MQ of the present invention comprises:
step S31: compiling a presto-main.jar file, and replacing the jar file under the corresponding directory with the compiled presto-main.jar file when presto service is deployed;
step S32: starting prest-server service, sending change information of a data source through a redis queue, and connecting the prest-server service by using a prest-cli client or prest-jdbc in a non-restarting mode;
step S33: and verifying whether the operated data source has been dynamically updated by executing the SQL statement.
Next, the present invention provides a computer readable storage medium having stored thereon a computer program which, when executed, performs the MQ-based presto data source hot-loading method described above.
Finally, the invention provides a computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the MQ-based presto data source hot loading method described above when executing the program.
The presto data source hot loading method based on the MQ has the following beneficial effects:
1. the data sources in the presto service comprise addition, deletion and modification updating, the real-time effective updating is realized, the updating is synchronously carried out on each node, the ductility and the flexibility of presto are improved, and the delay time is reduced.
2. And a manual creating of a category configuration file is not needed, so that the stability of the data source updating process and the system is remarkably improved.
3. The presto service is not required to be restarted, so that the development and debugging efficiency can be improved, and the operation and maintenance cost can be saved.
4. When the presto service is abnormally down or upgraded and deployed, the data cannot be lost after the presto service is started, all data sources before the down can be recovered, the operation and maintenance cost is saved, and the working efficiency is improved.
Detailed Description
Embodiments of the present invention will be described in detail below with reference to the accompanying drawings.
It should be noted that, without conflict, the following embodiments and features in the embodiments may be combined with each other; and, based on the embodiments in this disclosure, all other embodiments that may be made by one of ordinary skill in the art without inventive effort are within the scope of the present disclosure.
It is noted that various aspects of the embodiments are described below within the scope of the following claims. It should be apparent that the aspects described herein may be embodied in a wide variety of forms and that any specific structure and/or function described herein is merely illustrative. Based on the present disclosure, one skilled in the art will appreciate that one aspect described herein may be implemented independently of any other aspect, and that two or more of these aspects may be combined in various ways. For example, an apparatus may be implemented and/or a method practiced using any number of the aspects set forth herein. In addition, such apparatus may be implemented and/or such methods practiced using other structure and/or functionality in addition to one or more of the aspects set forth herein.
The terms involved in the following examples are explained as follows:
MQ: the Message Queue is used for Message notification and dynamic monitoring.
presto: the Facebook open-source distributed SQL query engine is used for interactively analyzing the query and cross-library query.
redis: the memory-based non-relational K-V database is mainly used as a database, a cache and a message middleware.
ConnectorManager class files: connector management class + metadata for managing data sources, partitions, indexes, configuration information, etc.
presto-main module: the presto starting module is used for the main program entrance of the server and the program initialization.
maven dependence: the project management tool is an object model for managing projects.
presto-server: presto server for executing cross-library inquiry and analyzing inquiry.
catalogMap: the data source set is identical to the set of all the current data sources.
source. Properties file: and the MQ and other configuration files are used for storing the configuration files and can be read when the device is started.
ConnectorFactory class file: connector factory class files for managing factory classes for all data source connectors.
connector object: connector objects, one category corresponding to one connector object.
Catalog object: a data source, one data source is a catalyst object.
CatalogManager class files: the data source defines a class file and performs a pruned management class on the data source.
Announcer class File: and the notification class is used for discovering the data source and triggering the update of the data source.
presto-cli client: presto client end is in use for connecting presto service, and submitting task.
presto-jdbc: the presto connects url, is used for connecting presto service through JDBC, submit the task.
FIG. 1 is a flow chart of a method for MQ-based prest data source hot loading in accordance with an exemplary first embodiment of the present invention. As shown in fig. 1, the method of the present embodiment is implemented as follows:
step S1: constructing a redis client, and initializing presto service through a redis queue;
step S2: acquiring data source configuration information from a redis client by monitoring a redis queue, and identifying a corresponding operation type according to the acquired data source configuration information;
step S3: calling a corresponding method of the ConnectorManager class file according to the identified operation type to operate the data source;
step S4: the presto service is deployed, and the operated data sources are verified.
The method of the embodiment can realize the update of the data sources including addition, deletion and modification in the prestock service, realize the real-time effectiveness of the update, synchronously update to each node, improve the ductility and flexibility of prestock and reduce the delay time. And a manual creating of a category configuration file is not needed, so that the stability of the data source updating process and the system is remarkably improved.
FIG. 2 is a flow chart of a method for hot loading of an MQ-based prest data source according to a second exemplary embodiment of the present invention, which is a preferred embodiment of the method of FIG. 1, as shown in FIG. 2, the step S1 of the method of the present embodiment is implemented as follows:
step S11: creating a configuration file in a presto service, writing redis configuration information in the created configuration file, adding maven dependence of a redis client in a presto-main module, writing a redis client class, reading the redis configuration information of the written configuration file, and constructing a redis client;
step S12: creating a redis queue, storing the total data source configuration information in the service system in a redis client through the created redis queue, sending the changed data source name in the service system to presto service, and updating the changed data source connection information to the redis client;
step S13: creating a catalogMap in prestoc service, storing the data source loaded by the current prestoc service, initializing prestoc service, acquiring the full data source configuration information from a redis client, and serializing the acquired full data source configuration information;
step S14: and sequentially traversing and reading each data source according to the serialized data source configuration information, loading the read data sources into a presto-server, and simultaneously storing the read data sources in the created catalogMap.
In practical application, in step S11 of the method of the present embodiment, a configuration file is created inside the presto service, and redis configuration information is written in the created configuration file, which is implemented in the following manner: creating a source.properties file under the pre-internal etc/directory, and writing the configuration information of redis in the created source.properties file in a key-value mode.
An exemplary third embodiment of the present invention provides a pre data source hot loading method based on MQ, which is a preferred embodiment of the method shown in fig. 1, and step S3 of the method of the present embodiment includes: and constructing an attribute object of the data source according to the identified operation type, and calling a corresponding method of the connectionmanager class file to operate the data source.
In practical application, step S3 of the method of the present embodiment is implemented as follows:
when the identified operation type is newly added, calling a connector factor file to create a connector object according to connection information, a data source name and data source type information in the data source configuration information, generating a catalyst object according to the created connector object, registering the generated catalyst object to a catalyst manager through the catalyst manager file to finish the addition of the data source, advertising the whole through the Announcer file, and updating a connector id set.
When the identified operation type is deletion, removing the registered catalogName data source name through a CatalogManager manager, removing metadata, authority control data, index data, schema mode configuration, table configuration and column configuration according to a connector id, advertising the global through an Announcer file, and updating the connector id set.
When the identified operation type is modification, deleting the data source, and then adding the data source according to the modification requirement.
FIG. 3 is a flowchart of a pre data source hot loading method based on MQ according to a fourth embodiment of the present invention, as shown in FIG. 3, and step S3 of the method of the present embodiment is implemented as follows:
step S31: compiling a presto-main.jar file, and replacing the jar file under the corresponding directory with the compiled presto-main.jar file when presto service is deployed;
step S32: starting prest-server service, sending change information of a data source through a redis queue, and connecting the prest-server service by using a prest-cli client or prest-jdbc in a non-restarting mode;
step S33: and verifying whether the operated data source has been dynamically updated by executing the SQL statement.
The method of the embodiment does not need to restart presto service, can improve development and debugging efficiency and saves operation and maintenance cost. When the presto service is abnormally down or upgraded and deployed, the data cannot be lost after the presto service is started, all data sources before the down can be recovered, the operation and maintenance cost is saved, and the working efficiency is improved.
The present invention provides a computer readable storage medium having stored thereon a computer program which, when executed, performs the MQ-based presto data source hot-loading method of the present invention described above.
As shown in fig. 4, the present invention also provides an apparatus comprising a processor 110, a communication interface 120, a memory 130 for storing a processor executable computer program, and a communication bus 140. The processor 110, the communication interface 120, and the memory 130 perform communication with each other through the communication bus 140. Processor 110 implements the MQ-based presto data source hot-loading method described above by running an executable computer program.
Wherein the computer program in the memory 130 may be implemented in the form of software functional units and may be stored in a computer readable storage medium when sold or used as a separate product. Based on such understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
The system embodiments described above are merely illustrative, in which elements illustrated as separate elements may or may not be physically separate, and elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected based on actual needs to achieve the purpose of the embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
From the above description of the embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by means of software plus necessary general hardware platforms, or of course may be implemented by means of hardware. Based on such understanding, the foregoing technical solutions may be embodied essentially or in part in the form of a software product, which may be stored in a computer-readable storage medium, such as a ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to perform the various embodiments or methods of some parts of the embodiments.
The foregoing is merely illustrative of the present invention, and the present invention is not limited thereto, and any changes or substitutions easily contemplated by those skilled in the art within the scope of the present invention should be included in the present invention. Therefore, the protection scope of the invention is subject to the protection scope of the claims.