CN115827593A - Database management method, system, electronic device, and computer-readable storage medium - Google Patents

Abstract

The application discloses a database management method, a system, an electronic device and a computer readable storage medium, wherein the method comprises the following steps: responding to the change of the metadata of the database sensed by a control module matched with the database, and receiving a metadata message corresponding to the metadata generated by the control module by a data center; the metadata of the database is related to the data tables in the database, and the metadata message is generated based on the structural information of the data tables in the database; generating a synchronization task corresponding to the metadata message, executing the synchronization task to determine a temporary data table in a database, and updating the structural information corresponding to the temporary data table to a storage module of a data center station; wherein the temporary data table is different from a predefined data table defined by the station in the data; and managing the temporary data table in the database in the data center station based on the structural information stored in the storage module. By the scheme, the convenience of database management can be improved, and the available memory space of the database is increased.

Description

Database management method, system, electronic device, and computer-readable storage medium

Technical Field

The present application relates to the field of data management technologies, and in particular, to a database management method, system, electronic device, and computer-readable storage medium.

Background

With the advent of the data age, massive data needs to be accommodated and managed by relying on databases, and therefore, the management of databases is increasingly emphasized. In the prior art, changes of the data tables in the database are difficult to obtain timely, so that the data tables are accumulated continuously to reduce the available memory space of the database, and when the space of the database needs to be released, the data tables need to be managed in a mode of invading the database, so that the convenience of managing the database is not high. In view of the above, how to improve the convenience of database management and increase the available storage capacity of the database becomes an urgent problem to be solved.

Disclosure of Invention

The technical problem mainly solved by the application is to provide a database management method, a database management system, an electronic device and a computer readable storage medium, which can improve the convenience of database management and increase the available storage capacity of a database.

To solve the above technical problem, a first aspect of the present application provides a database management method, including: responding to the control module matched with the database to sense that the metadata of the database changes, and receiving a metadata message corresponding to the metadata generated by the control module by a data center; the metadata of the database is related to a data table in the database, the metadata message is generated based on the structure information of the data table in the database, and the structure information at least comprises a database name, table information and a change type corresponding to the data table; generating a synchronization task corresponding to the metadata message, executing the synchronization task to determine a temporary data table in the database, and updating the structural information corresponding to the temporary data table to a storage module of the data center station; wherein the temporary data table is distinct from a predefined data table defined by the station in the data; and managing the temporary data table in the database in the data on the basis of the structural information stored in the storage module.

To solve the above technical problem, a second aspect of the present application provides a database management system, including: the control module is used for sensing the change of the metadata of the database and generating metadata information corresponding to the metadata; the metadata of the database is related to data tables in the database, the metadata message is generated based on structural information of the data tables in the database, and the structural information at least comprises database names, table information and change types corresponding to the data tables; the data center is used for receiving the metadata message generated by the control module; generating a synchronization task corresponding to the metadata message, executing the synchronization task to determine a temporary data table in the database, and updating the structural information corresponding to the temporary data table to a storage module; wherein the data center station comprises the storage module, the temporary data table is different from a predefined data table defined by the data center station, and the data center station can manage the temporary data table in the database based on the structural information stored in the storage module.

In order to solve the above technical problem, a third aspect of the present application provides an electronic device, including: a memory and a processor coupled to each other, wherein the memory stores program data, and the processor calls the program data to execute the method of the first aspect.

To solve the above technical problem, a fourth aspect of the present application provides a computer storage medium having stored thereon program data, which when executed by a processor implements the method of the first aspect.

According to the scheme, when a control module matched with a database senses that metadata of the database changes, a data center station receives metadata information corresponding to the metadata generated by the control module, the metadata in the database is related to data tables in the database, the metadata information is generated based on structure information of the data tables in the database, namely when the data tables in the database change, the metadata of the database is sensed by the control module after being changed, the control module generates the metadata information based on the structure information of the data tables so as to enable the data center station to receive the metadata information, wherein the structure information at least comprises database names, table information and change types corresponding to the data tables, and after the data center station obtains the metadata information, a synchronization task corresponding to the metadata information is generated so that the data center station executes the synchronization task to determine temporary data tables different from predefined data tables in the database, the structure information corresponding to the temporary data tables is updated in a storage module of the data center station, and the structure information comprises database names, table information and change types corresponding to the data tables, so that the temporary data tables can be managed in the storage module based on the structure information stored in the data tables, and the data base can be used for increasing the management convenience of the management of the data base.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts. Wherein:

FIG. 1 is a schematic flow chart diagram of one embodiment of a database management method of the present application;

FIG. 2 is a schematic flow chart diagram illustrating another embodiment of a database management method of the present application;

FIG. 3 is a schematic block diagram of an embodiment of a database management system of the present application;

FIG. 4 is a schematic diagram of a topology of an embodiment of a database management system of the present application;

FIG. 5 is a schematic structural diagram of an embodiment of an electronic device of the present application;

FIG. 6 is a schematic structural diagram of an embodiment of a computer-readable storage medium according to the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all the embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present application without making any creative effort belong to the protection scope of the present application.

The terms "system" and "network" are often used interchangeably herein. The term "and/or" herein is merely an association describing an associated object, meaning that three relationships may exist, e.g., a and/or B, may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the character "/" herein generally indicates that the former and latter related objects are in an "or" relationship. Further, the term "plurality" herein means two or more than two.

The database management method provided by the application is used for managing a database, such as a Hive database, and the database management method corresponds to a database management system to realize the database management method, wherein the database management system at least comprises a control module matched with the database and a data middle platform mutually independent from the control module, and the database management method provided by the application takes the data middle platform as an execution main body.

Referring to fig. 1, fig. 1 is a schematic flowchart illustrating an embodiment of a database management method according to the present application, the method including:

s101: responding to the fact that a control module matched with the database senses that metadata of the database changes, and receiving a metadata message corresponding to the metadata generated by the control module by a data center, wherein the metadata of the database is related to a data table in the database, the metadata message is generated based on structure information of the data table in the database, and the structure information at least comprises a database name, table information and a change type corresponding to the data table.

Specifically, when a control module matched with a database senses that metadata of the database changes, a data center station receives a metadata message corresponding to the metadata generated by the control module, the metadata in the database is related to a data table in the database, and the metadata message is generated based on structure information of the data table in the database, wherein the structure information at least comprises a database name, table information and a change type corresponding to the data table.

Further, when a data table in the database changes, the metadata of the database is sensed by the control module after changing, the control module generates a metadata message based on the structure information of the data table so that the data center station can receive the metadata message, and the data center station receives the metadata message generated by the control module, so that the data center station can conveniently acquire the metadata message in the database.

In an application mode, a control module corresponds to a plurality of databases, when a data table in any one or more databases changes, metadata of the database changes due to the change of the data table, wherein the change type of the data table includes but is not limited to an addition table, an edit table and a deletion table, the control module matched with the database senses the change of the metadata of the database, and generates a metadata message based on structural information corresponding to the changed data table, and a data center receives the metadata message, wherein the structural information includes a database name, table information and a change type.

In another application mode, the number of the control modules corresponds to the number of the databases, each control module corresponds to a matched database, when a data table in any database changes, metadata of the corresponding database changes due to the change of the data table, wherein the change type of the data table includes but is not limited to an additional table, an edit table and a delete table, and then the control module matched with the database with the changed data table senses that the metadata of the database changes, generates a metadata message based on the structural information corresponding to the changed data table, and the data console receives the metadata message, wherein the structural information includes a database name, table information and a change type.

In an application scenario, the database is a Hive database, the control module and the data center station transmit metadata messages through a card message queue, when a newly-built data table or a data table in the database is edited or deleted, the data table in the database changes and is sensed by the control module, the control module generates the metadata messages based on structural information corresponding to the changed data table, and the data center station receives the metadata messages sent by the control module in a producer mode in a consumer mode.

S102: and generating a synchronization task corresponding to the metadata message, executing the synchronization task to determine a temporary data table in a database, and updating the structural information corresponding to the temporary data table to a storage module of the data center station, wherein the temporary data table is different from a predefined data table defined by the data center station.

Specifically, after the data center station obtains the metadata message, a synchronization task corresponding to the metadata message is generated, so that the data center station executes the synchronization task to determine a temporary data table different from the predefined data table in the database, and the structure information corresponding to the temporary data table is updated in a storage module of the data center station, so that the data center station can update the structure information of the temporary data table in the database in time, and the management of the temporary data table is facilitated.

It should be noted that the predefined data table is preset, the predefined data table is a data table managed by a station in the data, and after the predefined data table is filtered, the storage module does not include the structural information of the predefined data table, so that the data security of the predefined data table is effectively protected.

In an application mode, the data center station generates a synchronization task based on the metadata message, sends the synchronization task to an idle thread for execution, analyzes the metadata message to determine a data table corresponding to the metadata message, compares the data table with a predefined data table, takes the data table different from the predefined data table in a database as a temporary data table, and updates the structural information corresponding to the analyzed temporary data table in a storage module of the data center station.

In another application mode, the metadata message comprises sub-messages corresponding to the data table, the data center station determines the sub-messages in the metadata message after obtaining the metadata message, and generates sub-tasks corresponding to the sub-messages based on the sub-messages, wherein the synchronization task comprises all the sub-tasks, all the sub-tasks are sent to idle threads in batch for execution, so that the metadata message is analyzed to determine the data table corresponding to the metadata message, the data table is compared with the predefined data table, the data table which is different from the predefined data table in the database is used as a temporary data table, and the structural information corresponding to the analyzed temporary data table is updated in a storage module of the data center station.

S103: and managing the temporary data table in the database in the data center station based on the structural information stored in the storage module.

Specifically, the structure information of the temporary data table can be called from the storage module by the data center station, so that the temporary data table in the database is consulted by the data center station, and the temporary data table in the database is deleted based on the structure information, so that the data table in the database is managed by the data center station, the storage space occupied by the temporary data table is effectively released, the convenience of database management is improved, and the available storage capacity of the database is increased.

In an application mode, the data volume of the temporary data table is determined based on the structure information stored in the data center, and when the data volume of the temporary data table exceeds a data volume threshold value, at least part of the temporary data table in the database is deleted to release the storage space occupied by at least part of the temporary data table, so that the available storage volume of the database is increased.

In another application mode, based on the structural information stored in the storage module, the data of the temporary data table and/or the data volume of the temporary data table are/is checked in the data; and sending a deleting instruction for deleting at least part of the temporary data table to the database in the data based on the structural information stored in the storage module. The data center station supports deleting the temporary data table, previewing data in the temporary data table and inquiring the storage amount of the data in the temporary data table, a user can check the data of the temporary data table, or check the data volume of the temporary data table, or check the data and the data volume of the temporary data table in the data center station, so that a selection result of the user is obtained, and based on the structural information stored in the storage module, a deleting instruction for deleting the temporary data table specified by the user is sent to the database in the data center station, so that customized service is provided for the user in the data center station, and the convenience of database management is improved.

In an application scenario, the functions supported by the data center station include, but are not limited to, viewing the temporary data table, deleting the temporary data table, previewing data in the temporary data table, querying storage amount of data in the temporary data table, querying a modification record of the temporary data table, and then a user can view and delete the temporary data table based on the functions so as to manage the temporary data table in the database.

According to the scheme, when a control module matched with a database senses that metadata of the database changes, a data center station receives metadata information corresponding to the metadata generated by the control module, the metadata in the database is related to data tables in the database, and the metadata information is generated based on structure information of the data tables in the database, namely when the data tables in the database change, the metadata of the database is sensed by the control module after changing, the control module generates the metadata information based on the structure information of the data tables so as to be received by the data center station, wherein the structure information at least comprises database names, table information and change types corresponding to the data tables, and after the data center station obtains the metadata information, a synchronization task corresponding to the metadata information is generated so that the data center station executes the synchronization task to determine temporary data tables different from predefined data tables in the database, the structure information corresponding to the temporary data tables is updated in a storage module of the data center station, and the structure information comprises the database names, the table information and the change types corresponding to the data tables, so that the temporary data tables stored in the database can be managed based on the structure information stored in the storage module, and the temporary data tables can be conveniently and the management of the database can be conveniently used for managing the data tables.

Referring to fig. 2, fig. 2 is a schematic flowchart illustrating another embodiment of a database management method according to the present application, the method including:

s201: the method comprises the steps that a control module responding to database matching senses that metadata of a database changes, a data center receives metadata information generated by the control module based on a preset protocol, wherein the metadata of the database is related to a data table in the database, the metadata information is generated based on structure information of the data table in the database, and the structure information at least comprises a database name, table information and a change type corresponding to the data table.

Specifically, the data center station is connected with at least one database, the control module monitors metadata of the database in real time, when a data table in any database changes, the control module generates metadata messages according to structural information corresponding to the changed data table based on a preset protocol, and each database with changed metadata corresponds to respective metadata message.

It should be noted that a control module is customized based on the type of the database, and the control module monitors the metadata of the database in real time, so as to improve the efficiency of obtaining the metadata message, when the data table in any database changes, the control module encodes the structure information corresponding to the changed data table based on a preset protocol to generate the metadata message, when the data table corresponds to a plurality of databases and the metadata of the plurality of databases changes, each database with changed metadata corresponds to a respective metadata message, so that the metadata messages corresponding to different databases can be handed over to different threads for execution, thereby improving the execution efficiency and reducing the probability of metadata message confusion among the databases.

Further, after the control module generates the metadata message, the data center receives the metadata message generated by the control module based on the preset protocol.

In an application scenario, a control module and a data center transmit a metadata message based on a card message queue, wherein the metadata message comprises at least one sub-message corresponding to at least one data table in a database, the sub-message is related to a change type, and the change type comprises a creation table, an edit table and a deletion table.

Further, receiving the metadata message generated by the control module based on the preset protocol includes: calling an interface of a control module to obtain a cluster address of a message queue; a plurality of sub-messages in the metadata message are obtained based on the cluster address, and a position index of the sub-messages in the message queue is determined.

Specifically, the control module and the data center station transmit the metadata message with high efficiency and low delay through a card-card message queue, the metadata message comprises at least one sub-message related to the change type corresponding to at least one data table in the database, such as a sub-message related to a creation table, a sub-message related to an edit table or a sub-message related to a deletion table, and when the data table corresponds to a plurality of sub-messages or the data tables respectively correspond to sub-messages, the sub-messages are sequentially arranged in the message queue and received by the data center station.

Further, the data center calls an interface of the control module to obtain a cluster address of the message queue, if the calling fails, retries are carried out at intervals of preset time length until the cluster address is successfully obtained, a plurality of sub-messages in the metadata message are obtained based on the cluster address, and position indexes of the sub-messages in the message queue are determined, so that when the synchronization task is interrupted, the synchronization task can be continued based on the position indexes, and the execution efficiency of the synchronization task is improved, wherein the preset time length can be 20 seconds or other time lengths.

In a specific application scenario, the structure information contained in the sub-message related to the table creation includes a database name, a table creation time, a type of time, metadata object information, table information, a field list, a partition information list, a storage format, a compression mode, a remark of the table, and a table name; the sub-message related to the edit table contains structure information including database name, edit table time, time type, metadata object information, table information, field list, partition information list, storage format, compression mode, table remark, table name, old table information and old table name; the sub-message related to the delete table contains structural information including a database name, a delete table time, a type of time, metadata object information, table information, and old table information.

S202: and generating a synchronization task corresponding to the metadata message, executing the synchronization task to determine a temporary data table in a database, and updating the structural information corresponding to the temporary data table to a storage module of the data center station.

Specifically, the control module and the data center station transmit the metadata message based on the card information queue, the metadata message comprises at least one sub-message corresponding to at least one data table in the database, the data center station corresponds to a predefined data table, after the data center station obtains the metadata message, a synchronization task is generated based on all the sub-messages in the metadata message, the data table in the database is compared with the predefined data table when the synchronization task is executed, a temporary data table in the database is determined, and the structure information corresponding to the temporary data table is updated to the storage module of the data center station.

In an application mode, based on the number of sub-messages in the metadata message, generating a synchronization sub-task corresponding to the metadata message, and sending the synchronization sub-task to an idle thread for execution; wherein the number of sub-messages in each synchronization subtask does not exceed a number threshold; and determining a temporary data table in the database based on the data table corresponding to the sub-message in the synchronization sub-task and the predefined data table, and updating the structural information of the temporary data table in a storage module of the data center station based on the sub-message corresponding to the temporary data table.

Specifically, based on the number of sub-messages in the metadata message, pulling the sub-messages in the metadata message in batches, wherein the number of the sub-messages pulled each time does not exceed a number threshold, generating a synchronization sub-task based on the pulled sub-messages, sending the synchronization sub-task to an idle thread for execution, so as to synchronize the metadata message corresponding to the data table in the database to the data center station, comparing the data table corresponding to the sub-message in the synchronization sub-task with a predefined data table of the data center station in a synchronization process, determining a temporary data table different from the predefined data table in the database, and updating the structure information of the temporary data table in a storage module of the data center station based on the sub-message corresponding to the temporary data table, so as to improve the accuracy of the structure information in the data center station.

In an application scenario, determining a temporary data table in a database based on a data table corresponding to a sub-message in a synchronization sub-task and a predefined data table, and updating structure information of the temporary data table in a storage module of a station in data based on the sub-message corresponding to the temporary data table, including: based on the predefined data table, filtering the data table which is repeated with the predefined data table in the data table corresponding to the sub-message to obtain a temporary data table in the database; and traversing all the sub-messages corresponding to the temporary data table, calling an updating mode corresponding to the change type based on the change type corresponding to the sub-messages, and updating the structural information of the temporary data table into the storage module.

Specifically, different change types correspond to respective updating modes, after a synchronization subtask is obtained, a data table corresponding to a subtask message in the synchronization subtask is compared with a predefined data table, a data table which is repeated with the predefined data table in the data table corresponding to the subtask message is filtered, a temporary data table in a database is obtained, and for the subtask message corresponding to each temporary data table, based on the change type corresponding to the subtask message, the updating mode corresponding to the change type is called, the structural information corresponding to the subtask message is analyzed, and the structural information of the temporary data table is updated to the storage module according to the updating mode matched with the change type.

In a specific application scenario, the quantity threshold is 200, when the remaining quantity of sub-messages in a metadata message exceeds the quantity threshold, 200 sub-messages are pulled in batch once, a synchronous sub-task is generated, the synchronous sub-task is sent to an idle thread to be executed, a predefined data table which is coincident with a data intermediate station is filtered, so that a temporary data table in a database is determined, based on a message type corresponding to the sub-messages of the temporary data table, when the message type is a creation table, it is determined that the current temporary data table is not in a storage module, an updating method which is matched with the creation table is called, structural information corresponding to the temporary data table is updated to the storage module of the data intermediate station, when the message type is an edit table, it is determined that the current temporary data table is in the storage module, an updating method which is matched with the edit table is called, structural information corresponding to the temporary data table is updated to the storage module of the data intermediate station, and when the message type is a delete table, it is determined that the current temporary data table is in the storage module, an updating method which is matched with the delete table is called, and structural information corresponding to the temporary data table is deleted from the storage module of the intermediate station is deleted.

S203: in response to the management service module issuing an event for the database, the data service module monitors the event and determines a task state of a synchronization task corresponding to the metadata message based on the event.

Specifically, the data center station further comprises a management service module and a data service module, the event comprises a binding database, an editing database and a unbinding database, and metadata of the database corresponding to the event changes after the event is executed. The event comprises the operation on the database, the binding of a new database to the database, the editing of a user name and/or a login password of the database, the corresponding modification of the database, and the unbinding of the binding relationship of the database, wherein the metadata of the database can be changed after the event is executed.

Further, when the management service module issues an event for the database, the data service module monitors the event, that is, after the data service module monitors the event, the database changes, and then a task state of a synchronization task corresponding to the metadata message needs to be determined based on the event, where the task state includes starting a new synchronization task, suspending the synchronization task, and terminating the synchronization task, so that when the database changes, the task state is timely confirmed to determine how to execute the synchronization task.

In an application scenario, determining a task state of a synchronization task corresponding to a metadata message based on an event includes: responding to the event that the event is a binding database, receiving the metadata message corresponding to the newly bound database generated by the control module from the beginning, and starting a synchronization task corresponding to the newly bound database; in response to the event that the database is edited, suspending a synchronization task corresponding to the edited database and recording the position index of the synchronized sub-message in the message queue; and in response to the event being the unbundling of the database, terminating the synchronization task corresponding to the unbundled database.

Specifically, when the event is a binding database, the metadata message corresponding to the newly bound database generated by the control module is received from the beginning, the synchronization task corresponding to the newly bound database is started, the synchronization task corresponding to the newly bound database is in an open state, when the event is an editing database, the synchronization task corresponding to the edited database is paused, the position index of the synchronized sub-message in the message queue is recorded, the synchronization task corresponding to the edited database is in a paused state, the synchronization task of the sub-message can be resumed based on the recorded position index after the editing of the database is completed, so that the execution efficiency of the synchronization task is improved, when the event is an unbinding database, the synchronization task corresponding to the unbound database is terminated, the synchronization task corresponding to the unbound database is in a terminated state, and the metadata message of the database is in a continuous updating process, the structure information corresponding to the unbound database table is deleted from the storage module, so that the execution efficiency and the rationality of the synchronization task are improved.

S204: and managing the temporary data table in the database in the data center station based on the structural information stored in the storage module.

Specifically, in the data center, the station can call the structure information of the temporary data table from the storage module, and manage the temporary data table based on the structure information stored in the storage module.

In an application scene, based on the structural information stored in the storage module, the data of the temporary data table and/or the data volume of the temporary data table are checked in the data center; and sending a deleting instruction for deleting at least part of the temporary data table to the database in the data based on the structural information stored in the storage module.

Specifically, the data center station supports deleting the temporary data table, previewing data in the temporary data table, and inquiring storage amount of data in the temporary data table, a user can check data of the temporary data table, or check data amount of the temporary data table, or check data and data amount of the temporary data table in the data center station, so that a selection result of the user is obtained, and based on the structural information stored in the storage module, a deletion instruction for deleting the temporary data table specified by the user is sent to the database in the data center station, so that customized service is provided for the user in the data center station, and convenience in database management is improved.

In this embodiment, a control module transmits metadata messages between a data center station and a data center station in a high-efficiency and low-delay manner through a card-card message queue, the metadata messages include at least one sub-message related to a change type and corresponding to at least one data table in a database, the sub-messages in the metadata messages are pulled in batch based on the number of the sub-messages in the metadata messages, the number of the sub-messages pulled each time does not exceed a number threshold, a synchronization sub-task is generated based on the pulled sub-messages, a temporary data table different from the predefined data table in the database is determined for the data table and the predefined data table of the data center station corresponding to the sub-messages in the synchronization sub-task, and then the structural information of the temporary data table is updated in a storage module of the data center station based on the sub-messages corresponding to the temporary data table, so as to improve the accuracy of the structural information in the data center station, when the management service module issues an event for the database, the data service module monitors the event, determines the task state of the synchronization task corresponding to improve the execution efficiency of the synchronization task, the rationality of the data table, and the management module increases the storage space of the temporary data table, and thus the management module can conveniently manage the temporary data table.

Referring to fig. 3, fig. 3 is a schematic structural diagram of an embodiment of a database management system 30 of the present application, including: the control module 300 is used for sensing the change of metadata of the database and generating metadata information corresponding to the metadata; the metadata of the database is related to the data tables in the database, the metadata message is generated based on the structure information of the data tables in the database, and the structure information at least comprises database names, table information and change types corresponding to the data tables; the data center 302 is used for receiving the metadata message generated by the control module 300; generating a synchronization task corresponding to the metadata message, executing the synchronization task to determine a temporary data table in the database, and updating the structural information corresponding to the temporary data table to the storage module 3020; the data center station 302 includes a storage module 3020, the temporary data table is different from the predefined data table defined by the data center station 302, and the data center station 302 can manage the temporary data table in the database based on the structure information stored in the storage module 3020.

Further, the database management system 30 can implement the method described in any of the above embodiments, and the description of the related contents refers to the detailed description of the above method embodiments, which is not repeated herein.

In a specific application scenario, please refer to fig. 4, where fig. 4 is a schematic diagram of a topology structure of an embodiment of a database management system of the present application, the data center station 302 further includes a data service module 3021 and a management service module 3022, the data center station 302 is connected to at least one database, the control module 300 monitors metadata of the databases in real time, when a data table in any database changes, the control module 300 generates a metadata message based on structural information corresponding to the changed data table according to a preset protocol, and each database with changed metadata corresponds to its own metadata message.

Further, the control module 300 and the data center 302 transmit the metadata message based on the card-card message queue, and the metadata message includes at least one sub-message corresponding to at least one data table in the database, and the sub-message is related to the change type, and the change type includes a creation table, an edit table, and a deletion table. The management service module 3022 is configured to issue an event for a database, and the data service module 3021 is configured to monitor the event and determine a task state of a synchronization task corresponding to a metadata message based on the event. The event comprises a binding database, an editing database and a unbinding database, and the metadata of the database corresponding to the event changes after the event is executed.

Optionally, the data service module 3021 specifically includes a metadata synchronization module 3023, a monitoring module 3024, an event generation module 3025, and a saving module 3026, where the metadata synchronization module 3023 is configured to receive and parse a metadata message sent by the control module 300 based on a card information queue, and call the saving module 3026, so that the saving module 3026 stores the structure information of the temporary data table in the storage module 3020, and the event generation module 3025 is configured to generate an event message corresponding to an event after the management service module 3022 issues the event, and then the monitoring module 3024 monitors the event message, thereby determining a task state of the synchronization task.

Referring to fig. 5, fig. 5 is a schematic structural diagram of an embodiment of an electronic device 50 of the present application, where the electronic device 50 includes a memory 501 and a processor 502 coupled to each other, where the memory 501 stores program data (not shown), and the processor 502 calls the program data to implement the method in any of the above embodiments.

Referring to fig. 6, fig. 6 is a schematic structural diagram of an embodiment of a computer-readable storage medium 60 of the present application, the computer-readable storage medium 60 stores program data 600, and the program data 600 is executed by a processor to implement the method in any of the above embodiments, and the related contents are described in detail with reference to the above method embodiments, which are not repeated herein.

It should be noted that, units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be substantially implemented or contributed to by the prior art, or all or part of the technical solution may be embodied in a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, a network device, or the like) or a processor (processor) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk, and various media capable of storing program codes.

The above description is only for the purpose of illustrating embodiments of the present application and is not intended to limit the scope of the present application, and all modifications of equivalent structures and equivalent processes, which are made by the contents of the specification and the drawings of the present application or are directly or indirectly applied to other related technical fields, are also included in the scope of the present application.

Claims (11)

1. A database management method, the method comprising:

responding to the control module matched with the database to sense that the metadata of the database changes, and receiving a metadata message corresponding to the metadata generated by the control module by a data center; the metadata of the database is related to a data table in the database, the metadata message is generated based on the structure information of the data table in the database, and the structure information at least comprises a database name, table information and a change type corresponding to the data table;

generating a synchronization task corresponding to the metadata message, executing the synchronization task to determine a temporary data table in the database, and updating the structural information corresponding to the temporary data table to a storage module of the data center station; wherein the temporary data table is distinct from a predefined data table defined by the station in the data;

and managing the temporary data table in the database in the data center station based on the structural information stored in the storage module.

2. The database management method according to claim 1,

the data center station is connected with at least one database in a butt joint mode, the control module monitors metadata of the databases in real time, when a data table in any one database changes, the control module generates the metadata message according to structural information corresponding to the changed data table on the basis of a preset protocol, and each database with changed metadata corresponds to the respective metadata message;

the receiving the metadata message corresponding to the metadata generated by the control module includes:

and receiving the metadata message generated by the control module based on the preset protocol.

3. The database management method according to claim 2, wherein the control module and the data center transmit the metadata message based on a card message queue, and the metadata message includes at least one sub-message corresponding to at least one of the data tables in the database, the sub-message being related to the change type, the change type including a creation table, an edit table, and a deletion table;

the receiving the metadata message generated by the control module based on the preset protocol includes:

calling an interface of the control module to obtain a cluster address of the message queue;

and obtaining a plurality of the sub-messages in the metadata message based on the cluster address, and determining the position indexes of the sub-messages in the message queue.

4. The database management method according to claim 3, wherein the generating a synchronization task corresponding to the metadata message, executing the synchronization task to determine a temporary data table in the database, and updating the structure information corresponding to the temporary data table to a storage module of the database console includes:

generating a synchronization subtask corresponding to the metadata message based on the number of the subtasks in the metadata message, and sending the synchronization subtask to an idle thread for execution; wherein the number of said sub-messages in each of said synchronization subtasks does not exceed a number threshold;

and determining the temporary data table in the database based on the data table corresponding to the sub-message in the synchronization subtask and the predefined data table, and updating the structural information of the temporary data table in the storage module of the data center station based on the sub-message corresponding to the temporary data table.

5. The database management method according to claim 4, wherein the determining the temporary data table in the database based on the data table corresponding to the sub-message in the synchronous sub-task and the predefined data table, and the updating the structure information of the temporary data table in the data by the storage module of the station based on the sub-message corresponding to the temporary data table comprises:

based on the predefined data table, filtering the data table which is repeated with the predefined data table in the data tables corresponding to the sub-messages to obtain the temporary data table in the database;

traversing all the sub-messages corresponding to the temporary data table, calling an updating mode corresponding to the change type based on the change type corresponding to the sub-messages, and updating the structural information of the temporary data table to the storage module.

6. The database management method of claim 3, wherein the data center further comprises a management service module and a data service module;

after the generating of the synchronization task corresponding to the metadata message, the executing of the synchronization task to determine the temporary data table in the database, and the updating of the structural information corresponding to the temporary data table to the storage module of the data center station, the method further includes:

responding to the event issued by the management service module to the database, monitoring the event by the data service module, and determining the task state of the synchronous task corresponding to the metadata message based on the event;

the event comprises a binding database, an editing database and a unbinding database, and the metadata of the database corresponding to the event changes after the event is executed.

7. The database management method according to claim 6, wherein said determining a task state of a synchronization task corresponding to the metadata message based on the event comprises:

responding to the event that the event is the binding database, receiving the metadata message corresponding to the newly bound database generated by the control module from the beginning, and starting a synchronization task corresponding to the newly bound database;

responding to the event that the event is the editing database, pausing a synchronization task corresponding to the edited database, and recording the position index of the synchronized sub-message in the message queue;

and in response to the event being the unbinding database, terminating the synchronization task corresponding to the unbound database.

8. The database management method according to any one of claims 1 to 7, wherein said managing, in the data, the temporary data table in the database based on the structural information stored in the storage module includes:

based on the structural information stored in the storage module, checking the data of the temporary data table and/or the data amount of the temporary data table in the data;

and sending a deleting instruction for deleting at least part of the temporary data table to the database in the data based on the structural information stored in the storage module.

9. A database management system, the system comprising:

the control module is used for sensing the change of the metadata of the database and generating metadata information corresponding to the metadata; the metadata of the database is related to data tables in the database, the metadata message is generated based on structural information of the data tables in the database, and the structural information at least comprises database names, table information and change types corresponding to the data tables;

the data center is used for receiving the metadata message generated by the control module; generating a synchronization task corresponding to the metadata message, executing the synchronization task to determine a temporary data table in the database, and updating the structural information corresponding to the temporary data table to a storage module;

wherein the data center station comprises the storage module, the temporary data table is different from a predefined data table defined by the data center station, and the data center station can manage the temporary data table in the database based on the structural information stored in the storage module.

10. An electronic device, comprising: a memory and a processor coupled to each other, wherein the memory stores program data that the processor calls to perform the method of any of claims 1-8.

11. A computer-readable storage medium, on which program data are stored, which program data, when being executed by a processor, carry out the method of any one of claims 1-8.

Images