CN111125101A - Data center table structure consistency monitoring method and system - Google Patents
Data center table structure consistency monitoring method and system Download PDFInfo
- Publication number
- CN111125101A CN111125101A CN201911291296.6A CN201911291296A CN111125101A CN 111125101 A CN111125101 A CN 111125101A CN 201911291296 A CN201911291296 A CN 201911291296A CN 111125101 A CN111125101 A CN 111125101A
- Authority
- CN
- China
- Prior art keywords
- table structure
- data center
- intermediate node
- consistency
- configuration information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Debugging And Monitoring (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a method for monitoring consistency of a data center table structure, which comprises the following steps: the first data center sends a first table structure to be monitored to the intermediate node according to the configuration information; the intermediate node sends query information to a second data center according to the configuration information; the second data center sends a second table structure to be monitored to the intermediate node according to the configuration information and the query information; the intermediate node detects consistency of the first table structure and the second table structure. Compared with the prior art, the method has the following advantages: the data centers are compared through the intermediate nodes, normal operation of the data centers is not affected, and meanwhile consistency of corresponding table structures among the data centers can be automatically detected.
Description
Technical Field
The application relates to the field of data synchronization, in particular to a method for monitoring consistency of a data center table structure.
Background
In the global service, in order to improve the access speed of users, the nearest data centers are selected according to user regions, the data of the data centers are isolated from each other, and the table structure and the data are kept consistent through synchronization. Synchronization anomalies or network jitter, etc., can cause table structures of different data centers to be inconsistent. At present, for a solution of a cloud manufacturer, a common method for synchronizing application data adopts a scheme of DDL (data definition language) synchronization, and a single data center table structure is changed and synchronized to all other data centers. However, the synchronization in the above scheme may fail, resulting in inconsistency; there is a problem of a single central execution failing, or missing execution.
Disclosure of Invention
The main objective of the present application is to provide a method for monitoring consistency of a table structure of a data center, including:
the first data center sends a first table structure to be monitored to the intermediate node according to the configuration information;
the intermediate node sends query information to a second data center according to the configuration information;
the second data center sends a second table structure to be monitored to the intermediate node according to the configuration information and the query information;
the intermediate node detects consistency of the first table structure and the second table structure.
Optionally, the intermediate node is a real-time distributed messaging platform.
Optionally, an alarm is given when the first table structure is inconsistent with the second table structure.
Optionally, the second data center is multiple.
Optionally, the first data center uploads the first table structure at regular time, when the first data center does not upload the first table structure within a preset time, the intermediate node sends an upload notification to the first data center, and the first data center sends the first table structure to the intermediate node according to the upload notification.
Optionally, the table structure contains a library table name, field information, or index information.
According to another aspect of the present application, there is also provided a data center table structure consistency monitoring system, including:
the first table structure sending module is used for sending the first table structure to be monitored to the intermediate node according to the configuration information;
the query information sending module is used for sending query information to a second data center according to the configuration information;
a second table structure sending module, configured to send a second table structure to be monitored to the intermediate node according to the configuration information and the query information;
a detection module to detect a correspondence of the first table structure and the second table structure. The application also discloses a computer device, which comprises a memory, a processor and a computer program stored in the memory and capable of being executed by the processor, wherein the processor realizes the method of any one of the above items when executing the computer program.
The application also discloses a computer-readable storage medium, a non-volatile readable storage medium, having stored therein a computer program which, when executed by a processor, implements the method of any of the above.
The present application also discloses a computer program product comprising computer readable code which, when executed by a computer device, causes the computer device to perform the method of any of the above.
Compared with the prior art, the method has the following advantages:
the data centers are compared through the intermediate nodes, normal operation of the data centers is not affected, and meanwhile consistency of corresponding table structures among the data centers can be automatically detected.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this application, serve to provide a further understanding of the application and to enable other features, objects, and advantages of the application to be more apparent. The drawings and their description illustrate the embodiments of the invention and do not limit it. In the drawings:
FIG. 1 is a schematic flow chart diagram of a data center table structure consistency monitoring method according to an embodiment of the present application;
FIG. 2 is a schematic diagram of a computer device according to one embodiment of the present application; and
FIG. 3 is a schematic diagram of a computer-readable storage medium according to one embodiment of the present application.
Detailed Description
In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only partial embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It should be noted that the terms "first," "second," and the like in the description and claims of this application and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It should be understood that the data so used may be interchanged under appropriate circumstances such that embodiments of the application described herein may be used. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict. The present application will be described in detail below with reference to the embodiments with reference to the attached drawings.
Referring to fig. 1, an embodiment of the present application provides a method for monitoring consistency of a table structure of a data center, including:
s2: the first data center sends a first table structure to be monitored to the intermediate node according to the configuration information;
s4: the intermediate node sends query information to a second data center according to the configuration information;
s6: the second data center sends a second table structure to be monitored to the intermediate node according to the configuration information and the query information;
s8: the intermediate node detects consistency of the first table structure and the second table structure.
For example, there are 10 data centers (1-10), each containing 5 table structures (a, b, c, d, e). The data center 1 sends the table structure c to the intermediate node according to the configuration information. The intermediate node discovers that the table structure c exists in the data centers 3, 6 and 8 according to the configuration information in addition to the data center 1. The intermediate node therefore sends a query message "needs to upload table structure c" to data centres 3, 6 and 8. And the data centers 3, 6 and 8 respectively find the table structure c in own databases according to the configuration information and upload the table structure c to the intermediate node. Finally, the intermediate node checks the consistency of the table structure c of the data centers 1, 3, 6 and 8. The first data center is a data center which actively transmits a table structure to be detected according to configuration information, and is not a data center with the number of 1. Therefore, the number of the first data centers may be any one or more. And similarly, the second data center returns the corresponding table structure according to the updated information. Thus, the second data center may also be one or more.
In an embodiment of the present application, the intermediate node is a real-time distributed messaging platform.
In an embodiment of the present application, an alarm is issued when the first table structure is inconsistent with the second table structure.
In an embodiment of the present application, the second data center is multiple.
In an embodiment of the present application, the first data center uploads the first table structure at regular time, when the first data center does not upload the first table structure within a preset time, the intermediate node sends an upload notification to the first data center, and the first data center sends the first table structure to the intermediate node according to the upload notification.
In an embodiment of the present application, the table structure includes a library table name, field information, or index information.
According to another aspect of the present application, there is also provided a data center table structure consistency monitoring system, including:
the first table structure sending module is used for sending the first table structure to be monitored to the intermediate node according to the configuration information;
the query information sending module is used for sending query information to a second data center according to the configuration information;
a second table structure sending module, configured to send a second table structure to be monitored to the intermediate node according to the configuration information and the query information;
a detection module to detect a correspondence of the first table structure and the second table structure.
Related concepts in the present application:
an intermediate node: and using NSQ (real-time distributed messaging platform) as message middleware to record the table structure information sent by each data center. And when the single data center does not upload the table structure within a certain time, issuing a table structure uploading notice to the data center. And comparing the table structures of the data centers according to the configured mapping relation, and alarming when the table structures are inconsistent.
A data center node: and the timing query table structure is arranged to the intermediate node and comprises a library table name, field information, index information and the like. Encrypted and sent to the intermediate node via the NSQ. And receiving the table structure uploading sent by the intermediate node and uploading the table structure.
Configuration information: and configuring the mapping relation of each data center base table. (the data center finds out which tables need to be compared according to the mapping relation, sends the tables needing to be compared to the intermediate node after finding out, the intermediate node finds out which tables needing to be compared exist in which data centers according to the mapping relation, and calls the corresponding tables of the data centers to compare)
The data center table structure consistency monitoring method provided by the application solves the table structure comparison scheme of the global data center and solves the problem that the table structures of all data centers are inconsistent. And each data center encrypts the table structure data and synchronizes the table structure data to the intermediate node, and the intermediate node records the table structure data of each data center after passing the authentication. The intermediate node monitors that certain data center table structure data is not received within a certain time, and sends a notice to the data center for uploading. And the data center receives the table structure uploaded by the intermediate node and uploads the table structure. And the intermediate node configuration center configures the mapping relation of each data center base table. And the intermediate node compares the table structures of the data centers according to the configured mapping relation, wherein the table structures comprise field names, types, lengths, whether the fields can be empty or not, default values and the like. Index name, field, type, data structure, etc. And alarming when the inconsistency exists.
Referring to fig. 2, the present application further provides a computer device including a memory, a processor, and a computer program stored in the memory and executable by the processor, wherein the processor implements the method of any one of the above methods when executing the computer program.
Referring to fig. 3, a computer-readable storage medium, a non-volatile readable storage medium, having stored therein a computer program which, when executed by a processor, implements any of the methods described above.
A computer program product comprising computer readable code which, when executed by a computer device, causes the computer device to perform the method of any of the above.
It will be apparent to those skilled in the art that the modules or steps of the present invention described above may be implemented by a general purpose computing device, they may be centralized on a single computing device or distributed across a network of multiple computing devices, and they may alternatively be implemented by program code executable by a computing device, such that they may be stored in a storage device and executed by a computing device, or fabricated separately as individual integrated circuit modules, or fabricated as a single integrated circuit module from multiple modules or steps. Thus, the present invention is not limited to any specific combination of hardware and software.
The above description is only a preferred embodiment of the present application and is not intended to limit the present application, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims (10)
1. A method for monitoring consistency of a data center table structure is characterized by comprising the following steps:
the first data center sends a first table structure to be monitored to the intermediate node according to the configuration information;
the intermediate node sends query information to a second data center according to the configuration information;
the second data center sends a second table structure to be monitored to the intermediate node according to the configuration information and the query information;
the intermediate node detects consistency of the first table structure and the second table structure.
2. The data center table structure consistency monitoring method according to claim 1, wherein the intermediate node is a real-time distributed messaging platform.
3. The data center table structure consistency monitoring method according to claim 2, characterized by alarming when the first table structure is inconsistent with the second table structure.
4. The data center table structure consistency monitoring method according to claim 3, wherein the number of the second data centers is multiple.
5. The method for monitoring consistency of a table structure of a data center according to claim 4, wherein the first data center uploads the first table structure at regular time, when the first data center does not upload the first table structure within a preset time, the intermediate node sends an upload notification to the first data center, and the first data center sends the first table structure to the intermediate node according to the upload notification.
6. The data center table structure consistency monitoring method according to claim 5, wherein the table structure contains library table names, field information, or index information.
7. A data center table structure consistency monitoring system is characterized by comprising:
the first table structure sending module is used for sending the first table structure to be monitored to the intermediate node according to the configuration information;
the query information sending module is used for sending query information to a second data center according to the configuration information;
a second table structure sending module, configured to send a second table structure to be monitored to the intermediate node according to the configuration information and the query information;
a detection module to detect a correspondence of the first table structure and the second table structure.
8. A computer device comprising a memory, a processor and a computer program stored in the memory and executable by the processor, wherein the processor implements the method of any one of claims 1-6 when executing the computer program.
9. A computer-readable storage medium, a non-transitory readable storage medium, having stored therein a computer program, characterized in that the computer program, when executed by a processor, implements the method according to any one of claims 1-6.
10. A computer program product comprising computer readable code that, when executed by a computer device, causes the computer device to perform the method of any of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911291296.6A CN111125101B (en) | 2019-12-16 | 2019-12-16 | Data center table structure consistency monitoring method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911291296.6A CN111125101B (en) | 2019-12-16 | 2019-12-16 | Data center table structure consistency monitoring method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111125101A true CN111125101A (en) | 2020-05-08 |
| CN111125101B CN111125101B (en) | 2023-10-13 |
Family
ID=70499071
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911291296.6A Active CN111125101B (en) | 2019-12-16 | 2019-12-16 | Data center table structure consistency monitoring method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111125101B (en) |
Citations (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20060248052A1 (en) * | 2005-04-29 | 2006-11-02 | Thomas Zurek | Data query verification |
| US20110093486A1 (en) * | 2009-10-15 | 2011-04-21 | Institute For Information Industry | Data query method, data query system and computer readable and writable recording medium |
| US8312037B1 (en) * | 2008-08-28 | 2012-11-13 | Amazon Technologies, Inc. | Dynamic tree determination for data processing |
| US20140173035A1 (en) * | 2011-08-02 | 2014-06-19 | Nec Corporation | Distributed storage system and method |
| CN104462435A (en) * | 2014-12-15 | 2015-03-25 | 同方知网数字出版技术股份有限公司 | Lateral extension method of distributed database |
| CN105262591A (en) * | 2015-09-08 | 2016-01-20 | 常熟理工学院 | Data-based network communication implementation method |
| CN106415532A (en) * | 2014-06-04 | 2017-02-15 | 株式会社日立制作所 | Medical care data search system |
| US20170169073A1 (en) * | 2015-12-14 | 2017-06-15 | International Business Machines Corporation | Executing Graph Path Queries |
| CN107515933A (en) * | 2017-08-29 | 2017-12-26 | 平安科技（深圳）有限公司 | Change method, server, system and the storage medium of source data database table structure |
| US20180150541A1 (en) * | 2016-11-28 | 2018-05-31 | Sap Se | Proxy Views for Extended Monitoring of Database Systems |
| CN108304413A (en) * | 2017-01-13 | 2018-07-20 | 北京京东尚科信息技术有限公司 | distributed data warehouse monitoring method, device, electronic equipment and storage medium |
| CN108874644A (en) * | 2018-06-06 | 2018-11-23 | 平安科技（深圳）有限公司 | Data monitoring method, device, computer equipment and storage medium |
| CN109828889A (en) * | 2019-01-31 | 2019-05-31 | 平安科技（深圳）有限公司 | Method, apparatus, computer equipment and the storage medium in monitoring data library |
| CN110109824A (en) * | 2019-04-09 | 2019-08-09 | 平安科技（深圳）有限公司 | Big data automatic regression test method, apparatus, computer equipment and storage medium |
| CN110209730A (en) * | 2019-04-25 | 2019-09-06 | 深圳壹账通智能科技有限公司 | Change synchronous method, device, computer equipment and the computer storage medium of data |
| CN110286883A (en) * | 2019-06-03 | 2019-09-27 | 中国建设银行股份有限公司 | Intelligence database encodes monitoring method, device and electronic equipment |
-
2019
- 2019-12-16 CN CN201911291296.6A patent/CN111125101B/en active Active
Patent Citations (18)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20060248052A1 (en) * | 2005-04-29 | 2006-11-02 | Thomas Zurek | Data query verification |
| US8312037B1 (en) * | 2008-08-28 | 2012-11-13 | Amazon Technologies, Inc. | Dynamic tree determination for data processing |
| US20110093486A1 (en) * | 2009-10-15 | 2011-04-21 | Institute For Information Industry | Data query method, data query system and computer readable and writable recording medium |
| US20140173035A1 (en) * | 2011-08-02 | 2014-06-19 | Nec Corporation | Distributed storage system and method |
| CN106415532A (en) * | 2014-06-04 | 2017-02-15 | 株式会社日立制作所 | Medical care data search system |
| CN104462435A (en) * | 2014-12-15 | 2015-03-25 | 同方知网数字出版技术股份有限公司 | Lateral extension method of distributed database |
| CN105262591A (en) * | 2015-09-08 | 2016-01-20 | 常熟理工学院 | Data-based network communication implementation method |
| US20170169073A1 (en) * | 2015-12-14 | 2017-06-15 | International Business Machines Corporation | Executing Graph Path Queries |
| US20180150541A1 (en) * | 2016-11-28 | 2018-05-31 | Sap Se | Proxy Views for Extended Monitoring of Database Systems |
| CN108304413A (en) * | 2017-01-13 | 2018-07-20 | 北京京东尚科信息技术有限公司 | distributed data warehouse monitoring method, device, electronic equipment and storage medium |
| CN107515933A (en) * | 2017-08-29 | 2017-12-26 | 平安科技（深圳）有限公司 | Change method, server, system and the storage medium of source data database table structure |
| WO2019041832A1 (en) * | 2017-08-29 | 2019-03-07 | 平安科技（深圳）有限公司 | Method, server and system for modifying source database table structure, and storage medium |
| CN108874644A (en) * | 2018-06-06 | 2018-11-23 | 平安科技（深圳）有限公司 | Data monitoring method, device, computer equipment and storage medium |
| WO2019232832A1 (en) * | 2018-06-06 | 2019-12-12 | 平安科技（深圳）有限公司 | Data monitoring method and device, computer device, and storage medium |
| CN109828889A (en) * | 2019-01-31 | 2019-05-31 | 平安科技（深圳）有限公司 | Method, apparatus, computer equipment and the storage medium in monitoring data library |
| CN110109824A (en) * | 2019-04-09 | 2019-08-09 | 平安科技（深圳）有限公司 | Big data automatic regression test method, apparatus, computer equipment and storage medium |
| CN110209730A (en) * | 2019-04-25 | 2019-09-06 | 深圳壹账通智能科技有限公司 | Change synchronous method, device, computer equipment and the computer storage medium of data |
| CN110286883A (en) * | 2019-06-03 | 2019-09-27 | 中国建设银行股份有限公司 | Intelligence database encodes monitoring method, device and electronic equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111125101B (en) | 2023-10-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112506659B (en) | Management method of edge equipment and related device | |
| CN111209011A (en) | Cross-platform container cloud automatic deployment system | |
| US20150278030A1 (en) | Distributed Database Synchronization Method and System | |
| US9477735B2 (en) | Cloud computing-based data sharing system and method | |
| CN109145060B (en) | Data processing method and device | |
| CN110334094B (en) | Data query method, system, device and equipment based on inverted index | |
| CN108322358B (en) | Method and device for sending, processing and consuming multi-live distributed messages in different places | |
| CN112015744A (en) | Configuration data access method, device, equipment, configuration center and storage medium | |
| CN110048896A (en) | A kind of company-data acquisition methods, device and equipment | |
| CN107180034A (en) | The group system of MySQL database | |
| CN110196780B (en) | Method, device, storage medium and electronic device for determining server state | |
| CN108063832B (en) | Cloud storage system and storage method thereof | |
| US20180123935A1 (en) | Network Configuration Management System | |
| CN115004662A (en) | Data synchronization method, data synchronization device, data storage system and computer readable medium | |
| CN113779153A (en) | Data synchronization method and device, electronic equipment and storage medium | |
| CN110545197A (en) | node state monitoring method and device | |
| CN111125101A (en) | Data center table structure consistency monitoring method and system | |
| US10275467B2 (en) | Multi-level high availability model for an object storage service | |
| CN104954505B (en) | Monitoring method and system for DNS data updating overall process | |
| US10860580B2 (en) | Information processing device, method, and medium | |
| CN105956141A (en) | Medicine data storage method based on Internet of things | |
| CN105681105A (en) | Network element configuration information synchronization method and device | |
| CN107231394B (en) | Construction method of data source address distribution tree and method for copying data | |
| JP2019508975A (en) | Neighbor monitoring in hyperscale environment | |
| CN107463484B (en) | Method and system for collecting monitoring records |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |