CN111125101B - Data center table structure consistency monitoring method and system - Google Patents
Data center table structure consistency monitoring method and system Download PDFInfo
- Publication number
- CN111125101B CN111125101B CN201911291296.6A CN201911291296A CN111125101B CN 111125101 B CN111125101 B CN 111125101B CN 201911291296 A CN201911291296 A CN 201911291296A CN 111125101 B CN111125101 B CN 111125101B
- Authority
- CN
- China
- Prior art keywords
- table structure
- data center
- intermediate node
- configuration information
- consistency
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Debugging And Monitoring (AREA)
Abstract
The application discloses a data center table structure consistency monitoring method, which comprises the following steps: the first data center sends a first table structure to be monitored to the intermediate node according to the configuration information; the intermediate node sends query information to a second data center according to the configuration information; the second data center sends a second table structure to be monitored to the intermediate node according to the configuration information and the query information; the intermediate node detects consistency of the first table structure with the second table structure. Compared with the prior art, the application has the following advantages: the intermediate nodes are used for comparison, so that the normal operation of the data centers is not affected, and the consistency of corresponding table structures among the data centers can be automatically detected.
Description
Technical Field
The application relates to the field of data synchronization, in particular to a data center table structure consistency monitoring method.
Background
In global service, in order to increase the access speed of users, the nearest data center is selected according to the user region, the data of the data centers are isolated from each other, and the table structure and the data are kept consistent through synchronization. Abnormal synchronization or network jitter, etc., can lead to inconsistent table structures for different data centers. Currently, for the solution of cloud manufacturers, a common method for synchronizing application data adopts the scheme that DDL (data definition language) is synchronized, and a single data center table structure change is synchronized to all other data centers. However, synchronization may fail in the above scheme, resulting in inconsistency; there are problems with single center execution failure, or missing execution.
Disclosure of Invention
The application mainly aims to provide a data center table structure consistency monitoring method, which comprises the following steps:
the first data center sends a first table structure to be monitored to the intermediate node according to the configuration information;
the intermediate node sends query information to a second data center according to the configuration information;
the second data center sends a second table structure to be monitored to the intermediate node according to the configuration information and the query information;
the intermediate node detects consistency of the first table structure with the second table structure.
Optionally, the intermediate node is a real-time distributed messaging platform.
Optionally, an alarm is given when the first table structure is inconsistent with the second table structure.
Optionally, the second data center is a plurality of.
Optionally, the first data center uploads the first table structure at regular time, when the first data center does not upload the first table structure within a preset time, the intermediate node sends an upload notification to the first data center, and the first data center sends the first table structure to the intermediate node according to the upload notification.
Optionally, the table structure contains a library table name, field information, or index information.
According to another aspect of the present application, there is also provided a data center table structure consistency monitoring system, including:
the first table structure sending module is used for sending the first table structure to be monitored to the intermediate node according to the configuration information;
the query information sending module is used for sending query information to a second data center according to the configuration information;
the second table structure sending module is used for sending a second table structure to be monitored to the intermediate node according to the configuration information and the query information;
and the detection module is used for detecting the consistency of the first table structure and the second table structure. The application also discloses a computer device comprising a memory, a processor and a computer program stored in the memory and executable by the processor, the processor implementing the method of any one of the above when executing the computer program.
The application also discloses a computer readable storage medium, a non-volatile readable storage medium having stored therein a computer program which, when executed by a processor, implements the method of any of the above.
The application also discloses a computer program product comprising computer readable code which, when executed by a computer device, causes the computer device to perform the method of any of the preceding claims.
Compared with the prior art, the application has the following advantages:
the intermediate nodes are used for comparison, so that the normal operation of the data centers is not affected, and the consistency of corresponding table structures among the data centers can be automatically detected.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application, are incorporated in and constitute a part of this specification. The drawings and their description are illustrative of the application and are not to be construed as unduly limiting the application. In the drawings:
FIG. 1 is a flow diagram of a data center table structure consistency monitoring method according to one embodiment of the application;
FIG. 2 is a schematic diagram of a computer device according to one embodiment of the application; and
FIG. 3 is a schematic diagram of a computer-readable storage medium according to one embodiment of the application.
Detailed Description
In order that those skilled in the art will better understand the present application, a technical solution in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in which it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present application without making any inventive effort, shall fall within the scope of the present application.
It should be noted that the terms "first," "second," and the like in the description and the claims of the present application and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate in order to describe the embodiments of the application herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
It should be noted that, without conflict, the embodiments of the present application and features of the embodiments may be combined with each other. The application will be described in detail below with reference to the drawings in connection with embodiments.
Referring to fig. 1, an embodiment of the present application provides a data center table structure consistency monitoring method, which includes:
s2: the first data center sends a first table structure to be monitored to the intermediate node according to the configuration information;
s4: the intermediate node sends query information to a second data center according to the configuration information;
s6: the second data center sends a second table structure to be monitored to the intermediate node according to the configuration information and the query information;
s8: the intermediate node detects consistency of the first table structure with the second table structure.
For example, there are 10 data centers (1-10) in total, each containing 5 table structures (a, b, c, d, e). The data center 1 sends the table structure c to the intermediate node according to the configuration information. The intermediate node finds out that the table structure c exists in the data center 3, 6 and 8 in addition to the data center 1 according to the configuration information. The intermediate node sends a query message to the data centers 3, 6 and 8 that "table structure c needs to be uploaded". The data centers 3, 6 and 8 find the table structure c in their own databases according to the configuration information, respectively, and upload to the intermediate node. Finally, the intermediate node detects the consistency of the table structure c of the data centers 1, 3, 6 and 8. The first data center refers to a data center actively transmitting a table structure to be detected according to configuration information, and does not refer to a data center with the number 1. Thus, the first data center may be any one or any plurality of first data centers. Similarly, the second data center is a data center which returns a corresponding table structure according to the new information. Thus, the second data center may also be one or more.
In an embodiment of the present application, the intermediate node is a real-time distributed messaging platform.
In one embodiment of the application, an alarm is given when the first table structure is inconsistent with the second table structure.
In an embodiment of the present application, the second data center is a plurality of data centers.
In an embodiment of the present application, the first data center uploads the first table structure at regular time, and when the first data center does not upload the first table structure within a preset time, the intermediate node sends an upload notification to the first data center, and the first data center sends the first table structure to the intermediate node according to the upload notification.
In one embodiment of the present application, the table structure includes a library table name, field information, or index information.
According to another aspect of the present application, there is also provided a data center table structure consistency monitoring system, including:
the first table structure sending module is used for sending the first table structure to be monitored to the intermediate node according to the configuration information;
the query information sending module is used for sending query information to a second data center according to the configuration information;
the second table structure sending module is used for sending a second table structure to be monitored to the intermediate node according to the configuration information and the query information;
and the detection module is used for detecting the consistency of the first table structure and the second table structure.
Related concepts in the present application:
intermediate node: and using NSQ (real-time distributed message delivery platform) as a message middleware to record the table structure information sent by each data center. When a single data center does not upload a table structure within a certain time, a downlink table structure uploads a notification to the data center. And comparing the table structures of the data centers according to the configured mapping relation, and alarming when the table structures are inconsistent.
Data center node: the lookup table structure is timed to the intermediate node, and contains the library table name, field information, index information and the like. The encrypted data is sent to the intermediate node through NSQ. And receiving the uploading of the table structure issued by the intermediate node, and uploading the table structure.
Configuration information: and configuring mapping relation of each data center database table. (the data centers find out which tables need to be compared according to the mapping relation, after finding out, the tables needing to be compared are sent to the intermediate node, the intermediate node finds out which data centers the tables needing to be compared exist in according to the mapping relation, and calls the corresponding tables of the data centers to be compared
The data center table structure consistency monitoring method provided by the application solves the problem of inconsistent table structures of all data centers in the world by solving the table structure comparison method of the global data center. Each data center encrypts the table structure data and synchronizes to the intermediate node, and each data center table structure data is recorded after the intermediate node passes the authentication. The intermediate node monitors that the data of a certain data center table structure is not received for a certain time, and issues a notification to the data center to notify uploading. And the data center receives the uploading of the table structure issued by the intermediate node and uploads the table structure. The intermediate node configuration center configures the mapping relation of each database table. And the intermediate node compares the table structures of the data centers according to the configured mapping relation, wherein the table structures comprise field names, types, lengths, whether the field names, the types, the lengths can be null, default values and the like. Index name, field, type, data structure, etc. And alarming when the two types of the data are inconsistent.
Referring to fig. 2, the present application further provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable by the processor, where the processor implements the method of any one of the above when executing the computer program.
Referring to fig. 3, a computer readable storage medium, a non-volatile readable storage medium, has stored therein a computer program which, when executed by a processor, implements the method of any of the above.
A computer program product comprising computer readable code which, when executed by a computer device, causes the computer device to perform the method of any of the preceding claims.
It will be apparent to those skilled in the art that the modules or steps of the application described above may be implemented in a general purpose computing device, they may be concentrated on a single computing device, or distributed across a network of computing devices, or they may alternatively be implemented in program code executable by computing devices, such that they may be stored in a memory device for execution by the computing devices, or they may be separately fabricated into individual integrated circuit modules, or multiple modules or steps within them may be fabricated into a single integrated circuit module. Thus, the present application is not limited to any specific combination of hardware and software.
The above description is only of the preferred embodiments of the present application and is not intended to limit the present application, but various modifications and variations can be made to the present application by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.
Claims (8)
1. A method for monitoring consistency of a data center table structure, comprising:
the first data center sends a first table structure to be monitored to the intermediate node according to the configuration information;
the intermediate node sends query information to a second data center according to the configuration information;
the second data center sends a second table structure to be monitored to the intermediate node according to the configuration information and the query information;
the intermediate node detects consistency of the first table structure and the second table structure;
the first data center uploads the first table structure at regular time, when the first data center does not upload the first table structure within preset time, the intermediate node sends an upload notification to the first data center, and the first data center sends the first table structure to the intermediate node according to the upload notification.
2. The method of claim 1, wherein the intermediate node is a real-time distributed messaging platform.
3. The data center table structure consistency monitoring method of claim 2, wherein an alarm is given when the first table structure is inconsistent with the second table structure.
4. The data center table structure consistency monitoring method of claim 3, wherein the second data center is a plurality of data centers.
5. The method of claim 1, wherein the table structure comprises a library table name, field information, or index information.
6. A data center table structure consistency monitoring system, comprising: the system comprises a first data center, a second data center and an intermediate node; wherein:
the first data center is used for sending a first table structure to be monitored to the intermediate node according to the configuration information; and uploading the first table structure at regular time, wherein when the first data center does not upload the first table structure within a preset time, the intermediate node transmits an uploading notification to the first data center, and the first data center transmits the first table structure to the intermediate node according to the uploading notification;
the intermediate node is used for sending query information to a second data center according to the configuration information;
the second data center is used for sending a second table structure to be monitored to the intermediate node according to the configuration information and the query information, and the intermediate node detects the consistency of the first table structure and the second table structure.
7. A computer device comprising a memory, a processor and a computer program stored in the memory and executable by the processor, wherein the processor implements the method of any of claims 1-5 when executing the computer program.
8. A computer readable storage medium, in which a computer program is stored which, when executed by a processor, implements the method according to any of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911291296.6A CN111125101B (en) | 2019-12-16 | 2019-12-16 | Data center table structure consistency monitoring method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911291296.6A CN111125101B (en) | 2019-12-16 | 2019-12-16 | Data center table structure consistency monitoring method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111125101A CN111125101A (en) | 2020-05-08 |
| CN111125101B true CN111125101B (en) | 2023-10-13 |
Family
ID=70499071
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911291296.6A Active CN111125101B (en) | 2019-12-16 | 2019-12-16 | Data center table structure consistency monitoring method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111125101B (en) |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8312037B1 (en) * | 2008-08-28 | 2012-11-13 | Amazon Technologies, Inc. | Dynamic tree determination for data processing |
| CN104462435A (en) * | 2014-12-15 | 2015-03-25 | 同方知网数字出版技术股份有限公司 | Lateral extension method of distributed database |
| CN105262591A (en) * | 2015-09-08 | 2016-01-20 | 常熟理工学院 | Data-based network communication implementation method |
| CN106415532A (en) * | 2014-06-04 | 2017-02-15 | 株式会社日立制作所 | Medical care data search system |
| CN107515933A (en) * | 2017-08-29 | 2017-12-26 | 平安科技（深圳）有限公司 | Change method, server, system and the storage medium of source data database table structure |
| CN108304413A (en) * | 2017-01-13 | 2018-07-20 | 北京京东尚科信息技术有限公司 | distributed data warehouse monitoring method, device, electronic equipment and storage medium |
| CN108874644A (en) * | 2018-06-06 | 2018-11-23 | 平安科技（深圳）有限公司 | Data monitoring method, device, computer equipment and storage medium |
| CN109828889A (en) * | 2019-01-31 | 2019-05-31 | 平安科技（深圳）有限公司 | Method, apparatus, computer equipment and the storage medium in monitoring data library |
| CN110109824A (en) * | 2019-04-09 | 2019-08-09 | 平安科技（深圳）有限公司 | Big data automatic regression test method, apparatus, computer equipment and storage medium |
| CN110209730A (en) * | 2019-04-25 | 2019-09-06 | 深圳壹账通智能科技有限公司 | Change synchronous method, device, computer equipment and the computer storage medium of data |
| CN110286883A (en) * | 2019-06-03 | 2019-09-27 | 中国建设银行股份有限公司 | Intelligence database encodes monitoring method, device and electronic equipment |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7610265B2 (en) * | 2005-04-29 | 2009-10-27 | Sap Ag | Data query verification |
| TW201113731A (en) * | 2009-10-15 | 2011-04-16 | Inst Information Industry | Data query method, data query system and computer readable and writable recording medium |
| WO2013018808A1 (en) * | 2011-08-02 | 2013-02-07 | 日本電気株式会社 | Distributed storage system and method |
| US10176220B2 (en) * | 2015-12-14 | 2019-01-08 | International Business Machines Corporation | Executing graph path queries |
| US10380142B2 (en) * | 2016-11-28 | 2019-08-13 | Sap Se | Proxy views for extended monitoring of database systems |
-
2019
- 2019-12-16 CN CN201911291296.6A patent/CN111125101B/en active Active
Patent Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8312037B1 (en) * | 2008-08-28 | 2012-11-13 | Amazon Technologies, Inc. | Dynamic tree determination for data processing |
| CN106415532A (en) * | 2014-06-04 | 2017-02-15 | 株式会社日立制作所 | Medical care data search system |
| CN104462435A (en) * | 2014-12-15 | 2015-03-25 | 同方知网数字出版技术股份有限公司 | Lateral extension method of distributed database |
| CN105262591A (en) * | 2015-09-08 | 2016-01-20 | 常熟理工学院 | Data-based network communication implementation method |
| CN108304413A (en) * | 2017-01-13 | 2018-07-20 | 北京京东尚科信息技术有限公司 | distributed data warehouse monitoring method, device, electronic equipment and storage medium |
| WO2019041832A1 (en) * | 2017-08-29 | 2019-03-07 | 平安科技（深圳）有限公司 | Method, server and system for modifying source database table structure, and storage medium |
| CN107515933A (en) * | 2017-08-29 | 2017-12-26 | 平安科技（深圳）有限公司 | Change method, server, system and the storage medium of source data database table structure |
| CN108874644A (en) * | 2018-06-06 | 2018-11-23 | 平安科技（深圳）有限公司 | Data monitoring method, device, computer equipment and storage medium |
| WO2019232832A1 (en) * | 2018-06-06 | 2019-12-12 | 平安科技（深圳）有限公司 | Data monitoring method and device, computer device, and storage medium |
| CN109828889A (en) * | 2019-01-31 | 2019-05-31 | 平安科技（深圳）有限公司 | Method, apparatus, computer equipment and the storage medium in monitoring data library |
| CN110109824A (en) * | 2019-04-09 | 2019-08-09 | 平安科技（深圳）有限公司 | Big data automatic regression test method, apparatus, computer equipment and storage medium |
| CN110209730A (en) * | 2019-04-25 | 2019-09-06 | 深圳壹账通智能科技有限公司 | Change synchronous method, device, computer equipment and the computer storage medium of data |
| CN110286883A (en) * | 2019-06-03 | 2019-09-27 | 中国建设银行股份有限公司 | Intelligence database encodes monitoring method, device and electronic equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111125101A (en) | 2020-05-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20150278030A1 (en) | Distributed Database Synchronization Method and System | |
| CN104980519B (en) | Multimachine room storage system | |
| CN101583024B (en) | Distributed network video monitoring system and registration control method thereof | |
| US10372504B2 (en) | Global usage tracking and quota enforcement in a distributed computing system | |
| CN106156318B (en) | System and method for realizing high availability of multi-node database | |
| CN112506659A (en) | Management method of edge device and related device | |
| CN108322358B (en) | Method and device for sending, processing and consuming multi-live distributed messages in different places | |
| WO2016183967A1 (en) | Failure alarm method and apparatus for key component, and big data management system | |
| CN112558997A (en) | Method and device for deploying applications | |
| CN109145060B (en) | Data processing method and device | |
| CN103001809A (en) | Service node state monitoring method for cloud storage system | |
| CN105701099A (en) | Method and device used for executing task in distributed environment, and distributed task execution system | |
| CN110196780B (en) | Method, device, storage medium and electronic device for determining server state | |
| CN105592139A (en) | HA implementation method and device of distributed file system management platform | |
| CN103856354A (en) | Method for achieving unified management of logs of cluster storage system | |
| CN108063832B (en) | Cloud storage system and storage method thereof | |
| WO2019178772A1 (en) | Database master-slave block consistency check method, device and system | |
| KR102225815B1 (en) | Managing data feeds | |
| CN108199912B (en) | Method and device for managing and consuming distributed messages of multiple activities in different places | |
| CN111125101B (en) | Data center table structure consistency monitoring method and system | |
| CN104954505B (en) | Monitoring method and system for DNS data updating overall process | |
| CN109510730A (en) | Distributed system and its monitoring method, device, electronic equipment and storage medium | |
| CN117130730A (en) | Metadata management method for federal Kubernetes cluster | |
| CN104618474B (en) | Facility information store method and device | |
| US10860580B2 (en) | Information processing device, method, and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |