JP2019212272A - ソースデータベースの負荷を最小化したデータ整合性検証方法およびシステム - Google Patents
ソースデータベースの負荷を最小化したデータ整合性検証方法およびシステム Download PDFInfo
- Publication number
- JP2019212272A JP2019212272A JP2018206576A JP2018206576A JP2019212272A JP 2019212272 A JP2019212272 A JP 2019212272A JP 2018206576 A JP2018206576 A JP 2018206576A JP 2018206576 A JP2018206576 A JP 2018206576A JP 2019212272 A JP2019212272 A JP 2019212272A
- Authority
- JP
- Japan
- Prior art keywords
- data
- consistency
- module
- change
- source database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims description 26
- 230000008859 change Effects 0.000 claims abstract description 74
- 238000012795 verification Methods 0.000 claims abstract description 18
- 238000013075 data extraction Methods 0.000 claims abstract description 17
- 238000011084 recovery Methods 0.000 claims description 26
- 239000000284 extract Substances 0.000 claims description 14
- 238000007726 management method Methods 0.000 claims description 10
- 238000013496 data integrity verification Methods 0.000 claims description 7
- 101000857759 Homo sapiens Probable G-protein coupled receptor 162 Proteins 0.000 claims 1
- 102100025358 Probable G-protein coupled receptor 162 Human genes 0.000 claims 1
- 238000012545 processing Methods 0.000 abstract description 7
- 230000008569 process Effects 0.000 description 10
- 230000006870 function Effects 0.000 description 6
- 230000005012 migration Effects 0.000 description 5
- 238000013508 migration Methods 0.000 description 5
- 230000010076 replication Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 230000008901 benefit Effects 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/275—Synchronous replication
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/0706—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation the processing taking place on a specific hardware platform or in a specific software environment
- G06F11/0709—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation the processing taking place on a specific hardware platform or in a specific software environment in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/0751—Error or fault detection not based on redundancy
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1471—Saving, restoring, recovering or retrying involving logging of persistent data for recovery
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/16—Error detection or correction of the data by redundancy in hardware
- G06F11/20—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
- G06F11/2053—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where persistent mass storage functionality or persistent mass storage control functionality is redundant
- G06F11/2056—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where persistent mass storage functionality or persistent mass storage control functionality is redundant by mirroring
- G06F11/2082—Data synchronisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2358—Change logging, detection, and notification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
- G06F16/285—Clustering or classification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2201/00—Indexing scheme relating to error detection, to error correction, and to monitoring
- G06F2201/80—Database-specific techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2201/00—Indexing scheme relating to error detection, to error correction, and to monitoring
- G06F2201/82—Solving problems relating to consistency
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Quality & Reliability (AREA)
- Computing Systems (AREA)
- Computer Security & Cryptography (AREA)
- Computer Hardware Design (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Debugging And Monitoring (AREA)
Abstract
Description
20:第1運営サーバー
22:ソースデータベース
24:DBMSトランザクションログ
30:第2運営サーバー
32:ターゲットデータベース
100:整合性検証サーバー
102:リポジトリ
110:変化データ抽出モジュール
120:パターン分析器モジュール
130:ルールエンジンモジュール
140 : 整合性実行モジュール
150:ダンプモジュール
160:比較モジュール
170:リカバリーモジュール
Claims (5)
- クライアントとソースデータベースを運営する運営サーバーとの間のパケットを抽出するか、トランザクションログやトリガー情報から変更データを抽出する変更データ抽出手段と、
前記変更データ抽出手段によって抽出した変更データをパターン分析して、変更情報が保存されたDML変更パターンビットセットデータを生成するパターン分析器と、
前記DML変更パターンビットセットデータから規則を探して整合性プロファイルを生成するルールエンジンモジュールと、
前記ルールエンジンモジュールの整合性プロファイルに従って整合性検証を実行する整合性実行モジュールと、
を含む、ソースデータベースの負荷を最小化したデータ整合性検証システム。 - 前記変更データ抽出手段は、
ネットワーク環境のスイッチまたはタップ設備からパケットデータをコピーしてSQL変更データを抽出するスニッフィングモジュール、ネットワークパケットを中継しつつSQL変更データを抽出するプロキシモジュール、第1運営サーバーのDBMSで復旧のために生成したトランザクションログを持ち込んで変更データを抽出するトランザクションログモジュール、変更データ履歴情報を残すことができるトリガー機能で変更データを抽出するモジュール、のうちいずれか一つであることを特徴とする、請求項1に記載のソースデータベースの負荷を最小化したデータ整合性検証システム。 - 前記パターン分析器は、
分析対象テーブル目録を持ち込んだ後、キューの保存領域から変更データを持ち込んでDML変更パターンビットセットデータを生成した後、リポジトリに保存することを特徴とする、請求項1に記載のソースデータベースの負荷を最小化したデータ整合性検証システム。 - 変更データ抽出手段と、パターン分析器、ルールエンジンモジュール、および整合性実行モジュールを含む整合性検証サーバーによるデータ整合性検証方法において、
前記変更データ抽出手段がクライアントとソースデータベースを運営する運営サーバーとの間のパケットを抽出するか、トランザクションログやトリガー情報から変更データを抽出する第1段階と、
前記パターン分析器が前記第1段階で抽出した変更データをパターン分析して変更情報が保存されたDML変更パターンビットセットデータを生成する第2段階と、
前記ルールエンジンモジュールが前記DML変更パターンビットセットデータから規則を探して整合性プロファイルを生成する第3段階と、
前記整合性実行モジュールが前記整合性プロファイルに従って整合性検証を実行する第4段階と、
を含む、ソースデータベースの負荷を最小化したデータ整合性検証方法。 - 前記第4段階は、
対象テーブル情報と整合性プロファイルを持ち込んだ後にソースデータベースの負荷を測定して実行可能の可否を確認した後、ダンプモジュールの並列度を設定した後にダンプモジュールを実行してソースおよびターゲットデータベースからデータを抽出してGRCAに基づいて整合性データを生成した後、比較モジュールを開始してデータ整合性をチェックし、不一致が検出され復旧データがある場合、リカバリーモジュールを実行してデータ同期化復旧を実行することを特徴とする、請求項4に記載のソースデータベースの負荷を最小化したデータ整合性検証方法。
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
KR1020180062876A KR101917807B1 (ko) | 2018-05-31 | 2018-05-31 | 원본 데이터베이스의 부하를 최소화한 데이터 정합성 검증 방법 및 시스템 |
KR10-2018-0062876 | 2018-05-31 |
Publications (2)
Publication Number | Publication Date |
---|---|
JP2019212272A true JP2019212272A (ja) | 2019-12-12 |
JP6711884B2 JP6711884B2 (ja) | 2020-06-17 |
Family
ID=64024429
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
JP2018206576A Active JP6711884B2 (ja) | 2018-05-31 | 2018-11-01 | ソースデータベースの負荷を最小化したデータ整合性検証方法およびシステム |
Country Status (4)
Country | Link |
---|---|
US (1) | US20190370368A1 (ja) |
JP (1) | JP6711884B2 (ja) |
KR (1) | KR101917807B1 (ja) |
GB (1) | GB2574282A (ja) |
Families Citing this family (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
KR102225258B1 (ko) | 2019-04-18 | 2021-03-10 | 주식회사 실크로드소프트 | 데이터베이스 시스템에서 효율적인 변경 데이터 캡쳐를 제공하기 위한 컴퓨터 프로그램 |
CN110990414B (zh) * | 2019-10-31 | 2023-06-16 | 口碑(上海)信息技术有限公司 | 一种数据处理方法以及装置 |
CN112231403B (zh) * | 2020-10-15 | 2024-01-30 | 北京人大金仓信息技术股份有限公司 | 数据同步的一致性校验方法、装置、设备和存储介质 |
CN112363873A (zh) * | 2020-11-27 | 2021-02-12 | 上海爱数信息技术股份有限公司 | 一种分布式一致性备份恢复系统及其备份方法 |
KR102463665B1 (ko) * | 2021-02-18 | 2022-11-09 | (주)알투비솔루션 | 원격 dbms 테이블간 고성능 테이블 데이터 정합성 검증 시스템 |
KR102669856B1 (ko) * | 2021-05-25 | 2024-05-29 | 손철규 | 서버간 네트워크가 단절된 망분리 환경에서의 dbms 테이블 정합성 검증 및 보정 시스템 |
KR20220159523A (ko) * | 2021-05-25 | 2022-12-05 | (주)알투비솔루션 | 서버간 네트워크가 단절된 망분리 환경에서의 cdc 방식의 db 복제 시스템 |
KR102431846B1 (ko) | 2022-01-26 | 2022-08-11 | (주) 다윈아이씨티 | 플랫폼 마이그레이션에 대한 검증 방법, 장치 및 시스템 |
Family Cites Families (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7257689B1 (en) * | 2004-10-15 | 2007-08-14 | Veritas Operating Corporation | System and method for loosely coupled temporal storage management |
US8751441B2 (en) * | 2008-07-31 | 2014-06-10 | Sybase, Inc. | System, method, and computer program product for determining SQL replication process |
US9171029B2 (en) * | 2013-01-31 | 2015-10-27 | International Business Machines Corporation | Performing batches of selective assignments in a vector friendly manner |
-
2018
- 2018-05-31 KR KR1020180062876A patent/KR101917807B1/ko active IP Right Grant
- 2018-09-17 US US16/133,415 patent/US20190370368A1/en not_active Abandoned
- 2018-09-20 GB GB1815308.0A patent/GB2574282A/en not_active Withdrawn
- 2018-11-01 JP JP2018206576A patent/JP6711884B2/ja active Active
Also Published As
Publication number | Publication date |
---|---|
JP6711884B2 (ja) | 2020-06-17 |
GB201815308D0 (en) | 2018-11-07 |
KR101917807B1 (ko) | 2018-11-13 |
GB2574282A (en) | 2019-12-04 |
US20190370368A1 (en) | 2019-12-05 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
JP6711884B2 (ja) | ソースデータベースの負荷を最小化したデータ整合性検証方法およびシステム | |
Deng et al. | The Data Civilizer System. | |
US8688622B2 (en) | Methods and systems for loading data into a temporal data warehouse | |
EP3120261B1 (en) | Dependency-aware transaction batching for data replication | |
US10452625B2 (en) | Data lineage analysis | |
CN108647357B (zh) | 数据查询的方法及装置 | |
AU2015206487B2 (en) | Database key identification | |
US8478797B2 (en) | Atomic deletion of database data categories | |
US20130041900A1 (en) | Script Reuse and Duplicate Detection | |
US20180075124A1 (en) | Consistent query execution in hybrid dbms | |
CN109522315B (zh) | 一种数据库处理方法及系统 | |
CN111259004B (zh) | 一种存储引擎中数据索引的方法以及相关装置 | |
US20080201290A1 (en) | Computer-implemented methods, systems, and computer program products for enhanced batch mode processing of a relational database | |
CN113420026A (zh) | 数据库表结构变更方法、装置、设备及存储介质 | |
CN115145943B (zh) | 多数据源元数据快速比对方法、系统、设备和存储介质 | |
CN112835918A (zh) | 一种MySQL数据库增量同步实现方法 | |
KR20100134355A (ko) | 데이터베이스의 논리적 데이터 오류 복구방법 | |
US11023449B2 (en) | Method and system to search logs that contain a massive number of entries | |
CN113961934A (zh) | 基于公开开源漏洞的多层级关联源代码方法 | |
CN109558153B (zh) | 版本一致性确定方法、装置及系统 | |
CN114153830B (zh) | 数据验证方法及其装置、计算机存储介质、电子设备 | |
Shaik et al. | Assimilating sense into disaster recovery databases and judgement framing proceedings for the fastest recovery | |
US20230222038A1 (en) | Continuous Data Protection Using Retroactive Backup Snapshots | |
Do et al. | Mining and creating a software repositories dataset | |
CN116881284A (zh) | 结构化查询语句的数据回补方法、装置、设备及存储介质 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
A621 | Written request for application examination |
Free format text: JAPANESE INTERMEDIATE CODE: A621 Effective date: 20181101 |
|
A131 | Notification of reasons for refusal |
Free format text: JAPANESE INTERMEDIATE CODE: A131 Effective date: 20191029 |
|
A521 | Request for written amendment filed |
Free format text: JAPANESE INTERMEDIATE CODE: A523 Effective date: 20200128 |
|
TRDD | Decision of grant or rejection written | ||
A01 | Written decision to grant a patent or to grant a registration (utility model) |
Free format text: JAPANESE INTERMEDIATE CODE: A01 Effective date: 20200512 |
|
A61 | First payment of annual fees (during grant procedure) |
Free format text: JAPANESE INTERMEDIATE CODE: A61 Effective date: 20200528 |
|
R150 | Certificate of patent or registration of utility model |
Ref document number: 6711884 Country of ref document: JP Free format text: JAPANESE INTERMEDIATE CODE: R150 |
|
R250 | Receipt of annual fees |
Free format text: JAPANESE INTERMEDIATE CODE: R250 |
|
R250 | Receipt of annual fees |
Free format text: JAPANESE INTERMEDIATE CODE: R250 |