CN110121694A - 一种日志管理方法、服务器和数据库系统 - Google Patents

一种日志管理方法、服务器和数据库系统 Download PDF

Info

Publication number
CN110121694A
CN110121694A CN201780055464.4A CN201780055464A CN110121694A CN 110121694 A CN110121694 A CN 110121694A CN 201780055464 A CN201780055464 A CN 201780055464A CN 110121694 A CN110121694 A CN 110121694A
Authority
CN
China
Prior art keywords
affairs
server
data
write
command log
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201780055464.4A
Other languages
English (en)
Other versions
CN110121694B (zh
Inventor
梁晓豪
张慧敏
王伟松
王铁英
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Huawei Technologies Co Ltd
Original Assignee
Huawei Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huawei Technologies Co Ltd filed Critical Huawei Technologies Co Ltd
Priority to CN202010704893.3A priority Critical patent/CN112015591A/zh
Publication of CN110121694A publication Critical patent/CN110121694A/zh
Application granted granted Critical
Publication of CN110121694B publication Critical patent/CN110121694B/zh
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
    • G06F3/065—Replication mechanisms
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/14—Error detection or correction of the data by redundancy in operation
    • G06F11/1402—Saving, restoring, recovering or retrying
    • G06F11/1446—Point-in-time backing up or restoration of persistent data
    • G06F11/1448—Management of the data involved in backup or backup restore
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/14—Error detection or correction of the data by redundancy in operation
    • G06F11/1402—Saving, restoring, recovering or retrying
    • G06F11/1471—Saving, restoring, recovering or retrying involving logging of persistent data for recovery
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/14—Error detection or correction of the data by redundancy in operation
    • G06F11/1402—Saving, restoring, recovering or retrying
    • G06F11/1446—Point-in-time backing up or restoration of persistent data
    • G06F11/1458—Management of the backup or restore process
    • G06F11/1464—Management of the backup or restore process for networked environments
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/14—Error detection or correction of the data by redundancy in operation
    • G06F11/1402—Saving, restoring, recovering or retrying
    • G06F11/1446—Point-in-time backing up or restoration of persistent data
    • G06F11/1458—Management of the backup or restore process
    • G06F11/1469—Backup restoration techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23—Updating
    • G06F16/2358—Change logging, detection, and notification
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23—Updating
    • G06F16/2379—Updates performed during online database operations; commit processing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0614—Improving the reliability of storage systems
    • G06F3/0619—Improving the reliability of storage systems in relation to data integrity, e.g. data losses, bit errors
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
    • G06F3/0659—Command handling arrangements, e.g. command buffers, queues, command scheduling
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/14—Error detection or correction of the data by redundancy in operation
    • G06F11/1402—Saving, restoring, recovering or retrying
    • G06F11/1474—Saving, restoring, recovering or retrying in transactions
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2201/00—Indexing scheme relating to error detection, to error correction, and to monitoring
    • G06F2201/84—Using snapshots, i.e. a logical point-in-time copy of the data

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Quality & Reliability (AREA)
  • Human Computer Interaction (AREA)
  • Databases & Information Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Computer Security & Cryptography (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

本申请提供了一种日志管理方法、服务器和数据库系统。该方法包括：服务器接收第一写事务，生成并存储第一写事务的命令日志，并检测所述第一写事务是否为多分区事务，在该服务器检测到第一写事务为多分区事务时，该服务器生成并存储第一写事务的数据日志。根据本申请公开的技术方案，服务器节点可以用过判断写事务的类型，从而采取不同的日志策略，如果写事务是多分区事务，则生成并存储该写事务的数据日志，后续通过数据日志进行多分区事务的数据恢复，从而保证数据恢复的效率，而命令日志可以用于标记多分区事务，避免了额外的标记信息，减少了系统的复杂性。

Description

PCT国内申请，说明书已公开。

Claims (26)

  1. PCT国内申请，权利要求书已公开。
CN201780055464.4A 2017-12-05 2017-12-05 一种日志管理方法、服务器和数据库系统 Active CN110121694B (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010704893.3A CN112015591A (zh) 2017-12-05 2017-12-05 一种日志管理方法、服务器和数据库系统

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2017/114647 WO2019109256A1 (zh) 2017-12-05 2017-12-05 一种日志管理方法、服务器和数据库系统

Related Child Applications (1)

Application Number Title Priority Date Filing Date
CN202010704893.3A Division CN112015591A (zh) 2017-12-05 2017-12-05 一种日志管理方法、服务器和数据库系统

Publications (2)

Publication Number Publication Date
CN110121694A true CN110121694A (zh) 2019-08-13
CN110121694B CN110121694B (zh) 2020-08-07

Family

ID=66750399

Family Applications (2)

Application Number Title Priority Date Filing Date
CN201780055464.4A Active CN110121694B (zh) 2017-12-05 2017-12-05 一种日志管理方法、服务器和数据库系统
CN202010704893.3A Pending CN112015591A (zh) 2017-12-05 2017-12-05 一种日志管理方法、服务器和数据库系统

Family Applications After (1)

Application Number Title Priority Date Filing Date
CN202010704893.3A Pending CN112015591A (zh) 2017-12-05 2017-12-05 一种日志管理方法、服务器和数据库系统

Country Status (4)

Country Link
US (1) US11748215B2 (zh)
EP (1) EP3696658B1 (zh)
CN (2) CN110121694B (zh)
WO (1) WO2019109256A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111190874A (zh) * 2019-12-27 2020-05-22 天津中科曙光存储科技有限公司 一种分布式存储的高可靠数据日志模块及其构建方法
CN112015591A (zh) * 2017-12-05 2020-12-01 华为技术有限公司 一种日志管理方法、服务器和数据库系统

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11567840B2 (en) * 2020-03-09 2023-01-31 Rubrik, Inc. Node level recovery for clustered databases
US20230195697A1 (en) * 2021-12-22 2023-06-22 Singlestore, Inc. Separation of storage and compute in database management systems

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050033779A1 (en) * 2001-02-19 2005-02-10 Hitachi, Ltd. Database management program, a database managing method and an apparatus therefor
CN1784677A (zh) * 2004-03-31 2006-06-07 微软公司 用于数据库备份的一致性检验的系统和方法
CN101464890A (zh) * 2008-12-30 2009-06-24 中兴通讯股份有限公司 一种混合日志生成方法和解析方法
CN101980203A (zh) * 2010-10-29 2011-02-23 中国科学院声学研究所 一种用于高清媒体的嵌入式文件系统
CN104980307A (zh) * 2015-06-29 2015-10-14 小米科技有限责任公司 数据访问请求的处理方法、装置及数据库服务器
CN106202305A (zh) * 2016-06-30 2016-12-07 北京北信源软件股份有限公司 一种日志处理方法、装置及数据库系统

Family Cites Families (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2003162439A (ja) * 2001-11-22 2003-06-06 Hitachi Ltd ストレージシステム及びその制御方法
CN100504905C (zh) * 2007-11-16 2009-06-24 中国科学院软件研究所 数据库恶意事务处理方法及其系统
US9092475B2 (en) * 2011-11-07 2015-07-28 Sap Se Database log parallelization
CN102662796B (zh) * 2012-03-28 2014-06-18 深圳万兴信息科技股份有限公司 一种数据恢复的方法及系统
CN103049355B (zh) * 2012-12-25 2015-06-17 华为技术有限公司 一种数据库系统恢复方法及设备
CN103729442B (zh) * 2013-12-30 2017-11-24 华为技术有限公司 记录事务日志的方法和数据库引擎
US9323569B2 (en) * 2014-09-10 2016-04-26 Amazon Technologies, Inc. Scalable log-based transaction management
CN104486107A (zh) * 2014-12-05 2015-04-01 曙光信息产业(北京)有限公司 一种日志采集装置及方法
US10459805B2 (en) * 2015-04-03 2019-10-29 Oath Inc. Method and system for data recovery in a data system
US10296632B2 (en) * 2015-06-19 2019-05-21 Sap Se Synchronization on reactivation of asynchronous table replication
US10268743B2 (en) * 2015-06-19 2019-04-23 Sap Se Distributed database transaction protocol
CN105159818B (zh) * 2015-08-28 2018-01-02 东北大学 内存数据管理中日志恢复方法及其仿真系统
EP3696658B1 (en) * 2017-12-05 2023-10-04 Huawei Technologies Co., Ltd. Log management method, server and database system

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050033779A1 (en) * 2001-02-19 2005-02-10 Hitachi, Ltd. Database management program, a database managing method and an apparatus therefor
CN1784677A (zh) * 2004-03-31 2006-06-07 微软公司 用于数据库备份的一致性检验的系统和方法
CN101464890A (zh) * 2008-12-30 2009-06-24 中兴通讯股份有限公司 一种混合日志生成方法和解析方法
CN101980203A (zh) * 2010-10-29 2011-02-23 中国科学院声学研究所 一种用于高清媒体的嵌入式文件系统
CN104980307A (zh) * 2015-06-29 2015-10-14 小米科技有限责任公司 数据访问请求的处理方法、装置及数据库服务器
CN106202305A (zh) * 2016-06-30 2016-12-07 北京北信源软件股份有限公司 一种日志处理方法、装置及数据库系统

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112015591A (zh) * 2017-12-05 2020-12-01 华为技术有限公司 一种日志管理方法、服务器和数据库系统
CN111190874A (zh) * 2019-12-27 2020-05-22 天津中科曙光存储科技有限公司 一种分布式存储的高可靠数据日志模块及其构建方法
CN111190874B (zh) * 2019-12-27 2023-10-03 天津中科曙光存储科技有限公司 一种分布式存储的高可靠数据日志模块及其构建方法

Also Published As

Publication number Publication date
WO2019109256A1 (zh) 2019-06-13
US11748215B2 (en) 2023-09-05
EP3696658A4 (en) 2020-11-18
CN112015591A (zh) 2020-12-01
EP3696658B1 (en) 2023-10-04
EP3696658A1 (en) 2020-08-19
US20200293412A1 (en) 2020-09-17
CN110121694B (zh) 2020-08-07

Similar Documents

Publication Publication Date Title
US11734306B2 (en) Data replication method and storage system
US11256715B2 (en) Data backup method and apparatus
US10216588B2 (en) Database system recovery using preliminary and final slave node replay positions
US11321291B2 (en) Persistent version control for data transfer between heterogeneous data stores
US20150213100A1 (en) Data synchronization method and system
JP5308403B2 (ja) データ処理の障害回復方法、システムおよびプログラム
US10067999B2 (en) High-performance database replication systems and methods
CN110121694A (zh) 一种日志管理方法、服务器和数据库系统
JPH07244597A (ja) 災害復旧機能を提供するために整合性グループを形成する方法および関連するシステム
JPH07239799A (ja) 遠隔データ・シャドーイングを提供する方法および遠隔データ二重化システム
US8762347B1 (en) Method and apparatus for processing transactional file system operations to enable point in time consistent file data recreation
US11409711B2 (en) Barriers for dependent operations among sharded data stores
JP2015527620A (ja) 計算機システム、サーバ、及び、データ管理方法
US20210165768A1 (en) Replication Barriers for Dependent Data Transfers between Data Stores
CN104965835B (zh) 一种分布式文件系统的文件读写方法及装置
CN110121712A (zh) 一种日志管理方法、服务器和数据库系统
WO2017122060A1 (en) Parallel recovery for shared-disk databases
US20230315713A1 (en) Operation request processing method, apparatus, device, readable storage medium, and system
Gibadullin et al. Realization of replication mechanism in PostgreSQL DBMS
CN105760456A (zh) 一种保持数据一致性的方法和装置
US11341159B2 (en) In-stream data load in a replication environment
US11468034B2 (en) Processing delete requests based on change feed of updates
CN116339609A (zh) 一种数据处理方法以及存储设备
CN103095767B (zh) 分布式缓存系统及基于分布式缓存系统的数据重构方法
KR101786874B1 (ko) 데이터의 일관성을 보장하는 상태기기 기반의 복제 및 체크포인트데이터와 복제로그를 이용한 분산 복구를 처리하는 데이터 복제 방법 및 데이터 저장 시스템

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant