CN111314479B - 一种数据处理方法和设备 - Google Patents

一种数据处理方法和设备 Download PDF

Info

Publication number
CN111314479B
CN111314479B CN202010118286.9A CN202010118286A CN111314479B CN 111314479 B CN111314479 B CN 111314479B CN 202010118286 A CN202010118286 A CN 202010118286A CN 111314479 B CN111314479 B CN 111314479B
Authority
CN
China
Prior art keywords
server
slave
data
log
data storage
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202010118286.9A
Other languages
English (en)
Chinese (zh)
Other versions
CN111314479A (zh
Inventor
阳振坤
赵裕众
师文汇
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Oceanbase Technology Co Ltd
Original Assignee
Beijing Oceanbase Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Oceanbase Technology Co Ltd filed Critical Beijing Oceanbase Technology Co Ltd
Priority to CN202010118286.9A priority Critical patent/CN111314479B/zh
Publication of CN111314479A publication Critical patent/CN111314479A/zh
Application granted granted Critical
Publication of CN111314479B publication Critical patent/CN111314479B/zh
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/18—File system types
    • G06F16/182—Distributed file systems
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/16—Error detection or correction of the data by redundancy in hardware
    • G06F11/1658—Data re-synchronization of a redundant component, or initial sync of replacement, additional or spare unit
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/16—Error detection or correction of the data by redundancy in hardware
    • G06F11/20—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
    • G06F11/202—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where processing functionality is redundant
    • G06F11/2023—Failover techniques
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/18—File system types
    • G06F16/182—Distributed file systems
    • G06F16/184—Distributed file systems implemented as replicated file system
    • G06F16/1844—Management specifically adapted to replicated file systems
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23—Updating
    • G06F16/2379—Updates performed during online database operations; commit processing
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/06—Management of faults, events, alarms or notifications
    • H04L41/0654—Management of faults, events, alarms or notifications using network fault recovery
    • H04L41/0668—Management of faults, events, alarms or notifications using network fault recovery by dynamic selection of recovery network elements, e.g. replacement by the most appropriate element after failure
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/30—Decision processes by autonomous network management units using voting and bidding
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2201/00—Indexing scheme relating to error detection, to error correction, and to monitoring
    • G06F2201/85—Active fault masking without idle spares

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Databases & Information Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Quality & Reliability (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Retry When Errors Occur (AREA)
  • Debugging And Monitoring (AREA)
  • Hardware Redundancy (AREA)
CN202010118286.9A 2016-06-20 2016-06-20 一种数据处理方法和设备 Active CN111314479B (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010118286.9A CN111314479B (zh) 2016-06-20 2016-06-20 一种数据处理方法和设备

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010118286.9A CN111314479B (zh) 2016-06-20 2016-06-20 一种数据处理方法和设备
CN201610444320.5A CN106899648B (zh) 2016-06-20 2016-06-20 一种数据处理方法和设备

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
CN201610444320.5A Division CN106899648B (zh) 2016-06-20 2016-06-20 一种数据处理方法和设备

Publications (2)

Publication Number Publication Date
CN111314479A CN111314479A (zh) 2020-06-19
CN111314479B true CN111314479B (zh) 2022-08-23

Family

ID=59190336

Family Applications (2)

Application Number Title Priority Date Filing Date
CN202010118286.9A Active CN111314479B (zh) 2016-06-20 2016-06-20 一种数据处理方法和设备
CN201610444320.5A Active CN106899648B (zh) 2016-06-20 2016-06-20 一种数据处理方法和设备

Family Applications After (1)

Application Number Title Priority Date Filing Date
CN201610444320.5A Active CN106899648B (zh) 2016-06-20 2016-06-20 一种数据处理方法和设备

Country Status (18)

Country Link
US (2) US10798168B2 (enExample)
EP (1) EP3474516B1 (enExample)
JP (1) JP6684367B2 (enExample)
KR (1) KR102145136B1 (enExample)
CN (2) CN111314479B (enExample)
AU (1) AU2017282817B2 (enExample)
BR (1) BR112018076689B1 (enExample)
CA (1) CA3028504C (enExample)
ES (1) ES2809174T3 (enExample)
MX (1) MX387988B (enExample)
MY (1) MY191234A (enExample)
PH (1) PH12018502712B1 (enExample)
PL (1) PL3474516T3 (enExample)
RU (1) RU2714098C1 (enExample)
SG (1) SG11201811389PA (enExample)
TW (1) TWI689181B (enExample)
WO (1) WO2017219857A1 (enExample)
ZA (1) ZA201900306B (enExample)

Families Citing this family (21)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111314479B (zh) 2016-06-20 2022-08-23 北京奥星贝斯科技有限公司 一种数据处理方法和设备
CN107105032B (zh) * 2017-04-20 2019-08-06 腾讯科技（深圳）有限公司 节点设备运行方法及节点设备
CN107908370B (zh) * 2017-11-30 2021-07-06 新华三技术有限公司 数据存储方法及装置
CN108616598B (zh) * 2018-05-10 2021-01-01 新华三技术有限公司成都分公司 数据同步方法、装置和分布式存储系统
US12013856B2 (en) 2018-08-13 2024-06-18 Amazon Technologies, Inc. Burst performance of database queries according to query size
CN109117312B (zh) * 2018-08-23 2022-03-01 北京小米智能科技有限公司 数据恢复方法及装置
CN110968561B (zh) * 2018-09-30 2024-02-13 北京国双科技有限公司 日志存储方法和分布式系统
CN111352943A (zh) * 2018-12-24 2020-06-30 华为技术有限公司 实现数据一致性的方法和装置、服务器和终端
US11308100B2 (en) * 2019-06-25 2022-04-19 Amazon Technologies, Inc. Dynamically assigning queries to secondary query processing resources
CN110275802B (zh) * 2019-06-27 2025-10-03 北京百度网讯科技有限公司 一种服务切换方法、装置及电子设备
CN110597673B (zh) * 2019-09-25 2021-04-30 腾讯科技（深圳）有限公司 存储系统的容灾方法、装置、设备及计算机可读存储介质
CN112835885B (zh) * 2019-11-22 2023-09-01 北京金山云网络技术有限公司 一种分布式表格存储的处理方法、装置及系统
CN111159292A (zh) * 2019-12-20 2020-05-15 深圳前海微众银行股份有限公司 一种数据存储方法及装置
CN111209333B (zh) * 2019-12-20 2023-06-20 中国平安财产保险股份有限公司 数据更新方法、装置、终端及存储介质
CN112434054A (zh) * 2020-12-09 2021-03-02 杭州海量存储技术有限公司 一种审计日志的更新方法及装置
CN112615944B (zh) * 2020-12-11 2023-07-11 互联网域名系统北京市工程研究中心有限公司 分布式dns主辅同步的方法和系统
CN113472469B (zh) * 2021-07-27 2023-12-05 厦门亿联网络技术股份有限公司 一种数据同步方法、装置、设备及存储介质
CN114090545A (zh) * 2021-11-22 2022-02-25 中国建设银行股份有限公司 分布式数据库的维护方法、装置、存储介质及程序产品
CN114201473A (zh) * 2021-12-15 2022-03-18 上海达梦数据库有限公司 日志管理方法及服务器集群
CN115547051B (zh) * 2022-10-13 2025-10-31 上海电机学院 一种基于边缘计算的实时车行目标检测和避障方法及系统
CN116095098B (zh) * 2022-11-28 2024-04-09 华南农业大学 数据的分布式存储方法及装置

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP2429134A1 (en) * 2009-06-09 2012-03-14 ZTE Corporation Method and apparatus for checking and synchronizing data block in distributed file system
US8301600B1 (en) * 2010-11-15 2012-10-30 Amazon Technologies, Inc. Failover recovery in a distributed data store
CN103546579A (zh) * 2013-11-07 2014-01-29 陈靓 一种用数据日志提高分布式存储系统可用性的方法
CN104506643A (zh) * 2014-12-30 2015-04-08 北京奇虎科技有限公司 分布式存储系统的服务器及服务器之间的数据同步方法
CN104657382A (zh) * 2013-11-21 2015-05-27 阿里巴巴集团控股有限公司 用于MySQL主从服务器数据一致性检测的方法和装置
CN105468297A (zh) * 2015-11-18 2016-04-06 临沂大学 一种云存储系统内主从设备数据快速同步的方法

Family Cites Families (25)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2005502957A (ja) * 2001-09-06 2005-01-27 ビーイーエイ　システムズ，　インコーポレイテッド 厳密に一回のキャッシュフレームワーク
KR100483742B1 (ko) * 2002-02-18 2005-04-18 주식회사 케이티프리텔 이동 통신 단말기를 이용한 대금 지불 및 개인 정보 관리방법
US7158985B1 (en) * 2003-04-09 2007-01-02 Cisco Technology, Inc. Method and apparatus for efficient propagation of large datasets under failure conditions
US7188273B2 (en) * 2003-11-24 2007-03-06 Tsx Inc. System and method for failover
US7577661B2 (en) * 2005-06-30 2009-08-18 Microsoft Corporation Extensible and automatically replicating server farm configuration management infrastructure
JP4998549B2 (ja) * 2007-02-28 2012-08-15 富士通株式会社 メモリミラー化制御プログラム、メモリミラー化制御方法およびメモリミラー化制御装置
KR100926880B1 (ko) * 2007-05-21 2009-11-16 엔에이치엔(주) Ｄｂｍｓ에서의 데이터 복제 방법 및 시스템
US7631214B2 (en) * 2007-05-31 2009-12-08 International Business Machines Corporation Failover processing in multi-tier distributed data-handling systems
KR101001946B1 (ko) 2008-10-22 2010-12-17 한국과학기술연구원 수소 저장 물질 및 그 제조 방법
KR20100044428A (ko) * 2008-10-22 2010-04-30 현대중공업 주식회사 다중서버 상위구조를 갖는 선박자동화 시스템
JP5039891B2 (ja) * 2009-10-19 2012-10-03 インターナショナル・ビジネス・マシーンズ・コーポレーション データベースの複製を生成する装置及び方法
CN101741911B (zh) 2009-12-18 2014-04-30 中兴通讯股份有限公司 基于多副本协同的写操作方法、系统及节点
US8572022B2 (en) * 2010-03-02 2013-10-29 Microsoft Corporation Automatic synchronization conflict resolution
US8510270B2 (en) * 2010-07-27 2013-08-13 Oracle International Corporation MYSQL database heterogeneous log based replication
US9251021B2 (en) * 2011-05-23 2016-02-02 Bradley Gene Calder Asynchronous replication in a distributed storage environment
US9348883B2 (en) * 2011-06-01 2016-05-24 Clustrix, Inc. Systems and methods for replication replay in a relational database
CN103580891A (zh) * 2012-07-27 2014-02-12 腾讯科技（深圳）有限公司 一种数据同步的方法、系统及服务器
JP6056453B2 (ja) * 2012-12-20 2017-01-11 富士通株式会社 プログラム、データ管理方法および情報処理装置
CN104346373B (zh) * 2013-07-31 2017-12-15 华为技术有限公司 分区日志队列同步管理方法及设备
CN103560906B (zh) * 2013-10-22 2017-01-25 珠海多玩信息技术有限公司 数据复制的方法及装置
JP6318902B2 (ja) * 2014-06-20 2018-05-09 富士通株式会社 冗長システムおよび冗長化方法
US9367410B2 (en) * 2014-09-12 2016-06-14 Facebook, Inc. Failover mechanism in a distributed computing system
CN105426439B (zh) * 2015-11-05 2022-07-05 腾讯科技（深圳）有限公司 一种元数据的处理方法和装置
CN105554142B (zh) * 2015-12-24 2019-02-01 北京奇虎科技有限公司 消息推送的方法、装置及系统
CN111314479B (zh) 2016-06-20 2022-08-23 北京奥星贝斯科技有限公司 一种数据处理方法和设备

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP2429134A1 (en) * 2009-06-09 2012-03-14 ZTE Corporation Method and apparatus for checking and synchronizing data block in distributed file system
US8301600B1 (en) * 2010-11-15 2012-10-30 Amazon Technologies, Inc. Failover recovery in a distributed data store
CN103546579A (zh) * 2013-11-07 2014-01-29 陈靓 一种用数据日志提高分布式存储系统可用性的方法
CN104657382A (zh) * 2013-11-21 2015-05-27 阿里巴巴集团控股有限公司 用于MySQL主从服务器数据一致性检测的方法和装置
CN104506643A (zh) * 2014-12-30 2015-04-08 北京奇虎科技有限公司 分布式存储系统的服务器及服务器之间的数据同步方法
CN105468297A (zh) * 2015-11-18 2016-04-06 临沂大学 一种云存储系统内主从设备数据快速同步的方法

Also Published As

Publication number Publication date
US10693965B2 (en) 2020-06-23
JP2019526106A (ja) 2019-09-12
CN106899648A (zh) 2017-06-27
RU2714098C1 (ru) 2020-02-11
PH12018502712A1 (en) 2019-07-29
MY191234A (en) 2022-06-10
MX2018016249A (es) 2019-08-29
US20190149605A1 (en) 2019-05-16
BR112018076689B1 (pt) 2021-09-08
ZA201900306B (en) 2021-01-27
ES2809174T3 (es) 2021-03-03
AU2017282817A1 (en) 2019-01-31
KR20190020099A (ko) 2019-02-27
JP6684367B2 (ja) 2020-04-22
AU2017282817B2 (en) 2020-01-02
BR112018076689A2 (pt) 2019-04-02
TWI689181B (zh) 2020-03-21
PH12018502712B1 (en) 2022-01-19
CN106899648B (zh) 2020-02-14
MX387988B (es) 2025-03-19
EP3474516A1 (en) 2019-04-24
PL3474516T3 (pl) 2021-02-08
US20200128076A1 (en) 2020-04-23
CA3028504A1 (en) 2017-12-28
WO2017219857A1 (zh) 2017-12-28
CA3028504C (en) 2021-02-16
CN111314479A (zh) 2020-06-19
TW201801495A (zh) 2018-01-01
EP3474516B1 (en) 2020-05-13
KR102145136B1 (ko) 2020-08-18
US10798168B2 (en) 2020-10-06
SG11201811389PA (en) 2019-01-30
EP3474516A4 (en) 2019-04-24

Similar Documents

Publication Publication Date Title
CN111314479B (zh) 一种数据处理方法和设备
CN116527696B (zh) 用于在区块链网络中删除节点的系统和方法
US10255343B2 (en) Initialization protocol for a peer-to-peer replication environment
CN112148798B (zh) 应用于分布式系统的数据处理方法及装置
US10402115B2 (en) State machine abstraction for log-based consensus protocols
EP3435604B1 (en) Service processing method, device, and system
US9798639B2 (en) Failover system and method replicating client message to backup server from primary server
CN105930498A (zh) 一种分布式数据库的管理方法及系统
US20190052709A1 (en) Clustered storage system synchronization
TW200805079A (en) Consolidating session information for a cluster of sessions in a coupled session environment
CN105183544B (zh) 一种非阻塞式容错的分布式事务提交方法及系统
CN106331081B (zh) 一种信息同步方法及装置
US20210218827A1 (en) Methods, devices and systems for non-disruptive upgrades to a replicated state machine in a distributed computing environment
US11500812B2 (en) Intermediate file processing method, client, server, and system
CN111488247A (zh) 一种管控节点多次容错的高可用方法及设备
US11288004B1 (en) Consensus-based authority selection in replicated network-accessible block storage devices
WO2021259240A1 (zh) 分布式事务处理方法、终端及计算机可读存储介质
WO2025246835A1 (zh) 基于快速同步机制的数据管理
WO2025246814A1 (zh) 高可用数据管理
CN117931830A (zh) 数据恢复方法、装置、电子设备、存储介质及程序产品
CN114090545A (zh) 分布式数据库的维护方法、装置、存储介质及程序产品

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
TA01 Transfer of patent application right

Effective date of registration: 20201012

Address after: Cayman Enterprise Centre, 27 Hospital Road, George Town, Grand Cayman, British Islands

Applicant after: Advanced innovation technology Co.,Ltd.

Address before: A four-storey 847 mailbox in Grand Cayman Capital Building, British Cayman Islands

Applicant before: Alibaba Group Holding Ltd.

Effective date of registration: 20201012

Address after: Cayman Enterprise Centre, 27 Hospital Road, George Town, Grand Cayman, British Islands

Applicant after: Innovative advanced technology Co.,Ltd.

Address before: Cayman Enterprise Centre, 27 Hospital Road, George Town, Grand Cayman, British Islands

Applicant before: Advanced innovation technology Co.,Ltd.

TA01 Transfer of patent application right
TA01 Transfer of patent application right

Effective date of registration: 20210129

Address after: 801-10, Section B, 8th floor, 556 Xixi Road, Xihu District, Hangzhou City, Zhejiang Province 310000

Applicant after: Ant financial (Hangzhou) Network Technology Co.,Ltd.

Address before: Ky1-9008 business centre, 27 Hospital Road, Georgetown, grand caiman, UK

Applicant before: Innovative advanced technology Co.,Ltd.

TA01 Transfer of patent application right
TA01 Transfer of patent application right

Effective date of registration: 20210902

Address after: 100020 unit 02, 901, floor 9, unit 1, building 1, No.1, East Third Ring Middle Road, Chaoyang District, Beijing

Applicant after: Beijing Aoxing Beisi Technology Co.,Ltd.

Address before: 801-10, Section B, 8th floor, 556 Xixi Road, Xihu District, Hangzhou City, Zhejiang Province 310000

Applicant before: Ant financial (Hangzhou) Network Technology Co.,Ltd.

TA01 Transfer of patent application right
GR01 Patent grant
GR01 Patent grant