WO2019128497A1 - Procédé de stockage hétérogène de données et dispositif de stockage hétérogène de données - Google Patents

Procédé de stockage hétérogène de données et dispositif de stockage hétérogène de données Download PDF

Info

Publication number
WO2019128497A1
WO2019128497A1 PCT/CN2018/115155 CN2018115155W WO2019128497A1 WO 2019128497 A1 WO2019128497 A1 WO 2019128497A1 CN 2018115155 W CN2018115155 W CN 2018115155W WO 2019128497 A1 WO2019128497 A1 WO 2019128497A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
storage
storage form
heterogeneous
definition
Prior art date
Application number
PCT/CN2018/115155
Other languages
English (en)
Chinese (zh)
Inventor
何东杰
Original Assignee
中国银联股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 中国银联股份有限公司 filed Critical 中国银联股份有限公司
Publication of WO2019128497A1 publication Critical patent/WO2019128497A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22—Indexing; Data structures therefor; Storage structures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/25—Integrating or interfacing systems involving database management systems
    • G06F16/258—Data format conversion from or to a database

Definitions

  • the present invention relates to data processing technologies, and in particular, to a data heterogeneous storage method and a data heterogeneous storage device.
  • the prior art In data heterogeneous storage, the prior art generally only uses multiple copies, and each copy is exactly the same form. If there is a problem with which copy, use another copy as a backup. At the same time, in order to cope with different data application scenarios, the prior art generally adopts different number storage forms to improve the efficiency of data processing. Therefore, for multiple different application scenarios, there will be multiple copies of multiple copies of data.
  • the present invention is directed to a further data heterogeneous storage method and a data heterogeneous storage device.
  • the data heterogeneous storage method of the present invention is for storing data having a plurality of copies, and is characterized by the following steps:
  • Definition step the form of data storage used for each copy of the data definition
  • the conversion storage step converts the data of the plurality of copies into the data storage form defined by the definition step and stores them.
  • the storage form includes: a line storage form, a column storage form, a row and column hybrid storage form, a compressed storage form, a key value pair storage form, and a user customized storage form.
  • a stored form identifier is set for the defined data storage form.
  • the data heterogeneous storage method of the present invention further comprises:
  • the association relationship storing step stores an association relationship between the data storage form and the copy.
  • the data heterogeneous storage device of the present invention is configured to store data having a plurality of copies, and is characterized by:
  • a conversion storage module configured to separately convert data of the plurality of copies into a data storage form defined by the definition step module and store the data.
  • the storage form includes: a line storage form, a column storage form, a row and column hybrid storage form, a compressed storage form, a key value pair storage form, and a user customized storage form.
  • the definition module sets a storage form identifier for the defined data storage form.
  • the conversion storage module is capable of mutually converting data between copies of different data storage forms.
  • the data heterogeneous storage device of the present invention is characterized by further comprising:
  • the association relationship storage module is configured to store an association relationship between the data storage form and the replica.
  • the computer readable medium of the present invention has stored thereon a computer program, characterized in that the computer program is executed by a processor to implement the above-described data heterogeneous storage method.
  • the computer device of the present invention includes a memory, a processor, and a computer program stored on the memory and operable on the processor, wherein the processor executes the computer program to implement the data heterogeneous storage method described above. step.
  • a heterogeneous form storage scheme of multiple copies of data is proposed, so that different applications can be satisfied while ensuring high data availability.
  • the ability to process data in a scenario can improve the efficiency of data storage.
  • heterogeneous storage of multiple copies of data can be configured in combination with actual needs, it can be flexibly configured and widely used.
  • FIG. 1 is a flow chart showing a data heterogeneous storage method of the present invention.
  • Fig. 2 is a block diagram showing the structure of a data heterogeneous storage device of the present invention.
  • the technical idea of the data heterogeneous storage method of the present invention is that for a multi-copy data of a distributed storage system, the storage form of each copy data is processed and stored according to application requirements, and corresponding data conversion functions are provided between different copies. To enable fast recovery from other copies after a disk or data failure. For applications, different replica settings have different identities, combined with application features, access to the corresponding efficient data storage form.
  • FIG. 1 is a flow chart showing a data heterogeneous storage method of the present invention.
  • the data heterogeneous storage method of the present invention includes the following steps:
  • Defining step S100 a data storage form employed for the data definition of each copy
  • the conversion storage step S200 separately converts data of the plurality of copies into a data storage form defined by the definition step and stores the data.
  • the storage form includes: a line storage form, a column storage form, a compressed storage form, a key value pair storage form, a user customized storage form, and the like, and Define the data storage form and set the storage form identifier.
  • the data heterogeneous storage party of the present invention further includes an association relationship storing step S300, in which the association relationship between the data storage format and the replica is stored.
  • Fig. 2 is a block diagram showing the structure of a data heterogeneous storage device of the present invention.
  • the data heterogeneous storage device of the present invention has:
  • a definition module 100 for using a data storage form for each copy of the data definition; and a conversion storage module 200 for converting the data of the plurality of copies into a data storage form defined by the definition step module and performing storage.
  • the storage form includes: a line storage form, a column storage form, a compressed storage form, a key value pair storage form, a user customized storage form, and the like, and
  • the definition module sets a storage form identifier for the defined data storage form.
  • the conversion storage module 200 is capable of mutually converting data between copies of different data storage forms.
  • the data heterogeneous storage device of the present invention further includes: an association storage module 300 for storing an association relationship between the data storage format and the replica.
  • each copy is defined according to the characteristics of the application requirements, and the copies may be the same or different;
  • the application system accesses the data, it specifies the data copy of the access to achieve higher data processing efficiency; the storage system can also automatically identify the optimal data copy according to the data access requirements of the application system, so that the application system does not need to be specified. A copy of the accessed data, shielding the underlying storage of the data.
  • different service scenarios correspond to multiple copies
  • a service profile corresponds to copy A1 A2 A3
  • B service scenario corresponds to copy B1 B2 B3
  • C service scenario corresponds to copy C1 C2 C3.
  • the data heterogeneous storage method of the present embodiment when the data is stored, different copies are backed up according to different service scenarios, so that in the three A B C service scenarios, the corresponding backup copy It is A1 B1 C1, A1 B1 C1 backup each other, saving storage space and improving storage efficiency.
  • the present invention also provides a computer device comprising a memory, a processor, and a computer program stored on the memory and operable on the processor, wherein the processor performs the above-described data heterogeneity when the computer program is executed The steps of the storage method.
  • the data heterogeneous storage method and the data heterogeneous storage device according to the present invention can propose a heterogeneous form storage scheme of multiple copies of data, so that different application scenarios can be satisfied while ensuring high data availability.
  • the ability to handle data needs.

Landscapes

  • Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

L'invention concerne un procédé de stockage hétérogène de données et un dispositif de stockage hétérogène de données. Le procédé de stockage hétérogène de données est utilisé pour stocker des données ayant de multiples copies, et est caractérisé en ce qu'il comprend les étapes suivantes : une étape de définition (S100) : définir une forme de stockage de données adoptée pour les données de chaque copie ; et une étape de conversion et de stockage (S200) : convertir respectivement les données des multiples copies en la forme de stockage de données définie à l'étape de définition et les stocker. Selon le procédé de stockage hétérogène de données et le dispositif de stockage hétérogène de données, dans les conditions où une haute disponibilité des données est garantie, l'exigence de traitement des données dans différentes scènes d'application peut être satisfaite.
PCT/CN2018/115155 2017-12-28 2018-11-13 Procédé de stockage hétérogène de données et dispositif de stockage hétérogène de données WO2019128497A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201711455543.2A CN108304471A (zh) 2017-12-28 2017-12-28 一种数据异构存储方法以及数据异构存储装置
CN201711455543.2 2017-12-28

Publications (1)

Publication Number Publication Date
WO2019128497A1 true WO2019128497A1 (fr) 2019-07-04

Family

ID=62867610

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2018/115155 WO2019128497A1 (fr) 2017-12-28 2018-11-13 Procédé de stockage hétérogène de données et dispositif de stockage hétérogène de données

Country Status (3)

Country Link
CN (1) CN108304471A (fr)
TW (1) TW201931121A (fr)
WO (1) WO2019128497A1 (fr)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108304471A (zh) * 2017-12-28 2018-07-20 中国银联股份有限公司 一种数据异构存储方法以及数据异构存储装置
CN113641673A (zh) * 2021-08-17 2021-11-12 山东勤成健康科技股份有限公司 一种数据异构存储方法以及数据异构存储装置

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090187600A1 (en) * 2008-01-23 2009-07-23 Omx Technology Ab Method of improving replica server performance and a replica server system
CN104572711A (zh) * 2013-10-18 2015-04-29 腾讯科技（深圳）有限公司 一种分布式文档形数据存取方法及装置
CN105095294A (zh) * 2014-05-15 2015-11-25 中兴通讯股份有限公司 一种分布式存储系统中管理异构副本的方法及装置
CN105740295A (zh) * 2014-12-12 2016-07-06 中国移动通信集团公司 一种分布式数据的处理方法及装置
CN108304471A (zh) * 2017-12-28 2018-07-20 中国银联股份有限公司 一种数据异构存储方法以及数据异构存储装置

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8255375B2 (en) * 2010-02-23 2012-08-28 Bank Of America Corporation Content service bus framework
CN107180031B (zh) * 2016-03-09 2021-04-09 华为技术有限公司 分布式存储方法及装置、数据处理方法及装置

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090187600A1 (en) * 2008-01-23 2009-07-23 Omx Technology Ab Method of improving replica server performance and a replica server system
CN104572711A (zh) * 2013-10-18 2015-04-29 腾讯科技（深圳）有限公司 一种分布式文档形数据存取方法及装置
CN105095294A (zh) * 2014-05-15 2015-11-25 中兴通讯股份有限公司 一种分布式存储系统中管理异构副本的方法及装置
CN105740295A (zh) * 2014-12-12 2016-07-06 中国移动通信集团公司 一种分布式数据的处理方法及装置
CN108304471A (zh) * 2017-12-28 2018-07-20 中国银联股份有限公司 一种数据异构存储方法以及数据异构存储装置

Also Published As

Publication number Publication date
TW201931121A (zh) 2019-08-01
CN108304471A (zh) 2018-07-20

Similar Documents

Publication Publication Date Title
US11182404B2 (en) Data replication technique in database management system
US11010103B2 (en) Distributed batch processing of non-uniform data objects
CN102541990B (zh) 利用虚拟分区的数据库重新分布方法和系统
US10860539B2 (en) De-duplication-based remote replication method, and apparatus
US9378233B2 (en) For all entries processing
US20160085633A1 (en) Automated data recovery from remote data object replicas
US9418094B2 (en) Method and apparatus for performing multi-stage table updates
JP6086463B2 (ja) ピアツーピアデータ複製用の方法、デバイス、およびシステム、ならびにマスタノード切替え用の方法、デバイス、およびシステム
WO2019105420A1 (fr) Interrogation de données
EP3432157A1 (fr) Procédé et appareil de traitement de modes de jonction de tables de données
WO2017028394A1 (fr) Procédé et appareil de récupération de données distribuées basée sur des exemples
US20150006589A1 (en) Apparatus and method for expanding a shared-nothing system
US20160306550A1 (en) Constructing a scalable storage device, and scaled storage device
US20150149481A1 (en) Table as query language parameter
WO2014089828A1 (fr) Procédé d'accès à un dispositif de stockage et dispositif de stockage
WO2019128497A1 (fr) Procédé de stockage hétérogène de données et dispositif de stockage hétérogène de données
CN107301214A (zh) 在hive中数据迁移方法、装置及终端设备
JP2017502539A (ja) マルチユーザの省電力コードブックの最適化を実現するためのシステムおよび方法
CN113961580A (zh) 数据查询方法、业务系统以及电子设备
WO2014010038A1 (fr) Système de traitement d'informations
KR101797483B1 (ko) 데이터베이스 관리 시스템에서 쿼리를 프로세싱하기 위한 기법
CN107491544B (zh) 一种增强非关系型数据库分析能力的数据处理平台
WO2018019310A1 (fr) Procédés et dispositifs de sauvegarde et de récupération de données de système big data, et support de stockage informatique
WO2014180395A1 (fr) Procédé et système de stockage de fusion de données massives
JP2006092503A (ja) マルチインスタンス・インメモリ・データベース

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18894738

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 18894738

Country of ref document: EP

Kind code of ref document: A1