WO2019128497A1

WO2019128497A1 - Data heterogeneous storage method and data heterogeneous storage device

Info

Publication number: WO2019128497A1
Application number: PCT/CN2018/115155
Authority: WO
Inventors: 何东杰
Original assignee: 中国银联股份有限公司
Priority date: 2017-12-28
Filing date: 2018-11-13
Publication date: 2019-07-04
Also published as: TW201931121A; CN108304471A

Abstract

A data heterogeneous storage method and a data heterogeneous storage device. The data heterogeneous storage method is used for storing data having multiple copies, and is characterized by comprising the following steps: a definition step (S100): defining a data storage form adopted for the data of each copy; and a conversion and storage step (S200): respectively converting the data of the multiple copies into the data storage form defined in the definition step and storing same. According to the data heterogeneous storage method and the data heterogeneous storage device, under the condition that high availability of the data is guaranteed, the requirement for processing the data under different application scenes can be met.

Description

Data heterogeneous storage method and data heterogeneous storage device

Technical field

The present invention relates to data processing technologies, and in particular, to a data heterogeneous storage method and a data heterogeneous storage device.

Background technique

In data heterogeneous storage, the prior art generally only uses multiple copies, and each copy is exactly the same form. If there is a problem with which copy, use another copy as a backup. At the same time, in order to cope with different data application scenarios, the prior art generally adopts different number storage forms to improve the efficiency of data processing. Therefore, for multiple different application scenarios, there will be multiple copies of multiple copies of data.

Although the prior art can achieve high availability of data, the storage space utilization needs to be improved.

Summary of the invention

In view of the problems, the present invention is directed to a further data heterogeneous storage method and a data heterogeneous storage device.

The data heterogeneous storage method of the present invention is for storing data having a plurality of copies, and is characterized by the following steps:

Definition step, the form of data storage used for each copy of the data definition;

The conversion storage step converts the data of the plurality of copies into the data storage form defined by the definition step and stores them.

Preferably, in the defining step, the storage form includes: a line storage form, a column storage form, a row and column hybrid storage form, a compressed storage form, a key value pair storage form, and a user customized storage form.

Preferably, in the defining step, a stored form identifier is set for the defined data storage form.

Preferably, in the conversion storage step, data conversion between the copies of different data storage forms is possible.

Preferably, the data heterogeneous storage method of the present invention further comprises:

The association relationship storing step stores an association relationship between the data storage form and the copy.

The data heterogeneous storage device of the present invention is configured to store data having a plurality of copies, and is characterized by:

a definition module for the data storage form used for the data definition for each copy;

And a conversion storage module, configured to separately convert data of the plurality of copies into a data storage form defined by the definition step module and store the data.

Preferably, in the definition module, the storage form includes: a line storage form, a column storage form, a row and column hybrid storage form, a compressed storage form, a key value pair storage form, and a user customized storage form.

Preferably, the definition module sets a storage form identifier for the defined data storage form.

Preferably, the conversion storage module is capable of mutually converting data between copies of different data storage forms.

Preferably, the data heterogeneous storage device of the present invention is characterized by further comprising:

The association relationship storage module is configured to store an association relationship between the data storage form and the replica.

The computer readable medium of the present invention has stored thereon a computer program, characterized in that the computer program is executed by a processor to implement the above-described data heterogeneous storage method.

The computer device of the present invention includes a memory, a processor, and a computer program stored on the memory and operable on the processor, wherein the processor executes the computer program to implement the data heterogeneous storage method described above. step.

As described above, according to the data heterogeneous storage method and the data heterogeneous storage device of the present invention, a heterogeneous form storage scheme of multiple copies of data is proposed, so that different applications can be satisfied while ensuring high data availability. The ability to process data in a scenario can improve the efficiency of data storage. Moreover, since heterogeneous storage of multiple copies of data can be configured in combination with actual needs, it can be flexibly configured and widely used.

DRAWINGS

1 is a flow chart showing a data heterogeneous storage method of the present invention.

Fig. 2 is a block diagram showing the structure of a data heterogeneous storage device of the present invention.

Detailed ways

The following are some of the various embodiments of the invention, which are intended to provide a basic understanding of the invention. It is not intended to identify key or critical elements of the invention or the scope of the invention.

The technical idea of the data heterogeneous storage method of the present invention is that for a multi-copy data of a distributed storage system, the storage form of each copy data is processed and stored according to application requirements, and corresponding data conversion functions are provided between different copies. To enable fast recovery from other copies after a disk or data failure. For applications, different replica settings have different identities, combined with application features, access to the corresponding efficient data storage form.

For example, in a big data scenario, we need both fast data query and fast analysis. Data row storage provides fast queries, while data column storage provides fast analysis. Obviously, if there are two copies of data, one uses row storage and the other uses columnar storage. For the data backup scenario, you can choose a more efficient compressed storage format, which further increases the storage space utilization. Therefore, if there are three copies of data, one copy can be in the form of row storage, one copy can be stored in columns, and then one can be used in compressed storage.

Next, the data heterogeneous storage method of the present invention will be described.

As shown in FIG. 1, the data heterogeneous storage method of the present invention includes the following steps:

Defining step S100: a data storage form employed for the data definition of each copy;

The conversion storage step S200: separately converts data of the plurality of copies into a data storage form defined by the definition step and stores the data.

Wherein, in the defining step, the storage form includes: a line storage form, a column storage form, a compressed storage form, a key value pair storage form, a user customized storage form, and the like, and Define the data storage form and set the storage form identifier.

On the other hand, in the conversion storage step, data conversion can be performed between the copies of different data storage forms.

Preferably, the data heterogeneous storage party of the present invention further includes an association relationship storing step S300, in which the association relationship between the data storage format and the replica is stored.

Next, a brief description will be given of the data heterogeneous storage device of the present invention.

As shown in FIG. 2, the data heterogeneous storage device of the present invention has:

a definition module 100 for using a data storage form for each copy of the data definition; and a conversion storage module 200 for converting the data of the plurality of copies into a data storage form defined by the definition step module and performing storage.

Specifically, in the definition module, the storage form includes: a line storage form, a column storage form, a compressed storage form, a key value pair storage form, a user customized storage form, and the like, and The definition module sets a storage form identifier for the defined data storage form.

On the other hand, the conversion storage module 200 is capable of mutually converting data between copies of different data storage forms.

In a preferred embodiment, the data heterogeneous storage device of the present invention further includes: an association storage module 300 for storing an association relationship between the data storage format and the replica.

Next, a specific embodiment of the data heterogeneous storage method and the data heterogeneous storage device of the present invention will be described.

The data heterogeneous storage method of this embodiment includes the following steps:

(1) During the data writing process, the format of each copy is defined according to the characteristics of the application requirements, and the copies may be the same or different;

(2) When data is written, through the data conversion function, other copies of the data are simultaneously generated, and at the same time, the data is stored, thereby forming a plurality of copies of the storage; for example, storing the data as a line, a column, Efficient compression, key-value pairing;

(3) Recording different data storage forms and stored locations in the metadata;

(4) When the application system accesses the data, it specifies the data copy of the access to achieve higher data processing efficiency; the storage system can also automatically identify the optimal data copy according to the data access requirements of the application system, so that the application system does not need to be specified. A copy of the accessed data, shielding the underlying storage of the data.

For example, in the prior art, different service scenarios correspond to multiple copies, for example, A service profile corresponds to copy A1 A2 A3, B service scenario corresponds to copy B1 B2 B3, and C service scenario corresponds to copy C1 C2 C3. On the other hand, in the data heterogeneous storage method of the present embodiment, when the data is stored, different copies are backed up according to different service scenarios, so that in the three A B C service scenarios, the corresponding backup copy It is A1 B1 C1, A1 B1 C1 backup each other, saving storage space and improving storage efficiency.

The present invention also provides a computer device comprising a memory, a processor, and a computer program stored on the memory and operable on the processor, wherein the processor performs the above-described data heterogeneity when the computer program is executed The steps of the storage method.

As described above, the data heterogeneous storage method and the data heterogeneous storage device according to the present invention can propose a heterogeneous form storage scheme of multiple copies of data, so that different application scenarios can be satisfied while ensuring high data availability. The ability to handle data needs.

The above examples mainly illustrate the data heterogeneous storage method and the data heterogeneous storage device of the present invention. Although only a few of the specific embodiments of the present invention have been described, it is understood that the invention may be embodied in many other forms without departing from the spirit and scope of the invention. Accordingly, the present invention is to be construed as illustrative and not restrictive, and the invention may cover various modifications without departing from the spirit and scope of the invention as defined by the appended claims With replacement.

Claims

A data heterogeneous storage method for storing data having multiple copies, characterized in that the method has the following steps:

Definition step, the form of data storage used for each copy of the data definition;

The conversion storage step converts the data of the plurality of copies into the data storage form defined by the definition step and stores them.
The data heterogeneous storage method according to claim 1, wherein

In the definition step, as the storage form, there are: a line storage form, a column storage form, a row and column hybrid storage form, a compressed storage form, a key value pair storage form, and a user customized storage form.
The data heterogeneous storage method according to claim 1, wherein

In the defining step, a storage form identifier is set for the defined data storage form.
The data heterogeneous storage method according to claim 1, wherein

In the conversion storage step, data conversion can be performed between the copies of different data storage forms.
The data heterogeneous storage method according to any one of claims 1 to 4, further comprising: an association relationship storing step of storing an association relationship between the data storage format and the replica.
A data heterogeneous storage device for storing data having a plurality of copies, characterized by comprising:

a definition module for the data storage form used for the data definition for each copy;

And a conversion storage module, configured to separately convert data of the plurality of copies into a data storage form defined by the definition step module and store the data.
The data heterogeneous storage device according to claim 6, wherein

In the definition module, as the storage form, there are: a line storage form, a column storage form, a row and column hybrid storage form, a compressed storage form, a key value pair storage form, and a user customized storage form.
The data heterogeneous storage device according to claim 6, wherein

The definition module sets a storage form identifier for the defined data storage form.
The data heterogeneous storage device according to claim 6, wherein

The conversion storage module is capable of mutually converting data between copies of different data storage forms.
The data heterogeneous storage device according to any one of claims 6 to 9, further comprising:

The association relationship storage module is configured to store an association relationship between the data storage form and the replica.
A computer readable medium having stored thereon a computer program, wherein the computer program is executed by a processor to implement the data heterogeneous storage method according to any one of claims 1 to 5.
A computer device comprising a memory, a processor, and a computer program stored on the memory and operable on the processor, wherein the processor executes the computer program to implement any one of claims 1 to 5. The steps of the data heterogeneous storage method.