WO2021224960A1 - Storage device, storage method, and program - Google Patents
Storage device, storage method, and program Download PDFInfo
- Publication number
- WO2021224960A1 WO2021224960A1 PCT/JP2020/018517 JP2020018517W WO2021224960A1 WO 2021224960 A1 WO2021224960 A1 WO 2021224960A1 JP 2020018517 W JP2020018517 W JP 2020018517W WO 2021224960 A1 WO2021224960 A1 WO 2021224960A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- data
- cluster
- storage
- feature amount
- unit
- Prior art date
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
Definitions
- the present invention relates to a storage device, a storage method, and a program.
- Parquet divides the column data into a plurality of blocks (LowGroup) and holds the feature amount of each block.
- the feature quantity is, for example, the minimum and maximum values of the data in the block, the hash value of the binary, or the vector representation of the character string.
- the column data that stores the age is divided into three blocks and stored, and the minimum and maximum values of the data in the first block are 19 and 31, and the minimum value of the data in the second block. It is assumed that the maximum values are 10 and 25, and the minimum and maximum values of the data in the third block are 35 and 70.
- the second block may be skipped and the first and third blocks may be read.
- the present invention has been made in view of the above, and an object of the present invention is to realize efficient reading and writing of data.
- the storage device of one aspect of the present invention is a storage device that divides a data group into a plurality of blocks and stores the data, and stores the data by dividing the data into the blocks according to the feature amount, and the characteristics of the input data.
- One aspect of the storage method of the present invention is a storage method in which a data group is divided into a plurality of blocks and stored, and the storage divides the data into the blocks according to the feature amount and stores the data.
- the feature amount of the input data is calculated, the data is temporarily held in the cluster corresponding to the feature amount, and the data held by the cluster is written to the corresponding block for each of the clusters at a predetermined timing. ..
- FIG. 1 is a diagram showing an example of the configuration of the data storage system of the present embodiment.
- FIG. 2 is a diagram for explaining the flow of data.
- FIG. 3 is a diagram showing an example of a storage data structure.
- FIG. 4 is a flowchart showing an example of the processing flow when new data is input.
- FIG. 5 is a diagram showing a state in which a data cluster is added.
- FIG. 6 is a flowchart showing an example of a flow of processing for storing data in the storage.
- FIG. 7 is a diagram showing a state in which a new block is added to the storage.
- FIG. 8 is a diagram showing an example of the hardware configuration of the cache device.
- the configuration of the data storage system of the present embodiment will be described with reference to FIG.
- the data storage system of FIG. 1 includes a cache device 10 and a storage 20, and stores column data having the same attributes. Further, the data storage system returns the data stored in the storage 20 in response to the query.
- the cache device 10 includes a feature amount calculation unit 11, a storage unit 12, and a management unit 13.
- the cache device 10 calculates the feature amount of the newly written input data, temporarily holds the feature amount in the data cluster according to the feature amount of the input data, and when a predetermined write condition is satisfied, for each data cluster, The data held by the data cluster is collectively written to the storage 20.
- the feature amount calculation unit 11 has a function of calculating the feature amount of the input data and distributing the input data to the data cluster based on the feature amount.
- the feature amount is information for allocating data to a plurality of data clusters.
- the feature quantity is, for example, information indicating a range of numerical values when the data is a numerical value, a hash value of the binary data when the data is binary data, and a vector representation of the character string when the data is a character string. ..
- the storage unit 12 has a plurality of areas for holding the cluster feature amount and the data cluster, and has a function of temporarily holding the input data.
- the cluster feature is the feature of the data held by the corresponding data cluster.
- the data cluster holds the data.
- the data cluster is made to correspond to the block held by the storage 20, and the cluster feature amount is made to correspond to the feature amount of the block.
- the storage unit 12 may use a memory in order to maintain the write performance. Any storage medium may be used as the storage unit 12.
- the management unit 13 has a function of writing the data held by the storage unit 12 to the storage 20 based on the write conditions.
- the writing conditions can be arbitrarily set according to the desired processing. For example, when obtaining write performance, the write condition is the passage of a certain period of time or the size of the data cluster. When real-time processing is required, the writing condition is the occurrence of a data search / extraction request.
- the storage unit 12 may adjust the data cluster according to the input data. For example, when the number of data held by a certain data cluster becomes unevenly large, the storage unit 12 may create a new data cluster for holding these data. When writing the data to the storage 20, the management unit 13 creates a block corresponding to the newly generated data cluster, and writes the data of the data cluster to the block.
- the storage 20 includes a metadata unit 21 and a data unit 22.
- the metadata unit 21 and the data unit 22 may be recorded in the same area in the storage 20.
- the data unit 22 holds data for each block. Data can be read from the data unit 22 in block units.
- the metadata unit 21 holds the feature amount of each block.
- the feature amount is the same as the cluster feature amount used by the cache device 10. In order to maximize the reading efficiency, the features should be ordered.
- the metadata unit 21 may hold an index of each data held by the data unit 22.
- the index is used for associating with the data of other data columns.
- FIG. 2 illustrates the input data, the data cluster of the storage unit 12 of the cache device 10, and the metadata unit 21 and the data unit 22 of the storage 20.
- the data storage system shall store numerical data indicating age.
- Input data is temporarily held in a data cluster according to the amount of data features.
- the cluster features indicate a numerical range. Specifically, the cluster features are classified into five categories: 15-19, 20-26, 20-29, 30-39, and 40-79. The corresponding data cluster holds data within the range indicated by the cluster features. If the cluster feature is in the numerical range represented by the minimum and maximum values, even if the number of data values minus the minimum value is saved in the data cluster as the minimum number of bits obtained by subtracting the minimum value from the maximum value. good. For example, for data stored in a data cluster with cluster features of 15-19, 15 is 0 (0b000), 16 is 1 (0b001), 17 is 2 (0b010), 18 is 3 (0b011), and 19 is 4. Save as (0b100) in the data cluster.
- the data in the data cluster is stored in the storage 20 at a predetermined timing.
- the metadata unit 21 holds the minimum and maximum values of the data stored in each block as feature quantities.
- the minimum value and the maximum value correspond to the cluster feature amount of the storage unit 12.
- the data unit 22 holds the data indicated by the feature amount of the metadata unit 21 for each block.
- the data temporarily held in the data cluster is collectively added to the corresponding block for each data cluster.
- FIG. 3 An example of the data structure of the storage 20 will be described with reference to FIG.
- the feature amount obtained by dividing a block into numerical data is set to the minimum value and the maximum value of the numerical value
- a block in which data of 20 or more and less than 30 is stored is assigned to an edge connecting 30 nodes and 20 nodes. You can add new blocks by adding nodes and edges.
- step S11 the cache device 10 calculates the feature amount of the input data.
- step S12 the cache device 10 compares the data feature amount with the cluster feature amount, and identifies the data cluster according to the feature amount. For example, when the data to be input is a numerical value and the cluster feature amount is indicated by a numerical range, the data cluster in the numerical range including the numerical value of the data is set as the data cluster for storing the data. When there are a plurality of applicable data clusters, the data cluster that stores the data may be specified according to the number of data in the data cluster, the number of data in the corresponding block, and the like.
- step S13 the cache device 10 determines whether or not there is a data cluster that stores the input data. Further, in step S13, the cache device 10 may determine that a new data cluster is created according to the data stored in the data cluster. For example, as shown in FIG. 5, when a large amount of data having a value of 70 or more is input, a new data cluster for storing 70 or more data is created.
- the cache device 10 creates a new data cluster in step S14. If the data cluster does not exist, the cache device 10 may change the cluster feature amount in step S14. For example, in the example of FIG. 5, when data having a value of 14 is input, the cluster feature amount of 15-19 is changed to 14-19.
- step S15 the cache device 10 stores the data in the corresponding data cluster.
- step S16 the cache device 10 updates the cluster feature amount using the data added to the data cluster.
- step S21 the cache device 10 writes the cluster feature amount to the metadata unit 21.
- the cluster feature amount of the new data cluster is written in the metadata unit 21.
- the data structure of the storage 20 is a binary search tree, add new nodes and edges as shown in FIG. In the example of FIG. 7, a node having a value of 70 was added, and an edge (block) for storing data having a minimum value of 70 and a maximum value of 79 was added.
- step S22 the cache device 10 adds data to the block of the data unit 22 for each data cluster.
- step S23 the cache device 10 deletes the data in the data cluster of the storage unit 12.
- the data storage system of the present embodiment is a data storage system that divides a data group into a plurality of blocks and stores the data.
- the data storage system temporarily stores the data in a storage 20 that stores the data in blocks according to the feature amount, a feature amount calculation unit 11 that calculates the feature amount of the input data, and a data cluster according to the feature amount.
- a storage unit 12 to be held in the data cluster and a management unit 13 to write the data held by the data cluster to the corresponding block for each data cluster at a predetermined timing are provided.
- the input data is temporarily held in the data cluster according to the feature amount of the data, and the data is collectively written to the storage 20 for each data cluster at a predetermined timing, so that the data writing disk I / O is performed. Can be reduced.
- the write performance can be obtained by writing data to the storage 20 by the cache device 10 after a certain period of time elapses or when the size of the data cluster becomes a predetermined value or more.
- the cache device 10 writes the data to the storage 20 to support real-time processing.
- the cache device 10 described above includes, for example, a central processing unit (CPU) 901, a memory 902, a storage 903, a communication device 904, an input device 905, and an output device 906, as shown in FIG.
- CPU central processing unit
- the cache device 10 is realized by the CPU 901 executing a predetermined program loaded on the memory 902.
- This program can be recorded on a computer-readable recording medium such as a magnetic disk, an optical disk, or a semiconductor memory, or can be distributed via a network.
Abstract
A data storage system according to the present embodiment divides a group of data into a plurality of blocks and stores the blocks. The data storage system comprises: storage 20 which divides data into blocks according to feature quantities and stores the blocks; a feature quantity calculation unit 11 which calculates feature quantities of input data; a storage unit 12 which temporarily retains data in data clusters according to feature quantities; and a management unit 13 which, for each data cluster, writes the data retained in the data cluster to a corresponding block with a prescribed timing.
Description
本発明は、保存装置、保存方法、およびプログラムに関する。
The present invention relates to a storage device, a storage method, and a program.
携帯端末の進化やソーシャルネットワーキングサービス（ＳＮＳ）の普及に伴い、多種多様で大量のデータが収集されるようになった。収集される多種多様で大量のデータはビッグデータと呼ばれる。ビッグデータを分析することで、人々の生活やビジネスに役立てることができる。
With the evolution of mobile terminals and the spread of social networking services (SNS), a wide variety of large amounts of data have come to be collected. The wide variety and large amount of data collected is called big data. By analyzing big data, it can be useful for people's lives and businesses.
ビッグデータ処理では様々なデータソースから収集されたデータに対して、分析タスクに応じてデータの探索・抽出を行う。ビッグデータ処理では、探索・抽出するデータは属性（列）単位で行うことが多い。そのため、ビッグデータ処理には、列方向に連続してデータを格納し、列単位でデータを取得できる列指向ストレージが適している。列指向ストレージとしては、非特許文献１の技術を実装したＰａｒｑｕｅｔが知られている。
In big data processing, data is searched and extracted according to the analysis task for the data collected from various data sources. In big data processing, the data to be searched / extracted is often performed in units of attributes (columns). Therefore, for big data processing, column-oriented storage that can store data continuously in the column direction and acquire data in column units is suitable. As a column-oriented storage, Parquet, which implements the technique of Non-Patent Document 1, is known.
列指向ストレージにおいて、列データ自体が大きくなると読み込み効率が低くなる。Ｐａｒｑｕｅｔは、列データを複数のブロック（ＲｏｗＧｒｏｕｐ）に分割し、各ブロックの特徴量を保持している。特徴量は、例えば、ブロック内のデータの最小値と最大値、バイナリーのハッシュ値、または文字列のベクトル表現などである。データ探索・抽出クエリーに含まれる特徴量をブロックの特徴量と比較することで、要求するデータがブロック内に存在するか否かを判定できる。要求するデータがブロック内に存在しない場合、ブロック全体をスキップできるため、読み込み効率の改善が期待できる。例えば、年齢を格納した列データが３つのブロックに分割されて格納されており、１番目のブロック内のデータの最小値と最大値が１９と３１、２番目のブロック内のデータの最小値と最大値が１０と２５、３番目のブロック内のデータの最小値と最大値が３５と７０であるとする。この列データから３１歳以上のデータを取得するとき、２番目のブロックは３１歳以上のデータを含まないので、２番目のブロックをスキップし、１番目と３番目のブロックを読み込めばよい。
In column-oriented storage, the larger the column data itself, the lower the reading efficiency. Parquet divides the column data into a plurality of blocks (LowGroup) and holds the feature amount of each block. The feature quantity is, for example, the minimum and maximum values of the data in the block, the hash value of the binary, or the vector representation of the character string. By comparing the feature amount included in the data search / extraction query with the feature amount of the block, it can be determined whether or not the requested data exists in the block. If the requested data does not exist in the block, the entire block can be skipped, so improvement in reading efficiency can be expected. For example, the column data that stores the age is divided into three blocks and stored, and the minimum and maximum values of the data in the first block are 19 and 31, and the minimum value of the data in the second block. It is assumed that the maximum values are 10 and 25, and the minimum and maximum values of the data in the third block are 35 and 70. When acquiring the data of 31 years old or older from this column data, since the second block does not include the data of 31 years old or older, the second block may be skipped and the first and third blocks may be read.
しかしながら、ブロックを一定サイズで単純に分割すると、ブロック間でデータの性質が均一になりやすく、スキップ効果が低いという問題があった。
However, if the blocks are simply divided into blocks of a certain size, the data properties tend to be uniform among the blocks, and there is a problem that the skip effect is low.
そのため、性質の近いデータを同じブロックにまとめることが有効であると考えられる。ところが、新しいデータが到着するたびに、そのデータを対応するブロックに振り分けて書き込むと、ディスクＩ／Ｏが大量に発生し、書き込み性能が低くなるという問題がある。
Therefore, it is considered effective to combine data with similar properties into the same block. However, if new data arrives, the data is distributed to the corresponding blocks and written, a large amount of disk I / O is generated, and the writing performance is deteriorated.
また、新しいデータを一時ファイルに保持し、一定の間隔でブロックを作り直すことが考えられるが、一時ファイルに保持したデータがブロック（ストレージ）に取り込まれるまで最新のデータを利用できず、リアルタイム分析をすることができない。
In addition, it is conceivable to keep new data in a temporary file and recreate blocks at regular intervals, but the latest data cannot be used until the data held in the temporary file is taken into the block (storage), so real-time analysis is performed. Can not do it.
本発明は、上記に鑑みてなされたものであり、データの効率的な読み出しと書き込みを実現することを目的とする。
The present invention has been made in view of the above, and an object of the present invention is to realize efficient reading and writing of data.
本発明の一態様の保存装置は、データ群を複数のブロックに分割して保存する保存装置であって、データを特徴量に応じた前記ブロックに分けて保存するストレージと、入力したデータの特徴量を計算する計算部と、前記データを前記特徴量に応じたクラスタに一時的に保持する保存部と、所定のタイミングで、前記クラスタごとに、当該クラスタの保持するデータを対応する前記ブロックに書き込む管理部を備える。
The storage device of one aspect of the present invention is a storage device that divides a data group into a plurality of blocks and stores the data, and stores the data by dividing the data into the blocks according to the feature amount, and the characteristics of the input data. A calculation unit for calculating the amount, a storage unit for temporarily holding the data in a cluster according to the feature amount, and a block corresponding to the data held by the cluster for each cluster at a predetermined timing. It has a management unit for writing.
本発明の一態様の保存方法は、データ群を複数のブロックに分割して保存する保存方法であって、ストレージがデータを特徴量に応じた前記ブロックに分けて保存しており、コンピュータが、入力したデータの特徴量を計算し、前記データを前記特徴量に応じたクラスタに一時的に保持し、所定のタイミングで、前記クラスタごとに、当該クラスタの保持するデータを対応する前記ブロックに書き込む。
One aspect of the storage method of the present invention is a storage method in which a data group is divided into a plurality of blocks and stored, and the storage divides the data into the blocks according to the feature amount and stores the data. The feature amount of the input data is calculated, the data is temporarily held in the cluster corresponding to the feature amount, and the data held by the cluster is written to the corresponding block for each of the clusters at a predetermined timing. ..
本発明によれば、データの効率的な読み出しと書き込みを実現できる。
According to the present invention, efficient reading and writing of data can be realized.
以下、本発明の実施の形態について図面を用いて説明する。
Hereinafter, embodiments of the present invention will be described with reference to the drawings.
図１を参照し、本実施形態のデータ保存システムの構成について説明する。図１のデータ保存システムは、キャッシュ装置１０とストレージ２０を備え、同じ属性を持つ列データを保存する。また、データ保存システムは、クエリーに応じてストレージ２０に保存されたデータを返却する。
The configuration of the data storage system of the present embodiment will be described with reference to FIG. The data storage system of FIG. 1 includes a cache device 10 and a storage 20, and stores column data having the same attributes. Further, the data storage system returns the data stored in the storage 20 in response to the query.
キャッシュ装置１０は、特徴量計算部１１、保存部１２、および管理部１３を備える。キャッシュ装置１０は、新たに書き込まれた入力データの特徴量を計算し、入力データの特徴量に応じたデータクラスタで一時的に保持し、所定の書き込み条件を満たす場合に、データクラスタごとに、当該データクラスタの保持するデータをまとめてストレージ２０に書き込む。
The cache device 10 includes a feature amount calculation unit 11, a storage unit 12, and a management unit 13. The cache device 10 calculates the feature amount of the newly written input data, temporarily holds the feature amount in the data cluster according to the feature amount of the input data, and when a predetermined write condition is satisfied, for each data cluster, The data held by the data cluster is collectively written to the storage 20.
特徴量計算部１１は、入力データの特徴量を計算し、特徴量に基づいて入力データをデータクラスタに振り分ける機能を有する。特徴量は、データを複数のデータクラスタに振り分けるための情報である。特徴量は、例えば、データが数値の場合は数値の範囲を示す情報であり、データがバイナリデータの場合はバイナリデータのハッシュ値であり、データが文字列の場合は文字列のベクトル表現である。
The feature amount calculation unit 11 has a function of calculating the feature amount of the input data and distributing the input data to the data cluster based on the feature amount. The feature amount is information for allocating data to a plurality of data clusters. The feature quantity is, for example, information indicating a range of numerical values when the data is a numerical value, a hash value of the binary data when the data is binary data, and a vector representation of the character string when the data is a character string. ..
保存部１２は、クラスタ特徴量とデータクラスタを保持する複数の領域を有し、入力データを一時的に保持する機能を有する。クラスタ特徴量は、対応するデータクラスタの保持するデータの特徴量である。データクラスタは、データを保持する。データクラスタをストレージ２０の保持するブロックに対応させ、クラスタ特徴量をブロックの特徴量に対応させる。保存部１２は、書き込み性能を維持するために、メモリを利用するとよい。保存部１２として任意の記憶媒体を利用してもよい。
The storage unit 12 has a plurality of areas for holding the cluster feature amount and the data cluster, and has a function of temporarily holding the input data. The cluster feature is the feature of the data held by the corresponding data cluster. The data cluster holds the data. The data cluster is made to correspond to the block held by the storage 20, and the cluster feature amount is made to correspond to the feature amount of the block. The storage unit 12 may use a memory in order to maintain the write performance. Any storage medium may be used as the storage unit 12.
管理部１３は、書き込み条件に基づき、保存部１２が保持するデータをストレージ２０に書き込む機能を有する。書き込み条件は、求める処理に応じて任意に設定できる。例えば、書き込み性能を求める場合は、一定時間の経過またはデータクラスタのサイズを書き込み条件とする。リアルタイム処理を求める場合は、データの探索・抽出リクエストの発生を書き込み条件とする。
The management unit 13 has a function of writing the data held by the storage unit 12 to the storage 20 based on the write conditions. The writing conditions can be arbitrarily set according to the desired processing. For example, when obtaining write performance, the write condition is the passage of a certain period of time or the size of the data cluster. When real-time processing is required, the writing condition is the occurrence of a data search / extraction request.
保存部１２は、入力データに応じてデータクラスタを調整してもよい。例えば、あるデータクラスタの保持するデータの数が偏って多くなる場合、保存部１２はこれらのデータを保持するための新たなデータクラスタを作成してもよい。管理部１３は、データをストレージ２０に書き込むときに、新たに生成したデータクラスタに対応するブロックを作成し、そのブロックにデータクラスタのデータを書き込む。
The storage unit 12 may adjust the data cluster according to the input data. For example, when the number of data held by a certain data cluster becomes unevenly large, the storage unit 12 may create a new data cluster for holding these data. When writing the data to the storage 20, the management unit 13 creates a block corresponding to the newly generated data cluster, and writes the data of the data cluster to the block.
ストレージ２０は、メタデータ部２１とデータ部２２を備える。メタデータ部２１とデータ部２２は、ストレージ２０内の同じ領域に記録されてもよい。
The storage 20 includes a metadata unit 21 and a data unit 22. The metadata unit 21 and the data unit 22 may be recorded in the same area in the storage 20.
データ部２２は、ブロックごとにデータを保持する。データ部２２からはブロック単位でデータを読み込むことができる。
The data unit 22 holds data for each block. Data can be read from the data unit 22 in block units.
メタデータ部２１は、各ブロックの特徴量を保持する。特徴量は、キャッシュ装置１０の利用するクラスタ特徴量と同じである。読み込み効率を最大化するには、特徴量は順序付きであるとよい。
The metadata unit 21 holds the feature amount of each block. The feature amount is the same as the cluster feature amount used by the cache device 10. In order to maximize the reading efficiency, the features should be ordered.
メタデータ部２１は、データ部２２の保持する各データのインデックスを保持してもよい。インデックスは、他のデータ列のデータとの対応付けに用いる。
The metadata unit 21 may hold an index of each data held by the data unit 22. The index is used for associating with the data of other data columns.
ここで、図２を参照し、入力されるデータの流れについて説明する。図２では、入力データ、キャッシュ装置１０の保存部１２のデータクラスタ、およびストレージ２０のメタデータ部２１とデータ部２２を図示している。データ保存システムは、年齢を示す数値データを保存するものとする。
Here, the flow of input data will be described with reference to FIG. FIG. 2 illustrates the input data, the data cluster of the storage unit 12 of the cache device 10, and the metadata unit 21 and the data unit 22 of the storage 20. The data storage system shall store numerical data indicating age.
入力データは、データの特徴量に応じたデータクラスタに一時的に保持される。図２の例では、クラスタ特徴量は数値の範囲を示す。具体的には、クラスタ特徴量は、１５－１９、２０－２６、２０－２９、３０－３９、および４０－７９の５つに分類される。対応するデータクラスタは、クラスタ特徴量で示される範囲内のデータを保持する。クラスタ特徴量が最小値と最大値で表される数値範囲の場合、データの値から最小値を引いた数を最大値から最小値を引いた数の最小ビット数でデータクラスタに保存してもよい。例えば、クラスタ特徴量が１５－１９のデータクラスタに格納されるデータについて、１５は０（０ｂ０００）、１６は１（０ｂ００１）、１７は２（０ｂ０１０）、１８は３（０ｂ０１１）、１９は４（０ｂ１００）としてデータクラスタに保存する。
Input data is temporarily held in a data cluster according to the amount of data features. In the example of FIG. 2, the cluster features indicate a numerical range. Specifically, the cluster features are classified into five categories: 15-19, 20-26, 20-29, 30-39, and 40-79. The corresponding data cluster holds data within the range indicated by the cluster features. If the cluster feature is in the numerical range represented by the minimum and maximum values, even if the number of data values minus the minimum value is saved in the data cluster as the minimum number of bits obtained by subtracting the minimum value from the maximum value. good. For example, for data stored in a data cluster with cluster features of 15-19, 15 is 0 (0b000), 16 is 1 (0b001), 17 is 2 (0b010), 18 is 3 (0b011), and 19 is 4. Save as (0b100) in the data cluster.
データクラスタのデータは、所定のタイミングでストレージ２０に保存される。
The data in the data cluster is stored in the storage 20 at a predetermined timing.
メタデータ部２１は、特徴量として、各ブロックに保存されるデータの最小値と最大値を保持する。この最小値と最大値は保存部１２のクラスタ特徴量に対応する。
The metadata unit 21 holds the minimum and maximum values of the data stored in each block as feature quantities. The minimum value and the maximum value correspond to the cluster feature amount of the storage unit 12.
データ部２２は、ブロックごとに、メタデータ部２１の特徴量で示されるデータを保持する。データクラスタで一時的に保持されたデータは、データクラスタごとにまとめて対応するブロックに追加される。
The data unit 22 holds the data indicated by the feature amount of the metadata unit 21 for each block. The data temporarily held in the data cluster is collectively added to the corresponding block for each data cluster.
図３を参照し、ストレージ２０のデータ構造の一例について説明する。数値データに対して、ブロックを分割した特徴量を数値の最小値と最大値とした場合、最小値と最大値のそれぞれをノード、ブロックをエッジとした二分探索木とすることが考えられる。例えば、図３の例では、３０のノードと２０のノードをつなぐエッジには、２０以上３０未満のデータを保存したブロックが割り当てられる。ノードとエッジを追加することで、新たにブロックを追加できる。
An example of the data structure of the storage 20 will be described with reference to FIG. When the feature amount obtained by dividing a block into numerical data is set to the minimum value and the maximum value of the numerical value, it is conceivable to use a binary search tree with each of the minimum value and the maximum value as a node and a block as an edge. For example, in the example of FIG. 3, a block in which data of 20 or more and less than 30 is stored is assigned to an edge connecting 30 nodes and 20 nodes. You can add new blocks by adding nodes and edges.
次に、図４のフローチャートを参照し、キャッシュ装置１０が新たなデータを入力したときの動作について説明する。図４の処理は、新たなデータを入力すると実行される。
Next, the operation when the cache device 10 inputs new data will be described with reference to the flowchart of FIG. The process of FIG. 4 is executed when new data is input.
ステップＳ１１にて、キャッシュ装置１０は、入力したデータの特徴量を計算する。
In step S11, the cache device 10 calculates the feature amount of the input data.
ステップＳ１２にて、キャッシュ装置１０は、データの特徴量とクラスタ特徴量とを比較し、特徴量に応じたデータクラスタを特定する。例えば、入力するデータが数値であり、クラスタ特徴量が数値範囲で示されている場合、データの数値を含む数値範囲のデータクラスタを当該データを格納するデータクラスタとする。該当するデータクラスタが複数存在する場合は、データクラスタ内のデータの数や対応するブロック内のデータの数などに応じて、データを格納するデータクラスタを特定してもよい。
In step S12, the cache device 10 compares the data feature amount with the cluster feature amount, and identifies the data cluster according to the feature amount. For example, when the data to be input is a numerical value and the cluster feature amount is indicated by a numerical range, the data cluster in the numerical range including the numerical value of the data is set as the data cluster for storing the data. When there are a plurality of applicable data clusters, the data cluster that stores the data may be specified according to the number of data in the data cluster, the number of data in the corresponding block, and the like.
ステップＳ１３にて、キャッシュ装置１０は、入力したデータを格納するデータクラスタが存在するか否か判定する。また、ステップＳ１３にて、キャッシュ装置１０は、データクラスタに格納されたデータに応じて新たなデータクラスタを作成すると判定してもよい。例えば、図５に示すように、値が７０以上のデータを大量に入力した場合に、７０以上のデータを格納する新たなデータクラスタを作成する。
In step S13, the cache device 10 determines whether or not there is a data cluster that stores the input data. Further, in step S13, the cache device 10 may determine that a new data cluster is created according to the data stored in the data cluster. For example, as shown in FIG. 5, when a large amount of data having a value of 70 or more is input, a new data cluster for storing 70 or more data is created.
データクラスタが存在しない場合、ステップＳ１４にて、キャッシュ装置１０は、新たなデータクラスタを作成する。また、データクラスタが存在しない場合、ステップＳ１４にて、キャッシュ装置１０は、クラスタ特徴量を変更してもよい。例えば、図５の例で、値が１４のデータを入力した場合に、１５－１９のクラスタ特徴量を１４－１９に変更する。
If the data cluster does not exist, the cache device 10 creates a new data cluster in step S14. If the data cluster does not exist, the cache device 10 may change the cluster feature amount in step S14. For example, in the example of FIG. 5, when data having a value of 14 is input, the cluster feature amount of 15-19 is changed to 14-19.
ステップＳ１５にて、キャッシュ装置１０は、データを対応するデータクラスタに格納する。
In step S15, the cache device 10 stores the data in the corresponding data cluster.
ステップＳ１６にて、キャッシュ装置１０は、データクラスタに追加されたデータを利用してクラスタ特徴量を更新する。
In step S16, the cache device 10 updates the cluster feature amount using the data added to the data cluster.
次に、図６のフローチャートを参照し、ストレージ２０にデータを保存する動作について説明する。図６の処理は、ストレージ２０にデータを書き込むタイミングになると実行される。
Next, the operation of saving data in the storage 20 will be described with reference to the flowchart of FIG. The process of FIG. 6 is executed at the timing of writing data to the storage 20.
ステップＳ２１にて、キャッシュ装置１０は、クラスタ特徴量をメタデータ部２１に書き込む。新たなデータクラスタが作成されていた場合は、新たなデータクラスタのクラスタ特徴量をメタデータ部２１に書き込む。ストレージ２０のデータ構造が二分探索木の場合、図７に示すように、新たなノードとエッジを追加する。図７の例では、７０の値を持つノードを追加し、最小値が７０で最大値が７９のデータを保存するエッジ（ブロック）を追加した。
In step S21, the cache device 10 writes the cluster feature amount to the metadata unit 21. When a new data cluster has been created, the cluster feature amount of the new data cluster is written in the metadata unit 21. If the data structure of the storage 20 is a binary search tree, add new nodes and edges as shown in FIG. In the example of FIG. 7, a node having a value of 70 was added, and an edge (block) for storing data having a minimum value of 70 and a maximum value of 79 was added.
ステップＳ２２にて、キャッシュ装置１０は、データクラスタごとに、データをデータ部２２のブロックに追加する。
In step S22, the cache device 10 adds data to the block of the data unit 22 for each data cluster.
ステップＳ２３にて、キャッシュ装置１０は、保存部１２のデータクラスタのデータを削除する。
In step S23, the cache device 10 deletes the data in the data cluster of the storage unit 12.
以上説明したように、本実施形態のデータ保存システムは、データ群を複数のブロックに分割して保存するデータ保存システムである。データ保存システムは、データを特徴量に応じたブロックに分けて保存するストレージ２０と、入力したデータの特徴量を計算する特徴量計算部１１と、データを特徴量に応じたデータクラスタに一時的に保持する保存部１２と、所定のタイミングで、データクラスタごとに、当該データクラスタの保持するデータを対応するブロックに書き込む管理部１３を備える。これにより、入力したデータをデータの特徴量に応じたデータクラスタに一時的に保持し、所定のタイミングで、データクラスタごとにまとめてデータをストレージ２０に書き込むため、データ書き込みのディスクＩ／Ｏを削減することができる。
As described above, the data storage system of the present embodiment is a data storage system that divides a data group into a plurality of blocks and stores the data. The data storage system temporarily stores the data in a storage 20 that stores the data in blocks according to the feature amount, a feature amount calculation unit 11 that calculates the feature amount of the input data, and a data cluster according to the feature amount. A storage unit 12 to be held in the data cluster and a management unit 13 to write the data held by the data cluster to the corresponding block for each data cluster at a predetermined timing are provided. As a result, the input data is temporarily held in the data cluster according to the feature amount of the data, and the data is collectively written to the storage 20 for each data cluster at a predetermined timing, so that the data writing disk I / O is performed. Can be reduced.
本実施形態のデータ保存システムは、一定の時間経過後またはデータクラスタのサイズが所定値以上になったときに、キャッシュ装置１０がストレージ２０にデータを書き込むことで、書き込み性能を求めることができ、データが要求されたときに、キャッシュ装置１０がストレージ２０にデータを書き込むことで、リアルタイム処理に対応できる。
In the data storage system of the present embodiment, the write performance can be obtained by writing data to the storage 20 by the cache device 10 after a certain period of time elapses or when the size of the data cluster becomes a predetermined value or more. When data is requested, the cache device 10 writes the data to the storage 20 to support real-time processing.
上記説明したキャッシュ装置１０には、例えば、図８に示すような、中央演算処理装置（ＣＰＵ）９０１と、メモリ９０２と、ストレージ９０３と、通信装置９０４と、入力装置９０５と、出力装置９０６とを備える汎用的なコンピュータシステムを用いることができる。このコンピュータシステムにおいて、ＣＰＵ９０１がメモリ９０２上にロードされた所定のプログラムを実行することにより、キャッシュ装置１０が実現される。このプログラムは磁気ディスク、光ディスク、半導体メモリ等のコンピュータ読み取り可能な記録媒体に記録することも、ネットワークを介して配信することもできる。
The cache device 10 described above includes, for example, a central processing unit (CPU) 901, a memory 902, a storage 903, a communication device 904, an input device 905, and an output device 906, as shown in FIG. A general-purpose computer system including the above can be used. In this computer system, the cache device 10 is realized by the CPU 901 executing a predetermined program loaded on the memory 902. This program can be recorded on a computer-readable recording medium such as a magnetic disk, an optical disk, or a semiconductor memory, or can be distributed via a network.
１０…キャッシュ装置
１１…特徴量計算部
１２…保存部
１３…管理部
２０…ストレージ
２１…メタデータ部
２２…データ部 10 ...Cache device 11 ... Feature calculation unit 12 ... Storage unit 13 ... Management unit 20 ... Storage 21 ... Metadata unit 22 ... Data unit
１１…特徴量計算部
１２…保存部
１３…管理部
２０…ストレージ
２１…メタデータ部
２２…データ部 10 ...
Claims (5)
- データ群を複数のブロックに分割して保存する保存装置であって、
データを特徴量に応じた前記ブロックに分けて保存するストレージと、
入力したデータの特徴量を計算する計算部と、
前記データを前記特徴量に応じたクラスタに一時的に保持する保存部と、
所定のタイミングで、前記クラスタごとに、当該クラスタの保持するデータを対応する前記ブロックに書き込む管理部を備える
保存装置。 A storage device that divides a data group into multiple blocks and stores it.
Storage that stores data by dividing it into the blocks according to the feature amount,
A calculation unit that calculates the features of the input data,
A storage unit that temporarily holds the data in a cluster according to the feature amount, and a storage unit.
A storage device including a management unit that writes data held by the cluster to the corresponding block at a predetermined timing for each cluster. - 請求項１に記載の保存装置であって、
前記管理部は、一定時間の経過、前記クラスタのサイズが所定値以上になったとき、またはデータが要求されたときに、前記クラスタの保持するデータを対応する前記ブロックに書き込む
保存装置。 The storage device according to claim 1.
The management unit is a storage device that writes data held by the cluster to the corresponding block when a certain period of time elapses, the size of the cluster exceeds a predetermined value, or data is requested. - 請求項１または２に記載の保存装置であって、
前記保存部は、前記特徴量に応じたクラスタを新たに追加し、
前記管理部は、新たに追加された前記クラスタに対応する新たなブロックに前記クラスタの保持するデータを書き込む
保存装置。 The storage device according to claim 1 or 2.
In the storage unit, a cluster corresponding to the feature amount is newly added.
The management unit is a storage device that writes data held by the cluster to a new block corresponding to the newly added cluster. - データ群を複数のブロックに分割して保存する保存方法であって、
ストレージがデータを特徴量に応じた前記ブロックに分けて保存しており、
コンピュータが、
入力したデータの特徴量を計算し、
前記データを前記特徴量に応じたクラスタに一時的に保持し、
所定のタイミングで、前記クラスタごとに、当該クラスタの保持するデータを対応する前記ブロックに書き込む
保存方法。 It is a saving method that divides a data group into multiple blocks and saves it.
The storage divides the data into the blocks according to the feature amount and stores it.
The computer
Calculate the features of the input data and
The data is temporarily held in a cluster according to the feature amount,
A storage method in which data held by the cluster is written to the corresponding block for each cluster at a predetermined timing. - 請求項１ないし３のいずれかに記載の保存装置の各部としてコンピュータを動作させるプログラム。 A program that operates a computer as each part of the storage device according to any one of claims 1 to 3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/JP2020/018517 WO2021224960A1 (en) | 2020-05-07 | 2020-05-07 | Storage device, storage method, and program |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/JP2020/018517 WO2021224960A1 (en) | 2020-05-07 | 2020-05-07 | Storage device, storage method, and program |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| WO2021224960A1 true WO2021224960A1 (en) | 2021-11-11 |
Family
ID=78467928
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| PCT/JP2020/018517 WO2021224960A1 (en) | 2020-05-07 | 2020-05-07 | Storage device, storage method, and program |
Country Status (1)
| Country | Link |
|---|---|
| WO (1) | WO2021224960A1 (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2012127988A1 (en) * | 2011-03-18 | 2012-09-27 | 日本電気株式会社 | Information processing device, distributed file system, client device, information processing method, and computer program |
| WO2013132779A1 (en) * | 2012-03-09 | 2013-09-12 | 日本電気株式会社 | Object placement device and method, and computer program |
-
2020
- 2020-05-07 WO PCT/JP2020/018517 patent/WO2021224960A1/en active Application Filing
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2012127988A1 (en) * | 2011-03-18 | 2012-09-27 | 日本電気株式会社 | Information processing device, distributed file system, client device, information processing method, and computer program |
| WO2013132779A1 (en) * | 2012-03-09 | 2013-09-12 | 日本電気株式会社 | Object placement device and method, and computer program |
Non-Patent Citations (1)
| Title |
|---|
| NISHIYAMA, TATSUYA ET AL.: "Bullet: Column-oriented query engine in distributed key value stores", PROCEEDINGS OF MULTIMEDIA, DISTRIBUTED, COOPERATIVE AND MOBILE SYMPOSIUMS (DICOM02014, vol. 2014, no. 1, 9 July 2014 (2014-07-09), pages 683 - 691 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| KR102564170B1 (en) | Method and device for storing data object, and computer readable storage medium having a computer program using the same | |
| US9672235B2 (en) | Method and system for dynamically partitioning very large database indices on write-once tables | |
| US10642515B2 (en) | Data storage method, electronic device, and computer non-volatile storage medium | |
| WO2016165509A1 (en) | Big data statistics at data-block level | |
| KR20200053512A (en) | KVS tree database | |
| US8719237B2 (en) | Method and apparatus for deleting duplicate data | |
| US10963440B2 (en) | Fast incremental column store data loading | |
| US10915533B2 (en) | Extreme value computation | |
| JP6642651B2 (en) | Storage method using user access preference model | |
| US11726743B2 (en) | Merging multiple sorted lists in a distributed computing system | |
| JP5790755B2 (en) | Database management apparatus and database management method | |
| US20180011897A1 (en) | Data processing method having structure of cache index specified to transaction in mobile environment dbms | |
| CN109325022B (en) | Data processing method and device | |
| US20220342888A1 (en) | Object tagging | |
| CN112241396B (en) | Spark-based method and system for merging small files of Delta | |
| CN116089414B (en) | Time sequence database writing performance optimization method and device based on mass data scene | |
| WO2021224960A1 (en) | Storage device, storage method, and program | |
| WO2012081165A1 (en) | Database management device and database management method | |
| KR101693108B1 (en) | Database read method and apparatus using t-tree index for improving read performance | |
| KR102354343B1 (en) | Spatial indexing method and apparatus for blockchain-based geospatial data | |
| US20150095284A1 (en) | Information storage system, information storage method, and computer-readable medium storing program | |
| CN112328587A (en) | Data processing method and device for ElasticSearch | |
| JP2010191903A (en) | Distributed file system striping class selecting method and distributed file system | |
| CN112965939A (en) | File merging method, device and equipment | |
| CN113312414B (en) | Data processing method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| 121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 20934245 Country of ref document: EP Kind code of ref document: A1 |
|
| NENP | Non-entry into the national phase |
Ref country code: DE |
|
| 122 | Ep: pct application non-entry in european phase |
Ref document number: 20934245 Country of ref document: EP Kind code of ref document: A1 |
|
| NENP | Non-entry into the national phase |
Ref country code: JP |