KR20160100752A - System and method for processing and analysing big data provding efficiently using columnar index data format - Google Patents

Abstract

Disclosed are a system and a method for efficiently processing and analyzing big data by using a columnar index data format. A method for processing big data in a processing system implemented with a computer comprises the steps of: generating a dictionary by arranging pieces of data in a column unit of the big data; classifying the arranged pieces of data as one or more data blocks every dictionary according to sizes of the pieces of data; generating an index including values of pieces of first data of the data blocks according to an order of the data blocks every dictionary; and generating a column identification for each column in a row order of the big data.

Description

[0001] SYSTEM AND METHOD FOR EFFICIENTLY PROCESSING AND ANALYZING BIG DATA BY USING COLUMN-INDEX DATA FORMAT [0002]

Embodiments of the present invention are directed to techniques for efficiently processing and analyzing big data using a column-index data format, and for efficiently processing and analyzing big data, an efficient column- To a processing system and a processing method capable of improving various query processing performance.

Recently, as demand for collection, storage, processing and analysis of Big Data surge, various big data solutions are emerging on the market in the form of open source or commercial products. No one can deny that these big data solutions have kept the vast amount of data that could not be covered by past technologies and infrastructures for a long period of time and derive value from them. The basic operation method of the prior art for executing a user-created program through a data scan scan (full scan) using a parallel process is suitable for various enterprise requirements.

However, all software systems have trade-offs. The conventional full scan scheme has a disadvantage in that it generates a considerable amount of disk and network input / output even for a simple query. In other words, it is a very inefficient system for querying only the necessary data to ensure fast latency. Also, due to the time-consuming nature of the run-time preparation for Map Reduce operations, the prior art approach is not suitable for fast response. Big data solution providers are actively researching the following two questions.

One. What data formats are appropriate for fast response times?

2. How do you create an efficient distributed query engine to replace the prior art mapping?

The distributed query engine establishes an optimal execution plan considering characteristics of data format through optimizer which is a core engine for efficiently processing queries such as SQL (Structured Query Language) in a distributed environment. Therefore, most distributed engine developers read only the data needed for query processing, and have implemented a column format data format that can reduce system resource usage on the disk or network and guarantee fast response time. However, there is still a problem in that some pool scanning methods do not perform as fast as expected.

The following is an example of a data format of the prior art.

1. RC File (Record Columnar File)

2. An ORC file (Optimized Record Columnar File)

3. Parquet

4. PowerDrill data format

1 is a diagram showing a configuration example of a row group and a column partition in the prior art. The data formats of the above-described prior arts are the same as those shown in the table 100 of FIG. 1, in which data is commonly divided into '1) row group (Row Group) and 2) column unit' I am taking. In this conventional technique, the columns belonging to the same row group are stored in the adjacent positions on the disk to perform efficient row reconstruction.

In addition, conventional data formats have the following common goals.

1. Fast data loading

2. Quick query processing

3. High efficiency storage system

4. Suitable for various workload patterns

Because Big Data is a task that requires a large amount of data to be processed, if the high-speed data loading is not done basically in a production environment, it is inadequate for the big data format. For example, when 20TB of data is imported every day, the resource usage of disk and network becomes very high. In this case, there is a lack of resources to perform the analysis query, which may cause difficulty in operation. Therefore, high-performance data loading is mandatory, while excluding row groups / columns that are not required for querying, it is necessary to guarantee fast response time during data processing. The above data formats are designed so that the data in the columns have the same characteristics, so that the compression rate can be increased and the disk can be efficiently used to quickly process various queries, not only optimized for some queries.

Based on this common goal, the RC file is organized into row groups, each row group is divided into columns, and unnecessary columns are excluded in the query processing. Based on this RC file, the ORC file further stores statistical information such as the minimum value and the maximum value of the row group, thereby improving the processing performance excluding the row group which is not used in the query processing. Parquette supports nested data models such as JavaScript Object Notation (JSON). The power drill further builds a global / local dictionary to reduce the storage capacity of the data by expressing the field value as a bit-wise integer and also improves the processing speed.

However, the common row group / column splitting methods of these formats have the following disadvantages.

1. Inefficiency of row group unit filtering

2. Unnecessary Sequential Access or Random Access frequently occurs to read a column.

Since the row group can not be precisely filtered out by the statistical information of the row group, a disk operation for reading unnecessary row groups occurs, and when the filtering through the global dictionary is applied to improve the efficiency, a large cost is incurred for managing the row group. The global dictionary approach should be able to process data in system memory when the number of unique values in the column is relatively small compared to the total column data. And the row group method is strictly a mixture of row layout and column layout. For pure column analysis, it is necessary to exclude unnecessary columns by random access or sequential reading. Performance degradation due to this behavior is severe. Therefore, a new data format capable of overcoming these drawbacks is required.

By efficiently processing and analyzing big data using the column-index data format, we are able to quickly analyze data from various perspectives in the current situation of simply storing large amounts of data and providing slow search / analysis in the big data era Visualization to provide a processing system and method that can help a lot of people gain insights and create value.

A method of processing big data in a computer-implemented processing system, the method comprising: generating a local dictionary by arranging data in columns of the big data; Classifying the sorted data into at least one data block for each dictionary according to the size of the data; And generating, for each dictionary, an index including values of first data for each of the data blocks according to the order of data blocks; And generating a column ID for each column in the row order of the big data. These column IDs can be used for row reconstruction or for fast column scans.

A processing system for processing big data by being implemented by a computer, the processing system comprising: a dictionary generation unit for generating a local dictionary by arranging data in units of columns of the big data; A data classifying unit classifying the sorted data into at least one data block according to a size of data; And an index generator for generating, for each dictionary, an index including the values of the first data for each data block according to the order of the data blocks; And a column ID generator for generating a column ID for each column in the row order of the big data.

By efficiently processing and analyzing big data using the column-index data format, we are able to quickly analyze data from various perspectives in the current situation of simply storing large amounts of data and providing slow search / analysis in the big data era Visualization can help a lot of people gain insights and create value.

1 is a diagram showing a configuration example of a row group and a column partition in the prior art.
2 is a diagram showing an example of a column-index data format in an embodiment of the present invention.
3 is a diagram showing an example of big data in an embodiment of the present invention.
4 is a diagram showing an example of a dictionary configuration in an embodiment of the present invention.
5 is a diagram showing an example of an index configuration in an embodiment of the present invention.
6 is a diagram showing an example of the configuration of a column ID in an embodiment of the present invention.
7 is a block diagram for explaining an internal configuration of a processing system in an embodiment of the present invention.
Figs. 8 to 11 are flowcharts showing the processing method in the embodiments of the present invention. Fig.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

In the Relational Database Management System (RDBMS), data is stored in a row-type layout for fast transaction processing or data is stored in a column form for quick analysis. Due to the nature of disks, frequent random access has a bad effect on performance. To reduce these adverse effects, data is stored in a column form in order to store the data to be read at the same time. In other words, in a database where frequent processing such as updating or deleting a row is performed, data is stored in a row unit, and when aggregation or statistical data is extracted for each column, data is stored in a column unit. In the data format for big data analysis, it is a relatively analysis-oriented operation rather than row-by-row processing, so a data format that can be basically stored in a column form can be designed.

The column-index file (CI File) to be described in the embodiments of the present invention is based on this column format, so that the data analysts frequently watch the operations performed by the data analysts in the business, The following two goals can be considered as the focus.

1. Fast data retrieval through index

2. Rapid aggregation analysis with column layout

The column index data format can have three key components for a column:

1. Dictionary

One dictionary can correspond to one column and can consist of sorted data blocks. Compression rates are good for columns that are dictionaries encoded with few unique values, and you can have low IDs to reconstruct the row corresponding to each value. For example, the order of values in one data block does not represent the order of the rows because the data in the data block is sorted. Thus, each of the values may have row IDs so that they know the original row. The row ID can also be compressed by a compression technique such as a variable integer.

2. Index

An index can consist of an offset pointing to a data block of a dictionary to quickly locate the data in the dictionary. The index may be loaded into memory and used to quickly locate a block of data that contains data upon retrieval.

3. Column ID

Column IDs are stored in numerical order in row order and can be pointed to the dictionary or, if the column is an integer column, the integer value of the integer column can be used to point to the dictionary. The column ID can be compressed as needed on a block-by-block basis.

2 is a diagram showing an example of a column-index data format in an embodiment of the present invention.

The dictionary can organize the data of one column into a plurality of aligned data blocks. The divided data can be stored in a plurality of repositories for distributed parallel processing, and desired query results can be obtained quickly using data stored in a plurality of repositories.

Through the index, it is possible to find out which data block among the plurality of data blocks included in the dictionary contains a value to be searched. To this end, the index may be configured to include the first values of each data block. It is possible to determine between which values the desired value is included among the values arranged in the block, thereby extracting the desired data.

Column IDs can store numeric values in row order. For example, the first value in a column ID block can indicate the position in the dictionary of the data of the first row in that column. For example, it may indicate that the data in the first row of the column is the tenth of the sorted data in the dictionary.

If the corresponding column is a column containing integer values, the values of the column ID block can store the values in a row order. For example, if the first value of the column ID block is 5, it is immediately known that the data value of the first row of the column is 5.

3 is a diagram showing an example of big data in an embodiment of the present invention.

The first dashed line box 310 corresponds to the region, the second dashed line box 320 corresponds to the sex, the third dashed box 330 corresponds to the age, and the fourth dashed box 330 corresponds to the age. (340) may represent a column corresponding to a hobby, respectively. At this time, the database may be configured such that each column corresponds to one dictionary.

4 is a diagram showing an example of a dictionary configuration in an embodiment of the present invention.

4, the first dictionary, which is a column corresponding to the area indicated by the first dotted line box 310 of FIG. 3 described above, includes a fourth dotted line box 410, a fifth dotted line box 420, 430), as shown in FIG. For example, the processing system may sort the data of the column corresponding to the region by value, and sort the sorted data into three data blocks to construct a first dictionary having three data blocks. At this time, each of the data included in the first dictionary may have a low ID. For example, a row ID "3 " corresponding to" Gwangju " may indicate that "Gwangju" is data of the third row in the column corresponding to the region. This low id may be used to reconstruct the rows of the original big data through the first dictionary. For example, if only the data having a low ID of "3" in all the dictionaries are collected and listed in column order, the third row in the big data of FIG. 3 can be obtained.

5 is a diagram showing an example of an index configuration in an embodiment of the present invention.

Here, FIG. 5 shows an example of an index composed of the first data values of the three data blocks described with reference to FIG. The first value 510, "Gwangju", represents the first value of the data block shown in the fourth dashed box 410 in FIG. 4, and the second value 520 "Seoul" 420 represents the first value of the data block and the third value 530 represents the first value of the data block shown in the sixth dotted line box 430 in FIG. For example, when searching for a place name beginning with "part ", the" part "is located between the first value 510" Quot; is stored in the data block shown in the fourth dotted line box 410 of FIG. Because these indexes are organized in memory, the processing system can very quickly provide the desired search results through the desired block of data. For example, it is possible to find the necessary data block immediately for the query "number of users in Busan" and to read the continuous data contained in one data block without moving between columns or between rows, You can get the value very quickly.

In addition, as already described, it is possible to identify the row of each data through the row ID since the data can be directly ascertained through the row ID of the original several rows.

6 is a diagram showing an example of the configuration of a column ID in an embodiment of the present invention. The column ID block 600 may store numerical values in the order of rows. For example, the first value 610 '5001' may indicate where the 'Seoul' data of the first row of the column corresponding to the place name in the big data of FIG. 3 is located in the dictionary. For example, if it is assumed that each of the data blocks of FIG. 4 includes 5000 data, it can be immediately confirmed that the data represented by '5001' is the first data of the data block indicated by the fifth dotted line box 620. In addition, it can be immediately confirmed that the last value 620 '5000' is the last data of the data block shown in the fourth dotted line box 610. Such a column ID makes it possible to directly access a data block including data of a specific row in the corresponding column.

For a column whose data value is an integer type, the value of the data may be the value of the column ID block. For example, the column corresponding to the age indicated through the third dashed box 330 in FIG. 3 may be a column ID block. As a more specific example, the first value of the column ID block may be '25', which is the first value of the corresponding column. In this case, the processing system can retrieve the desired row value directly from the column ID block without reading the data block.

FIG. 7 is a block diagram for explaining an internal configuration of a processing system in an embodiment of the present invention, and FIGS. 8 to 11 are flowcharts showing processing methods in the embodiments of the present invention.

The processing system 700 according to the present embodiment may be a system for storing, managing, and processing big data, and may be composed of one computer device, but may also be composed of a plurality of computer devices. For example, if the big data can not be stored in one computer device, the processing system 700 can be configured to be interconnected by separate computer devices configured for each dictionary or data block of the dictionary.

The processing system 700 may include a dictionary generator 710, a data classifier 720, an index generator 730, and a column ID generator 740, as shown in FIG. At this time, the dictionary generating unit 710, the data classifying unit 720, the index generating unit 730, and the column ID generating unit 740 may perform the steps of FIG. 8 (steps 810 to 840) Can be implemented.

In step 810, the dictionary generator 710 may sort the data in column units of the big data and generate a dictionary. For example, Hangul can be arranged in alphabetical order (or in reverse order), alphabetical order (or reverse order), and numbers can be sorted in ascending order (or descending order), respectively. A dictionary can correspond to columns in big data, and can be created as many as the number of columns. An example of big data has been described with reference to FIG. 3, and an example of a dictionary has been described with reference to FIGS.

In step 820, the data classifier 720 may classify the sorted data into at least one data block for each dictionary according to the size of the data. In the case of big data, the data of one column is generally too large to be included in one storage. In this case, the data classifier 720 may classify the data included in one column into a plurality of data blocks. These data blocks have been described with reference to FIG.

In step 830, the index generator 730 may generate, for each dictionary, an index including the values of the first data for each data block according to the order of the data blocks. In other words, the first value of the index may contain the value of the first data of the first data block of the dictionary.

In step 840, the column ID generator 740 may generate a column ID for each column in the row order of the big data. The column ID may be an offset value that points to the dictionary. For numeric columns, the actual values of each column can be stored as column IDs. For example, a column ID is a value for identifying the position or order of data in a dictionary, and may be stored in the column ID block in a row order in the big data.

The processing system 700 may further include an index loading unit 751, a data block determination unit 742, and a response generation unit 753 in another embodiment. The index loading unit 751, the data block determining unit 742 and the response generating unit 753 are the same as the steps in the processing of FIG. 9 (steps 910 to 930 )). ≪ / RTI >

In step 910, the index loading unit 751 may load the index into memory. Since the index includes only the first data of the data blocks, its size is relatively small as compared with the big data. The processing system 700 may use this index to quickly find data blocks containing the desired data.

In step 920, the data block determination unit 752 may determine the data block including the determined data by comparing the data determined through the received query with the first data of the data blocks included in the index. As already explained, the data in the dictionary are sorted by value, and the index contains the first data of the data blocks. Therefore, the data block determination unit 752 can identify which data of the index or the data included among the data. Accordingly, when the data of the index is specified, the data block determining unit 752 can determine the data block having the data as the first data as a data block including the determined data.

In step 930, the response generator 753 may read the determined data block and generate a response to the query using the data block. Big data is stored through storage that takes a very long time to read, such as a hard disk, relative to memory. The processing system 700 can ensure fast processing time by reading only necessary data (data of the determined data block) without performing full scan of such big data as a whole.

In addition, the processing system 700 may further include a low reconstruction unit 761 in another embodiment. This row reconstruction unit 761 may be implemented to perform the step 1010 of FIG. 10, which may further include the processing method in another embodiment.

First, in the present embodiment, the dictionary may store a row ID that identifies a row order in the big data of the data included in the data blocks together with each of the data.

At this time, in step 1010, the row reconstructing unit 761 may reconstruct the row of the big data using the row ID of the data included in the dictionary. For example, if only the data of the same row ID is extracted from the dictionaries, one row can be reconstructed.

In addition, the processing system 700 may further include a data extracting unit 771 in another embodiment. The data extracting section 771 may be implemented to perform the step 1110 of FIG. 11, which may further include the processing method in another embodiment.

In step 1110, the data extracting unit 772 extracts the column ID of the row order requested in the column ID block of the dictionary corresponding to the received query, and extracts the position or order data identified by the column ID extracted from the dictionary Can be extracted.

7 and 8, the description of FIGS. 2 to 6 can be referred to.

Thus, according to embodiments of the present invention, by efficiently processing and analyzing the big data using the column-index data format, it is possible to simply store a large amount of data in the big data era and provide slow search / analysis In the present situation, we can quickly analyze and visualize data from various perspectives, which can help many people gain insights and create value.

The apparatus described above may be implemented as a hardware component, a software component, and / or a combination of hardware components and software components. For example, the apparatus and components described in the embodiments may be implemented within a computer system, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA) , A programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For ease of understanding, the processing apparatus may be described as being used singly, but those skilled in the art will recognize that the processing apparatus may have a plurality of processing elements and / As shown in FIG. For example, the processing unit may comprise a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of the foregoing, and may be configured to configure the processing device to operate as desired or to process it collectively or collectively Device can be commanded. The software and / or data may be in the form of any type of machine, component, physical device, virtual equipment, computer storage media, or device , Or may be permanently or temporarily embodied in a transmitted signal wave. The software may be distributed over a networked computer system and stored or executed in a distributed manner. The software and data may be stored on one or more computer readable recording media.

The method according to an embodiment may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions to be recorded on the medium may be those specially designed and configured for the embodiments or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. For example, it is to be understood that the techniques described may be performed in a different order than the described methods, and / or that components of the described systems, structures, devices, circuits, Lt; / RTI > or equivalents, even if it is replaced or replaced.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (13)

A method for processing big data in a computer-implemented processing system,
Generating a dictionary by arranging data in a column of the big data;
Classifying the sorted data into at least one data block for each dictionary according to the size of the data;
Generating, for each dictionary, an index including values of first data for each of the data blocks according to the order of data blocks; And
A step of generating a column ID for each column in a row order of the big data
≪ / RTI > The method according to claim 1,
Loading the index into a memory;
Comparing the data determined through the received query with the first data of the data blocks included in the index, and determining a data block including the determined data; And
Reading the determined data block, and generating a response to the query using the data block
≪ / RTI > The method according to claim 1,
Wherein the dictionary stores a row ID that identifies a row order in the big data of the data contained in the data blocks together with each of the data. The method of claim 3,
Reconstructing a row of the big data using the row ID of the data included in the dictionary
≪ / RTI > The method according to claim 1,
Generating a column ID block storing a column ID, which is a value for identifying a position or order of data of a corresponding dictionary, in a row order in the big data;
≪ / RTI > 6. The method of claim 5,
Extracting the column ID of the row order requested in the column ID block of the dictionary corresponding to the received query, and extracting data of a position or order identified by the column ID extracted from the dictionary
≪ / RTI > A computer-readable recording medium having recorded thereon a program for executing the method according to any one of claims 1 to 6. 1. A processing system for processing large data, which is implemented by a computer,
A dictionary generating unit for generating a dictionary by arranging data in column units of the big data;
A data classifying unit classifying the sorted data into at least one data block according to a size of data;
An index generating unit for generating, for each dictionary, an index including the values of the first data for each of the data blocks according to the order of the data blocks; And
A column ID generator for generating a column ID for each column in a row order of the big data;
≪ / RTI > 9. The method of claim 8,
An index loading unit loading the index into a memory;
A data block determining unit for comparing the data determined through the received query with the first data of the data blocks included in the index to determine a data block including the determined data; And
A response generator for reading the determined data block and generating a response to the query using the data block,
≪ / RTI > 9. The method of claim 8,
Wherein the dictionary is stored together with a row ID identifying a row order in the big data of the data contained in the data blocks in association with each of the data. 11. The method of claim 10,
A row reconstruction unit for reconstructing a row of the big data using the row ID of the data included in the dictionary,
≪ / RTI > 9. The method of claim 8,
And a column ID block for storing a column ID, which is a value for identifying a position or order of data of a corresponding dictionary, in a row order in the big data,
≪ / RTI > 13. The method of claim 12,
Extracts the column ID of the row order requested in the column ID block of the dictionary corresponding to the received query, and extracts data of the position or order identified by the column ID extracted from the dictionary,
≪ / RTI >

Images