KR101515304B1 - Reduce-side join query processing method for hadoop-based reduce-side join processing system - Google Patents

Abstract

The present invention relates to a reduce-side join query processing method of a Hadoop-based reduce-side join processing system, wherein the method comprises the steps of: (a) generating bitmap section filter data based on a result key value extracted from a result of a join query and storing the bitmap section filter data in a BIF database; (b) when a new join query is input, searching the BIF database for bitmap section filter data corresponding to the new join query; (c) when the bitmap section filter data is searched in the step (b), transmitting the searched bitmap section filter data to each mapper of the Hadoop system; (d) transforming, by each of the mappers, a record which is filtered based on the bitmap section filter data to a reducer of the Hadoop system; and (e) outputting, by the reducer, results for the new join query on the basis of the record transmitted from each of the mappers.

Description

[0002] REDUCE-SIDE JOIN QUERY PROCESSING METHOD FOR HADOOP-BASED REDUCE-SIDE JOIN PROCESSING SYSTEM OF REDUCE-

The present invention relates to a reduction-side join query processing method of a Hadoop-based redundancy-side join processing system, and more particularly to a method and apparatus for processing a network and a computing cost generated by records that are not related to a join query, Side join query processing system of the Hadoop based reduction-side join processing system capable of optimizing the memory usage by expressing the interval filter in bits.

Recently, interest and research on Big data are increasing. Previously, Big Data could not be utilized because of the lack of technology to analyze and manage big data and the excessive cost of analysis and management. However, with the recent development of cloud computing technology, it became easier and cheaper to manage and analyze big data. In line with this, the scope of application of Big Data is expanding to various fields such as politics, society, economy and culture.

Cloud computing technologies are also being applied to existing data warehouses. The data warehouse operates a large database that is extracted and integrated from databases of various operating systems. The main features of the data warehouse are that there are many repeated pattern queries and a large amount of data is uploaded at regular intervals. Data warehouse systems also use Hadoop and Hive to process large amounts of structured and unstructured data.

Apache Open source project Hadoop consists of Mapreduce, which distributes a large amount of data with typical cloud computing technology, and Hadoop Distributed File System (HDFS), which stores a large amount of data. And Hive makes it easier to use and manage MapReduce through HiveQL, which is based on Hadoop, similar to SQL.

However, since Hadoop and Hive have a structure different from that of a relational DBMS (database management system), it takes a lot of money to process the data operations that have been processed in the existing relational DBMS. The MapReduce, which acts as data processing in Hadoop and the hive, has a shared nothing structure. While MapReduce has the advantage of receiving various data as input, it has a simple structure for processing data.

The process of MapReduce can be divided into Map and Reduce. The distributed data is processed in the mapper of each map node, and the result of the process is transmitted by a small number of Reduce and processed through Reduce. In this case, the bottleneck occurs in the process because the data is moved only at the stage of transition from maple deuce to distributed map.

FIG. 1 is a diagram for explaining a data processing process of a MapReduce constituting Hadoop. Hereinafter, a process of join query will be described with reference to FIG. 1. The process of data processing of the lumpidus consists of three processes. In the map process, which is the first process, the distributed data is read and processed into a pair of key and value.

The shuffling process between the mapper and the Reduce classifies the data output from the map by the Hadoop framework into a list of values after they are sorted according to the key value and sends them to the Reduce. The redescription process processes the data in the form of key and list and outputs the result.

While Hadoop provides the flexibility to process a wide variety of data, it requires learning from the data analysts as well as programmers wishing to use Hadoop to construct distributed processing programs that Hadoop requires.

Hive is a framework that provides data handling convenience to these users in a manner similar to the SQL language. Hive is Hadoop's internal project, a data warehouse system based on Hadoop built on facebook. The concept and structure of Hadoop is unfamiliar to users who have used existing database tools and requires a lot of time and effort. Research has been conducted to provide easier scripting languages and SQL types to existing users. Hive's HiveQL language, which is similar to SQL, makes MapReduce programming easier.

On the other hand, Hadoop's MapReduce provides a map-side join method and a reduce-sign join method in connection with the processing of join query.

A map-side join is a join method in which a join is performed in a mapper. Since the join is performed in the mapper, there is no redescing step, which is faster than a reduce-side join. However, the map-side join method can only be used if it meets certain requirements. Each input data to be used in the join must be divided into the same number of partitions, and only constraints that can be used if they are sorted by the same join key within the same data .

On the other hand, the reduction-side join method is a join used in the redescription process in a more general case. Referring to FIG. 2, in a map, a map reads records from a block, and a tag containing table information is attached to each record with a key value as a key, and the value is output to a value.

The result of such map is sorted and temporarily stored in the local disk of each map. When the processing of each map is completed, all the results are copied to the local disk of each reducer through the network by the join key. Then, in Reduce, after the temporary file is merged / sorted, a join is performed using the tags that are collected by each join key.

However, the Reduce-Side join method of Hadoop's MapReduce does not require any unnecessary network and computing costs because the records that are not related to the join query are sent to the reducer through the network, There is a problem that causes waste.

FIG. 3 is a view for explaining a process of consuming network and computing costs in the method of reducing-side joining Hadoop's MapReduce. Referring to FIG. 3, as described above, all the records of Table 1 and Table 2 are read during the mapping process and sent to the redescription process. Then, in the redescription process, it knows that it has a common key B in both tables and outputs a record with join key B as a result.

However, in this process, the records of the join key A that are not joined and the records of the join key C are transmitted to the redescription process, and the records having the join key A and the join key C are processed in the shuffling process and the re- It can be seen that the unnecessary cost is consumed.

SUMMARY OF THE INVENTION The present invention has been made in view of the above problems, and it is an object of the present invention to provide a Hadoop-based reduced-side memory which can reduce network and computing costs caused by records that are not correlated with join queries, And to provide a method for processing a redo-side join query of a join processing system.

The above object is achieved according to the present invention by a method for processing a reduction-side join query of a Hadoop-based reduced-side join processing system, the method comprising: (a) comparing the result of the join query in the Hadoop- (B) if a new join query is input to the Hadoop-based redox-side join processing system, the Hadoop-based Side join filter data corresponding to the new join query is retrieved from the BIF database in the redesed-side join processing system of the BSS; (c) if bitmap interval filter data is retrieved in the step (b) And transmitting the retrieved bitmap interval filter data to each of the mapper of the Hadoop; and (d) And (e) outputting a result for the new join query based on the record sent from each mapper, wherein the reducer comprises: ; The step (a) performed by the Hadoop-based reduction-side join processing system includes the steps of: (a1) extracting the result key value from the result of the join query; (a2) dividing the join key domain into a plurality of BIF intervals in units of BIF intervals; (a3) checking whether the extracted result key value belongs to each BIF section; (a4) bitmap interval filter data including information on the plurality of BIF sections and a BIF value as to whether the extracted result key value belongs to each BIF section is generated and stored in the BIF database ; In the step (d), the mapper transmits to the reducer a record in a BIF section to which a join key value of a table record included in the new join query among the plurality of BIF sections of the bitmap section filter data belongs. Side join query processing method of a Hadoop based reduced-side join processing system.

Herein, in the step (a2), the BIF interval may be expressed by the following equation: L = D / R where L is the BIF interval, D is the number of elements of the join key domain, R is the number ). ≪ / RTI >

In addition, the bitmap interval filter data further includes a BIF interval and a start join key value of the join key domain, and in the step (d), the mapper adds a join key value of a table record included in the new join query, Wherein the offset value is calculated on the basis of an interval between the start join key values, and based on the join key value, the offset value, and the BIF interval of the table record included in the new join query, The BIF section to which the join key value belongs is extracted and based on the extracted BIF value of the extracted BIF section, it is possible to determine whether the record in the inquired BIF section is transmitted to the reducer.

In addition, the mapper may be expressed by a formula I = (V-Base) / L, where I is the index of the BIF section to which the join key value of the table record included in the new join query belongs and V is included in the new join query A base key is a start join key value of the join key domain, and L is the BIF interval), the BIF section to which the join key value of the table record included in the new join query belongs is extracted can do.

If the bitmap section filter data is not retrieved in step (b), the result of the new join query is output through a processing method previously set by the Hadoop-based redis-side join processing system ; (f2) extracting a result key value from the result of the new join query output in the step (f1) by the Hadoop-based reduction-side join processing system; (f3) The steps (a2) to (a4) are performed on the resultant key value extracted in the step (f2) by the Hadoop-based reduction-side join processing system, And the bitmap interval filter data is generated and stored in the BIF database.

According to the present invention as described above, according to the present invention, it is possible to reduce the network and computing costs caused by records that are not correlated with the join query, and to reduce the memory usage of the Hadoop-based reduced- A deuce-side join query processing method is provided.

FIG. 1 is a diagram for explaining a data processing process of a MapReduce constituting Hadoop,
FIG. 2 is a diagram for explaining a reduction-sign joining method of Hadoop's MapReduce,
FIG. 3 is a view for explaining a process of consumption of network and computing costs in the method of reducing-side joining Hadoop's MapReduce,
4 is a view for explaining a reduction-side join query processing method of the Hadoop-based reduction-side join processing system according to the present invention,
5 is a diagram for explaining a process of generating bitmap interval filter data in a method of processing a reduction-side join query of a Hadoop-based reduction-side join processing system according to the present invention,
FIG. 6 is a diagram for explaining the effect of filtering using bitmap interval filter data according to the present invention, and FIG.
FIG. 7 is a diagram illustrating a relationship among elements of bitmap interval filter data according to the present invention,
FIG. 8 is a diagram illustrating an example of a hive system for implementing a method for processing a redo-side join query of a redo-side join processing system based on the Hadoop system according to the present invention,
9 to 12 are diagrams showing experimental results of a method for processing a reduction-side join query in the Hadoop-based reduction-side join processing system according to the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. Here, the BIF described in this specification is an abbreviation of 'Bitmap for Interval Filter' and is a name arbitrarily defined for expressing the technical idea of the present invention.

4 is a diagram for explaining a reduction-side join query processing method of the Hadoop-based reduction-side join processing system according to the present invention. Referring to FIG. 4, when a query is input to the redo-side join processing system based on Hadoop (S40), it is determined whether the inputted query is a join query (S41). Here, the present invention will be described by taking as an example a Hadoop-based reduction-side join processing system being a Hadoop-based hive system.
If it is determined in step S41 that the query input by the Hadoop based redundancy-side join processing system is not a join query, the existing Hadoop-based redes-side join processing system, for example, the query of the Hadoop based hive system Processing is performed (S50), and the result is output (S46).

On the other hand, if it is determined that the query input in step S41 is a join query, the Hadoop-based redes-side join processing system determines whether the query inputted to the BIF database, that is, bitmap section filter data corresponding to the join query exists (S42). Here, the bitmap interval filter data is determined according to whether the join query input at the time of generating the bitmap interval filter data and the join query input at step S40 are similar queries. Here, the similar query means a query joining the same tables to the same join condition. The select clause, the group by clause, and the having clause of the sql statement may be different, and other conditions for comparing constants may be added to the where clause.

Here, the bitmap interval filter data stored in the BIF database is generated using the result of the join query in the previous join query process, and is registered and updated in the BIF database. Hereinafter, referring to FIG. 5, A process of generating bitmap interval filter data in a Hadoop-based reduction-side join query processing method of a redundancy-side join processing system based on the present invention will be described in detail.

First, the result key value is extracted from the result of the join query (S60). Here, the result key value is the result of the join query, that is, the join key values of the result records. Then, the join key domain is divided into a plurality of BIF intervals on a BIF interval basis (S62). Here, the BIF interval means the length of one BIF interval, i.e., the length of the BIF interval.

Here, the BIF interval for dividing the BIF interval can be registered according to the setting of the user. In the present invention, the BIF interval is calculated based on Equation (1) based on the number of elements of the key domain in which the BIF interval is joined, As shown in Fig.

[Equation 1]

L = D / R

In Equation (1), L is the BIF interval, D is the number of elements of the join key domain, and R is the number of extracted result key values.

Then, it is checked whether or not the result key value belongs to each divided BIF section, that is, whether there is a result key value in the corresponding BIF section (S63). Here, if the result key value exists in the corresponding BIF section, the BIF value for whether the result key value exists in the corresponding BIF section is set to 1 (S64). That is, the bit value of the BIF value is set to 1. On the other hand, if the result key value does not exist in the corresponding BIF section, the BIF value for the corresponding BIF section is set to 0 (S65). That is, the bit value of the BIF value is set to zero.

In the above process, until the setting of all the BIF sections is completed (S68), the search for the next BIF section is performed (S68), and the BIF value for the BIF section is set.

Hereinafter, an example in which bitmap interval filter data is generated through the above process will be described with reference to Table 1.

[Table 1]

[Table 1] shows an example in which a join query is performed on two tables. When joining table A and table B, the join key domain of the join key values of the two tables has a value of 0 to 39.

1, 5, 9, 19, 23, 29, 36, 37, and 39, the join key values of the query result records belonging to both tables among the join key values on the join key domain do. That is, the join key values of the output records are extracted as the result key values.

Here, the BIF interval is calculated by dividing the number of elements of the join key domain by the number of result key values. In this example, 40/10 = 4. Then, if the entire join key domain is divided using the BIF interval, it is divided into 10 BIF sections as shown in [Table 2].

[Table 2]

In Table 2, the index value is an ID for identifying the BIF section, and the BIF value indicates whether the result key values 0, 1, 5, 9, 19, 23, 29, 36, 37, 39 exist in each BIF section As shown in FIG.

The BIF value is set to 1 when the result key value exists in the corresponding BIF section, and to 0 when the result key value does not exist. The BIF value enables to check the interval of the join key value which is not correlated with the result record for the join query.

In this way, when the BIF interval is filtered using the BIF interval of 0, the join keys of 5, 7, 12, 21, 23, 27, and 35 in Table A and 4, 5, 23, 26, You can remove records with values.

Through the above process, bitmap section filter data including information on a plurality of BIF sections and a BIF value as to whether the result key value belongs to each BIF section is generated (S69) and stored in the BIF database . In the present invention, the BIF interval and the start join key value of the join key domain, that is, the join key value '0' in [Table 1] can be generated in the bitmap interval filter data.

FIG. 6 is a diagram for explaining the effect of filtering using bitmap interval filter data according to the present invention. Referring to FIG. 6, the basic concept of bitmap interval filter data according to the present invention is a method of filtering by using an interval of join key values rather than a single join key value, as described above. That is, the entire join key domain is divided by a predetermined interval, for example, the BIF interval, and whether a result key value exists in the corresponding BIF interval is checked.

By storing the bitmap interval filter data in a bitmap form and storing the interval rather than each of the join key values, information of a larger amount of the join key value can be stored in the memory, do.

Referring again to FIG. 4, if the bitmap section filter data corresponding to the join quality inputted in step S40 is retrieved, the bitmap section filter data is transmitted to each mapper of Hadoop (S43). Here, it is assumed that distribution of bitmap interval filter data is performed by Hadoop DistributedCache.

Then, each mapper uses the bitmap section filter data to filter records of its own table (S44). Here, each mapper transmits only the records in the BIF section to which the join key value of the table record included in the join query belongs in the plurality of BIF sections of the bitmap section filter data to the reducer, and filters the remaining records.

Hereinafter, a method of filtering a mapper according to the present invention will be described with reference to FIG. FIG. 7 is a diagram illustrating a relationship among elements of bitmap interval filter data according to the present invention.

In Fig. 7, D represents the join key domain, and Base is the start join key value of the join key domain. L is the BIF interval, and I is the index value of the BIF interval. The offset value Offset represents the distance between the start join key value and the join key value, and the join key value of the table record included in the currently input join query is defined as V. [

First, the mapper calculates the offset value using the interval between the join key value and the start join key value of the table record included in the currently input join query. The joining key value V is 37, the BIF interval is 4, and the joining key value is 37. The joining key value V is 37 and the joining key value of the table record included in the current joining query is 37 and the example of [Table 1] The start join key value Base is zero.

The offset value is calculated by the distance between V and Base, that is, the deviation, and is calculated as 37. Here, the mapper extracts the BIF section to which the join key value V belongs using the calculated offset value and the BIF interval. That is, when the BIF interval is 4 and the offset value is 37, the BIF interval to which the join key value V belongs can be extracted using the quotient obtained by dividing the offset value by the BIF interval.

Such a method can be defined as [Equation 2].

&Quot; (2) "

I = (V-Base) / L

Here, I is the index of the BIF section to which the join key value of the table record included in the currently input join query belongs, V is the join key value of the table record included in the currently input join query, Start join key value, and L is the BIF interval).

In this example, the join key value 37 is extracted as belonging to the BIF section having an index value of 9, and the index value starts from 0, for example, and the calculation may be changed according to the index number.

When the BIF section to which the current join key value belongs is extracted as described above, the mapper determines whether to transmit the record belonging to the BIF section based on the BIF value of the extracted BIF section, that is, the bit value, . That is, when the BIF value is 1, the records belonging to the corresponding BIF section are transmitted to the reducer (S45). When the BIF value is 0, the records are filtered and removed.

Through the above process, each mapper transmits only the records of the BIF section having the BIF value of 1 to the reducer, thereby eliminating the network load due to the transmission of the records irrelevant to the inputted join query.

Also, in the case of the reducer, only the records filtered by the mapper are received, the join query is processed, and the result is output (S46), thereby reducing the computation cost by processing fewer records in the existing re- .

In the filtering process by the mapper, the BIF section in which the join key value belongs is searched, and the time required to find the BIF value, that is, the bit value, can be quickly processed irrespective of the size of the filter, .

If the bitmap section filter data is not found in step S42 of FIG. 4, the Hadoop-based reduced-side join processing system according to the present invention can perform join query processing using an existing method, for example, a conventional Hadoop- (S47). For example, the result of the join query is generated through the Reduce-Sine join method, the map-side join method, or a combination thereof of the existing MapReduce.

At this time, the result key value is extracted from the result of the join query processed through the existing method (S48), and bitmap interval filter data for the corresponding join query is generated through the process shown in FIG. 5 Stored in the BIF database (S49), so that it can be utilized in inputting similar similar join queries thereafter.

8 is a diagram illustrating an example of a hive system for implementing a Hadoop-based reduction-side join query processing method according to the present invention. A QueryMatcher, a JoinkeyDistributer, a RecordFilter, a JoinkeyExtractor, and a BIF Generator are added to an existing Hadoop based hive system to implement a join query processing method according to the present invention.

QueryMatcher determines whether or not the query received from the user is a query. If the query is a join query, it searches the BIF database for bitmap interval filter data.

If there is bitmap interval filter data, the JoinkeyDistributer distributes the retrieved bitmap interval filter data to each mapper. Then, the RecordFilter implemented in each mapper performs the above-described filtering process using the bitmap interval filter data.

On the other hand, if the bitmap interval filter data does not exist, the JoinkeyExtractor extracts the result key value, and the BIF generator generates the filtered bitmap data of the corresponding join query using the extracted result key value.

In FIG. 8, an example is shown in which a QueryMatcher is implemented in a SemanticAnalyzer that analyzes a query, a JoinkeyDistributer is implemented in ExecDrier, a RecordFilter is implemented in a mapper, i.e., ExecMapper, and a JoinkeyExtrator is implemented in ExecReducer. Here, the BIF Generator is designed to operate independently of the query flow of the join query.

Hereinafter, referring to FIGS. 9 to 12, a description will be given of an experiment result of applying the Hadoop-based reduction-side join query processing method according to the present invention to the hive HIVE.

In the experiment, the processing operation of the Hadoop based reduction-side join query processing method according to the present invention was classified into two processes. If there is no interval filter mit map data corresponding to the user query, the result key value is extracted in the query processing. If there is segment map filter map map data, the segment filter map map data corresponding to the query is distributed to each mapper, and records having no correlation with the join result are removed. Experimental results show that the query processing performance of these two processes is compared with that of existing hives.

Experiments were divided into two cases: binary joins, that is, joining with one mapping task, and multiway joining, ie joining with two or more MapReduce tasks. The data model used in the experiment uses the data model of TPC-H which is mainly used for measuring the performance of the data warehouse system. Among them, the binary join uses the LineItem table and the Orders table, and the multiway join uses the customer table of the two tables. The SF (Scale Factor) used here is a coefficient that determines the number of record generation per table.

The join query used in the experiment is shown in [Table 3]. Because TPC-H has many influences on other factors such as aggregation function, the following join condition and simple where condition are used for the query. The reason for attaching the where clause is that it contains not only the join condition but also the where clause condition because the join key is extracted from the join result. Also, the where clause condition is used to give the join selectivity. In the binary join, orderdate is used as the condition of the where clause. In addition, the mktsegment condition is added to the where clause condition in the multiway join.

orderdate: 1992-01-01 ~ 1998-12-31

mktsegment: AUTOMOBILE, BUILDING, FURNITURE, MACHINERY, HOUSEHOLD

[Table 3]

9 is a graph illustrating a query query processing time according to the number of nodes in a binary join. In describing the graph, the case where the bitmap interval filter data is absent is defined as 'without BIF' and the case where bitmap interval filter data is defined as 'using BIF'.

Referring to FIG. 9, it can be seen that the query processing time decreases as the number of nodes increases in both graphs. If there is no BIF, the result key value is extracted during the query processing. Therefore, as shown in FIG. 9, it can be seen that the process of extracting the join key value has no significant effect on performance.

In the case of using BIF and experiment results of existing hive, we can confirm that the query processing time of BIF is improved by 20% ~ 26% compared with the existing hive. This is because filtering in the mapper removes records that are not related to the join query, reducing file I / O and network costs in the shuffle process.

FIG. 10 is a graph illustrating a query query processing time according to an increase in the number of nodes in a multi-way join, FIG. 11 is a graph illustrating a query query processing time according to a data size in a binary join, This is a graph measuring the query processing time according to the data size.

As shown in FIGS. 10 to 12, when the BIF is used, that is, when the Hadoop-based redox-side join query processing method according to the present invention is applied, it can be confirmed that the performance is improved as compared with the existing hive.

Although several embodiments of the present invention have been shown and described, those skilled in the art will appreciate that various modifications may be made without departing from the principles and spirit of the invention . The scope of the invention will be determined by the appended claims and their equivalents.

Claims (5)

A method for processing a redo-side join query of a redo-side join processing system based on Hadoop,
(a) generating bitmap interval filter data based on a resultant key value extracted from a result of a join query in the Hadoop-based reduction-side join processing system and storing the bitmap interval filter data in a BIF database;
(b) if a new join query is input to the Hadoop-based redundancy-side join processing system, the redo-side join processing system of the Hadoop classifies the bitmap section filter corresponding to the new join query into the BIF database Data is retrieved,
(c) when the bitmap interval filter data is retrieved in the step (b), transmitting the retrieved bitmap interval filter data to each mapper of Hadoop;
(d) transmitting, by the respective mapper, a filtered record through filtering based on the bitmap interval filter data to the Hadoop reducer;
(e) outputting a result of the new join query based on the record transmitted from each mapper by the reducer;
The step (a) performed by the Hadoop-based reduction-side join processing system
(a1) extracting the result key value from the result of the join query;
(a2) dividing the join key domain into a plurality of BIF intervals in units of BIF intervals;
(a3) checking whether the extracted result key value belongs to each BIF section;
(a4) bitmap interval filter data including information on the plurality of BIF sections and a BIF value as to whether the extracted result key value belongs to each BIF section is generated and stored in the BIF database ;
In the step (d), the mapper transmits to the reducer a record in a BIF section to which a join key value of a table record included in the new join query among the plurality of BIF sections of the bitmap section filter data belongs. A method for processing a redo-side join query of a Hadoop-based redo-side join processing system.
The method according to claim 1,
In step (a2), the BIF interval may be expressed by the following equation: L = D / R (where L is the BIF interval, D is the number of elements of the join key domain, and R is the number of the extracted result key values) Side joining processing system of the Hadoop based reduction-side join processing system.
3. The method of claim 2,
Wherein the bitmap interval filter data further includes a BIF interval and a start join key value of the join key domain,
In the step (d), the mapper
An offset value is calculated based on an interval between a join key value of the table record included in the new join query and the start join key value,
Extracts a BIF section to which a join key value of a table record included in the new join query belongs based on a join key value of the table record included in the new join query, the offset value, and the BIF interval,
Side join processing of the Hadoop based reduction-side join processing system, based on the BIF value of the extracted BIF section, whether or not a record in the extracted BIF section is transmitted to the reducer Query processing method.
The method of claim 3,
The mapper may be represented by equation
I = (V-Base) / L
(Where I is an index of a BIF section to which a join key value of a table record included in the new join query belongs, V is a join key value of a table record included in the new join query, The join key value is a start join key value, and L is the BIF interval), the BIF section to which the join key value of the table record included in the new join query belongs is extracted. Reduce - Side join query processing method.
5. The method according to any one of claims 1 to 4,
(f1) outputting the result of the new join query through a predetermined processing method by the Hadoop-based redes-side join processing system when the bitmap section filter data is not retrieved in the step (b) ;
(f2) extracting a result key value from the result of the new join query output in the step (f1) by the Hadoop-based reduction-side join processing system;
(f3) The steps (a2) to (a4) are performed on the resultant key value extracted in the step (f2) by the Hadoop-based reduction-side join processing system, And generating bitmap interval filter data and storing the bitmap interval filter data in the BIF database. The method of claim 1,

Images