KR101400499B1 - Apparatus and method of parallel processing of linked big data - Google Patents

Abstract

The present invention relates to a parallel processing apparatus and a parallel processing method for large-sized linked data. The parallel processing apparatus for large-sized linked data can extract a schema triple for an input file and repeatedly apply a MapReduce process on the extracted schema triple to process the transitivity reasoning for resource description framework (RDF) ontology.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a parallel processing apparatus and method for large-

Embodiments of the present invention implement a parallel processing apparatus and method for large amount of linked data by effectively performing transitivity reasoning on an RDF (Resource Description Framework) ontology by repeatedly applying MapReduce It is about technical thought.

A distributed file system is a client / server based application that allows users to easily access and manage the same distributed files by physically connecting different computers to each other by networking. It is mainly used for adding file servers and modifying file locations, It can be used when accessing multiple sites or objects, when an organization has an internal or external website, and so on.

Recently, in this distributed file system, Hadoop distributed framework for analyzing big data is used to improve the processing time of large mobile lifelog data. Hadoop is an open source distributed processing framework that clusters multiple computers and provides a reliable shared storage, Hadoop Distributed File System (HDFS) and map / resume, analysis system.

Recently, in the analysis of big data, researches on how to solve fixed-point solution by applying MapDeuse repeatedly based on Hadoop are being studied.

In particular, the transitivity reasoning for the Resource Description Framework (RDF) ontology is based on a join to triple, a repetitive maple that continuously generates a monotone increasing dataset It can be implemented as a MapReduce Job.

In this case, if the size of the output dataset generated by the Reduce task of the n-th iteration increases, it becomes the input of the (n + 1) -th map task, Mapper) and Reducer nodes are increasing rapidly and the processing cost is very high.

A method of operating a parallel processing apparatus for massively linked data according to an exemplary embodiment includes extracting a schema triple for an input file, repeatedly applying mapreduce to the extracted schema triple, And processing the transitivity reasoning for the Resource Description Framework (RDF) ontology.

The step of processing the transitivity reasoning according to an embodiment includes defining the transitivity reasoning as a monotone fixed-point iteration model . ≪ / RTI >

The processing of transitivity reasoning in accordance with an embodiment includes the steps of storing a data partition processed in a previous iteration through a redess task in a local file system, And assigning a next redundancy task on the basis of the second redundancy task.

The method of operating the apparatus for parallelizing large capacity linked data according to an exemplary embodiment may further include removing a duplicate triple, and processing inheritance reasoning.

The step of extracting a schema triple for the input file according to an exemplary embodiment may include setting a property for the input file as a key value.

The step of processing transitivity reasoning for an RDF (Resource Description Framework) ontology by iteratively applying a mapreduce to the extracted schema triple according to an embodiment includes: Checking whether a new matter is generated according to jitter ratio re-establishment, and if a new matter is generated as a result of the checking, And repeatedly performing transitivity reasoning.

A method of operating a massively parallel linked data parallel processing apparatus according to an exemplary embodiment of the present invention is a method in which a schema triple scheme (schema triple) in an output file.

The operation method of the massively parallel linked data parallel apparatus according to an embodiment may further include storing an instance triple obtained by extracting a schema triple of the input file.

The apparatus for parallel processing large amount of linked data according to an exemplary embodiment includes a schema triple extracting unit for extracting a schema triple for an input file, and a mapreduce is repeatedly applied to the extracted schema triple A transitivity bottleneck processing unit for processing transitivity reasoning for a Resource Description Framework (RDF) ontology, a duplicate triple in a schema triplet processed in the transit bit depth list, And an inheritance reasoning processing unit for processing inheritance reasoning for the schema triples from which the redundant triples have been removed. have.

The transitivity bottleneck processing unit according to an exemplary embodiment may define the transitivity reasoning as a monotone fixed-point iteration model.

The transit bit listening processor according to an exemplary embodiment may store a data partition processed in a previous iteration through a redo task in a local file system and allocate a next redo task based on the location of the stored data partition can do.

According to an embodiment of the present invention, it is possible to effectively perform transitivity reasoning on an RDF (Resource Description Framework) ontology by repeatedly applying MapReduce using Hadoop.

According to an embodiment of the present invention, a transitivity reasoning for a resource description framework (RDF) ontology is defined as a monotone fixed-point iteration model, and an efficient parallel processing You can implement a model based on Hadoop.

According to one embodiment of the present invention, the redisplay task stores the data partition processed in the previous iteration in the local file system and allocates the next redisplay task based on the locality of the data partition, And the amount of calculation can be reduced.

(델타)-조인 알고리즘을 설명하는 도면이다.
도 9는 본 발명의 일실시예에 따른 트랜지터비티 리즈닝 처리부에서의 맵 테스크를 설명하는 도면이다.
도 10는 본 발명의 일실시예에 따른 트랜지터비티 리즈닝 처리부에서의 리듀스 테스크를 설명하는 도면이다.
도 11은 본 발명의 일실시예에 따른 트랜지터비티 리즈닝 처리부에서의 맵 테스크를 설명하는 도면이다.
도 12은 본 발명의 일실시예에 따른 트랜지터비티 리즈닝 처리부에서의 리듀스 테스크를 설명하는 도면이다.
도 13은 본 발명의 일실시예에 따른 트랜지터비티 리즈닝 처리부에서의 맵 테스크를 설명하는 도면이다.
도 14는 본 발명의 일실시예에 따른 트랜지터비티 리즈닝 처리부에서의 리듀스 테스크를 설명하는 도면이다.
도 15은 본 발명의 일실시예에 따른 중복 트리플 제거부에서의 맵 테스크를 설명하는 도면이다.
도 16은 본 발명의 일실시예에 따른 중복 트리플 제거부에서의 리듀스 테스크를 설명하는 도면이다.
도 17는 본 발명의 일실시예에 따른 인헤리턴스 리즈닝 처리부에서의 맵 테스크를 설명하는 도면이다.
도 18은 본 발명의 일실시예에 따른 인헤리턴스 리즈닝 처리부에서의 리듀스 테스크를 설명하는 도면이다.Fig. 1 is a view for explaining a map deuce.
FIG. 2 is a diagram for explaining a monotone fixed-point iteration model. FIG.
Figures 3 and 4 are diagrams illustrating an improved monotone fixed-point iteration model.
5 is a block diagram illustrating an apparatus for parallel processing large capacity linked data according to an embodiment of the present invention.
6 is a view for explaining a map task in a schema triple extracting unit according to an embodiment of the present invention.
7 is a view for explaining a reduction task in a schema triple extracting unit according to an embodiment of the present invention.
Figure 8

(Delta) -join algorithm.
FIG. 9 is a view for explaining a map task in a transitivity feedback processing unit according to an embodiment of the present invention.
10 is a view for explaining a reduction task in a transitivity bottleneck processing unit according to an embodiment of the present invention.
11 is a view for explaining a map task in a transitivity bottleneck processing unit according to an embodiment of the present invention.
FIG. 12 is a view for explaining a reduction task in a transitivity bottleneck processing unit according to an embodiment of the present invention.
FIG. 13 is a view for explaining a map task in a transitivity bottleneck processing unit according to an embodiment of the present invention.
FIG. 14 is a view for explaining a reduction task in a transitivity bottleneck processing unit according to an embodiment of the present invention.
15 is a view for explaining a map task in a duplicate triple elimination according to an embodiment of the present invention.
16 is a view for explaining a reduction task in a duplicate triple elimination according to an embodiment of the present invention.
17 is a view for explaining a map task in an inheritance listening processor according to an embodiment of the present invention.
18 is a view for explaining a reduction task in the inheritance listening processor according to an embodiment of the present invention.

Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings.

In the following description of the present invention, detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear. The terminologies used herein are terms used to properly represent preferred embodiments of the present invention, which may vary depending on the user, the intent of the operator, or the practice of the field to which the present invention belongs. Therefore, the definitions of these terms should be based on the contents throughout this specification. Like reference symbols in the drawings denote like elements.

Fig. 1 is a view for explaining a map deuce.

To process the MapReduce, the input data is split and divided into several pieces.

For example, data cleaved from input data can be sent to each Map () and output in the form of <key (key), value> in Map ().

Then, the outputs from each Map () can be combined with the numbers output from other Map (), and the partitioning and soring processes can occur using the key values output from Map ().

After the map process is finished, the redescription process can be performed and the output values having the same key value are collected during the redescription process.

This process is called shuffle. The shuffled data is preceded by the key value followed by each value.

Reduce () processing can be done through this shuffled data.

Map () is responsible for the tasks that need to be done on a record-by-record basis, while Reduce () is responsible for handling the related data together. Therefore, between Map () and Reduce () operations, we need to partition the data to process the data processed by Map () in Reduce ().

Reference numeral 110 denotes that the input file is generated as an output file through the mapping process, and reference numeral 120 denotes a process of repeatedly processing the mapping process in the distributed file system.

Reference numeral 120 can be used to obtain a fixed-point solution by repeatedly applying MapReduce in analysis of recent big data.

2 is a diagram for explaining a monotone fixed point repetition model 200. Fig.

An output value is generated through a process of Map and Reduce for an input file. Such a process can be represented by a job.

x은 T_n으로부터 새롭게 획득되는 RDF 트리플들의 세트로 해석될 수 있다.In Figure 2, x _n can be interpreted as a set of RDF schema triples for the n-th listening loop,

x can be interpreted as a set of RDF triples newly obtained from T _n .

p [k] _n may be interpreted as a data partition for the key (k) fed back to the residue at the nth iteration.

and n = 1, 2, 3, can be applied to f (x _n) = x _n be f (x _n) = x _n online endpoint repeated model, which is represented by +1 until.

x _n에 의해서 단조 증가할 수 있다.The 'f' in the monotone fixed point repetition can be applied to x _n , the set of input files, and the size of the output triple set x _{n + 1} is determined from the size of the input triple set x _n ,

x _n . &lt; / RTI &gt;

도 함께 증가할 수 있다.In the monotone repetitive point repetition model of FIG. 2, as the repetition progresses, the amount of data re-input to the map test increases recursively, and the total amount of data

Can also increase.

Figure 3 is a diagram illustrating an improved monotone Pulsepoint repetition model 300.

x _n을 생성할 수 있는데, |

x _n|이 '0'일 때까지 반복할 수 있다.The monotone Fulldeppe repetition model 300 can be represented by &quot; f &quot; in n = 1, 2, 3,

x _n , where |

It can be repeated until x _n | is '0'.

x_n에 의해서 단조 증가하지 않는 n까지 반복을 계속할 수 있다.In other words, 'f' is the newly created triple set

it is possible to continue the repetition up to n which does not increase monotone by x _n .

의 조건을 만족하는

x만 다시 회귀 입력(recursive input)으로 반복하여 전송되는 데이터 량을 대폭 줄일 수 있다.What is a monotone Phillips repeat model 300? _If x _n-1 does not affect the generation of _n ,

Satisfy the condition of

the amount of data to be transmitted repeatedly through recursive input can be greatly reduced.

인 경우, 즉

x_n을 생성하는데 x_n-1이 영향을 주는 경우에는 트랜지터비티 리즈닝의 적용이 불가하다.But,

In other words,

When x _{n -1} affects x _n , it is not possible to apply transient bit listening.

4 is a diagram for explaining an improved monotone Fulldep Repetition model 400. FIG.

The monotone Puls point repetition model 400 may store a data set, i.e., a partition, fed back from the map task to the resume task in the nth iteration, in the local file system and HDFS (Hadoop Distributed File System).

That is, black and white Phelps endpoint repeated model 400 is the key (k) the set of data feedback from each of Li deuce task the n th iteration of the (p [k] _n) own local file system and the HDFS (Hadoop Distributed File System) can be stored (∪ _{1 ≤ i ≤ n} p [k] _i ).

Also, the monotone Puls point repetition model 400 can use a file stored in the local file system in the (n + 1) -th iteration.

The monotone repetitive point repetition model 400 may consider data locality unlike the conventional redundancy task scheduling.

The monotone fill point repetition model 400 can allocate the data locality of the partition file in preference to the allocation of the redo task for the key k in the nth iteration.

For example, the monotone fill-point repetition model 400 can handle the following cases when an error occurs in the resume task for the key k.

First, the monotone Puls point repetition model 400 reassigns the redescription task to the new node according to the locality scheduling, and causes the new node to copy the partition file corresponding to its key from the HDFS to the LFS , It is possible to re-execute the resume task using the partition file copied to the LFS.

In addition, the monotone punctual repetition model 400 can delete the existing node information in which the error occurred from the list of the locality scheduling, and add new node information.

5 is a block diagram illustrating an apparatus 500 for parallel processing large capacity linked data according to an embodiment of the present invention.

The apparatus 500 for parallel processing large amount of linked data according to an embodiment of the present invention performs transitivity reasoning on an RDF (Resource Description Framework) ontology by repeatedly applying MapReduce .

In addition, the apparatus 500 for parallel processing large amount of linked data according to an embodiment of the present invention may perform a transitivity reasoning on a resource description framework (RDF) ontology using a monotone fixed- point iteration model, and implement a model based on Hadoop for efficient parallel processing.

In addition, the large-capacity linked data parallel processor 500 according to an embodiment of the present invention stores the data partition processed in the previous iteration in the local file system by the resume task, and based on the locality of the data partition By assigning the next Reduce task, you can reduce the amount of data traffic and calculation between tasks.

To this end, the apparatus for parallel processing large capacity linked data 500 according to an embodiment of the present invention includes a schema triple extracting unit 510, a transistor bit depth recognition processing unit 520, a redundant triple eliminating unit 530, And a processing unit 540 for processing the received signal.

The schema triple extractor 510 according to an embodiment of the present invention can extract a schema triple for an input file.

The schema triple of an ontology can be interpreted as representing the concepts of two entities, the relationship between these concepts, or the attributes that a concept can have.

An ontology can contain a large number of instance triples that follow a schema triple, and these instance triples can appear in natural-language sentences in the real world.

The schema triple extractor 510 according to an embodiment of the present invention can extract a given schema triple using the instance triple information of the input file.

The ontology can be divided into schema level and instance level.

A triple at a schema level represents concepts of objects, their semantic relations and possible attributes, and an instance-level triple can appear as a variety of sentences including this meaning in real-world natural language as fact information.

FIG. 6 is a view for explaining a map task in the schema triple extracting unit 510 according to an embodiment of the present invention, and FIG. 7 is a view for explaining a reduce task in the schema triple extracting unit according to an embodiment of the present invention FIG.

Map and Reduce can be interpreted as a software framework that distributes and processes jobs in a computer cluster environment in distributed processing of big data.

The job used in this specification can be interpreted as a unit of a job requested by the user. Jobs can be distributed on several nodes. In this case, the detailed work unit performed by each node is called a task. Once the Map tasks are executed first, Reduce tasks can take the results and process them.

Map and Reduce are functions used to process list data in functional languages, and they are divided into Map functions and Reduce functions.

First, in FIG. 6, a schema triple extractor 510 according to an embodiment of the present invention can perform a map task 600 from an input file.

The schema triple extracting unit 510 according to an embodiment of the present invention can take charge of operations to be performed in units of records in the map task 600. [

Specifically, the schema triple extractor 510 according to an exemplary embodiment of the present invention receives a triple from an input file and extracts a schema triple in the form of a <key> value.

That is, when a triple of "C1 rdfs: subClassOf C2" is inputted, the schema triple extracting unit 510 according to an embodiment of the present invention extracts "rdfs: subClassOf" as a key value and "C1 rdfs : subClassOf C2 "can be extracted.

Likewise, when a triple of "C2 rdfs: subClassOf C3" is input, the schema triple extractor 510 according to the embodiment of the present invention extracts "rdfs: subClassOf" as a key value and "C2 rdfs : subClassOf C3 "can be extracted.

When a triple of "a rdf: type C1" is inputted, the schema triple extracting unit 510 according to an embodiment of the present invention extracts "rdf: type" as a key value and "a rdf : type C1 "can be extracted.

When a triple of "b rdf: type C2" is inputted, the schema triple extracting unit 510 according to an embodiment of the present invention extracts "rdf: type" as a key value and "a rdf : type C2 "can be extracted.

When a triple of "C3 rdfs: subClassOf C4" is inputted, the schema triple extracting unit 510 according to an embodiment of the present invention extracts "rdfs: subClassOf" as a key value and "C3 rdfs : subClassOf C4 "can be extracted.

In addition, when a triple of "c rdf: type C3" is input, the schema triple extracting unit 510 according to an embodiment of the present invention extracts "rdf: type" as a key value and "a rdf : type C2 "can be extracted.

Next, the schema triple extractor 510 according to an embodiment of the present invention may perform a reduce task 700 as shown in FIG.

The schema triple extracting unit 510 according to an embodiment of the present invention can process jobs to be related to each other by processing related data in the redess task 700. [

Between the map task and the redess task, a process of partitioning data processed by the map task 600 in the redissue task 700 is required. In the present invention, Data can be classified.

That is, the schema triple extractor 510 according to an embodiment of the present invention can sort the schema triples extracted by the key value.

That is, the extracted schema triples can be classified into key values of "rdfs: subClassOf" and "rdf: type ", and the schema triples can be sorted according to key values,

The schema triple extracting unit 510 according to an embodiment of the present invention can extract the schema triple by using the instance triple information of the input file and dividing the ontology into the schema level and the instance level.

The schema triple extracting unit 510 according to an embodiment of the present invention may assign the same key value to related data in the map task 600 and then transmit the data having the same key value to the reduction task 700. [

The schema triple extractor 510 according to an embodiment of the present invention can sort and store the same key values when storing the result data.

At this time, it is possible to process partitioning for collecting and arranging data to be processed in the same reduction task 700 again for the data having the same key values.

In addition, the schema triple extractor 510 according to an embodiment of the present invention performs a shuffle process of fetching classification data held by a child in the reduction task 700 when the map task 600 is finished, You can merge and sort by value.

Hereinafter, the schema triple extractor 510 according to an embodiment of the present invention may process schema triples in a pair of <key and value>, and process the schema triples in the reduce task 300.

The schema triple extractor 510 extracts schema triples of "C1 rdfs: subClassOf C2", "C2 rdfs: subClassOf C3", and "C3 rdfs: subClassOf C4" according to an embodiment of the present invention, / "Output directory.

The result of the reduction task 700 processing may be stored in a result directory designated by the user in the parallel processing unit 500 of the large capacity linked data.

The transitivity recognition processing unit 520 according to an embodiment of the present invention repeatedly applies a mapreduce to the extracted schema triple to generate a transit bit list for the RDF (Resource Description Framework) ontology, Transitivity reasoning can be handled.

For this purpose, the transit bit depth recognition processing unit 520 according to an embodiment of the present invention may define a monotone fixed-point iteration model.

In addition, the transitivity listening processing unit 520 according to an exemplary embodiment of the present invention stores the data partition processed in the previous iteration through the redess task in the local file system, and based on the position of the stored data partition, Reduce tasks can be assigned.

x_n-1) = f(x_n-1) ∪f(

x_n-1)"의 조건을 만족하는 경우에, 즉

x_n을 생성하는데 x_n-1이 영향을 주지 않는 경우의

x만 다시 반복되는 입력(recursive input)으로 반복하여 전송되는 데이터 량을 대폭 줄일 수 있다.The transit bit boundary recognition processing unit 520 according to an embodiment of the present invention calculates the bit error rate of the bit stream according to "f (x _n ) = f (x _n-

x _n-1 ) = f (x _n-1 )? f

x _n-1 ) &quot;, that is,

If x _{n -1} does not affect the generation of x _n

The amount of data that is repeatedly transmitted with only x recursive input can be greatly reduced.

To this end, the transitivity listening processing unit 520 according to an embodiment of the present invention transmits data sets or partitions fed from the map task to the resume task at the n-th iteration to the local file system and the large capacity linked data In parallel processing apparatuses.

At this time, each resume task for the key value k continuously stores the data set (p [k] _n ) fed from the n-th iteration in its own local file system and the parallel processing unit of massively linked data (∪1 _{= i = n} p [k] _i ).

Thus, the local file stored in the local file system and the parallel processing device of the large amount of linked data can be used in the (n + 1) th iteration.

The transitivity listening processing unit 520 according to an embodiment of the present invention may preferentially consider the data locality of the partition file when assigning the resume task for the key value k in the nth iteration.

The transitivity listening processing unit 520 according to an embodiment of the present invention is a node that stores a partition file up to the (n-1) th iteration of the key k in the local file system, and other nodes, System and the node to which the resume task is assigned that is copying the partition file stored in the parallel processing unit of the large amount of linked data.

The transit bit listening processing unit 520 according to an embodiment of the present invention can allocate a resume task to a new node according to local scheduling when an error occurs in the resume task for the key value k.

In addition, the transit bit listening processing unit 520 according to an embodiment of the present invention controls the node so that a partition file corresponding to the key value k of the large amount of linked data is multiplied into the local file system , You can use the partition file copied to the local file system to redo the task.

In addition, the transitivity listening processing unit 520 according to an exemplary embodiment of the present invention may delete the existing node information in which the error occurred from the list of the localized schedules, and add new node information.

-조인 알고리즘(800)을 설명하는 도면이다.Figure 8

- &lt; / RTI &gt; join algorithm 800 of FIG.

-조인 알고리즘(800)을 살펴보면, 클래스들의 세트 C와, 프로퍼티들의 세트 P가 주어졌을 때,

- Join algorithm 800, given a set of classes C and a set of properties P,

Triple t = (s p o), where s ∈ C ∧ p ∈ P ∧ o ∈ C,

Triple set T ⊂ {(s p o) | s ∈ C ∧ p ∈ P ∧ o ∈ C}

The triple <k, t> = <k, (s p o)> mapped by the key value k,

The set of triples mapped by key value k is Tk = {<k, t> | t ∈ T}.

-조인 알고리즘(800)은 트리플들의

-조인(join)을 이용하는 리듀스 함수(

-Join)를 구동시킬 수 있다.

The join algorithm 800 determines the

- a reduction function that uses a join (

-Join) can be driven.

-조인 알고리즘(800)은 리듀스 함수(

-Join)를 통해 리듀스 테스크로부터 새로 수신된 맵 처리된 트리플들로서, Tk를 정의하고, Tk로부터 추출된 트리플들의 세트로서 T를 정의하며, S =

, O =

, L =

로 정의할 수 있다.

The join algorithm 800 can be used to determine the &lt; RTI ID = 0.0 &gt;

-Join), defines Tk as the newly received mapped triples from the Reduce task, defines T as the set of triples extracted from Tk, and S =

, O =

, L =

.

-조인 알고리즘(800)은 리듀스 함수(

-Join)를 통해 "localfile_k"로부터 L에 트리플들을 리딩할 수 있다.Also,

The join algorithm 800 can be used to determine the &lt; RTI ID = 0.0 &gt;

You can read triples from "localfile_k" to L via -Join.

-조인 알고리즘(800)은 T를 "localfile_k"와 "hdfsfile_k"에 추가할 수 있다.If the condition of the leading result "| L | &gt;0" is satisfied,

The join algorithm 800 may add T to "localfile_k" and "hdfsfile_k &quot;.

-조인 알고리즘(800)은 "hdfsfile_k"로부터 L에 트리플들을 리딩하며, "|L| > 0"의 조건을 만족하지 않으면서 "|L| == 0"의 조건을 만족하면, "hdfsfile_k"를 생성하고, T를 "hdfsfile_k"에 추가하고, "|L| > 0"의 조건을 만족하지 않으면, "localfile_k"를 생성하고, (L ∪ T)를 "localfile_k"에 추가할 수 있다.If the condition of "| L | &gt;0" is not satisfied,

- Join algorithm 800 reads triples in L from " hdfsfile_k &quot;, and if it satisfies the condition of " | L | == 0 & And add T to "hdfsfile_k", and if "L |>0" is not satisfied, "localfile_k" can be created and (L ∪ T) can be added to "localfile_k".

-조인 알고리즘(800)은 "|L| > 0"의 조건을 만족하면, L로부터 's' rdfs:subClassOf k를 만족하는 's'를 선택하여 S에 치환하고, L로부터 k rdfs:subClassOf 'o'를 만족하는 'o'를 선택하여 O에 치환하며, T로부터 's' rdfs:subClassOf k를 만족하는 's'를 선택하여

S에 치환하고, T로부터 k rdfs:subClassOf 'o'를 만족하는 'o'를 선택하여

O에 치환할 수 있다.Also,

- Join algorithm 800 selects's' that satisfies's' rdfs: subClassOf k from L and substitutes it into S, if k satisfies the condition of | L |> 0, and k rdfs: subClassOf ' o 'that satisfies'o' is selected and replaced with O, and 's'satisfying's' rdfs: subClassOf k is selected from T

S, and selects 'o' that satisfies k rdfs: subClassOf 'o' from T,

O &lt; / RTI &gt;

-조인 알고리즘(800)은 s ∈

S이고, o ∈

O이면, "temp/"에 "s rdfs:subClassOf o"를 출력할 수 있고, o ∈

O이고, s ∈ S이면, "temp/"에 "output (s rdfs:subClassOf o)"를 출력할 수 있다.Also,

The join algorithm (800)

S, and o ∈

O, you can output "s rdfs: subClassOf" to "temp /", and o ∈

O, and s ∈ S, "output (s rdfs: subClassOf o)" can be output to "temp /".

-조인 알고리즘(800)은 리듀스 함수(

-Join)를 통해서 "localfile_k"에 저장된 트리플들 끼리의 조인은 제외시킬 수 있다.

The join algorithm 800 can be used to determine the &lt; RTI ID = 0.0 &gt;

-Join), you can exclude joins between triples stored in "localfile_k".

-Join)는 새로 입력된, 즉

x_n-1에서 획득된 트리플에 대해서만 조인을 수행할 수 있다.That is,

-Join) is a new input

The join can be performed only for the triples obtained at x _n-1 .

-조인 알고리즘(800)에 따르면, n번째 반복의 리듀스 테스크에서 조인해야 하는 전체 트리플의 개수M_n에 대해서 Σ_1≤i≤n|p[k]_i| 로 표현할 수 있다.

According to the join algorithm 800, for the number of all triples M _n to be joined in the reduction task of the n-th iteration,? _1? I _{? N} | p [k] _i | .

-조인 알고리즘(800)은 전체 M_n개의 트리플 중 "localfile_k"에 저장된 트리플의 개수를 Σ_1≤i≤n-1|p[k]_i| = |L|로 표현할 수 있고, 그 중에 1/2이 k가 오브젝트 또는 서브젝트인 트리플이라고 가정하면, |S| = |O| = |L|/2으로 표현할 수 있다.Also,

- Join algorithm 800 determines the number of triples stored in "localfile_k" of all M _n triples to be Σ _{1 ≤ i ≤ n-1} | p [k] _i | = | L |, and half of them are triples, where k is an object or a subject, the | S | = | O | = | L | / 2.

-조인 알고리즘(800)에 따르면, 새로 입력된, 다시 말해

x_n-1에서 획득된 트리플의 개수를 |p[k]_n| = |T| = M_n - |L|로 표현할 수 있고, 그 중에서 1/2이 k가 오브젝트 또는 서브젝트인 트리플이라고 가정하면, |

S| = |

O| = |T|/2 = (Mn-|L|)/2로 표현될 수 있다.

According to the join algorithm 800, a new input, i. E.

Let p [k] _n | be the number of triples acquired in x _n-1 . = | T | = M _n - | L |, and half of them are triples, where k is an object or a subject.

S | = |

O | = | T | / 2 = (Mn- | L |) / 2.

-조인 알고리즘(800)에서의 전체 조인 횟수는 다음과 같다.

The total number of joins in the join algorithm 800 is as follows.

_{((M n - | L |} ) / 2) * ((M n - | L |) / 2) + 2 * ((M n - | L |) / 2) * (| L | / 2)

_{= ((M n - | L} |) / 2) * (((M n - | L |) / 2) + 2 * (| L | / 2)) = ((M n - | L |) / 2 ) * ((M _n + | L |) / 2)

= (M _n - | L |) * (M _n + | L)) / 4 = (M _n ² - | L | ² ) / 4 = M _n ² / ^2.4

-조인 알고리즘(800)의 각 반복에서 발생하는 전체 조인의 수는 로컬 트리플 수의 제곱 만큼씩 줄어 들고, |L| = 0인 경우에 조인의 횟수는 'M_n ²/4'로 표현될 수 있다.

The total number of joins occurring in each iteration of the join algorithm 800 decreases by the square of the number of local triples, The number of times of the join in the case of = 0 can be represented by 'M _n ^2/4'.

The transitivity listening processing unit 520 according to an embodiment of the present invention repeats the mapreduce for the extracted schema triples to process the transitivity reasoning can do.

FIG. 9 is a view for explaining a map task 900 in a transitivity listening processing unit 520 according to an embodiment of the present invention.

Specifically, "C1 rdfs: subClassOf C2", "C2 rdfs: subClassOf C3", and "C3 rdfs: subClassOf C4" extracted as a schema triple are transmitted through the map task 900 of FIG. Value &gt; &lt; / RTI &gt;

Quot ;, " C1, C1 rdfs: subClassOf C2> "and" C1 rdfs: subClassOf C2 &quot; for the schema triple of "C1 rdfs: subClassOf C2 " through the map task 900 according to an embodiment of the present invention. <C2, C2 rdfs: subClassOf C3> "and" <C3, C2 rdfs: subClassOf C2> "for the schema triple of" C2 rdfs: C3 rdfs: subClassOf C4> "and" C4, C3 rdfs: subClassOf C4> "for the schema triple of" C3 rdfs: subClassOf C4 ".

FIG. 10 is a view for explaining a reduction task 1000 in a transitivity bottleneck processing unit according to an embodiment of the present invention.

The transitivity listening processing unit 520 according to an embodiment of the present invention, when there are three RDF classes,

( x rdf: type rdfs: Class),

( y rdf: type rdfs: Class),

( z rdf: type rdfs: Class)

Based on the class transitivity rule below, the reduce task 1000 can be processed.

&Lt; Class Transistor Bitty Rule >

"(x rdfs: subClassOf y) ∧ (y rdfs: subClassOf z) ⇒ (x rdfs: subClassOf z)"

Specifically, the transit bit depth recognition processing unit 520 according to an embodiment of the present invention includes a map task 900, a &quot; C1, C1 rdfs: subClassOf C2> C3 rdfs: subClassOf C3> "," <C3, C2 rdfs: subClassOf C3> "," <C3, C2 rdfs: C1 rdfs: subClassOf C3 "and" C2 rdfs: subClassOf C4 "can be extracted through the redescription task 1000.

The transitivity listening processing unit 520 according to an exemplary embodiment of the present invention checks whether a new matter is generated according to the transitivity listening and if a new matter is generated , Transitivity reasoning for the Resource Description Framework (RDF) ontology can be repeated.

FIG. 11 is a diagram illustrating a map task 1100 in a transitivity bit processing unit 520 according to an embodiment of the present invention. FIG. 12 is a block diagram of a transit bit rate processing unit 1100 according to an embodiment of the present invention. Lt; RTI ID = 0.0 &gt; 1200 &lt; / RTI &gt;

Concretely, "C1 rdfs: subClassOf C3" and "C2 rdfs: subClassOf C4" extracted through the reduction task 1000 are stored in the map task 1100 of FIG. 11 as a key, a value, &Lt; / RTI >

Quot; C1 rfs: subClassOf C3 "and" C1 rdfs: subClassOf C3 " for schema triples of "C1 rdfs: subClassOf C3 &quot; through the map task 1100 according to an embodiment of the present invention. C2 rdfs: subClassOf C4> "and" <C4, C2 rdfs: subClassOf C4> "for the schema triple of" C2 rdfs: subClassOf C4 " ".

Next, the transit bit listening processing unit 520 according to an embodiment of the present invention transmits the "<C1, C1 rdfs: subClassOf C2> C2 rdfs: subClassOf C3> "," <C2, C1 rdfs: subClassOf C2> "," <C2, C2 rdfs: C3 rdfs: subClassOf C4> "," <C4, C2 rdfs: subClassOf C4> ", and" <C4, C3 rdfs: subClassOf C3> "," < C4 rdfs: subClassOf C4 "and" C1 rdfs: subClassOf C4 "

The transit bit listening processing unit 520 according to an embodiment of the present invention can extract "C1 rdfs: subClassOf C4" and "C1 rdfs: subClassOf C4" in the reduction task 1200 of FIG.

FIG. 13 is a view for explaining a map task 800 in a transitivity bottleneck processing unit according to an embodiment of the present invention.

C1 rdfs: subClassOf C4 "and" C1 rdfs: subClassOf C4 "extracted through the reduction task 1200 of FIG. 12 are transmitted through the map task 800, C4 rdfs: subClassOf C4> "," <C1, C1 rdfs: subClassOf C4> ", and" <C1, C1 rdfs: subClassOf C4> ".

Next, in the same manner as in Fig. 14,

인 조건을 만족하므로 트랜지터비티 리즈닝 과정을 종료할 수 있다.The size of

The transient biting process can be terminated.

The redundant triple eliminator 530 according to an embodiment of the present invention may remove a duplicate triple in the schema triple that has been subjected to the bit-by-residue processing.

FIG. 15 illustrates a map task 1500 in a redundant triple eliminator 530 according to an embodiment of the present invention, and FIG. 16 is a diagram illustrating a reduction task 1600. Referring to FIG.

C1 rdfs: subClassOf C4 "," C2 rdfs: subClassOf C4 "," C1 rdfs: subClassOf C4 "through the map task 1500 of FIG. 15 according to an exemplary embodiment of the present invention. C1 rdfs: subClassOf C4> "," <C1, C1 rdfs: subClassOf C3> "," <C2, C2 rdfs: "&Lt; C1, C1 rdfs: subClassOf C4 &gt; ".

Next, the redundant triple elimination unit 530 according to an embodiment of the present invention includes the classified "<C1, C1 rdfs: subClassOf C3>>, <C2, C2 <C1, C1 rdfs: subClassOf C4> "among the" rdfs: subClassOf C4> "," <C1, C1 rdfs: subClassOf C4> "and" <C1, C1 rdfs:

C1 rdfs: subClassOf C4 "and" C2 rdfs: subClassOf C4 "may be output to the output directory" schema_out / ".

The inheritance recognition processing unit 540 according to an embodiment of the present invention may process inheritance reasoning for a schema triple in which the overlapping triples are removed.

The inheritance recognition processing unit 540 according to an embodiment of the present invention is configured to execute the inferential matching processing unit 540 according to the map task 1700 of FIG. C3 rdf: type C2> "," <C3, c rdf: type C3> "for the instance of" C3 ".

Also, the inheritance recognition processing unit 540 according to an exemplary embodiment of the present invention includes the map task 1700 shown in FIG. 17, and outputs the C1 rdfs: subClassOf C2, C2 rdfs: subClassOf C3, C3 rdfs: subClassOf C2> "," <C2, C2 rdfs: subClassOf C3> "and" <C3, C3 rdfs: subClassOf C4> "for the schema input of" .

Also, the inheritance recognition processing unit 540 according to an exemplary embodiment of the present invention includes C1 rdfs: subClassOf C3, C1 rdfs: subClassOf C4, C2 rdfs C1 rdfs: subClassOf C4> "," <C1, C1 rdfs: subClassOf C4> "and" <C2, C2 rdfs: subClassOf C4> "for the schema output of" .

Next, the inheritance recognition processing unit 540 according to an embodiment of the present invention performs the inference processing based on the C1, a rdf: type C1, C1, C1 rdfs: subClassOf C2, C2, C2 rdfs: subClassOf C3>, <C2, C2 rdfs: subClassOf C4>, <C1, C1 rdfs: Type C3, a rdf: type C4, b rdf: type C3, b rdf: type C2, a rdf: type C3, for <C3, c rdf: type C3>, <C3, C3 rdfs: subClassOf C4> type C4, c rdf: type C4 as an instance, and output it to the output directory of "instance_out /".

The method according to an embodiment may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions to be recorded on the medium may be those specially designed and configured for the embodiments or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. For example, it is to be understood that the techniques described may be performed in a different order than the described methods, and / or that components of the described systems, structures, devices, circuits, Lt; / RTI &gt; or equivalents, even if it is replaced or replaced.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

500: Parallel processing device of large capacity linked data
510: schema triple extraction unit
520: Transit ratio bit processing unit 130: Duplicate triple elimination unit
540: Inheritance rezoning processor

Claims (17)

A method of operating a parallel processing device,
Receiving a triple from an input file and extracting a schema triple including a key value and a value from the inputted triple;
The extracted schema triple is processed by a map task, the data processed by the map task is sorted using the key value, and the sorted data is subjected to a reduction task, ; &Lt; / RTI &gt;
The processing of the Map task and the processing of the Reduce task on the extracted schema triple are repeated to perform transitive bit listening (RDF) on the RDF (Resource Description Framework) ontology transitivity reasoning
Capacity parallel data parallel processing unit. The method according to claim 1,
Wherein processing the transitivity reasoning comprises:
Defining the transitivity reasoning as a monotone fixed-point iteration model;
Capacity parallel data parallel processing unit. The method according to claim 1,
Wherein processing the transitivity reasoning comprises:
Storing a data partition processed in a previous iteration through a redess task in a local file system; And
Assigning a next redundancy task based on the location of the stored data partition
Capacity parallel data parallel processing unit. The method of claim 3,
Wherein the step of assigning a next redundancy task based on the location of the stored data partition comprises:
Performing joins on newly entered triples except joins between triples stored in the local file system
Capacity parallel data parallel processing unit. The method of claim 3,
Wherein the step of assigning a next redundancy task based on the location of the stored data partition comprises:
Assigning the next redundancy task so that the number of total joins occurring in each iteration is reduced by the square of the number of local triples
Capacity parallel data parallel processing unit. The method according to claim 1,
Removing a duplicate triple; And
Step of processing inheritance reasoning
Further comprising the steps of: (a) The method according to claim 1,
Wherein extracting a schema triple for the input file comprises:
Setting a property for the input file to a key value
Capacity parallel data parallel processing unit. The method according to claim 1,
The step of processing transitivity reasoning for the Resource Description Framework (RDF) ontology comprises:
Confirming whether new matter is generated according to the transistor bit listening; And
As a result of the checking, if new matter is generated, repeating the transitivity reasoning for the Resource Description Framework (RDF) ontology
Capacity parallel data parallel processing unit. The method according to claim 1,
Storing a schema triple subjected to transitivity reasoning on the Resource Description Framework (RDF) ontology in an output file;
Further comprising the steps of: (a) The method according to claim 1,
Extracting a schema triple for the input file and storing an instance triple obtained by extracting a schema triple for the input file
Further comprising the steps of: (a) Storing large amount of linked data in nodes of a distributed file system;
Allocating a distributed parallel task for processing the divided linked stored data to a node of each distributed file system to perform processing on the data and storing the distributed processing results in the distributed file system; And
When a re-process is required for the distributed processing result, a redescription task necessary for re-processing is assigned to each node of each distributed file system storing the processing result, and the result of each process is transmitted to each of the distributed files Steps to save to the system
Capacity parallel data parallel processing unit. 12. The method of claim 11,
The large amount of linked data includes a schema and an instance triple,
Storing the schema and instance triples in nodes of the distributed file system; And
Constructing a distributed parallel task for inference processing of the divided schema and instance triple as a MapReduce task
Further comprising the steps of: (a) 13. The method of claim 12,
The distributed parallel inference processing is performed by assigning the map task and the reduce task for inference of the divided stored schemas and instance triples to the data nodes of the distributed file system in which the schema and the instance triples are distributed and stored, Storing it again in a distributed file system; And
If it is necessary to re-process the inference result, a mapping task necessary for reprocessing is assigned to the data nodes of the distributed file systems storing the inference results, and the respective processing results are sent back to the distributed file system Steps to save
Further comprising the steps of: (a) A schema triple extracting unit for extracting a schema triple for an input file;
A transitivity bottleneck processing unit for processing transitivity reasoning on an RDF (Resource Description Framework) ontology by repeatedly applying a mapreduce to the extracted schema triple;
A duplicate triple eliminating unit for removing a duplicate triple in the schema triple processed in the transitory bit listening process; And
An inheritance reasoning processing unit for processing inheritance reasoning for a schema triple in which the redundant triples are removed;
Capacity parallel data parallel processing unit. 15. The method of claim 14,
Wherein the transistor bit-
Wherein said transitivity reasoning is defined as a monotone fixed-point iteration model. 15. The method of claim 14,
Wherein the transistor bit-
Wherein the data partition processed in the previous iteration is stored in the local file system through the redundancy task and the next redundancy task is allocated based on the position of the stored data partition. A computer-readable recording medium having recorded thereon a program for performing the method according to any one of claims 1 to 13.

Images