CN106980901A - Streaming RDF data parallel reasoning algorithm - Google Patents

Abstract

The invention provides a parallel reasoning algorithm for streaming RDF data: construct a regular pseudo-bidirectional network, and establish an intermediate node if there is a connection variable of a class in the rule node; regularly acquire batches of new data in the Streaming data stream and data generated by previous reasoning as Input data, classify the input data or create a corresponding node and store it in the corresponding Redis cluster; for the input triplet data combined with the pseudo-bidirectional network, judge whether the antecedents monitored by the corresponding intermediate node or rule node are all satisfied, Then infer the rules to generate inference data; by deleting duplicate inference data in real time and saving all the data generated by this inference into the Redis cluster as the input data for the next inference, the OWL Horst rule of RDF data can be fully and efficiently realized Parallel streaming inference.

Description

Streaming RDF Data Parallel Reasoning Algorithm

technical field

The invention belongs to the technical field of semantic web, and specifically relates to a parallel reasoning algorithm for streaming RDF data.

Background technique

In recent years, researchers have gradually realized the importance of research on parallel reasoning algorithms for real-time streaming data, but there are still few related algorithms proposed in this field, and further research is needed. At the same time, there are quite a lot of researches on reasoning in intelligent technology, such as knowledge discovery and case reasoning. It has become a consensus in academia and industry to solve large-scale RDF streaming data related problems through distributed parallel computing.

Research on RDFS/OWL streaming parallel reasoning is a relatively new field at present. Barbieri D F et al. proposed an incremental reasoning algorithm based on streaming and rich background knowledge. This algorithm adds expiration time information to each RDF triple, and when new streaming data arrives, reasoning calculations are performed on new data. , and terminate explicit facts as well as remove invalid triples. The IDRM algorithm can perform RDFS reasoning on incremental data in an efficient and scalable manner. Since the IDRM algorithm performs special modeling on RDFS rules, the efficiency of OWL Horst rule reasoning is not high. Chevalier J et al. proposed an effective incremental reasoner (Slider), which reasoned through the intrinsic features in the semantic data stream, thus implementing a scalable batch reasoner for streaming data. However, since Slider is only designed for RDFS rules, it is not suitable for complex OWL Horst rule reasoning.

Today's challenges in large-scale RDF file reasoning are: distributed data on the network is difficult to obtain appropriate triples; the growing amount of data requires scalable computing power for large data sets; existing reasoning methods are Designed for static ontologies where data is usually changing in the real world. The existing distributed reasoning methods mainly focus on the reasoning of static data, and the research on parallel reasoning of streaming RDF data is a relatively new field at present.

Technical issues to be resolved:

1. Solve how to combine RDF data ontology and OWL Horst rules to construct a pseudo-bidirectional network of rules, which contains class nodes and rule nodes corresponding to pattern triples, so that all OWL Horst rules can be efficiently completed in the case of large-scale streaming data reasoning.

2. Combined with the proposed streaming scheme, a corresponding parallel reasoning scheme is proposed, so as to meet the needs of distributed parallel reasoning for large-scale streaming data.

Contents of the invention

In order to solve the above problems, the present invention provides a parallel reasoning algorithm for streaming RDF data. Aiming at the OWL Horst rule and combining the advantages of the HAL algorithm, the PRAS algorithm (Parallel Reasoning Algorithm for Streaming RDF Data) is proposed. In the case of large-scale streaming data, the algorithm can efficiently construct and maintain a pseudo-bidirectional network, and perform inference correctly and completely.

To achieve the above object, the present invention adopts the following technical solutions: a streaming RDF data parallel reasoning algorithm, characterized in that it comprises the following steps: S1: load rule nodes and pattern triples P _j _RDD and _Ok _RDD and save to Redis cluster, build the middle node midnode of the connection variable in the rule, skip to S2; S2: regularly read the batch new data new_data in the data stream and the data itr_data generated by the previous reasoning; if it is a pattern triplet (S _i , P _i , O _i ), then skip to S3; if it is an instance triplet (s _i , p _i , o _i ), then skip to S5; if new_data is empty and itr_data is empty, then the algorithm ends; S3: If the corresponding class node P _j _RDD or _Ok _RDD exists, classify it into the corresponding class node; if it does not exist, create a corresponding class node and save it to the Redis cluster; if its predicate belongs to Symmetric Property, then Skip to S4; otherwise, skip to S6; Symmetric Property is a set used to identify that the predicates in the pattern triples have a symmetrical relationship. SymTriples, a collection of symmetric attribute triples, is defined as follows: ; Among them, P _j _RDD is a set of pattern triples; S4: classify and reason the input data; S5: store and deduplicate the triples generated by reasoning.

Compared with the prior art, the present invention has the following advantages:

1. Combining OWL Horst rules and RDF ontology files to construct a pseudo-bidirectional network structure, which improves the efficiency of stream reasoning.

2. Combining with the storage strategy designed by Redis cluster, triples are deduplicated and iterative data is stored, which reduces the storage space and reasoning time of repeated triples, thereby improving the efficiency of reasoning.

Description of drawings

Fig. 1 is a general framework schematic diagram of the present invention.

Figure 2 is a construction diagram of a pseudo-bidirectional network.

Figure 3 loads rule and ontology data and constructs a pseudo-bidirectional network.

Figure 4 is a relationship diagram of OWL Horst rules.

detailed description

The present invention will be further explained below in conjunction with the accompanying drawings and specific embodiments.

The streaming parallel reasoning proposed by the present invention is mainly divided into three parts: constructing a pseudo-bidirectional network, classifying streaming data and reasoning of OWLHorst rules. According to the characteristics of Spark Streaming and Redis, combined with the HAL algorithm, OWLHorst rules and RDF data ontology, a regular pseudo-bidirectional network is constructed, which contains class nodes and rule nodes corresponding to pattern triples. If there are class connection variables in the rule nodes, then Establish an intermediate node; then, regularly obtain batches of new data in the Streaming data stream and the data generated by the previous reasoning as input data, classify the input data or create a new corresponding node and store it in the corresponding Redis cluster; then, for the input The triplet data combined with the pseudo-two-way network judges whether the antecedents monitored by the corresponding intermediate node or rule node are all satisfied, and then infers the rule to generate inference data. Finally, by deleting duplicate inference data in real time and saving all data generated by this inference into the Redis cluster as the input data for the next inference, the parallel streaming inference of RDF data OWL Horst rules can be realized completely and efficiently.

See Figure 1 for the overall frame diagram.

A streaming RDF data parallel reasoning algorithm, which comprises the following steps:

S1: Load the rule node and pattern triplets P _j _RDD and _Ok _RDD and save them to the Redis cluster, build the intermediate node midnode that connects the variables in the rule, and skip to S2;

S2: Regularly read the batch new data new_data in the data stream and the data itr_data generated by the previous reasoning; if it is a pattern triplet (S _i , P _i , O _i ), skip to S3; if it is instance 3 tuple (s _i , p _i , o _i ), skip to S5; if new_data is empty and itr_data is empty, the algorithm ends;

S3: If the corresponding class node P _j _RDD or _Ok _RDD exists, classify it into the corresponding class node; if it does not exist, create a corresponding class node and save it to the Redis cluster; if its predicate belongs to Symmetric Property , then skip to S4; otherwise, skip to S6; Symmetric Property is a set used to identify that the predicates in the pattern triples have a symmetrical relationship. SymTriples, a collection of symmetric attribute triples, is defined as follows:

;

Among them, P _j _RDD is a set of pattern triples; for example, SymTriples={sameAs, inverseOf, equivalentClass, equivalentProperty} in OWL Horst rules;

S4: Classify and reason the input data;

S5: Store and deduplicate the triples generated by reasoning.

Wherein S4 includes the following steps: S41: If the input triplet data is a pattern triplet (S _i , P _i , O _i ), then the input triplet data is represented by P _i +”_”+S _i is the key, O _i is the value, P _i + "_"+O _i is the key, S _i is the value, construct the bidirectional relationship between S _i and O _i in the triplet, and save it to the Redis cluster, skip to S43;

S42: If the input triplet data is an instance triplet (s _i , p _i , o _i ), construct <s _i ,(p _i ,o _i )>, < p _i from the input triplet data , (s _i , o _i )> and < o _i , (s _i , p _i )> three key-value pairs, and save them in the Redis cluster, skip to S43;

S43: Check the pseudo-two-way network corresponding to new_data or itr_data, and determine whether new_data or itr_data contains the Rule _{m _link_RDD monitored by the regular node or the intermediate node. If it is the Rule m _link_RDD} monitored by the intermediate node, skip to S44. If it is the Rule monitored by the regular node _m _link_RDD skips to S45, otherwise skips to S2; Pseudo-two-way network refers to establishing a rule node Rule _i _node for a certain rule Rule _i , the class involved in the rule builds a class node Class _i _node, if the precondition of the rule contains The connection variable establishes the intermediate node mid _i _node; the connection variable of Rule _i refers to the pattern triplet item used to connect the two preceding items in Rule _i , and the connection variable information of each rule is represented by <key,value> The form is stored in Rule _{m_link_RDD} , where the key stores all the pattern triplet items used for the connection of the former part of the rule, and the value stores the pattern triplet items in the conclusion part of the rule; the construction process of the pseudo-bidirectional network is shown in Figure 2.

S45: Judging whether all monitored Rule _{m_link_RDDs} are satisfied, if so, skip to S46, otherwise, skip to S2;

S46: Determine whether all the antecedents corresponding to the rule node are satisfied, if so, execute the inference of the rule to generate a triple, and skip to S5; otherwise, skip to S2.

S5 includes the following specific steps: for the triples generated by reasoning, save them in a set named itr_data in the Redis cluster, and perform deduplication operations on repeated triples, and then use the itr_data set as part of the next reasoning input data, Jump to S2 if there is no command to stop.

According to the characteristics of Spark RDD and Redis cluster, the PRAS algorithm of the present invention combines the principle of HAL algorithm and OWLHorst rules and RDF ontology data, and adopts the pseudo-bidirectional network of rules to construct. First, for the pattern triplet (S _i , P _j , Ok) Construct the corresponding class node O _k _RDD or P _{j _RDD} and save it to the Redis cluster. If P belongs to the symmetric attribute, then build a bidirectional relationship between S and O in the triple and save it to the Redis cluster. In order to quickly judge whether all the antecedents in the rules are satisfied, establish corresponding rule nodes for all rules. If the rule contains the connection variable link_var, then create an intermediate node midnode, save the test condition information in the intermediate node and set the intermediate node and the rule node Two-way communication between; if there is no connection variable, the class node is directly connected to the rule node, and the test conditions are stored in the class node. Taking rule 8a in Figure 2 as an example, the schematic diagram is shown in Figure 3 . Through the heuristic information between nodes and the construction of symmetric attributes, combined with the efficient access of the Redis cluster, the required triples are read from the Redis cluster in the form of queries, reducing the reading and transmission of irrelevant triples , thereby improving the overall inference efficiency.

The Map stage mainly completes data classification and reasoning: if the batch stream data new_data in the Streaming data stream or the data itr_data generated by the previous reasoning are regularly obtained as ontology data, it will be classified into the corresponding class node and the Redis cluster will be updated. The value corresponding to the node; if its attribute is a symmetric attribute, then use "symm_"+S and "symm_"+O as keys respectively to construct a bidirectional relationship between S and O in the triplet and save it in the Redis cluster. If new_data or itr_data is instance data, then for the instance triplet (s _i , p _i , o _i ), construct < s _i , (p _i , o _i )>, < p _i , (s _i , o _i ) > and <o _i , (s _i ,p _i )> three key-value pairs, and save them in the Redis cluster. Then check the pseudo-two-way network corresponding to new_data or itr_data, and judge whether all the antecedents (may include multiple intermediate nodes) corresponding to the connection variables monitored by the intermediate node corresponding to new_data or itr_data (may contain multiple intermediate nodes) are all satisfied, and if so, execute the rule Reasoning generates triples and outputs the result to the Reduce stage; if it is partially satisfied, the state of the corresponding condition is modified. The specific steps of the data classification and reasoning algorithm proposed in this paper are as follows:

Map phase algorithm

Input streaming triplet data and triplets generated by previous inference

output<"new", >

Step1 For the input triplet data, ∀(S _i , P _j , O _k )∈SchemaTriple is classified into the corresponding class node and the Redis cluster is updated; if P _j is a symmetric attribute, P _j +” _”+ S _i is the key, O is the value and P+”_”+O is the key, S is the value, construct the bidirectional relationship between S and O in the triplet, and save it in the Redis cluster. Skip to Step3.

Step2 For the input triplet data, ∀(s _i ,p _j ,ok ) _{∈InstanceTriple} , then construct < s _i , (p _j ,ok _k ) for the instance triplet (s _i ,p _j , _ok ) ) >, < p _j , (s _i , _ok ) > and < _{ok , (s i ,} p _j ) > three key-value pairs are stored in the Redis cluster. Skip to Step3.

Step3 Check the pseudo-bidirectional network corresponding to (s _i , p _j , _ok ), read the required data from the Redis cluster, and determine the connection variable monitored by the intermediate node corresponding to (s _i , p _j , _ok ) Or whether all the antecedents corresponding to the rule node (may include multiple intermediate nodes) are all satisfied, if all are satisfied, skip to Step4. If some of them are not satisfied, modify the monitoring information of intermediate nodes or class nodes in conjunction with (S _i , P _j , _Ok ).

Step4 According to the conclusion of the current rule, get the triples generated by reasoning and output <"new", >.

Taking rules 8a and 8b (inverseOf) in Figure 4 as an example, the pseudocode is described as follows:

Input: (S ₁ , P ₁ , O ₁ )

Output: <"new", >

Begin

If (S ₁ , P ₁ , O ₁ ) ∈SchemaTriple //This triple is a schema triple, which is classified and saved

{

If P1 equal “type”

sadd O ₁ S ₁

else {

sadd P ₁ (S ₁ ,O ₁ )

If P ₁ ∈ SymmetriProperty { /*The predicate is a symmetric property, which is to construct a symmetric relationship between the subject S ₁ and O ₁ */

sadd P ₁ +”_”+S ₁ O ₁

sadd P ₁ +”_”+O ₁ S ₁

}

}

} else { /*Construct three key-value pairs for instance triples*/

sadd S ₁ (P ₁ ,O ₁ )

sadd P ₁ (S ₁ ,O ₁ )

sadd O ₁ (S ₁ ,P ₁ )

}

/* Read the collection of inverseOf_S ₁ and inverseOf_O ₁ in the Redis cluster to inverseOf*/

inverseOf smembers ("inverseOf_"+S ₁ )

∪smembers ("inverseOf_"+O ₁ )

If(inverseOf != null){

yield(“new”,( O ₁ ,P ₁ , S ₁ ))

For (inverse in inverseOf. value){

yield(“new”,( O ₁ , inverse, S ₁ ))

}

}

end

Assume that the current input batch stream data contains a pattern triplet T(memberOf, owl:inverseOf, member) and an instance triplet t(GraduateStudent0, memberOf, University0_Department0). First, for the pattern triplet T, judge whether inverseOf_RDD exists, if not, create a new inverseOf_RDD and save (memberOf, member) to inverseOf_RDD; if it exists, directly save it to inverseOf_RDD. Next, since inverseOf is a symmetric attribute, use inverseOf_memberOf as key, member as value and inverseOf_member as key, memberOf as value to build a bidirectional relationship between memberOf and member and save it in the Redis cluster. For instance triplet t, construct <GraduateStudent0, (memberOf, University0_Department0)>, <memberOf, (GraduateStudent0, University0_Department0)>, <University0_Department0, (GraduateStudent0, memberOf )> and save to Redis cluster. Finally, read the collection of inverseOf_ memberOf and inverseOf_ member in the Redis cluster to inverseOf, traverse inverseOf and output (GraduateStudent0, member, University0_Department0).

Similar to rules 8a and 8b that contain symmetric attributes, through the two-way relationship between the construction and storage of symmetric attributes, the relevant triples can be quickly found in the Redis cluster, thereby improving the efficiency of reasoning.

Taking rule 15 (someValuesFrom) in Figure 4 as an example, the pseudocode is described as follows:

Input: (S ₁ , P ₁ , O ₁ )

Output: <"new", >

Begin

If (S ₁ , P ₁ , O ₁ ) ∈SchemaTriple {//The triplet is a schema triplet, which is classified and saved

If P ₁ equal “type”

sadd O ₁ S ₁

else {

sadd P ₁ (S ₁ ,O ₁ )

If P ₁ ∈ SymmetriProperty {//The predicate is a symmetric property, which is to construct a symmetric relationship between the subject S1 and O1

sadd P ₁ +”_”+S ₁ O ₁

sadd P ₁ +”_”+O ₁ S ₁

}

}

} else { //construct three key-value pairs for instance triplet

sadd S ₁ (P ₁ ,O ₁ )

sadd P ₁ (S ₁ ,O ₁ )

sadd O ₁ (S ₁ ,P ₁ )

}

someValuesFrom_set smembers("someValuesFrom") /*Read the collection of someValuesFrom in the Redis cluster*/

onProperty_set smembers("onProperty")

For (svf in someValuesFrom_set) {

For (op in onProperty_set) {

If(svf.v equals op.v){

temp_w smembers (svf.w)

x_type_w= temp_w.filter(x => x.p==”type”) /* Filter out the triples in temp_w where p is type*/

u_p_x smembers (op.p)

result = u_p_x. filter(t=>

t.x==x_type_w.x

yield("new",(t.u,type,svf.v))) /*connect the antecedents in the rules to generate inference results*/

}

}

}

end

Assume that the currently input batch stream data contains pattern triplets T1(Chair, owl:someValuesFrom,Department), T2(Chair, owl:onProperty, headOf) and instance triplets t1(FullProfessor7,headOf, University0_Department0), t2( University0_Department0, rdf:type, Departmentmment). First, for the pattern triplets T1 and T2, determine whether someValuesFrom_RDD and onProperty_RDD exist. If not, create someValuesFrom_RDD and onProperty_RDD and save (Chair, Department) to someValuesFrom_RDD and save (Chair, headOf) to onProperty_RDD respectively; if they exist, directly Save to someValuesFrom_RDD or onProperty_RDD. For instance triplet t1, construct <FullProfessor7, (headOf, University0_Department0)>, < headOf, (FullProfessor7, University0_Department0)>, <University0_Department0 , (FullProfessor7, headOf)> and save it to the Redis cluster, and t2 is similar to the above operation. Next, read the collections of someValuesFrom and onProperty in the Redis cluster to someValuesFrom_set and onProperty_set respectively, and traverse someValuesFrom_set and onProperty_set. At this time, the Chair in someValuesFrom_set is the same as the Chair in onProperty_set, and then use the Department as the key and headOf as the key to obtain the Redis cluster. Two collections; finally connect FullProfessor7 with Chair and output (FullProfessor7, rdf:type, Chair).

Similar to the rule 15 of multi-connection variables, the mode triplet can be quickly obtained from the Redis cluster through the key of the class node; for the associated instance triplet, use the storage strategy of the instance triplet in Redis to pass the connection variable The value of σ finds out relevant instance triplets, thus improving the inference efficiency.

The Reduce stage mainly saves the data generated by reasoning. For the triplets generated by inference, save them in a collection named "itr_data" in the Redis cluster, and perform deduplication operations on repeated triplets, and then use the "itr_data" collection as part of the next inference input data. The specific steps of the data deduplication and storage algorithm proposed in this paper are as follows:

Reduce Algorithm

Input <"new", Iterator<String> values>

output null

Step1. Save the input SchemaTriple and InstanceTriple with itr_data as the collection name in the Redis cluster for reading in the next inference.

In order to clarify the deduplication and storage of input data in the Reduce phase, the pseudocode is described as follows:

Input: <”new”, Iterator <String> values>

Output: null

Begin:

del itr_data

itr for each value

sadd itr_data itr.value /* traverse the values in values and add them to the itr_data collection of Redis cluster*/

end

From the above pseudo code, it can be obtained that in the Reduce stage, the input triples are deduplicated and stored through the Redis collection, so as to prepare the data for the next reasoning.

Algorithm complexity analysis is an important index to measure the efficiency of an algorithm, and the complexity analysis of the PRAS algorithm of the present invention is different from the centralized algorithm. When analyzing the complexity of the PRAS algorithm, it can be decomposed into two stages of Map and Reduce for algorithm complexity analysis. Assuming that the experimental data set contains N triples, the time to read Redis data is set to t, and the parallel number of Map tasks in the MapReduce process is set to k, the number of instance triples received in the Reduce phase is set to m, and the The number of parallelism is set to x. Since the PRAS algorithm scans each input triplet in the Map stage, combined with class nodes or intermediate nodes, it can judge whether the triplet can participate in certain rule reasoning. If it can participate in subsequent rule reasoning, it can be read by reading The antecedent data in Redis are then inferred to obtain inference results. Therefore, the time complexity of the Map phase is: O(t*N/k). In the Reduce stage, each input triplet is classified, therefore, the time complexity of the Reduce stage is: O(m/x).

The above are the preferred embodiments of the present invention, and all changes made according to the technical solution of the present invention, when the functional effect produced does not exceed the scope of the technical solution of the present invention, all belong to the protection scope of the present invention.

Claims (3)

1. a kind of streaming RDF data parallel reasoning algorithm, it is characterised in that comprise the following steps：

S1：Loading rule node and pattern triple P_j_ RDD and O_k_ RDD is simultaneously saved in Redis clusters, builds in rule and connects The intermediate node midnode of variable, skips to S2；

S2：The data itr_data that batch new data new_data and previous reasoning in timing reading data flow are produced；If its For pattern triple (S_i,P_i,O_i), then skip to S3；If it is example triple (s_i,p_i,o_i), then skip to S5；If new_data It is empty for empty and itr_data, then algorithm terminates；

S3：If its corresponding class node P_j_ RDD or O_k_ RDD is present, then is referred to corresponding class node；If being not present, Newly-built corresponding class node is simultaneously saved in Redis clusters；If its predicate belongs to Symmetric Property, S4 is skipped to；It is no Then skip to S6；Symmetric Property are the set for predicate in markers triple with symmetric relation.Symmetrically Attribute triplet sets SymTriples is defined as follows：

；

Wherein, P_j_ RDD is pattern triplet sets；

S4：Data to input are sorted out and reasoning；

S5：Stored and duplicate removal for the triple that reasoning is produced.

2. a kind of streaming RDF data parallel reasoning algorithm according to claim 1, it is characterised in that：S4 includes following step Suddenly：S41：If the triple data of input are pattern triple (S_i,P_i,O_i), then by the triple data of input respectively with P_i +”_”+S_iFor key, O_iFor value and P_i+”_”+O_iFor key, S_iFor value, S in triple is built_iAnd O_iBidirectional relationship, And Redis clusters are saved in, skip to S43；

S42：If the triple data of input are example triple (s_i,p_i,o_i), then the triple data of input are built< s_i,(p_i,o_i)>、< p_i , (s_i, o_i)>With< o_i , (s_i,p_i)>Three key-value pairs, and Redis clusters are stored in, skip to S43；

S43：The pseudo- bilateral network corresponding to new_data or itr_data is checked, and whether judges new_data or itr_data The Rule monitored comprising regular node or intermediate node_m_ link_RDD, if the Rule that intermediate node is monitored_m_ link_RDD is then S44 is skipped to, if the Rule that regular node is monitored_m_ link_RDD then skips to S45, otherwise skips to S2；Pseudo- bilateral network is referred to To certain rule Rule_iSet up regular node Rule_iThe class being related in _ node, rule builds class node Class_i_ node, such as In the regular former piece of fruit intermediate node mid is then set up comprising link variable_i_node；Regular Rule_iLink variable refer to Rule_i In be used for the pattern triple that connects two former pieces, by the link variable information of each rule with<key,value>Shape Formula is stored in Rule_m_ link_RDD, wherein key store all pattern triples connected for former piece of the rule, value Store the pattern triple of the rule conclusion part；

S45：Judge the Rule monitored_mWhether _ link_RDD all meets, if then skipping to S46, otherwise skips to S2；

S46：Whether the corresponding all former pieces of judgment rule node all meet, if then the reasoning of executing rule produces ternary Group, skips to S5；Otherwise S2 is skipped to.

3. a kind of streaming RDF data parallel reasoning algorithm according to claim 1, it is characterised in that：S5 includes following tool Body step：The triple produced for reasoning, is stored in the set of entitled itr_data in Redis clusters, and to repetition Triple carries out deduplication operation, then itr_data is gathered to the part as next reasoning input data, if do not stopped Order only then skips to S2.