CN109885579B

CN109885579B - Hierarchical counting-based Datalog parallel increment maintaining method

Info

Publication number: CN109885579B
Application number: CN201910027398.0A
Authority: CN
Inventors: 秦小麟; 项哲慧; 高雅
Original assignee: Nanjing University of Aeronautics and Astronautics
Current assignee: Nanjing University of Aeronautics and Astronautics
Priority date: 2019-01-11
Filing date: 2019-01-11
Publication date: 2023-06-06
Anticipated expiration: 2039-01-11
Also published as: CN109885579A

Abstract

The invention provides a hierarchical counting-based datalog parallel increment maintaining method, and belongs to the field of knowledge bases. The invention firstly carries out layering on the instantiation process of the datalog, maps the iterative datalog calculation process into a designed hierarchical structure, and counts each deduced fact by combining a counting method in the hierarchical structure. The method converts recursive datalog calculation into a plurality of non-recursive calculation in series, thereby not only solving the defect that the counting method cannot be applied to the recursive program, but also retaining the efficient characteristic of the counting method. The increment triggered by the epitaxial database update is maintained, and each hierarchy is updated in turn in a counting process. On the basis, the invention further introduces an epitaxial data state sequence, and provides access to different versions of the external database for parallel operation by using a small amount of space. The method can process a plurality of updating operations at the same time, and greatly improves the instantiation efficiency of the epitaxial database which is frequently and dynamically changed.

Description

Hierarchical counting-based Datalog parallel increment maintaining method

Technical Field

The invention discloses a hierarchical counting-based data log (declarative logic programming language) parallel increment maintaining method, and belongs to the technical field of knowledge bases.

Background

Datalog is a declarative language similar to SQL (structured query language) language in that a developer only needs to care what needs to be done and not what. Such language may allow the developer to focus more on the task itself, regardless of how it is implemented. Also, because Datalog has more recursive semantics than SQL, such languages are widely used for data analysis applications such as web ranking, social network computing, and semantic web analysis. To improve the efficiency of datalog queries, pre-instantiating an inlier database is a common technique given the inlier database and the datalog rule program. Meanwhile, the updating of the epitaxial database can influence the consistency of the epitaxial database and the internal epitaxial database, and the re-instantiation cost is quite high, so that the existing method adopts an increment maintenance mode to update the locally updated epitaxial data on the basis of the original instantiated internal epitaxial database.

Existing incremental maintenance methods are all single threaded and can only handle one update operation at a time, and the update process depends on the integrity of the deferred database before the update operation. Many cases encounter data updates that are centralized, either without updates or with multiple updates for a short period of time, such as continuously adding or deleting data that meets certain criteria. Most algorithms at present focus on accelerating one-time updating operation, for example, the B/F algorithm reduces the condition of excessive deletion by early backward query, the B/F-C algorithm and the DRed-C algorithm replace the costly reverse deduction process involved in the backward query stage of the original algorithm by a counting method, and the algorithms all have a default internal extension database integrity premise, under the premise, the algorithms cannot process a plurality of updates simultaneously theoretically. There is therefore a need for a method that can support concurrent update operations that speeds up and provides a multi-version epitaxial database that supports concurrent update processing.

Disclosure of Invention

In order to overcome the defect that the traditional increment maintaining method cannot be parallel when frequent updating operation is processed, the invention provides a hierarchical counting-based datalog parallel increment maintaining method, which can correctly update an internal delay database based on the updating information of the external delay database on the basis of an incomplete internal delay database, so that the updating operation can be processed in parallel, and further the efficiency of maintaining the increment of the internal delay database is improved.

The invention adopts the following technical scheme for solving the technical problems:

a method for maintaining a datalog parallel increment based on hierarchical counting comprises the following steps:

step one, in the process of instantiating an internal delay database, dynamically constructing a counting structure of a corresponding level according to the iteration times of a calculation process, counting internal delay data derived in the iteration process, and recording in the counting structure of the corresponding level; starting a thread process every time of updating, calculating a constructed counting structure layer by the thread, wherein each layer of counting structure can only be processed by one thread at a time, and the multi-layer structure is processed by a plurality of threads simultaneously;

step two, when calculating the counting structure of each layer, increasing and decreasing the data sets of the internal data olds and the news of the structure according to the external data add_i and Del_i, and dividing the data sets of the olds and the news according to specific data states; add and delete del data in the counting structure, and respectively judge and migrate the data of the olds and news parts in the counting structure to finish the updating maintenance operation of the counting structure;

thirdly, deducing the increment of the news data set when calculating the counting structure of each layer; deducing the increment part of the divided news data set according to a semi-naive algorithm, and taking the deduction result as increment data of a lower layer counting structure;

and step four, when the multithreading accesses the multi-version database, constructing a multi-version database structure and accessing the multi-version database, and designing an epitaxial database with a state sequence to replace direct copying of data.

In the first step, the process of dynamically constructing the hierarchical counting structure and simultaneously processing the multi-layer structure by a plurality of threads includes the following processes:

when the epitaxial database is updated, a thread is started to process the counting structure layer by layer; each layer of counting structure has a mutual exclusion lock, when a thread needs to process the counting structure, the lock of the structure needs to be acquired first, and before the thread releases the lock of the current counting structure, the thread needs to acquire the lock of the next layer of technical structure first.

In the second step, when the counting structure of each layer is calculated, the increase and decrease adjustment is performed on the data sets of the internal data olds and the news according to the external data add_i and del_i, and the data sets of olds and news are segmented according to the specific data states, which comprises the following steps:

(1) Add_i to News or Olds: acquiring the state S (d) of the data d in the Add_i from an internal database, if S (d) > i, attempting to update the state of d, so that S (d) =i, if updating is successful, adding the data d to News, if updating is failed, continuing the attempt until the updating is successful, or until the condition S (d) > i is no longer satisfied, if the latter is the case, returning the updating failure, and putting the data d into Olds; the data added with News in the process needs to be cached for subsequent deduction;

(2) Del_i to News or Olds: deleting d from News if the data d in Del_i belongs to News, and deleting d from Olds if the data d belongs to Olds; if the data d does not exist after the d is deleted in the News, the data d needs to be cached for subsequent deduction;

(3) News to Olds: judging the data d in News, acquiring the state S (d) of d from the internal delay database, if S (d) < i, indicating that the data d is obtained by deduction before the ith iteration after the previous update operation, wherein the data d is not the data deduced for the first time in the ith iteration any more, and the data d needs to be moved to Olds and cached for subsequent deduction;

(4) Olds to News: judging the data d in the Olds, acquiring the state S (d) of d from the inner delay database, if S (d) > i, indicating that the data d obtained by the previous updating operation in a certain operation before the ith iteration cannot be obtained by the deleting operation, and at this time, the data d obtained by the ith iteration becomes the first derivation of d, so that the data d needs to be moved from Olds to News and cached for subsequent derivation processing.

Step three, deducing the increment of the news data set when calculating the counting structure of each layer; the specific procedure for deriving the delta portion of the segmented news dataset according to the semi-naive algorithm is as follows:

for deleted data and newly added data in News, the results derived from the data rule of the difference data are respectively put into Del_i+1 and Add_i+1; wherein the newly added data in News are from Add_i and Olds, del_i and the migrated data of News to Olds are deleted data in News; the thread needs to carry Add_i+1 and Del_i+1 to enter an i+1 layer counting structure, and News and Olds in the i+1 layer technical structure are subjected to increasing and decreasing treatment by the Add_i+1 and Del_i+1; if the lower counting structure is not constructed, an empty counting structure needs to be constructed first, and then the processing data is entered.

The method for constructing the multi-version epitaxial database and accessing the multi-version database in the fourth step comprises the following steps:

determining whether data d exists at time T by replacing a copy of the data with a sequence of states, where N is the size of the sequence of states of data d, at the time complexity of log N; searching for a state smaller than or equal to the time T by two parts, if the state U (d) is true, the data d exists, otherwise, the data does not exist; the size of each sequence of data states is much smaller than the total number of updates and as updates accumulate, only the state at the time of the unprocessed, but each data must retain at least one state.

The beneficial effects of the invention are as follows:

(1) Processing multiple update operations simultaneously by multiple threads can significantly speed up incremental maintenance efficiency of the deferred database under frequent updates.

(2) Simultaneous processing of multiple update operations may, in some cases, cause the processing of a single update to end early. For example, in a common revocation situation, the data d is newly added by one update, but immediately after the revocation, the operation of deleting d is processed at the same time, and the new operation can indirectly acquire the information of the revocation operation by internally extending the update of the data state, so that the update operation is finished early. And on the basis of concurrency, the updating efficiency is further improved.

(3) The designed epitaxial database with the state sequence replaces high-proportion data copy with low-proportion state marks, so that the storage cost of the multi-version epitaxial database is greatly reduced, and the influence of the memory size on the algorithm is reduced.

Drawings

FIG. 1 is a flow chart of a hierarchical count based method for maintaining parallel increments of datalog.

FIG. 2 is a diagram illustrating the dynamic construction of a counting structure and internal data.

FIG. 3 is a flow diagram of an extended database weak consistency maintenance data.

FIG. 4 is an extensive database structure diagram and version access flow chart of a strip sequence.

Detailed Description

The invention will be described in further detail with reference to the accompanying drawings.

In order to solve the problem of efficiency of incremental maintenance of an deferred database in the case of multiple updates, the present invention provides a parallel incremental maintenance algorithm, which simultaneously processes multiple update operations, and the overall flow is shown in fig. 1.

The present invention is described in further detail below with reference to the accompanying drawings.

Implement one

As shown in FIG. 1, the hierarchical count structure is constructed by instantiating the epitaxial database in dependence upon the datalog rules, and has a layer count equal to the number of iterations of the instantiation process at the epitaxial database. The data quantity of the epitaxial database is 0 in the initial state, the layer number of the corresponding counting structure is 0, and the counting structure of the corresponding iteration times is dynamically constructed in the processing process of updating operation along with the increase and decrease of the data in the epitaxial database. The main flow of data update is as follows:

(1) Starting a thread: when the epitaxial database is updated, a thread is started to process the counting structure layer by layer from the layer 1 counting structure.

(2) Acquisition lock: each layer of counting structure has a mutual exclusion lock, when a thread needs to process the counting structure, the lock of the structure needs to be acquired first, so as to prevent a plurality of threads from simultaneously processing the same counting structure and making mistakes.

(3) Releasing the lock: before the thread releases the lock of the current counting structure, the lock of the next technical structure is required to be acquired, so that the situation that the update sequence is disordered when the thread for processing the subsequent update operation exceeds the thread for processing the current update operation can be avoided, as shown in fig. 1, the k thread acquires the j+1 layer lock before releasing the j layer lock, and the k+1 thread can acquire the j layer lock after the k thread acquires the j+1 layer lock and releases the j layer lock, so that the k+1 thread is always on the upper layer of the k thread, and the problem that the update sequence is disordered due to competition when the multithreading processes a plurality of update operations is avoided.

(4) Ending the thread: if the current count structure is processed and no new or deleted data is generated for the lower layer and the lower layer structure is empty, then only the lock for the current count structure may be released and the thread may be ended.

Implement two

The internal data structure of each layer of counting structure is shown in fig. 2, taking the i-layer counting structure as an example, wherein the processed data can be divided into two parts: internal data and external data. The internal data consists of two parts, olds and News. The external data consists of two parts add_i and del_i. The external data is carried by the thread into the i-layer counting structure, and the internal data is private in the counting structure. These 4 data structures store the following data:

olds: the data derived in the ith iteration, which has been derived prior to the ith iteration, is stored.

News: the data derived in the ith iteration is stored, the data being derived for the first time in the ith iteration.

Add_i: and storing the newly added data of the lower layer caused by the change of the data in the i-1 layer counting structure.

Del_i: the deleted data of the lower layer caused by the change of the data in the i-1 layer technical structure is stored.

In the process of processing the i-layer counting structure, the data deleted in News is put into Del_i+1 after being deduced by a datalog rule, and the data newly added in News is put into Add_i+1 after being deduced. Where the newly added data in News comes from Add_i and Olds, del_i and the migration of data from News to Olds causes the deletion of data in News. While the modification of data in Olds by add_i and del_i does not cause the addition or deletion of underlying data. The two sets of add_i+1 and Del_i+1 obtained through processing are empty, so that processing on lower layer data is not needed, and threads can be ended; otherwise, the lower layer data needs to be processed, the thread needs to carry add_i+1 and del_i+1 to enter the i+1 layer counting structure, and at this time, if the lower layer counting structure is not yet constructed, an empty counting structure needs to be constructed first, and then the processing data needs to be entered.

Implementation three

The data flow process between the four data structures mentioned above, as shown in fig. 3, can be divided into two main steps: data flow process and derivation process. The data flow can be divided into the following steps:

(1) Add_i to News or Olds: the state S (d) of the data d in add_i is obtained from the in-range database, and if S (d) > i, the state of d is attempted to be updated, so that S (d) =i, if the update is successful, the data d is added to News, if the update is failed, the attempt is continued until the update is successful, or the condition S (d) > i is no longer satisfied, if the latter is the case, the update is returned to be failed, and the data d is put into Olds. Adding News data in this process may cause an increase in the lower layer data, requiring buffering for subsequent deduction.

(2) Del_i to News or Olds: the data d in Del_i is deleted from News if it belongs to News, and d is deleted from Olds if it belongs to Olds. Since the number of d in News may be more than 1, if there is no more data d after d is deleted in News, this may cause deletion of lower layer data, requiring buffering for subsequent derivation.

(3) News to Olds: judging the data d in News, acquiring the state S (d) of d from the internal delay database, if S (d) < i, indicating that the data d can be deduced before the ith iteration after the previous update operation, wherein the data d is no longer the data deduced for the first time in the ith iteration, and the data d needs to be moved into Olds. This data flow may cause deletion of lower layer data, requiring buffering for subsequent derivation.

(4) Olds to News: judging the data d in the Olds, acquiring the state S (d) of d from the inner delay database, if S (d) > i, indicating that the data d obtained by the previous update operation in a certain operation before the ith iteration may not be obtained by the deletion operation, and the data d obtained by the ith iteration may become the first derivation of d, so the data d needs to be moved from Olds to News. This data flow may cause an increase in the lower layer data, requiring buffering for subsequent derivation.

The derivation process of the data buffered after the above 4 data flows will now be described:

(1) The cached data are divided into two types according to the caching conditions: data causing deletion of the lower layer and data causing addition of the lower layer.

(2) And (3) respectively substituting the two data obtained in the step (1) into a datalog rule to carry out deduction, generating corresponding lower-layer data, deleting the data deduction result and putting the data deduction result into Del_i+1, and putting the new data deduction result into Add_i+1.

(3) Carrying Del_i+1 and Add_i+1 by the thread to enter an i+1 layer counting structure for data processing.

Implement four

The invention designs a multi-version epitaxial database with low storage cost, as shown in figure 4, wherein one part of data in the deduction process is from the internal epitaxial data cached in the data flow process, the other part of data is from the epitaxial database, threads for processing different updating operations are accessed in different versions. Each epitaxial data di maintains a state sequence, and records the modified process of the data di: s (ti: j, u) indicates that the data di is updated at the time ti: j, that di is added to the epitaxial database when u=true/T, and that di is deleted from the epitaxial database when u=false/F.

Because the epitaxial data is dissimilar, it is meaningless to delete non-existing data or add existing data, the state sequence of each data must begin with an added state, the added and deleted states alternate, and the time instants of the state sequences are strictly monotonically increasing. As in fig. 4, the state sequence of di is a state sequence in which the values of the S state sequence u are TFT sequentially, the adding and deleting states occur alternately, and the state moments in the sequence are 2,3,5 sequentially and are strictly monotonically increasing, so that such a state sequence is an effective state sequence. In such a state sequence, it is determined whether the data di exists in a certain version, the time complexity of which is log N, where N is the state sequence size of the data di, n=3 in the figure, and the process of determining the existence of di is shown in the flow chart on the right side of fig. 4. Now assume that it is determined whether data di exists in version 4.

(1) Determining the position of di data through indexes, and acquiring a state sequence of di;

(2) Finding states at the moment less than or equal to 4 through binary search, namely states S (3, F);

(3) The resulting u value for state S is F indicating that data di is not present in the extended database of version 4.

Claims

1. The method for maintaining the datalog parallel increment based on the hierarchical counting is characterized by comprising the following steps of:

2. The method for maintaining the parallel increment of datalog based on hierarchical counting according to claim 1, wherein the dynamically constructing the hierarchical counting structure in the step one and the process that the multi-layer structure is processed by a plurality of threads simultaneously includes the following processes:

3. The method for maintaining the parallel increment of datalog based on hierarchical counting according to claim 1, wherein when calculating the counting structure of each layer in the second step, the increment and decrement adjustment is performed on the data sets of the internal data olds and news according to the external data add_i and del_i, and the partitioning is performed on the data sets of olds and news according to the specific data state, comprising the following steps:

4. The hierarchical count based datalog parallel increment maintenance method according to claim 1, wherein the step three is characterized by deriving news dataset increments when calculating each layer count structure; the specific procedure for deriving the delta portion of the segmented news dataset according to the semi-naive algorithm is as follows:

5. The hierarchical count based datalog parallel increment maintenance method according to claim 1, wherein the steps of constructing the multi-version epitaxial database and accessing the multi-version database in the fourth step include the following steps: