CN103412883A - Semantic intelligent information publishing and subscribing method based on P2P technology - Google Patents

Abstract

A method for publishing and subscribing semantic intelligent information based on P2P technology, the steps of which are: (1) Construct the system topology and set up super nodes; (2) Normalize the high-dimensional attribute space; (3) Partition high-dimensional data and establish and maintain Global index tree; (4) Dimensionality reduction of high-dimensional data: the node organizes the data aggregation area assigned to itself into a multi-dimensional hypercube in the high-dimensional attribute space, and finally uses the pyramid dimensionality reduction method to map the high-dimensional data objects to one-dimensional Data space, and use B+ tree to index; (5) Use i-Chord method to manage data objects; (6) Realize semantic-based intelligent information subscription and release. The invention has the advantages of simple principle, easy realization and popularization, improvement of system fault tolerance, dynamicity, information distribution efficiency and the like.

Description

Semantic Intelligent Information Publishing and Subscribing Method Based on P2P Technology

technical field

The invention mainly relates to the field of semantic-based intelligent information interaction methods in large-scale information networks, in particular to a P2P technology-based semantic intelligent information publishing and subscribing method.

Background technique

With the rapid development and wide application of computer network technology, the digital resources in the network are showing an exponential growth trend, the forms of expression are becoming more and more diversified, and the user's demand for information acquisition is also increasing. Obtaining information that users are "interested in" has become a problem that people pay more and more attention to.

The network environment presents the characteristics of large scale, decentralized control, loose coupling, autonomy, and dynamics. For this reason, researchers have proposed the Publish/Subscribe (hereinafter referred to as P/S) technology. The P/S system consists of three parts: publisher, subscriber and event agent. The publisher refers to the object that generates the event, that is, the information producer; the subscriber refers to the object that consumes the event, that is, the information consumer; and the event agent is the middleware for publishing/subscribing, and the publisher publishes information to the event in the form of an "event" Agents, subscribers subscribe interested events to the event agent, and the event agent routes the published events to interested subscriptions in a timely and reliable manner. P/S technology is an asynchronous communication mode that can provide one-to-many and many-to-many information interaction at the same time. It can completely decouple the participants of information interaction in time, space and control flow, and also has anonymous communication. and other features, which can well meet the needs of loose communication in large-scale distributed network systems.

The existing publish/subscribe technology research is still in the development stage, there are certain problems in terms of reliability and information distribution efficiency, and there are still many key technologies to be solved urgently. For example, in terms of topology, the existing P/S system middleware is usually designed in a centralized or unstructured P2P form. The centralized topology relies on a single server to intermediary publishers and subscribers (such as SIENA of the University of Colorado and Gryphon of IBM Research Center and JEDI, etc.), but its disadvantage is that it is easy to cause performance bottlenecks. If the server fails, the entire system cannot work; non-structural P2P topology (such as Hermes proposed by Cambridge University) often uses flooding, Gossiping or random walk (random walk) algorithms to route information, but due to its structureless and node dynamics, event routing is difficult to maintain, and the system can Poor scalability.

On the other hand, in an open network environment, information resources have different forms of expression, and information structure heterogeneity generally exists (different users use different structures to represent the same event, for example, some events are in Map format, and some events are in XML format ) and semantic heterogeneity (different users use different vocabulary (terms) to represent the same event, or use the same vocabulary to represent different concepts). However, the existing P/S systems (such as CORBA, Scribe, Bayeux, JEDI, etc.) still have a lot of shortcomings in terms of expressiveness. They describe events according to their structural information and lack understanding of the semantics of events themselves. The matching algorithm between subscriptions belongs to exact matching, which is easily interfered by synonyms or similar words during the matching process, and may return a large number of erroneous results that deviate from user semantics, and intelligent matching based on information semantics cannot be realized.

In order to enhance the semantic expression ability of the system and realize intelligent matching based on information semantics, various digital resources in the information network can be abstracted into points or feature vectors in the high-dimensional attribute space, and the distance between high-dimensional data points or The angle between feature vectors is used to measure the semantic similarity between data objects. The proposal of high-dimensional attribute space also leads to the problem of "dimension disaster", which shows that the data distribution in high-dimensional attribute space is sparse and close to the surface distribution of high-dimensional space, which leads to the high cost of semantic similarity search and the low search efficiency. not tall. High-dimensional data dimensionality reduction technology can map data objects from difficult-to-manage high-dimensional spaces to low-dimensional spaces, effectively reducing the search space and improving data retrieval efficiency. It is one of the effective means to solve the "dimension disaster" problem. For example: the technical scheme described in the Chinese patent application titled "An Embedded Dimensionality Reduction Method Based on Image Data Structure Protection" performs similar subset and For the division of non-similar subsets, different embedding operations are performed for different subsets to achieve the purpose of distance conversion, and then the new distance matrix is projected to achieve the purpose of dimensionality reduction. However, the data objects in the real information network have complex components and various types, and their expressions and semantic attributes are constantly changing, so it is difficult to abstract them all into fixed-dimensional and fixed-type vectors; Many kinds of attributes can be defined, but many attributes in the actual search process have nothing to do with the search (for example, the concept in medicine cannot appear in computer science), so it is necessary to map various data objects to a fixed-structure attribute space And appropriately reduce the attributes irrelevant to the search, thereby reducing the amount of calculation in the semantic similarity search and further improving the search efficiency.

To sum up, the existing publish-subscribe system has certain deficiencies in terms of dynamics, fault tolerance, self-organization, etc. At the same time, the system still has defects in expressive ability, lack of understanding of the semantics of the event itself, and cannot realize semantic-based intelligent information interaction.

Contents of the invention

The technical problem to be solved by the present invention is: aiming at the technical problems existing in the prior art, the present invention provides a P2P technology-based semantic Intelligent information publish and subscribe method.

In order to solve the problems of the technologies described above, the present invention adopts the following technical solutions:

A method for publishing and subscribing semantic intelligent information based on P2P technology, the steps of which are as follows:

(1) Construct the system topology and set up super nodes: use structured P2P technology to construct the topology of multiple event agents in the P/S system into a Chord ring structure, and set a super node on the ring to extract information from the network The attributes of the data resources in the information network and construct the attribute space, the data resources in the information network are abstracted as points or vectors in the high-dimensional attribute space;

(2) High-dimensional attribute space normalization processing: use the vector space model on the super node to represent the data information in the network as a high-dimensional point vector in the high-dimensional attribute space, and map all the data information in the network to a high-dimensional attribute space In , the mathematical form is expressed as a high-order matrix; the latent semantic index is used to remove the information attributes that have little relevance to information retrieval, and an attribute subspace is used to approximately replace the original high-dimensional attribute space;

(3) Partition high-dimensional data and establish and maintain a global index tree: the super node SN divides the high-dimensional data in the attribute space into different data aggregation areas, and assigns each data aggregation area to different nodes. The super node also maintains An index tree constructed from all node information, called a global index tree, is used to assign event information to nodes in the Chord ring and determine the proxy nodes that subscription requests need to visit;

(4) Dimensionality reduction of high-dimensional data: the node organizes the data aggregation area assigned to itself into a multi-dimensional hypercube in the high-dimensional attribute space, and finally uses the pyramid dimensionality reduction method to map the high-dimensional data objects to the one-dimensional data space, and uses B+ tree for indexing;

(5) Use the i-Chord method to manage data objects: After mapping high-dimensional data sets to one-dimensional data space, use the Chord protocol to organize and maintain network data information; set an order-preserving function to map data in one-dimensional data space to Chord resources Identifier space; each node of Chord corresponds to a data aggregation area, stores and manages high-dimensional data in the corresponding data aggregation area, and each node maintains a routing table to facilitate quick information query;

(6) Realize semantic-based intelligent information subscription and publishing: each node in the system maintains a subscription table to record subscription information related to the node semantics; when a subscriber sends a subscription request to the system, it first searches the global index tree Determine the proxy node that is semantically related to the subscription request, send the request to the proxy node, add a subscription record to the subscription table of the proxy node, register the association between the subscription request and the node, and then the proxy node Determine the precise search range in the high-dimensional attribute space, and accurately find the event information with the same semantics as the subscription request within the precise search range and return it to the subscriber; when the publisher sends event information to the system, it first determines the event information by searching the global index tree The host node related to the semantics of the event sends the event information to the host node and consults the subscription table of the host node. If the event information successfully matches the semantics of a subscription information in the subscription table, the event is actively pushed to the user.

As a further improvement of the present invention: the specific steps of the step (1) are:

(1.1) There may be multiple event agents in the P/S system. Multiple event agents are organized into an event agent network according to the Chord ring structure. The event agents correspond to each node in the network, and each event agent stores information in the network according to certain rules. The data resources in , and save the information of some other nodes;

(1.2) Select a node with the strongest capability in the Chord ring as the super node SN, the super node SN regularly checks the capabilities of other nodes in the ring, selects candidate super nodes, and the candidate super nodes back up important information on the SN;

(1.3) The super node SN is responsible for extracting the attributes of different forms of data resources and constructing a multi-dimensional attribute space. The data resources in the information network are abstracted into points or vectors in the high-dimensional attribute space.

As a further improvement of the present invention: the specific steps of the step (2) are:

(2.1) According to the idea of the vector space model, the high-dimensional data information in the network is described as an attribute vector, and then the system organizes all the high-dimensional information in the network into a matrix, such as d high-dimensional data described by t attributes in the information network The information is described as a t*d matrix A, each column of the matrix represents a high-dimensional information data, and the matrix element a _ij is the attribute value of the attribute i in the data object j, representing the importance of the attribute i in the object j, if high Dimension information j does not have attribute i, then a _ij is 0;

(2.2) Note that most of the elements in the matrix are 0, indicating that most of the attributes of a high-dimensional information in information retrieval are useless information, and a low-rank matrix is used to approximate the original matrix; assume that the high-dimensional information set in the network is expressed as a matrix A, the rank of A is r, using the singular value decomposition of the matrix to decompose A into the product of three matrices:

A=UΣV ^T

where U=(u _l , u ₂ , u _r ) is a t*r matrix, Σ=diag(σ ₁ ,...,σ _r ) is a r*r diagonal matrix, V=(v _l ,...,v _r ) is a d*r matrix, σ _i is the singular value of A, σ ₁ ≥σ ₂ ≥...≥σ _r ;

(2.3) Only keep one of the largest matrix singular values, omit other singular values, and approximate the matrix A with rank r into matrix A 1 with rank ₁ :

A ₁ = U ₁ Σ ₁ V ₁ ^T

where U ₁ =(u _l , u ₂ ,...,U ₁ ), Σ ₁ =diag(σ ₁ ,...,σ ₁ ), V ₁ =(V _l ,...,V ₁ ), The rows of V ₁ Σ ₁ are semantic vectors of high-dimensional information.

As a further improvement of the present invention: the specific steps of the step (3) are:

(3.1) The data resources in the information network are abstracted as points or vectors in the high-dimensional attribute space. According to the principle that data resources with similar distances in the attribute space have similar semantics, the super node SN aggregates the high-dimensional data distributed in the multi-dimensional attribute space. Class partitioning, which divides high-dimensional data into multiple disjoint data aggregation areas, and assigns each data aggregation area to different nodes;

(3.2) In addition to its own routing table, the super node SN also maintains an index tree constructed from the resource identifier range information of all nodes on the ring, called the global index GI;

(3.2.1) Quickly assign event information to nodes; each node in the ring is responsible for a data aggregation area, and all data information in the aggregation area is allocated to this node. When an event information request is added, it is determined by querying GI Which data aggregation area the event information belongs to, that is, the host node of the event information is determined, and then the identifier of the host node is used as the search key, and the event information is distributed to the host node by using the Chord routing protocol;

(3.2.2) Determine the node that the subscription request needs to visit; when a node inputs a subscription request, it first sends the request to SN to search for GI, determines which nodes are responsible for the data aggregation area that intersects with the semantic space of the subscription request, and returns these The identifier of the node, the returned node is called a proxy node, and then the identifier of the proxy node is used as the search key to route the subscription request to these nodes to further realize semantic matching.

As a further improvement of the present invention: the specific steps of the step (4) are:

(4.1) Organize each data aggregation area into a multi-dimensional hypercube with dimension d in the high-dimensional data space, and normalize the multi-dimensional hypercube, that is, the side length of each dimension is 1, and the center of the hypercube Points are expressed as (0.5, 0.5, ..., 0.5), and then the center of the hypercube is used as the vertex, and the (d-1) dimensional hyperplane of the data aggregation area is used as the base, and each d-dimensional data aggregation area is divided into 2d a pyramid;

(4.2) Divide each pyramid into several sub-divisions parallel to the base, each sub-division corresponds to a B+ tree data page, and then map the high-dimensional data in the data aggregation area according to the distance from the high-dimensional data point to the pyramid base It is a one-dimensional value, and the data after dimension reduction is organized and managed by B+ tree; the dimension reduction formula is:

y _v =i*C+(j+h _v )=i*c+(0+|0.5-h _v |)

Among them, the number of cubes where a high-dimensional data object v is located is i, the number of pyramids is j, and the distance between v and the base plane of the pyramid is h _v ', then the distance to the plane where the apex of the pyramid is located is h _v =∣0.5-h _v '∣; After dimensionality reduction, the one-dimensional value of each high-dimensional data object in a high-dimensional cube is limited to the interval [i*c, (i+1)*c], c is a constant large enough to ensure that each Data objects in a data partition have different index keys than other partitions.

As a further improvement of the present invention: the specific steps of the step (5) are:

(5.1) Assuming that the Chord system resource identifier interval size is 2 ^m , use an order-preserving function h to map the one-dimensional values after dimensionality reduction to the interval [0,2 ^m ) in sequence; that is, for a certain height in the attribute space Dimension data point v, the i-Chord resource identifier of point v is:

key _v =ichord(v)=h(y _v )=h(i*c+(0+h _v ))∈[0, 2 ^m )

(5.2) Each node N _i on the Chord ring is responsible for storing and managing the high-dimensional data information of a data aggregation area. Let the N _i node identifier Nkey _i , then the resource identifiers of the high-dimensional data information in the aggregation area are distributed in the interval ( Nkey _i-1 , Nkey _i ];

(5.3) Each node N _i maintains a routing table, that is, a pointer table, pointing to other nodes on the ring; there are m entries in the routing table, and the kth (1≤k≤m) row entry is Chord The first node whose identifier on the ring is equal to or greater than (Nkey _i +2 ^k-1 )mod2 ^m is the successor((Nkey _i +2 ^k-1 )mod2 ^m ); When requesting, first check whether its own node is equal to the key, and if so, return it directly; otherwise, the node searches its routing table, finds the first node in the table that is the largest but does not exceed the key, and forwards the query request to this node, and repeats this process until the request reaches a node N _k , and when the key is located between N _k and N _k ’s subsequent node N _k+1 , the node N _k reports its subsequent node N _k+1 as the response to the request;

(5.4) When the high-dimensional data in a data aggregation area is too dense and the nodes cannot effectively manage it, split the aggregation area into two or more new aggregation areas, and select idle nodes on the Chord ring accordingly In order to share the load of the original node; when the data in the adjacent data aggregation area is sparse and the node resources are wasted, the aggregation area is merged, and one of the two execution nodes is selected as the execution node of the new data aggregation area, and the other is set as free node.

As a further improvement of the present invention: the specific steps of the step (6) are:

(6.1) Each node maintains a subscription table, which records all subscription information related to the node. When the node has event information, query the subscription table of the node. If the event information matches the semantics of a certain subscription information in the subscription table If successful, the event is actively pushed to the user;

(6.2) When a subscriber submits subscription request information to a node N _i on the ring, the node searches the global search tree to determine the proxy node related to the subscription request semantics, routes the subscription request to the proxy node, and Execute a similar search algorithm based on subscription semantics. If the event information with the same semantics as the subscription request is searched, the event information will be passed back to N _i ; at the same time, the proxy node will store the subscription request, subscription conditions, routing path and search range information in its subscription form;

(6.3) When the publisher publishes event information to a node N _j on the ring, the node searches the global search tree to determine the host node related to the semantics of the event information, and routes the event information to and stores it in the host node according to the Chord routing protocol At the same time, the host node semantically matches the event attribute information with the subscription request information in the subscription table, and if the match is successful, it will directly push the event information to the subscriber;

(6.4) When a subscriber publishes unsubscribe information to a node on the ring, the node searches the global search tree to determine the proxy node related to the semantics of the subscription information, and routes the request information to the corresponding proxy node. In the proxy node subscription table Find and delete the corresponding subscription information.

Compared with the prior art, the present invention has the advantages of:

The principle of the present invention is simple, easy to implement and popularize. On the basis of structured P2P technology, a semantic-based publish/subscribe system can be constructed, which improves the fault tolerance, dynamics and information distribution efficiency of the system; the present invention combines event information and subscription The information is mapped to the multi-dimensional attribute space. The system supports the understanding of the semantics of the information itself, which enhances the expressive ability of the system and supports intelligent matching based on the information semantics. At the same time, it eliminates the The disaster of dimensionality brought by multi-dimensional attribute space can efficiently realize data organization and intelligent matching based on information semantics.

Description of drawings

Fig. 1 is a schematic flow chart of the method of the present invention.

Fig. 2 is a schematic diagram of mapping high-dimensional data from high-dimensional space to one-dimensional space in the present invention.

Fig. 3 is a schematic diagram according to the search data process and the subscription table in the present invention.

Fig. 4 is a schematic diagram of a subscription request processing process in the present invention.

FIG. 5 is a schematic diagram of processing range queries based on information semantics in the present invention.

Detailed ways

The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

As shown in Figure 1, the method for publishing and subscribing semantic intelligent information based on P2P technology of the present invention, the specific steps are:

(1) Construct the system topology and set up super nodes: use structured P2P technology to construct the topology of multiple event agents in the P/S system into a Chord ring structure, and set a super node on the ring to extract information from the network The attributes of the data resources in the information network and construct the attribute space, and the data resources in the information network are abstracted as points or vectors in the high-dimensional attribute space.

(2) High-dimensional attribute space normalization processing: In view of the fact that there are many types of information in the current information network and different information attributes are inconsistent, the vector space model (VSM) is used on the super node to represent the data information in the network as a high-dimensional attribute space The high-dimensional point vector in the network maps all data information in the network to a high-dimensional attribute space, which is expressed as a high-order matrix in mathematical form; in order to reduce noise and synonym interference in the process of information indexing, latent semantic indexing (LSI ) to remove information attributes that have little correlation with information retrieval, and use an attribute subspace to approximately replace the original high-dimensional attribute space.

(3) Partition high-dimensional data and establish and maintain a global index tree: the super node SN divides the high-dimensional data in the attribute space into different data aggregation areas, and assigns each data aggregation area to different nodes. The super node also maintains An index tree constructed from all node information, called a global index tree, is used to assign event information to nodes in the Chord ring and determine the proxy nodes that subscription requests need to access.

(4) Dimensionality reduction of high-dimensional data: the node organizes the data aggregation area assigned to itself into a multi-dimensional hypercube in the high-dimensional attribute space, and finally uses the pyramid dimensionality reduction method to map the high-dimensional data objects to the one-dimensional data space, and uses B+ tree for indexing.

(5) Use the i-Chord method to manage data objects: After mapping the high-dimensional data set to a one-dimensional data space, use the Chord protocol to organize and maintain network data information. Aiming at the mismatch between the one-dimensional data space and the Chord resource identifier space, set an order-preserving function to map the data in the one-dimensional data space to the Chord resource identifier space; each node of Chord corresponds to a data aggregation area, and stores and manages the corresponding data aggregation High-dimensional data in the area, and each node maintains a routing table to facilitate quick information query; when the amount of data managed by a node is too large, the load balance of the system can be maintained through operations such as node insertion and departure.

(6) Realize semantic-based intelligent information subscription and publishing: each node in the system maintains a subscription table to record subscription information related to the semantics of the node. When a subscriber sends a subscription request to the system, it first determines the proxy node related to the subscription request semantics by searching the global index tree, sends the request to the proxy node, adds a subscription record to the subscription table of the proxy node, and registers the subscription request Then the proxy node determines the precise search range in the high-dimensional attribute space according to the subscription request and subscription conditions, and accurately finds the event information with the same semantics as the subscription request within the precise search range and returns it to the subscriber. When the publisher sends event information to the system, it first determines the host node related to the event semantics by searching the global index tree, sends the event information to the host node, and consults the subscription table of the host node. If the semantic matching of the subscription information is successful, the event will be actively pushed to the user.

In this embodiment, the specific steps of the above step (1) are:

(1.1) There may be multiple event agents in the P/S system. Multiple event agents are organized into an event agent network according to the Chord ring structure. The event agents correspond to each node in the network. Each event agent follows certain rules (event Semantic information) stores data resources in the information network, serves a certain number of clients, and saves some information of other nodes, so that subscription and event information can reach semantically related nodes at low cost, efficiently and reliably.

(1.2) In the Chord ring, select a node with the strongest capability (computing power, storage space, online time, bandwidth, etc.) as the super node SN, and the super node SN regularly checks the capabilities of other nodes in the ring, from which Select a candidate super node, and the candidate super node backs up important information on the SN, so that when the SN fails, it will replace the SN and become a new super node.

(1.3) In order to obtain the semantic information of data resources in the network, the super node SN is responsible for extracting the attributes of different forms of data resources and constructing a multi-dimensional attribute space. The data resources in the information network are abstracted as points or vectors in the high-dimensional attribute space.

In this embodiment, the specific steps of the above step (2) are:

(2.1) According to the idea of the vector space model, the high-dimensional data information in the network is described as an attribute vector. The attributes can be elements such as concepts, keywords, and terms related to the given high-dimensional information, and then the system combines all the high-dimensional information in the network Organized as a matrix, such as d high-dimensional data information that can be described by t attributes in an information network can be described as a t*d matrix A, each column of the matrix represents a high-dimensional information data, and matrix elements a _ij are data The attribute value of attribute i in object j represents the importance of attribute i in object j (for example, it can be calculated by the frequency of a term appearing in a document). If attribute i does not exist in high-dimensional information j, then a _ij is 0.

(2.2) Note that most of the elements in the matrix are 0, indicating that most of the attributes of a high-dimensional information in information retrieval are useless information. In order to reduce unnecessary calculations in the process of information semantic retrieval, a low-rank matrix is used to approximate instead of the initial matrix. Assuming that the high-dimensional information set in the network can be expressed as a matrix A, the rank of A is r, and the singular value decomposition of the matrix is used to decompose A into the product of three matrices:

A=U∑V ^T

Where U=(u ₁ ,u ₂ ,...,u _r ) is a t*r matrix, ∑=diag(σ ₁ ,...,σ _r ) is a r*r diagonal matrix, V=( v ₁ ,...,v _r ) is a d*r matrix, σ _i is the singular value of A, σ ₁ ≥σ ₂ ≥...≥σ _r .

(2.3) In order to reduce the amount of calculation, speed up information retrieval, and avoid the interference of synonyms in the process of semantic matching, only the l largest singular values of the matrix are reserved, and other singular values are omitted, and the matrix A with rank r is approximately simplified is a matrix A _l of rank l:

A ₁ = U ₁ Σ ₁ V ₁ ^T

where U ₁ =(u _l , u ₂ ,...,U ₁ ), Σ ₁ =diag(σ ₁ ,...,σ ₁ ), V ₁ =(V _l ,...,V ₁ ), The rows of V ₁ Σ ₁ are semantic vectors of high-dimensional information. In this way, processing on high-dimensional information retrieval can be performed within this low-rank semantic matrix.

In this embodiment, the specific steps of the above step (3) are:

(3.1) The data resources in the information network are abstracted as points or vectors in the high-dimensional attribute space. According to the principle that data resources with similar distances in the attribute space have similar semantics, the super node SN aggregates the high-dimensional data distributed in the multi-dimensional attribute space. Class partitioning divides high-dimensional data into multiple disjoint data aggregation areas, and assigns each data aggregation area to different nodes.

(3.2) In addition to its own routing table, the super node SN also maintains an index tree constructed from the resource identifier range information of all nodes on the ring, called the global index GI. GI serves two purposes:

(3.2.1) Quickly assign event information to nodes. Each node in the ring is responsible for a data aggregation area, and all data information in the aggregation area is allocated to this node. When an event information request is added, it is determined which data aggregation area the event information belongs to by querying GI, that is, the event is determined. The host node of the information, then uses the identifier of the host node as a search key, and uses the Chord routing protocol to distribute the event information to the host node.

(3.2.2) Determine the nodes that the subscription request needs to visit. When a node inputs a subscription request, it first sends the request to SN to search GI to determine which nodes are responsible for the data aggregation area intersecting with the semantic space of the subscription request and return the identifiers of these nodes. The returned nodes are called proxy nodes. It may contain subscription results, and then use the identifier of the proxy node as the search key to route the subscription request to these nodes to further achieve semantic matching.

In this embodiment, the specific steps of the above step (4) are:

(4.1) Organize each data aggregation area into a multi-dimensional hypercube with dimension d in the high-dimensional data space, and normalize the multi-dimensional hypercube, that is, the side length of each dimension is 1, and the center of the hypercube Points are expressed as (0.5, 0.5, ..., 0.5), and then take the center of the hypercube as the vertex, and the (d-1) dimensional hyperplane of the data aggregation area as the base, and each d-dimensional data aggregation area (hypercube ) is divided into 2d pyramids.

(4.2) Divide each pyramid into several sub-divisions parallel to the base, each sub-division corresponds to a B+ tree data page, and then map the high-dimensional data in the data aggregation area according to the distance from the high-dimensional data point to the pyramid base It is a one-dimensional value, and the data after dimension reduction is organized and managed by B+ tree. The dimensionality reduction formula is:

y _v =i*C+(j+h _v )=i*c+(j+|0.5-h _v |)

Among them, the number of cubes where a certain high-dimensional data object v is located is i (to distinguish different data aggregation areas), the number of pyramids is j, and the distance between v and the base plane of the pyramid where v is located is h _v ', then the distance to the plane where the apex of the pyramid is located is h _v =∣0.5-h _v'∣ . After dimensionality reduction, the one-dimensional value of the high-dimensional data object in each high-dimensional cube is limited to the interval [i*c, (i+1)*c], c is a constant large enough to ensure that each data Data objects in a partition have different index keys than other partitions.

In this embodiment, the specific steps of the above step (5) are:

(5.1) Assuming that the resource identifier interval size of the Chord system is 2 ^m , in order to use the resource identifier of Chord to manage the one-dimensional data after dimension reduction, an order-preserving function h is used to map the one-dimensional values after dimension reduction to the interval [0 ,2 ^m ). That is, for a certain high-dimensional data point v in the attribute space (v belongs to the hypercube of cube digit i), the i-Chord resource identifier of point v is:

key _v =ichord(v)=h(y _v )=h(i*c+(j+h _v ))∈[0, 2 ^m )

(5.2) Each node N _i on the Chord ring is responsible for storing and managing the high-dimensional data information of a data aggregation area. Let the N _i node identifier Nkey _i , then the resource identifiers of the high-dimensional data information in the aggregation area are distributed in the interval ( Nkey _i-1 , Nkey _i ].

(5.3) Each node N _i maintains a routing table, that is, a finger table, pointing to other nodes on the ring. There are m (number of identifiers) entries in the routing table, among which the entry k (1≤k≤m) is the identifier on the Chord ring equal to or greater than (Nkey _i +2 ^k-1 )mod2 ^m The first node of , namely successor((Nkey _i +2 ^k-1 )mod2 ^m ). When any node receives a request whose keyword is key, it first checks whether its own node is equal to key, and if so, it returns directly; otherwise, the node searches its routing table, finds the first node in the table that is the largest but does not exceed key, and Forward the query request to this node, and repeat this process until the request reaches a node N _k , and when the key is located between N _k and N _k ’s subsequent node N _k+1 , node N _k reports its successor node N _k+1 in response to a request.

(5.4) Nodes are responsible for limited resource identifiers, so nodes cannot maintain infinite high-dimensional data information. When the high-dimensional data in a data aggregation area is too dense and the nodes cannot effectively manage it, the aggregation area is split into two or more new aggregation areas, and correspondingly select idle nodes on the Chord ring to share the original data. The load of the node; when the data in the adjacent data aggregation area is sparse and the node resources are wasted, the aggregation area is merged, and one of the two execution nodes is selected as the execution node of the new data aggregation area, and the other is set as an idle node. By performing the above operations, the system load balance is effectively realized.

In this embodiment, the specific steps of the above step (6) are:

(6.1) In order to realize the semantic-based information publishing and subscription function of the system, each node maintains a subscription table to record all subscription information related to the node. When the node has event information, query the subscription table of the node. If the event If the information successfully matches the semantics of a piece of subscription information in the subscription table, the event will be actively pushed to the user.

(6.2) When a subscriber submits subscription request information to a node N _i on the ring, the node searches the global search tree to determine the proxy node related to the subscription request semantics, routes the subscription request to the proxy node, and Execute a similar search algorithm based on subscription semantics (such as point query, range query, k-nearest neighbor query, nearest neighbor query), and if the event information with the same semantics as the subscription request is found, the event information will be reversely passed to N _i ; at the same time, The agent node stores information such as subscription request, subscription condition, routing path and search scope in its subscription table.

(6.3) When the publisher publishes event information to a node N _j on the ring, the node searches the global search tree to determine the host node related to the semantics of the event information, and routes the event information to and stores it in the host node according to the Chord routing protocol At the same time, the host node semantically matches the event attribute information with the subscription request information in the subscription table, and if the match is successful, it directly pushes the event information to the subscribers.

(6.4) When a subscriber publishes unsubscribe information to a node on the ring, the node searches the global search tree to determine the proxy node related to the semantics of the subscription information, and routes the request information to the corresponding proxy node. In the proxy node subscription table Find and delete the corresponding subscription information.

The present invention will be described in detail below with a specific application example, and its detailed implementation steps are:

1) Network data collection: There are a large number of digital resources in various forms (such as text, images, music, videos, etc.) in the network, and the system cannot intuitively understand the semantic information of the above resources. Chemical processing for machine identification and further processing.

2) Construct the system topology and set up super nodes: organize multiple event agents into an event agent network according to the Chord ring structure, the event agents correspond to each node in the network, and each event agent stores information according to certain rules (semantic information of events) The data resources in the network serve a certain number of clients and store some information of other nodes, so that subscription and event information can reach semantically related nodes at low cost, efficiently and reliably. Set up the most capable node in the system (computing power, online time, storage space, bandwidth and other considerations) super node SN, and determine the backup node of the super node to ensure the normal operation of the system when the super node fails. In order to obtain the semantic information of the data resources in the network, the super node SN is responsible for extracting the attributes of different forms of data resources and obtaining the multi-dimensional attribute space, and the data resources in the information network are abstracted as points or vectors in the high-dimensional attribute space.

3) High-dimensional attribute space normalization processing: According to the vector space model (VSM), the attribute information of the data object is extracted, and the data object in the network is described as a high-dimensional attribute vector, and the network digital resources are mapped to the high-dimensional attribute space. A collection of network data resources can be expressed as a matrix, and the element value in the vector indicates the importance of an attribute in the data object. For example, a collection of d data objects in the network can be described as a matrix A of t*d by t attributes.

In the process of information retrieval, the network data resources represented by the high-dimensional attribute space have many attributes irrelevant to the query (many elements in the matrix are 0), in order to speed up the query and avoid synonyms and noise interference, the matrix is simplified, using A low-rank matrix approximates the original matrix. Assuming that the data resource set in the network is expressed as a matrix A, and the rank of A is r, use the singular value decomposition of the matrix to decompose A into the product of three matrices: A=U∑V ^T , where U=(u ₁ ,u ₂ , ...,u _r ) is a t*r matrix, ∑=diag(σ ₁ ,...,σ _r ) is a r*r diagonal matrix, V=(v ₁ ,...,v _r ) is a d*r matrix, σ _i is the singular value of A, σ ₁ ≥σ ₂ ≥...≥σ _r ; then only keep the l largest matrix singular values, omit other singular values, and rank r The matrix A of is approximated as a matrix A l with rank _l : A _l = U _l ∑ _l V _l ^T , where U _l =(u ₁ ,u ₂ ,...,u _l ),∑ _l =diag( σ ₁ ,...,σ _l ), V _l = (v ₁ ,...,v _l ), the row of V _l ∑ _l is the semantic vector of high-dimensional information, then the network resource can be expressed as an l-dimensional attribute space High-dimensional vectors in .

4) Data aggregation and partitioning and establishment of a global index tree: According to the principle that data objects with close distances in the attribute space are more likely to have similar semantics, the data objects in the attribute space are aggregated and partitioned, and the high-dimensional data in the space are divided into multiple disjoint The data aggregation area, try to ensure that the data in each aggregation area tends to be evenly distributed.

In addition, the super node maintains a global index tree GI, which contains information on the range of data aggregation areas corresponding to each node. When a new data object is added to the system, GI quickly determines which node the data object should belong to according to the coordinates of the data object, so that Quickly assign newly added data objects to this node; when there is a new subscription request, determine the semantic space range of the subscription request, query GI to quickly confirm the data aggregation area intersected with the subscription request, and send the request to the relevant node for further Accurate query speeds up the query speed. In addition, the global index GI should be refreshed regularly to avoid the impact of system node changes.

5) Dimensionality reduction processing of high-dimensional data: In order to eliminate the impact of the “curse of dimensionality” on information retrieval in high-dimensional attribute spaces, dimensionality reduction processing is performed on high-dimensional data information, as shown in Figure 2.

The data aggregation area obtained by data partitioning may be irregular in shape, and each data aggregation area is normalized into a high-dimensional hypercube. Each hypercube determines a cube number i, and performs normalization processing so that the side length of the cube is 1, and then The center of the hypercube is the vertex, and the (d-1)-dimensional hyperplane of the data aggregation area is used as the base. Each d-dimensional data aggregation area (hypercube) is divided into 2d pyramids, and a pyramid value i is assigned to each pyramid.

Divide each pyramid into several sub-divisions parallel to the base, each sub-division corresponds to a B+ tree data page, and then map the high-dimensional data in the data aggregation area to one-dimensional according to the distance from the high-dimensional data point to the pyramid base Value, as shown in Figure 2, and use B+ tree to organize and manage the data after dimension reduction. The dimension reduction formula is: y _v =i*c+(j+h _v )=i*c+(j+∣0.5-h _v '∣), after dimension reduction, the one-dimensional value of the high-dimensional data object in each high-dimensional cube It is limited in the interval [i*c, (i+1)*c], where c is a constant large enough to ensure that the data objects in each data partition have index keys different from other partitions.

6) Manage data objects with the i-Chord method: use the Chord method to store and manage network data information, realize distributed processing of information retrieval in information retrieval, and improve system efficiency. First, construct the resource identifier of the data object based on the one-dimensional data generated in Part 5. Assume that the resource identifier interval size of the Chord system is 2 ^m , and use an order-preserving function h to map the one-dimensional data values to the interval [0, 2 ^m ), the mapping formula is: key _v =ichord(v)=h(y _v )=h(i*c+(j+h _v ))∈[0,2 ^m ).

In order to facilitate data query, each node on the Chord ring stores and manages data objects in a data aggregation area, and the data information corresponding to a resource identifier on the ring is stored on its successor nodes, such as the node identifier Nkey _i of node N _i , Then the high-dimensional data object whose resource identifier is in the interval (Nkey _i-1 , Nkey _i ] is stored on the node N _i . Each node maintains a routing table, and the system searches for the data corresponding to the given resource identifier according to the routing table object (Figure 3 demonstrates the process of finding key=28 from node N4). At the same time, in order to maintain system load balance, when the data objects in the data aggregation area are too dense or too sparse, split or merge the data aggregator and execute the corresponding node operation to ensure stable system performance and resource optimization.

7) Publish and subscribe information based on semantics: the process of processing subscription requests is shown in Figure 4. Each node maintains a subscription table (as shown in Figure 3). When a user sends a subscription request to a node, the system checks the subscription table , if there is no same subscription request, the system searches the global index tree to determine the proxy nodes related to the subscription request semantics and routes the subscription request to these proxy nodes, and executes a similar search algorithm on these semantically related nodes to find out the semantics related to the subscription request The same event information is returned to the user, and relevant information such as user subscription requests and search results are added to the subscription table in the proxy node. If there is the same user request in the subscription table, the results are quickly found according to the subscription table and the subscription table is updated. ; When new event information is added to the system, store it on the corresponding host node according to its resource identifier, and check the subscription table of the host node to determine whether there is a subscription that matches its semantics, and if so, actively push the data object To the corresponding user; when the user sends an unsubscribe message, the system searches the subscription table and deletes the corresponding subscription information.

Similar search methods are divided into four categories: point query (Point Query), to find the same target object p as a given query point q in the data space S; range query (Range Query), for a given query point q and valve Value r, find all target objects p satisfying d(p,q)≤r in the data space S; nearest neighbor query (Nearest Neighbor Query), find the closest target to the given query point q in the data space S Object p; k-nearest neighbor query (KNN Query), find the k target objects p closest to a given query point q in the data space S.

Taking range query as an example, as shown in Figure 5 (taking two-dimensional space as an example), firstly, the query range (circular area in the figure) is obtained according to the coordinates of the query point and its query radius, and it is judged whether it is consistent with the data in the data space The aggregation area intersects. If there is a data aggregation area that intersects the query range, determine the two boundary index keywords y _low and y _high that each pyramid in the aggregation area intersects with the query range, and perform ranges in the B+ tree species based on these two index keywords Search to obtain a point set (the point set in the shaded area in the figure), and then accurately judge whether the data points in the point set belong to the query range, and finally output the result.

The above are only preferred implementations of the present invention, and the protection scope of the present invention is not limited to the above-mentioned embodiments, and all technical solutions under the idea of the present invention belong to the protection scope of the present invention. It should be pointed out that for those skilled in the art, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.

Claims (7)

1. the semantic intelligent information distribution subscription method based on the P2P technology, is characterized in that, step is:

(1) constructing system topological structure set up super node: utilize the structural P 2 P technology that the topology constructing of a plurality of event agents in the P/S system is become to the Chord ring structure, and on ring, set a super node, for attribute the structure attribute space of information extraction network data resource, in information network data resource abstract be point or vector in the higher-dimension attribute space;

(2) normalization of higher-dimension attribute space is processed: on super node, utilize vector space model that data message in network is expressed as to the higher-dimension point vector in the higher-dimension attribute space, all data messages in network are mapped in a higher-dimension attribute space, on mathematical form, are expressed as a high level matrix; Utilize potential semantic indexing to remove the information attribute very little with Relevance in Information Retrieval, by the original higher-dimension attribute space of the approximate replacement of an attribute subspace;

(3) the global index tree is safeguarded in high dimensional data subregion foundation: super node SN is divided into different data gathering districts by the high dimensional data in attribute space, and each data gathering is distinguished and is fitted on different nodes, super node is also safeguarded an index tree be configured to by all nodal informations, be called the global index tree, be used for as node dispense event information in the Chord ring and determine the agent node that subscribe request need to be accessed;

(4) high dimensional data dimensionality reduction: the data gathering district consolidation that node will be distributed to oneself is the multidimensional hypercube in the higher-dimension attribute space, finally utilizes the pyramid dimension reduction method that the high dimensional data object map is arrived to the one-dimensional data space, and carries out index with the B+ tree;

(5) utilize i-Chord management by methods data object: after High Dimensional Data Set is mapped to the one-dimensional data space, utilize Chord agreement organizations maintaining network data message; Arrange an isotonic function by data-mapping in the one-dimensional data space to Chord resource identifier space; The corresponding data accumulation area of each node of Chord, the high dimensional data in storage administration corresponding data accumulation area, and routing table fast and easy Query Information of each node maintenance;

(6) intelligent information that realizes semantic-based is subscribed to and issue: subscription table of each node maintenance in system, record and the semantic relevant subscription information of this node; When the subscriber sends subscribe request to system, at first by search global index tree, determine and the semantic relevant agent node of this subscribe request, request is sent to agent node, in the subscription table of agent node, increase by one and subscribe to record, register the incidence relation of this subscribe request and this node, then agent node is determined the precise search scope in the higher-dimension attribute space according to subscribe request and subscription condition, in the precise search scope, accurately searches with the semantic identical event information of subscribe request and returns to the subscriber; When the publisher sends event information to system, at first by search global index tree, determine the host node relevant to this event semantics, event information is sent to host's node, and the subscription table of consulting host's node, if certain subscription information semantic matches success in event information and subscription table, by the event active push to the user.

2. the semantic intelligent information distribution subscription method based on the P2P technology according to claim 1, is characterized in that, the concrete steps of described step (1) are:

(1.1) in the P/S system, may there be a plurality of event agents, a plurality of event agents are organized into to event agent's network according to the Chord ring structure, the event agent is to each node in should network, each event agent is according to the data resource in certain rale store information network, and the information of preserving other nodes of part;

(1.2) in the Chord ring, select the node that ability is the strongest as super node SN, super node SN makes regular check on the ability of other nodes in ring, therefrom selects candidate's super node, the upper important information of candidate's super node backup SN;

(1.3) super node SN is responsible for extracting the attribute of multi-form data resource, structure multidimensional property space, in information network data resource abstract be point or vector in the higher-dimension attribute space.

3. the semantic intelligent information distribution subscription method based on the P2P technology according to claim 1, is characterized in that, the concrete steps of described step (2) are:

(2.1) thought according to vector space model is described as an attribute vector by high dimensional data information in network, then system is a matrix by all higher-dimension Information Organizations in network, if in information network, be described as the matrix A of a t*d by d high dimensional data information of t attribute description, each row of matrix represent a higher-dimension information data, matrix element a _IjFor the property value of attribute i in data object j, represented the importance of attribute i in object j, if higher-dimension information j does not exist attribute i, a _IjBe 0;

(2.2) notice that in matrix, most element is 0, in the descriptive information retrieval, the most attribute of a higher-dimension information is garbage, replaces initial matrix with a low-rank approximate matrix; Suppose that in network, the higher-dimension information aggregate is expressed as matrix A, the order of A is r, utilizes the svd of matrix A to be decomposed into to the product of three matrixes:

A=U∑V ^T

U=(u wherein ₁, u ₂..., u _r) be a t*r matrix, ∑=diag (σ ₁..., σ _r) be a r*r diagonal matrix, V=(v ₁..., v _r) be a d*r matrix, σ _iThe singular value of A, σ ₁>=σ ₂>=...>=σ _r

(2.3) only retain l maximum singular values of a matrix, dispense other singular value, the approximate abbreviation of matrix A that is r by order is that order is the matrix A of l _l:

A _l=U _l∑ _lV _l ^T

U wherein _l=(u ₁, u ₂..., u _l), ∑ _l=diag (σ ₁..., σ _l), V _l=(v ₁..., v _l), V _l∑ _lRow be the semantic vector of higher-dimension information.

4. the semantic intelligent information distribution subscription method based on the P2P technology according to claim 1, is characterized in that, the concrete steps of described step (3) are:

(3.1) in information network data resource abstract be point or vector in the higher-dimension attribute space, according to the close data resource of distance in attribute space, has the principle of similar semanteme, super node SN carries out the cluster subregion to the high dimensional data distributed in the multidimensional property space, high dimensional data is divided into to a plurality of mutually disjoint data gatherings district, and each data gathering is distinguished and is fitted on different nodes;

(3.2) super node SN, except the routing table of self, also safeguards the index tree that a structure of the resource identifier range information by the upper all nodes of ring forms, and is called the GI of global index;

(3.2.1) be fast node dispense event information; In ring, each node is responsible for a data accumulation area, in this accumulation area, all data messages are all distributed to this node, when an event information request adds fashionable, GI determines which data gathering district this event information belongs to by inquiry, namely determined host's node of event information, then the identifier of host's node of take is search key, utilizes the Chord Routing Protocol, and this event information is distributed to host's node;

(3.2.2) determine the node that subscribe request need to be accessed; When certain node input subscribe request, at first request is sent to SN search GI, determine that the semantic space of data gathering district which node is responsible for and this subscribe request intersects and return the identifier of these nodes, the node returned is called agent node, then the identifier of agent node of take is search key, subscribe request is routed to these nodes, further realizes semantic matches.

5. the semantic intelligent information distribution subscription method based on the P2P technology according to claim 1, is characterized in that, the concrete steps of described step (4) are:

(4.1) by each data gathering district consolidation, be that in high-dimensional data space, dimension is the multidimensional hypercube of d, and the multidimensional hypercube is carried out to normalized, the length of side that is every one dimension is 1, the central point of hypercube be expressed as (0.5,0.5 ..., 0.5), then take the hypercube center is summit, and (d-1) dimension lineoid in data gathering district, as base, is divided into 2d pyramid by each d dimension data accumulation area;

(4.2) each pyramid being divided into to several height parallel with base divides, each height is divided corresponding with a B+ tree data page, then the distance of putting the pyramid base according to high dimensional data is mapped as one dimensional numerical by the high dimensional data in the data gathering district, and by data after B+ tree organization and administration dimensionality reduction; The dimensionality reduction formula is:

y _v=i*c+(j+h _v)=i*c+(j+∣0.5-h _v’∣)

Wherein the cube number at certain high dimensional data object v place is i, and the pyramid number is j, and v is h to place pyramid base plane distance _v', the distance that arrives plane, place, pyramid summit is h _v=∣ 0.5-h _v’ ∣; After dimensionality reduction, in each High Dimension Cubes, the one dimension value of high dimensional data object is limited at [i*c, (i+1) * c] in interval, c is enough large constant, guarantees that the data object in each data partition has the index key that is different from other subregion.

6. the semantic intelligent information distribution subscription method based on the P2P technology according to claim 1, is characterized in that, the concrete steps of described step (5) are:

(5.1) suppose that between Chord system resource identifier field, size is 2 ^m, utilize an isotonic function h that one dimensional numerical after dimensionality reduction is mapped to interval [0,2 in order ^m) in; Namely for certain the high dimensional data point v in attribute space, the i-Chord resource identifier of some v is:

key _v=ichord(v)=h(y _v)=h(i*c+(j+h _v))∈[0,2 ^m)

(5.2) each node N on the Chord ring _iBe responsible for the high dimensional data information of a data accumulation area of storage administration, establish N _iNode identifier Nkey _i, in this accumulation area, the resource identifier of high dimensional data information is distributed in interval (Nkey _I-1, Nkey _i] in;

(5.3) each node N _iSafeguard a routing table, namely pointer gauge, point to other nodes on ring; M list item arranged in routing table, wherein (it is that on the Chord ring, identifier is equal to or greater than (Nkeyi+2 that 1≤k≤m) goes list item to k ^K-1) mod2 ^mFirst node, i.e. successor ((Nkey _i+ 2 ^K-1) mod2 ^m); Any one node receives when key word is the request of key, checks that at first whether self node equals key, if it is directly returns; Otherwise node is searched its routing table, find in table maximum but be no more than first node of key, and inquiry request is transmitted to this node, repeat this process, until request arrives a node N _k, meet key and be positioned at N _kAnd N _kSubsequent node N _K+1Between the time, node N _kReport its descendant node N _K+1As replying of request;

(5.4) when high dimensional data in certain data gathering district is too intensive while causing node effectively to it, not manage, this accumulation area is split into to two or more new accumulation area, correspondingly on the Chord ring, selects idle node to share the load of origin node; When Sparse in the data gathering district of closing on causes node resource when waste, merge accumulation area, select the XM in the new data gathering district of a conduct in two XM, and another is set to idle node.

7. the semantic intelligent information distribution subscription method based on the P2P technology according to claim 1, is characterized in that, the concrete steps of described step (6) are:

(6.1) subscription table of each node maintenance, record all subscription information relevant to this node, when node has event information to arrive, the subscription table of inquiring about this node, if certain subscription information semantic matches success in event information and subscription table, by the event active push to the user;

(6.2) as the subscriber to upper certain the node N of ring _iWhile submitting subscribe request information to, this node searching global search tree is determined and the semantic relevant agent node of subscribe request, this subscribe request is routed to agent node, and carry out based on subscribing to semantic similarity algorithm on agent node, if search and the semantic identical event information of subscribe request, by the event information back transfer to N _iSimultaneously, agent node is stored in subscribe request, subscription condition, routed path and hunting zone information in its subscription table;

(6.3) as the publisher to upper certain the node N of ring _jDuring the issue event information, this node searching global search tree is determined and the semantic relevant host's node of event information, according to the Chord Routing Protocol, this event information is routed to and is stored on host's node; Simultaneously, host's node carries out semantic matches by the subscribe request information in event attribute information and its subscription table, if the match is successful directly event information is pushed to the subscriber;

(6.4) when the subscriber cancels subscriptions information to upper certain the node issue of ring, this node searching global search tree is determined and the semantic relevant agent node of subscription information, and solicited message is routed to corresponding agent node, in the agent node subscription table, find and delete corresponding subscription information.

Images