CN101320370A - Deep layer web page data source sort management method based on query interface connection drawing - Google Patents

Abstract

The present invention discloses a deep webpage data source classifying management method based on a query interface connecting picture. The method includes the following procedures: (1) acquiring the form assembly of a deep webpage query interface; (2) automatically extracting the characteristic values of the query interface form acquired in procedure (1) which comprises a form label name and attribute values; (3) constructing characteristic vectors of the form; (4) acquiring an associated adjacent matrix related with label, attribute value and the combination of label and attribute value, through the similarity comparison among vectors; (5) constructing a connecting picture of the query interface form assembly which can be expressed by the associated adjacent matrix; (6) utilizing a clustering method to cluster weighted undirected graphs; (7) acquiring the clustering result of the deep webpage data source. The present invention increases the automatic classifying management performance of the large-scale deep webpage data source through effectively constructing the query interface connecting picture of the deep webpage data source and combining with the graph mining.

Description

Deep layer net page data source sort management method based on the query interface connection layout

Technical field

The present invention relates to the automatic sort management method of a kind of information, be specifically related to a kind of sort management method that is applied to deep layer net page data source.

Background technology

Along with the widespread use of network data base, network quickens " in-depth ".It is dynamically to be produced by background data base that a large amount of pages are arranged on the internet, this part information can not directly be obtained by static linkage, can only submit to inquiry to obtain by filling in list,, can't obtain these pages because traditional web crawlers (Crawler) does not have the ability of filling in list.Therefore, existing search engine searches does not go out this part page info, thus cause this part information to the user be hide, sightless, we are referred to as deep layer Webpage (Deep Web is called Invisible Web again, Hidden Web).Deep Web is one and the corresponding notion of SurfaceWeb, is proposed in 1994 by Dr.Jill Ellsworth at first, refers to those are difficult to find its information content by the general search engine Webpage.Deep Web information generally is stored in the database, and compare common quantity of information with static page bigger, and theme is more single-minded, and information quality is better, and message structureization is better, and growth rate is faster.Studies show that Deep Web information is 500 times of Surface Web information, nearly 450,000 Deep Web websites are arranged.Realize that extensive Deep Web data integration is an effective way that is user-friendly to Deep Web information.

Deep Web data source has isomerism, dynamic, content and covers features such as the field is wide, the Deep Web information of magnanimity can't be obtained by the reptile Automatic Program, can only be with the unique inlet of query interface as the Web database, the user submits inquiry to by filling in list, obtains the inquiry correlated results.In order to effectively utilize the affluent resources that freely are distributed on the Web and in addition integrated automatically, to find the suitable data storehouse easily and retrieve hiding information to help the user, the automatic Classification Management of Deep Web data source seems particularly important.Utilizing manual mode to come organizational information obviously is a very thing of difficulty, research to the query interface feature at present provides opportunity for our present work, research method to the query interface feature awaits further deeply, seeks new method and realizes the automatic Classification Management of Deep Web data source.Classification and cluster are the important method of data source Classification Management in the data integration, at present the approach of query interface research is confined to the Web feature in the taxonomic clustering method and the excavation of text message mostly, do not relate to new approach, being introduced as us and providing new approach to the research of Deep Web data source Classification Management of graph model, this is the field that the researcher does not set foot in as yet.

To scheme to excavate and apply in the Deep Web data source Classification Management research, has following advantage: at first, Deep Web outstanding behaviours is isomerism and autonomy, a large amount of Deep Web inside relate to a plurality of fields, therefore belong to the association that just exists between the Web in same field to a certain extent, the construction process that meets a Web graph model fully, and the definition of each online database is different, be isomery each other, this has just brought inconvenience for our information integration, utilization figure excavation means just can excavate Web each other linked character and classify.Secondly, figure excavates and can excavate a lot of theme features of hiding, and utilizes graph structure so, and we also can find to be hidden in a plurality of fields theme among the Deep Web.

In order to effectively utilize the affluent resources that freely are distributed on the Web, help the user to find the suitable data storehouse easily and retrieve hiding information, need such technology, can realize carrying out automatic Classification Management at extensive Deep Web data source.

Summary of the invention

The object of the invention provides a kind of automatic deep layer net page data source sort management method, utilize the feature-rich of deep layer net page (Deep Web) data source query interface and the method that figure excavates, improve the automatic Classification Management performance of deep layer net page heterogeneous data source, be beneficial to the integrated realization of large-scale data.

For achieving the above object, main design of the present invention is:

We are primarily aimed at structurized query interface and carry out cluster analysis, and structurized query interface comprises a plurality of attribute informations, the list that query interface is normally represented with HTML in Web page.The control that list comprises can divide three major types: INPUT control, SELECT control and TEXTAREA control, the TYPE attribute description of INPUT control the types of elements of input, eight kinds of Text, CheckBox, Radio, Submit, Reset, Radio, Image, Hidden are arranged.But the query interface formalization is defined as F={a ₁, a ₂..., a _n, a _iRepresent the control property on the list.Each control all has corresponding label (Label) to describe, and promptly describes text for one, and each control can have one or more values (value).For example a drop-down list has a plurality of values to select for the user, and radio button and check box have a value usually.Say that in logic a control label related with it has constituted an attribute (attribute), a corresponding field in the Deep Web background data base.A common attribute comprises a label, one or more form controls.We can regard the title (attribute name) of attribute as label in the attribute, and we can regard the value (attribute value) of attribute as the form controls in the attribute.Query interface can formalized description be F={ (L then ₁, V ₁) ..., (L _n, V _n), L wherein _iRepresent label value, V _i={ E _j... E _kThe representative one or more control property values corresponding with label.

Deep layer net page (Deep Web) data source has stronger isomerism and interconnectivity, and therefore, whole Deep Web can abstractly be the graph structure of the many relations of an isomery.Node among the figure is represented the query interface list, and the incidence relation between the query interface list is represented on the limit among the figure, and we represent correlation degree between the two with associated weight value.

According to above-mentioned design, the technical solution used in the present invention is: a kind of deep layer net page data source sort management method based on the query interface connection layout comprises the following steps:

(1) obtains the set of deep webpage query interface list;

(2) eigenwert of the query interface list that obtains of Automatic Extraction step (1), described eigenwert comprises the title and the property value of form tags;

(3) structure form feature vector comprises, with the title of the label that extracts and property value structural attitude Space L S and VS respectively, the characteristic of correspondence vector of characteristic set structure to each list among LS and the VS forms obtains the vector set thus;

(4) in the vector set that step (3) obtains, to calculating the query interface connection layout that obtains about label, property value, label and property value combination by similarity between each vector, available respectively LableMatrix, ValueMatrix, LableValueMatrix adjacency matrix represent that the degree of association computing method between the query interface are:

In calculating based on the list degree of association of label, utilize the quantity of the same characteristic features item of label to weigh, the column criterionization of going forward side by side,

${\mathrm{Sim}}_L(F_1,F_2)=\frac{\mathrm{sw}}{\mathrm{len}}$

Wherein, sw represents list F ₁And F ₂Have the number of same label, len represents F ₁And F ₂The average length of middle label proper vector, both are divided by and carry out standardization, Sim _L(F ₁, F ₂) represent based on label (Label, list F L) ₁And F ₂Associated weight value;

The list degree of association based on property value and label and property value combination is calculated, and we utilize the similarity function between its vector to calculate,

${\mathrm{Sim}}_v(F_1,F_2)=\frac{\underset{k=1}{\overset{n}{\mathrm{\Σ}}}{W}_{1k}\×W_{2k}}{\sqrt{\left(\underset{k=1}{\overset{n}{\mathrm{\Σ}}}{W}_{1k}^2\right)\left(\underset{k=1}{\overset{n}{\mathrm{\Σ}}}{W}_{2k}^2\right)}}$

In the formula, W _1kAnd W _2kRepresent list F respectively ₁And F ₂(Value, the vector representation of V) set formation utilizes vectorial cosine formula to calculate based on the list F1 of property value and the associated weight value Sim of F2 to middle property value _V(F ₁, F ₂).Based on label and property value (Label﹠amp; Value, LV) Zu He list associated weight value Sim _LV(F ₁, F ₂) and Sim _V(F ₁, F ₂) computing method similar, but W wherein _1kAnd W _2kRepresent list F respectively ₁And F ₂Middle label and property value constitute the vector representation of set jointly;

(5) structure query interface connection layout:

With step (4) obtain three matrix L ableMatrix, ValueMatrix, LableValueMatrix be weighted merging, be about to similarity value in above-mentioned three matrixes and be weighted summation as the associated weight value between the query interface list of interconnection in twos; According to the building method of the undirected connection layout of cum rights, as a node among the figure, for setting up a nonoriented edge between the query interface that has certain degree of association, this associated weight value is just as the weights on limit each query interface;

Sim(F ₁，F ₂)＝ω ₁＊Sim _L(F ₁，F ₂)+ω ₂＊SimV(F ₁，F ₂)+ω ₃＊Sim _LV(F ₁，F ₂)

Wherein, ω 1, ω 2, ω 3 are expressed as the weight coefficient that each degree of association component distributes, and its span ω 1 is 0.25～0.35, and ω 2 is 0.15～0.25, and ω 3 is 0.45～0.55, the available optimal value of determining these weights based on genetic algorithm.Sim (F ₁, F ₂) expression query interface list F ₁And F ₂Associated weight value, form a deep webpage query interface connection layout thus, available adjacency matrix FormLinkMatrix represents;

(6) utilize clustering method that the undirected connection layout of query interface list cum rights is carried out cluster;

(7) obtain the deep layer net page data source cluster result, finish.

In the technique scheme, in the described step (4), generate list incidence matrix LableMatrix from Label space vector, dependency value space vector generates list incidence matrix ValueMatrix, generates list incidence matrix LableValueMatrix from label and the property value space vector that combines;

In the technique scheme, in the described step (5), from form tags incidence matrix LableMatrix, list property value incidence matrix ValueMatrix and form tags and the property value incidence matrix LableValueMatrix generated query interface list connection layout that combines.

Because the technique scheme utilization, the present invention compared with prior art has following advantage:

The present invention in conjunction with the figure digging technology, has improved the performance of the automatic Classification Management of extensive Deep Web data source by constructing Deep Web data source query interface connection layout effectively.

Description of drawings

Fig. 1 is based on the deep layer net page Classification Management workflow synoptic diagram of query interface connection layout in the embodiment of the invention;

Fig. 2 is based on the automatic Classification Management process flow diagram of the deep layer net page of query interface connection layout among the embodiment;

Fig. 3 is a query interface form tags incidence matrix structure process flow diagram among the embodiment;

Fig. 4 is a query interface list property value incidence matrix structure process flow diagram among the embodiment;

Fig. 5 is query interface form tags and a property value incidence matrix structure process flow diagram among the embodiment;

Fig. 6 is the organigram of query interface connection layout among the embodiment.

Embodiment

Below in conjunction with drawings and Examples the present invention is further described:

Embodiment one: to shown in the accompanying drawing 6, a kind of deep layer net page data source sort management method based on the query interface connection layout comprises the following steps: referring to accompanying drawing 1

(1) obtains the set of deep webpage query interface list;

(2) eigenwert of the query interface list that obtains of Automatic Extraction step (1), described eigenwert comprises the title and the property value of form tags;

(3) structure form feature vector comprises, with the title of the label that extracts and property value structural attitude Space L S and VS respectively, the characteristic of correspondence vector of characteristic set structure to each list among LS and the VS forms obtains the vector set thus;

(4) in the vector set that step (3) obtains, to calculating the query interface connection layout that obtains about label, property value, label and property value combination by similarity between each vector, available respectively LableMatrix, ValueMatrix, LableValueMatrix adjacency matrix represent that the degree of association computing method between the query interface are:

In calculating based on the list degree of association of label, utilize the quantity of the same characteristic features item of label to weigh, the column criterionization of going forward side by side,

${\mathrm{Sim}}_L(F_1,F_2)=\frac{\mathrm{sw}}{\mathrm{len}}$

Wherein, sw represents list F ₁And F ₂Have the number of same label, len represents F ₁And F ₂The average length of middle label proper vector, both are divided by and carry out standardization, Sim _L(F ₁, F ₂) represent based on label (Label, list F L) ₁And F ₂Associated weight value;

The list degree of association based on property value and label and property value combination is calculated, and we utilize the similarity function between its vector to calculate,

${\mathrm{Sim}}_v(F_1,F_2)=\frac{\underset{k=1}{\overset{n}{\mathrm{\Σ}}}{W}_{1k}\×W_{2k}}{\sqrt{\left(\underset{k=1}{\overset{n}{\mathrm{\Σ}}}{W}_{1k}^2\right)\left(\underset{k=1}{\overset{n}{\mathrm{\Σ}}}{W}_{2k}^2\right)}}$

In the formula, W _1kAnd W _2kRepresent list F respectively ₁And F ₂(Value, the vector representation of V) set formation utilizes vectorial cosine formula to calculate based on the list F1 of property value and the associated weight value Sim of F2 to middle property value _V(F ₁, F ₂).Based on label and property value (Label﹠amp; Value, LV) Zu He list associated weight value Sim _LV(F ₁, F ₂) and Sim _V(F ₁, F ₂) computing method similar, but W wherein _1kAnd W _2kRepresent list F respectively ₁And F ₂Middle label and property value constitute the vector representation of set jointly;

(5) structure query interface connection layout:

With step (4) obtain three matrix L ableMatrix, ValueMatrix, LableValueMatrix be weighted merging, be about to similarity value in above-mentioned three matrixes and be weighted summation as the associated weight value between the query interface list of interconnection in twos; According to the building method of the undirected connection layout of cum rights, as a node among the figure, for setting up a nonoriented edge between the query interface that has certain degree of association, this associated weight value is just as the weights on limit each query interface;

Sim(F ₁，F ₂)＝ω ₁＊Sim _L(F ₁，F ₂)+ω ₂＊SimV(F ₁，F ₂)+ω ₃＊Sim _LV(F ₁，F ₂)

Wherein, ω 1, ω 2, ω 3 are expressed as the weight coefficient that each degree of association component distributes, and its span ω 1 is 0.25～0.35, and ω 2 is 0.15～0.25, and ω 3 is 0.45～0.55, the available optimal value of determining these weights based on genetic algorithm.Sim (F ₁, F ₂) expression query interface list F ₁And F ₂Associated weight value, form a deep webpage query interface connection layout thus, available adjacency matrix FormLinkMatrix represents;

(6) utilize clustering method that the undirected connection layout of query interface list cum rights is carried out cluster;

Traditional division methods is a kind of hard method, being divided in certain class of each pending object strictness, C-mean algorithm for example, its degree of membership are not 1 to be exactly 0, the division of this strictness uncertain membership between the object and class of failing truly to reflect reality in the world.In the fuzzy clustering algorithm of promoting, in membership function, introduced the weight index.We apply to fuzzy clustering (FCM) method in the cluster of query interface connection layout, and its calculating is simple and speed is fast, has geometric meaning more intuitively.With the input of adjacency list form, and the weights of opposite side carry out regularization, through the FCM cluster calculation, obtain the clustering cluster of query interface list with the list connection layout that generated.

(7) obtain the deep layer net page data source cluster result, finish.

Claims (3)

1. the deep layer net page data source sort management method based on the query interface connection layout is characterized in that, comprises the following steps:

(1) obtains the set of deep webpage query interface list;

(2) eigenwert of the query interface list that obtains of Automatic Extraction step (1), described eigenwert comprises the title and the property value of form tags;

(3) structure form feature vector comprises, with the title of the label that extracts and property value structural attitude Space L S and VS respectively, the characteristic of correspondence vector of characteristic set structure to each list among LS and the VS forms obtains the vector set thus;

(4) in the vector set that step (3) obtains, to calculating the query interface connection layout that obtains about label, property value, label and property value combination by similarity between each vector, available respectively LableMatrix, ValueMatrix, LableValueMatrix adjacency matrix represent that the degree of association computing method between the query interface are:

In calculating based on the list degree of association of label, utilize the quantity of the same characteristic features item of label to weigh, the column criterionization of going forward side by side,

${\mathrm{Sim}}_L(F_1,F_2)=\frac{\mathrm{sw}}{\mathrm{len}}$

Wherein, sw represents list F ₁And F ₂Have the number of same label, len represents F ₁And F ₂The average length of middle label proper vector, both are divided by and carry out standardization, Sim _L(F ₁, F ₂) represent based on label (Label, list F L) ₁And F ₂Associated weight value;

The list degree of association based on property value and label and property value combination is calculated, and we utilize the similarity function between its vector to calculate,

${\mathrm{Sim}}_v(F_1,F_2)=\frac{\underset{k=1}{\overset{n}{\mathrm{\Σ}}}{W}_{1k}\×W_{2k}}{\sqrt{\left(\underset{k=1}{\overset{n}{\mathrm{\Σ}}}{W}_{1k}^2\right)\left(\underset{k=1}{\overset{n}{\mathrm{\Σ}}}{W}_{2k}^2\right)}}$

In the formula, W _1kAnd W _2kRepresent list F respectively ₁And F ₂(Value, the vector representation of V) set formation utilizes vectorial cosine formula to calculate based on the list F1 of property value and the associated weight value Sim of F2 to middle property value _V(F ₁, F ₂).Based on label and property value (Label﹠amp; Value, LV) Zu He list associated weight value Sim _LV(F ₁, F ₂) and Sim _V(F ₁, F ₂) computing method similar, but W wherein _1kAnd W _2kRepresent list F respectively ₁And F ₂Middle label and property value constitute the vector representation of set jointly;

(5) structure query interface connection layout:

With step (4) obtain three matrix L ableMatrix, ValueMatrix, LableValueMatrix be weighted merging, be about to similarity value in above-mentioned three matrixes and be weighted summation as the associated weight value between the query interface list of interconnection in twos; According to the building method of the undirected connection layout of cum rights, as a node among the figure, for setting up a nonoriented edge between the query interface that has certain degree of association, this associated weight value is just as the weights on limit each query interface;

Sim(F ₁，F ₂)＝ω ₁＊Sim _L(F ₁，F ₂)+ω ₂＊SimV(F ₁，F ₂)+ω ₃＊Sim _LV(F ₁，F ₂)

Wherein, ω 1, ω 2, ω 3 are expressed as the weight coefficient that each degree of association component distributes, and its span ω 1 is 0.25～0.35, and ω 2 is 0.15～0.25, and ω 3 is 0.45～0.55, the available optimal value of determining these weights based on genetic algorithm.Sim (F ₁, F ₂) expression query interface list F ₁And F ₂Associated weight value, form a deep webpage query interface connection layout thus, available adjacency matrix Form LinkMatrix represents;

(6) utilize clustering method that the undirected connection layout of query interface list cum rights is carried out cluster;

(7) obtain the deep layer net page data source cluster result, finish.

2. the deep layer net page data source sort management method based on the query interface connection layout according to claim 1, it is characterized in that: in the described step (4), generate list incidence matrix LableMatrix from the Label space vector, dependency value space vector generates list incidence matrix ValueMatrix, generates list incidence matrix LableValueMatrix from label and the property value space vector that combines;

3. the deep layer net page data source sort management method based on the query interface connection layout according to claim 1, it is characterized in that: in the described step (5), from form tags incidence matrix LableMatrix, list property value incidence matrix ValueMatrix and form tags and the property value incidence matrix LableValueMatrix generated query interface list connection layout that combines.

Images