CN103324705B - Extensive vector field data processing method - Google Patents

Abstract

The present invention relates to a kind of extensive vector field data processing method, comprise the steps:, from External memory equipment, vector field data are read to internal memory one by one; Utilize streaming K-means algorithm to process and obtain several cluster centres to the vector field data in internal memory; Build the neighborhood of cluster centre; According to the neighborhood of cluster centre, cluster centre is carried out to hierarchical clustering. Above-mentioned extensive vector field data processing method, from External memory equipment, vector field data are read to internal memory one by one, avoid the disposable all vector field data internal memory that is all written into, and utilize streaming K-means algorithm to process the vector field data in internal memory and according to the neighborhood of cluster centre, cluster centre carried out hierarchical clustering and also greatly reduced the time complexity of cluster, therefore performance, the request memory of above-mentioned extensive vector field data processing method to computer is all lower, and can process quickly extensive vector field data.

Description

Extensive vector field data processing method

Technical field

The present invention relates to areas of information technology, particularly relate to a kind of extensive vector field data processing method.

Background technology

Along with the fast development of scientific and technical development, particularly computer technology, the mankind produce and obtain numberAccording to ability become the order of magnitude and increase. Wherein Fluid Mechanics Computation simulation, environmental science and material science etc.Extensive vector field data visual that application produces be all the time complexity facing of people andDifficult task. At present relate generally to following two sides for the visual and drafting of extensive vector field dataFace:

One, the processing of large data: mainly adopt the technology based on design of Parallel Algorithms at present, rationally utilizeLimited computational resource, processes and analyzes the characteristic of specific data set efficiently. Towards large-scale dataParallel visual work in relate generally to data flow linearize, tasks in parallel, pipeline parallelization, data alsoFour kinds of basic fundamentals of rowization.

Its two, the expression of vector field: well-known, in the time processing large-scale data, cluster has played to closingImportant effect. The one that wherein people such as AlexandruTelea proposes is carried out hierarchical clustering to vectorial field dataMethod is to process the most classical method of vector field data. This clustering method is adjacent with it by calculating each classThe similitude of class, then carries out cluster successively according to similitude height, finally obtains the cluster result of stratification.The method has been simplified the expression of vector field and to the professional dependence of user and use compared to method in the pastFamily can be by regulating simple parameter to produce flexibly different representative results.

But the large data of parallel processing cannot win very high to the performance requirement of computer, common computerAppoint. And when existing clustering method is processed extensive vector field data, need to be by all large dataAll be written into internal memory, and most computers internal memory is limited, cannot meet this requirement at all. In addition, supposeThe quantity that hierarchical clustering is crossed middle need object to be processed is N, so the time complexity of hierarchical clustering up toO (NlogN) requires a great deal of time in the time processing large data, is not suitable for extensive vectorial number of fieldsAccording to real-time processing drafting present.

Summary of the invention

Based on this, be necessary to provide a kind of lower and process extensive vector to computing power, request memoryField data is extensive vector field data processing method faster.

A kind of extensive vector field data processing method, comprises the steps:

From External memory equipment, vector field data are read to internal memory one by one;

Utilize streaming K-means algorithm to process and obtain several clusters to the vector field data in internal memoryCenter;

Build the neighborhood of cluster centre;

According to the neighborhood of cluster centre, cluster centre is carried out to hierarchical clustering.

In an embodiment, the described streaming K-means algorithm that utilizes is to the vector field data in internal memory thereinThe step of processing and obtain several cluster centres comprises:

(1), from internal memory, read vector field data;

(2), judge whether described vector field data are first vector field data that read, if so,Using described vector field data as new cluster centre, if not, described vector field data and institute calculatedThere is the similarity of cluster centre, and find out the cluster centre the highest with described vector field data similarity;

(3), judge described vector field data and with the highest cluster centre of described vector field data similarityWhether similarity within predetermined threshold value scope, if so, is attached to described vector field data with describedThe highest cluster centre of vector field data similarity, if not, using described vector field data as newCluster centre;

(4), repeating step (1), step (2) and step (3), until the vector field data quilt in internal memoryAll read.

In an embodiment, the described streaming K-means algorithm that utilizes is to the vector field data in internal memory thereinThe step of processing and obtain several cluster centres comprises:

After vector field data in internal memory have all been read, calculate the sum of cluster centre.

Therein in an embodiment, the similarity of the described vector field data of described calculating and all cluster centres,And the step of finding out the cluster centre the highest with described vector field data similarity comprises:

A selected accidental projection vector;

Each cluster centre is obtained to an array with described accidental projection multiplication of vectors respectively, and by described numberAll elements in group sorts by size, and obtains a subordinate ordered array;

Vector field data and described accidental projection multiplication of vectors are obtained to vector field data projection value;

In subordinate ordered array, find out and the immediate element of described vector field data projection value, find out with describedThe highest cluster centre of vector field data similarity.

Therein in an embodiment, if vector field data and with the highest poly-of described vector field data similarityClass center, within predetermined threshold value scope, is attached to described vector field data and described vector field data phaseLike the highest cluster centre of degree.

In an embodiment, the step of the neighborhood of described structure cluster centre comprises therein:

Vector field data in traversal internal memory, as long as adjacent vector field data belong to different cluster centres,Two cluster centres are formed to neighbouring relations.

Therein in an embodiment, describedly according to the neighborhood of cluster centre, cluster centre is carried out to levelThe step of cluster comprises:

Calculate the similarity of all adjacent cluster centres;

Get a pair of cluster centre that similarity is the highest and carry out cluster and obtain a new cluster centre, and upgrade shouldThe similarity of new cluster centre and the cluster centre adjacent with this new cluster centre;

Repeatedly get two cluster centres that similarity is the highest and carry out cluster, until be finally polymerized to a cluster centre.

In an embodiment, before the neighborhood step of described structure cluster centre, also comprise thereinFollowing steps:

Judge whether cluster centre sum is greater than preset value, if so, carry out described structure cluster centreThe step of neighborhood and according to the neighborhood of cluster centre, cluster centre is carried out the step of hierarchical clustering,If not, directly finish, no longer carry out subsequent step.

In an embodiment, described similarity adopts similarity function to calculate, described similarity function thereinAdopt oval isopleth method construct.

Above-mentioned extensive vector field data processing method is read vector field data one by one from External memory equipmentBe taken to internal memory, avoided being all written into internal memory by disposable all vector field data, and utilized streamingK-means algorithm is processed the vector field data in internal memory and according to the neighborhood pair of cluster centreCluster centre carries out hierarchical clustering has also greatly reduced the time complexity of cluster, therefore above-mentioned on a large scale toPerformance, the request memory of amount field data processing method to computer is all lower, and can process quicklyExtensive vector field data.

Brief description of the drawings

Fig. 1 is the algorithm flow chart of the extensive vector field data processing method of an embodiment;

Fig. 2 be an embodiment utilize streaming K-means algorithm to the vector field data place in internal memoryManage and obtain the algorithm flow chart of several cluster centres;

Fig. 3 be an embodiment according to the neighborhood of cluster centre, cluster centre is carried out to hierarchical clusteringAlgorithm flow chart;

Fig. 4 be an embodiment according to the neighborhood of cluster centre, cluster centre is carried out to hierarchical clusteringExemplary plot;

Fig. 5 is the schematic diagram of the oval isopleth method construct of the employing similarity function of an embodiment;

Fig. 6 is the effect that the vector field data of an embodiment obtain through extensive vector field data processing methodFruit figure.

Detailed description of the invention

For solve the extensive vector field data of current processing to computing power, request memory higher and efficiencyLow problem, present embodiment provides a kind of extensive vector field data processing method. Below in conjunction with toolThe embodiment of body, is specifically described extensive vector field data processing method.

Please refer to Fig. 1, the extensive vector field data processing method that present embodiment provides, mainly comprises:

Step S110: from External memory equipment, vector field data are read to internal memory one by one. In this step,Based on stream mode one by one the vector field data in External memory equipment are read to computer inDeposit.

Step S120: utilize streaming K-means algorithm that the vector field data in internal memory are processed and obtainedSeveral cluster centres. Utilize the large-scale vector field data of processing that streaming K-means algorithm can streaming,And the final result obtaining is similar to and utilizes the extensive vector field data of traditional K-Means algorithm process to obtainThe result arriving. K-means algorithm is the very typical clustering algorithm based on distance, adopts distance as similitudeEvaluation index, think that the distance of two objects is nearer, its similarity is just larger. And streaming K-meansAlgorithm, in the time of the similarity of compute vector field data and cluster centre, has been introduced an accidental projection vector, usesIn the complexity that reduces cluster. Please refer to Fig. 2, step S120 specifically comprises:

Step S121: read vector field data from internal memory.

Step S122: judge whether vector field data are first vector field data that read, if vector fieldData are first vector field data that read, using vector field data as new cluster centre, if notBe, the similarity of compute vector field data and all cluster centres, and find out and vector field data similarityThe highest cluster centre. In the present embodiment, the similarity of compute vector field data and all cluster centres,And the step of finding out the cluster centre the highest with vector field data similarity is to realize based on accidental projection vector, be simply described below at this:

Selected a certain accidental projection vector. Each cluster centre is obtained to one with accidental projection multiplication of vectors respectivelyIndividual array. Then all elements in array is sorted by size, obtain a subordinate ordered array. Again by vectorField data and accidental projection multiplication of vectors obtain vector field data projection value. Finally in subordinate ordered array, find outWith the immediate element of vector field data projection value, find out the cluster centre the highest with vector field data similarity.Suppose that n cluster centre obtains an array through after sequence with accidental projection multiplication of vectors respectively here,The subordinate ordered array obtaining is A[A1, A2 ... An], and by vector field data and accidental projection multiplication of vectors obtain toAmount field data projection value is y, as long as use dichotomy can be at subordinate ordered array A[A1, A2 ... An] in fastFind out with the immediate elements A i(Ai of vector field data projection value y and represent i in subordinate ordered array AElement). If vector field data and with the similarity of the highest cluster centre of vector field data similarity defaultWithin threshold range, vector field data should be the cluster centre that Ai is corresponding by attached cluster centre.

Step S123: judge that whether the similarity of vector field data and nearest cluster centre is in predetermined threshold value scopeWithin, if so, vector field data are attached to this nearest cluster centre. In the present embodiment,That vector field data are attached to the cluster centre that Ai is corresponding. If not, using vector field data as newlyCluster centre. By this step, vector field data can be attached to certain cluster centre as much as possible,Merge thereby reach vector field data, that is the object of cluster.

Step S124: repeating step S121, step S122 and step S123, until the vector field in internal memoryData have all been read. By this step, the whole vector field data in internal memory can be processed,And all vector field data are carried out to cluster, finally obtain several cluster centres.

Step S125: after the vector field data in internal memory have all been read, calculate cluster centreSum. Here the sum that obtains cluster centre is mainly used in determining whether carrying out subsequent builds cluster centreNeighborhood step and cluster centre is carried out to hierarchical clustering step according to the neighborhood of cluster centre.

Step S130: judge whether cluster centre sum is greater than preset value, if so, carry out subsequent stepS140 and step S150, if not, directly finish, no longer carry out subsequent step. That is vectorial number of fieldsAccording to not reaching preset value through the cluster centre sum obtaining after abovementioned steps, there is no need vectorField data is further processed, and computer just can be processed and result is informed to user it;If but vector field data have exceeded preset value through the cluster centre sum obtaining after abovementioned steps, thatWe also need the current cluster centre obtaining to be further processed, concrete further processing methodPerform step exactly S140 and step S150.

Step S140: the neighborhood that builds cluster centre. Owing to utilizing streaming K-means algorithm to vectorThe result that field data is carried out cluster does not have neighborhood, need to determine vectorial number of fields and carry out hierarchical clusteringAccording between hierarchical relationship. So we are carrying out needing to build before hierarchical clustering the neighborhood of cluster centre.

The step that builds the neighborhood of cluster centre comprises: the vector field data in traversal internal memory, as long as phaseAdjacent vector field data belong to different cluster centres, these two cluster centres are formed to neighbouring relations. ByTo be endowed neighborhood, institute in based on process uniform grid and that reading in original vector field dataOnly need travel through again one time original vector field data with us, as long as adjacent original vector vector field data belong toDifferent cluster centres, these two cluster centres are adjacent.

Step S150: cluster centre is carried out to hierarchical clustering according to the neighborhood of cluster centre. Please refer to figure3, in present embodiment, according to the neighborhood of cluster centre, cluster centre is carried out wrapping in hierarchical clustering stepDraw together:

Step S151: the similarity of calculating all adjacent cluster centres. Here calculate all adjacent cluster centresSimilarity with compute vector field data is consistent with the method for the similarity of cluster centre before, no longer superfluous at thisState.

Step S152: get a pair of cluster centre that similarity is the highest and carry out cluster and obtain a new cluster centre,And upgrade the similarity of this new cluster centre and the cluster centre adjacent with this new cluster centre. That is everyAfter a pair of cluster centre completes cluster, all need to upgrade immediately new cluster centre and the cluster new with thisThe similarity of the adjacent cluster centre in center.

Step S153: repeatedly get two cluster centres that similarity is the highest and carry out cluster, until be finally polymerized to oneIndividual cluster centre.

Please refer to Fig. 4, according to step S150, cluster centre is carried out to the process of hierarchical clustering and can manage like thisSeparate: suppose to have at first 16 cluster centres, in Fig. 4, represent that with 16 " 0 " initial 16 are gatheredClass center, according to step S151, for these 16 cluster centres, calculates the phase of adjacent cluster centre between twoLike degree. Again according to step S152, the result of calculating from step S151, choose similarity the highest oneCluster centre is carried out to cluster and obtain a new cluster centre. In Fig. 4, suppose that result of calculation shows itIn two adjacent " 0 " (representing two adjacent cluster centres) there is the highest similarity, by this twoIndividual " 0 " is merged into a new cluster centre, and this new cluster centre represents with " 1 ". Then according to stepRapid S153, repeatedly gets two cluster centres that similarity is the highest and carries out cluster, until be finally polymerized to a clusterCenter. In Fig. 4, the cluster centre being finally polymerized to represents with " 15 ".

In the present embodiment, step S120 and step S140 similarity used adopts similarity functionCalculate, similarity function has been considered length, direction and the position of vector field data. We use ellipse etc.Value line method structure similarity function. Please refer to Fig. 5, the v vector shown in Fig. 5 (b) is along x shaft lengthFor L, build oval isopleth equation $f(x,y,t)={\text{\[}\frac{x-c\left(t\right)}{a\left(t\right)}\text{\]}}^2+{\[\frac{y}{b\left(t\right)}\]}^2-1,$ Wherein

$\begin{array}{c}a\left(t\right)=\mathrm{\αt}\\ b\left(t\right)=\mathrm{\βt}\\ c\left(t\right)=L+\mathrm{\γt}\end{array},$ (a, b, c represent that respectively oval major axis, minor axis and center of circle abscissa deduct vVector length, as shown in Fig. 5 (a)).

If a given vectorial w=[x, y], the similarity of this w vector sum v vector can be represented by t $t=\frac{\sqrt{({(x-L)}^2+\frac{{\mathrm{\α}}^2}{{\mathrm{\β}}^2}{y}^2)({\mathrm{\α}}^2-{\mathrm{\γ}}^2)+{\mathrm{\γ}}^2{(x-L)}^2}}{{\mathrm{\α}}^2-{\mathrm{\γ}}^2}-\frac{r(x-L)}{{\mathrm{\α}}^2-{\mathrm{\γ}}^2},$ This formula has been considered vectorial length and sideTo.

In addition, note also vectorial position, such as w3 in Fig. 5 (c) and w1 with v location similarity areEquate. The position of supposing w is (xs, ys). Definition position similarity function is:Wherein d and e control oval ratio.

Finally obtain comprehensive similarity function: L (s, t)=As+ (1-A) t, parameter A is set by the user for adjustingJoint length, direction, the weighing factor of position in similarity.

Please refer to Fig. 6, after extensive vector field data are processed through step S110～step S150, dataAmount will greatly be simplified, and finally computer utilization is specially for the friendship of extensive vector field data visualizationThe software of formula analysis just can be processed extensive vector field data mutually, and extensive vector field data are enteredAfter row clustering processing, paint with arrow or sweeping style and show, efficiently inform in real time client.

Above-mentioned extensive vector field data processing method is read vector field data one by one from External memory equipmentBe taken to internal memory, avoided being all written into internal memory by disposable all vector field data, and utilized streamingK-means algorithm is processed the vector field data in internal memory and according to the neighborhood pair of cluster centreCluster centre carries out hierarchical clustering has also greatly reduced the time complexity of cluster, therefore above-mentioned on a large scale toPerformance, the request memory of amount field data processing method to computer is all lower, and can process quicklyExtensive vector field data.

The above embodiment has only expressed several embodiment of the present invention, and it describes comparatively concrete and detailed,But can not therefore be interpreted as the restriction to the scope of the claims of the present invention. It should be pointed out that for this areaThose of ordinary skill, without departing from the inventive concept of the premise, can also make some distortion andImprove, these all belong to protection scope of the present invention. Therefore, the protection domain of patent of the present invention should be with appendedClaim is as the criterion.

Claims (7)

1. an extensive vector field data processing method, is characterized in that, comprises the steps:

From External memory equipment, vector field data are read to internal memory one by one;

Utilize streaming K-means algorithm to process and obtain several clusters to the vector field data in internal memoryCenter, comprising:

(1), from internal memory, read vector field data;

(2), judge whether described vector field data are first vector field data that read, if so,Using described vector field data as new cluster centre, if not, selected accidental projection vector, willEach cluster centre obtains an array with described accidental projection multiplication of vectors respectively, and by described arrayAll elements sorts by size, and obtains a subordinate ordered array, by vector field data and described accidental projection vectorMultiply each other and obtain vector field data projection value, in subordinate ordered array, find out with described vector field data projection valueApproaching element, finds out the cluster centre the highest with described vector field data similarity;

(3), judge described vector field data and with the highest cluster centre of described vector field data similarityWhether similarity within predetermined threshold value scope, if so, is attached to described vector field data with describedThe highest cluster centre of vector field data similarity, if not, using described vector field data as newCluster centre;

(4), repeating step (1), step (2) and step (3), until the vector field data quilt in internal memoryAll read;

Build the neighborhood of cluster centre;

According to the neighborhood of cluster centre, cluster centre is carried out to hierarchical clustering.

2. extensive vector field data processing method according to claim 1, is characterized in that, described inUtilize streaming K-means algorithm to process and obtain several cluster centres to the vector field data in internal memoryStep comprise:

After vector field data in internal memory have all been read, calculate the sum of cluster centre.

3. extensive vector field data processing method according to claim 1, is characterized in that, if toAmount field data and with the highest cluster centre of described vector field data similarity within predetermined threshold value scope,Described vector field data are attached to the cluster centre the highest with described vector field data similarity.

4. extensive vector field data processing method according to claim 1, is characterized in that, described inThe step that builds the neighborhood of cluster centre comprises:

Vector field data in traversal internal memory, as long as adjacent vector field data belong to different cluster centres,Two cluster centres are formed to neighbouring relations.

5. extensive vector field data processing method according to claim 4, is characterized in that, described inThe step of cluster centre being carried out to hierarchical clustering according to the neighborhood of cluster centre comprises:

Calculate the similarity of all adjacent cluster centres;

Get a pair of cluster centre that similarity is the highest and carry out cluster and obtain a new cluster centre, and upgrade shouldThe similarity of new cluster centre and the cluster centre adjacent with this new cluster centre;

Repeatedly get two cluster centres that similarity is the highest and carry out cluster, until be finally polymerized to a cluster centre.

6. extensive vector field data processing method according to claim 2, is characterized in that, in instituteBefore stating the neighborhood step that builds cluster centre, also comprise the steps:

Judge whether cluster centre sum is greater than preset value, if so, carry out described structure cluster centreThe step of neighborhood and according to the neighborhood of cluster centre, cluster centre is carried out the step of hierarchical clustering,If not, directly finish, no longer carry out subsequent step.

7. according to the extensive vector field data processing method described in claim 1,2,3 or 5, its featureBe, described similarity adopts similarity function to calculate, and described similarity function adopts oval isopleth methodStructure.