CN104699659A - Data processing method, device and system - Google Patents
Data processing method, device and system Download PDFInfo
- Publication number
- CN104699659A CN104699659A CN201310685350.1A CN201310685350A CN104699659A CN 104699659 A CN104699659 A CN 104699659A CN 201310685350 A CN201310685350 A CN 201310685350A CN 104699659 A CN104699659 A CN 104699659A
- Authority
- CN
- China
- Prior art keywords
- matrix
- row
- segmentation
- data processing
- node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Abstract
The embodiment of the invention provides a data processing method and device and a data processing system. The data processing device comprises a first determination unit, a first copying unit and a first calculation unit, wherein the first determination unit is used for segmenting a left matrix and a right matrix to determine the segment information of segments according to the positions of elements and the numbers of the elements in the left and right matrixes; the first copying unit is used for copying the left and right matrixes according to the segment information; the first calculation unit is used for performing multiplication calculation on the left and right matrixes according to a copying result. According to the embodiment of the invention, matrix multiplication expandability can be improved, and matrix multiplication efficiency can be improved.
Description
Technical field
The present invention relates to technical field of data processing, particularly relate to a kind of data processing method, device and system.
Background technology
Matrix multiple calculates and applies in numerous areas, and under large data cases especially instantly, large matrix is multiplied and can be applied to webpage sorting (Page-Rank) algorithm of improvement Search Results, and is applied to the various recommendation service etc. based on large data.
The matrix multiple computing method of one quasi-tradition are that ranks are multiplied method, are namely multiplied by right matrix column with the row of left matrix; Another kind of popular matrix multiple method is block multiplication method, and the ranks that the essence of these class methods remains traditional are multiplied method.
Above it should be noted that, just conveniently to technical scheme of the present invention, clear, complete explanation is carried out to the introduction of technical background, and facilitate the understanding of those skilled in the art to set forth.Only can not think that technique scheme is conventionally known to one of skill in the art because these schemes have carried out setting forth in background technology part of the present invention.
Summary of the invention
The present inventor finds when implementing prior art: to be multiplied method for traditional ranks, memory size and the storage size of single computing node limit the scale of two matrixes be multiplied, in addition, traditional ranks are multiplied the element one_to_one corresponding that method needs in the element in left matrix and right rectangular array, and extensive matrix sparse matrix often, so just cause the significant wastage of computational resource; For popular block multiplication method, because ranks still traditional are in essence multiplied method, therefore, still there is the invalid computation problem in sparse matrix, in addition, the method is also very loaded down with trivial details in logic control.
The embodiment of the present invention provides a kind of data processing method, device and system, its object is to improve the extensibility of matrix multiple and improve the efficiency of matrix multiple.
According to the first aspect of the embodiment of the present invention, provide a kind of data processing equipment, this device comprises:
First determining unit, it is for according to the position of element and the quantity of element in left matrix and right matrix, to described left matrix and described right matrix segmentation, to determine the segment information of described segmentation;
First copied cells, it, for according to described segment information, copies described left matrix and described right matrix;
First computing unit, its result for copying described in basis, to be multiplied with described right matrix calculating to described left matrix.
According to the second aspect of the embodiment of the present invention, provide a kind of data handling system, it has the data processing equipment as described in the first aspect of the embodiment of the present invention.
According to the third aspect of the embodiment of the present invention, provide a kind of data processing method, the method comprises:
According to the position of element and the quantity of element in left matrix and right matrix, to described left matrix and described right matrix segmentation, to determine the segment information of described segmentation;
According to described segment information, described left matrix and described right matrix are copied;
According to the described result copied, described left matrix to be multiplied with described right matrix calculating.
Beneficial effect of the present invention is: left matrix and right matrix are carried out to segmentation and copy, and carries out matrix multiple calculating according to the result copied, and thus, improves the extensibility of matrix multiple, and improves the efficiency of matrix multiple.
With reference to explanation hereinafter and accompanying drawing, disclose in detail particular implementation of the present invention, specifying principle of the present invention can adopted mode.Should be appreciated that, thus embodiments of the present invention are not restricted in scope.In the spirit of claims and the scope of clause, embodiments of the present invention comprise many changes, amendment and are equal to.
The feature described for a kind of embodiment and/or illustrate can use in one or more other embodiment in same or similar mode, combined with the feature in other embodiment, or substitutes the feature in other embodiment.
Should emphasize, term " comprises/comprises " existence referring to feature, one integral piece, step or assembly when using herein, but does not get rid of the existence or additional of one or more further feature, one integral piece, step or assembly.
Accompanying drawing explanation
Included accompanying drawing is used to provide the further understanding to the embodiment of the present invention, which constituting a part for instructions, for illustrating embodiments of the present invention, and coming together to explain principle of the present invention with text description.Apparently, the accompanying drawing in the following describes is only some embodiments of the present invention, for those of ordinary skill in the art, under the prerequisite not paying creative work, can also obtain other accompanying drawing according to these accompanying drawings.In the accompanying drawings:
Fig. 1 is the composition schematic diagram of the data processing equipment of the embodiment of the present invention 1;
Fig. 2 is the composition schematic diagram of the first determining unit of the embodiment of the present invention 1;
Fig. 3 is the interrecord structure schematic diagram of text before pre-service in the embodiment of the present invention 1;
Fig. 4 is the interrecord structure schematic diagram of text after pre-service in the embodiment of the present invention 1;
Fig. 5 is the schematic diagram being stored in the left matrix of the second storage unit and the information of right matrix segmentation;
Fig. 6 is the composition schematic diagram of the first copied cells 1 of the embodiment of the present invention 1;
Fig. 7 is the schematic diagram that the downward first nodes of even higher level of node of the embodiment of the present invention 1 transmits control signal;
Fig. 8 is the composition schematic diagram of the first computing unit of the embodiment of the present invention 1;
Fig. 9 is the left matrix of the embodiment of the present invention 1 and the schematic diagram of right matrix multiple;
Figure 10 is the formation schematic block diagram of the data handling system of the embodiment of the present invention 2;
Figure 11 is the process flow diagram of the data processing method of the embodiment of the present invention 3;
Figure 12 is the method flow diagram of the determination segment information of the embodiment of the present invention 3;
Figure 13 is the method flow diagram copied matrix of the embodiment of the present invention 3;
Figure 14 is the method flow diagram be multiplied to matrix of the embodiment of the present invention 3.
Embodiment
With reference to accompanying drawing, by instructions below, aforementioned and further feature of the present invention will become obvious.In the specification and illustrated in the drawings, specifically disclose particular implementation of the present invention, which show the some embodiments that wherein can adopt principle of the present invention, will be appreciated that, the invention is not restricted to described embodiment, on the contrary, the present invention includes the whole amendments fallen in the scope of claims, modification and equivalent.
Embodiment 1
The embodiment of the present invention 1 provides a kind of data processing equipment, and as shown in Figure 1, data processing equipment 100 comprises the first determining unit 101, first copied cells 102 and the first computing unit 103.
Wherein, the first determining unit 101, for according to the position of element and the quantity of element in the left matrix be input in data processing equipment 100 and right matrix, to described left matrix and described right matrix segmentation, to determine the segment information of this segmentation; First copied cells 102, for according to this segment information, copies left matrix and right matrix; First computing unit 103, for according to the result that copies, to be multiplied with right matrix calculating to left matrix.
The data processing equipment of the embodiment of the present invention 1 can be applied in the multiplication operation of extensive matrix, wherein, and the matrix of extensive matrix can be number of elements be 10,000,000,000 magnitudes.
The data processing equipment of the embodiment of the present invention 1 carries out segmentation to the matrix be multiplied, and obtains the segment information of segmentation, and wherein, this segment information can be the hop count that each row of left matrix are segmented, and the hop count that every a line of right matrix is segmented; By each segmentation is copied as many parts, can be multiplied by parallel fragmentation on multiple distributed computing node, thus, the segmentation that large-scale matrix multiple is decomposed on distributed computational nodes is multiplied, improve scale and the counting yield of matrix multiple.
Visible, according to embodiments of the invention 1, left matrix and right matrix carried out segmentation and copied, and carries out matrix multiple calculating according to the result copied, the extensibility of matrix multiple can be improved, and improving the efficiency of matrix multiple.
Fig. 2 is the composition schematic diagram of the first determining unit 101 of the embodiment of the present invention 1.As shown in Figure 2, the first determining unit 101 comprises, pretreatment unit 201, second determining unit 202 and the 3rd determining unit 203.
Wherein, pretreatment unit 201 is for the row number according to element in this left matrix, and the line number of element in this right matrix, carries out pre-service to this left matrix and right matrix; Second determining unit 202, for the result according to pretreatment unit 201, determines the quantity of this each column element of left matrix and the quantity of this right each row element of matrix; 3rd determining unit 203 is for the determination result according to the second determining unit 203, to each row of this left matrix and every a line segmentation of this right matrix, to determine the second information of the first information to each row segmentation of this left matrix and the every a line segmentation to this right matrix.
In the present embodiment, pretreatment unit 201 can according to the expression-form of element in left matrix and right matrix, determine line number and the row number of element, and according to element row number in the row number of element in left matrix and right matrix, carries out pre-service to left square person and right matrix.
In an embodiment, be input to left matrix in data processing equipment 100 and right entry of a matrix element can be stored in text, wherein, a line item of text file can represent an element of left matrix or right matrix.Fig. 3 is the interrecord structure schematic diagram of text before pre-service, and as shown in Figure 3, each line item of text illustrates the position of this element and the numerical value of this element, such as, for record L-1-2
, L-1-2 represents that this element is positioned at left matrix the first row secondary series,
represent the concrete numerical value of this element; In like manner, R-2-1 is recorded
represent the position of element and the numerical value of element that are positioned at the second row first row of right matrix.In addition, when left matrix or right matrix are sparse matrixes, numerical value be 0 element can not be recorded in text file.
In the present embodiment, pretreatment unit 201 can according to each line item of text file, determine the line number of each element in the row number of each element in left matrix and right matrix, and according to the row determined number and line number, change the expression-form of each line item in text file, thus pre-service is carried out to this left matrix and right matrix.Fig. 4 is through the interrecord structure schematic diagram of pretreated text, and as shown in Figure 4, through pre-service, in left matrix, the row number of element are extracted, as index (key) value of this line item in text with the line number of element in right matrix.Such as, for the record L-1-2 in text
pretreatment unit 201 extracts the row number " 2 " of the left matrix element representated by this line item as index, and this index is placed on the beginning of this line item.
It should be noted that, above-mentioned explanation is citing just, element in left matrix and right matrix can have other storage mode and expression-form, also other expression-form can be had through pretreated left matrix and right matrix, as long as the line number of element in the row number of element in left matrix and right matrix can be indicated easily, all meet the description of the embodiment of the present invention.
In the present embodiment, the second determining unit 202 according to pretreated left matrix and right matrix, can determine the quantity of each column element of left matrix and the quantity of right each row element of matrix.Second determining unit 202 can travel through each line item of text, by adding up index (key) value of each line item, determines the quantity of each row element in the quantity of each column element in left matrix and right matrix.Such as, if the second determining unit 202 counts, in the record representing left matrix element, the number of times that index value " k " occurs is M
k, then determine that the quantity of left matrix kth column element is M
k, equally, in the record representing right matrix element, if the number of times that index value " k " occurs is N
k, the quantity of so right matrix row k element is N
k.
In the present embodiment, 3rd determining unit 203 is according to the quantity of the quantity of each column element in left matrix with each section of element preset, segmentation is carried out to each row of left matrix, to determine the first information to each row segmentation of left matrix, and, 3rd determining unit 203 also according to the quantity of each row element in right matrix and the quantity of each section of element preset, to every a line segmentation of right matrix, to determine the second information of the every a line segmentation of right matrix.Wherein, this first information can be the number to each row segmentation of left matrix, and above-mentioned second information can be the number to the every a line segmentation of right matrix.
Such as, when the quantity of each section of element preset is w, 3rd determining unit 203 can travel through each line item of text, in the record representing left matrix kth column element, the element of every w line item representative is divided into one section, thus, the number of the kth column element segmentation determined left matrix is
in addition, the tables of data of i-th segmentation of kth row in left matrix can also be shown as the form as shown in the formula (1):
Wherein, element_list represents the element set in this segmentation;
represent this segment data to need to be replicated in follow-up operation
part, concrete analysis sees below.
In like manner, the number of row k element segmentation that the 3rd determining unit 203 can be determined right matrix is
further, the tables of data of a jth segmentation of row k in right matrix can also be shown as the form of (2) as follows:
Wherein,
represent this segment data to need to be replicated in follow-up operation
part, concrete analysis sees below.
In addition, in the present embodiment, data processing equipment 100 can also have the first storage unit (scheming not shown) and the second storage unit (scheming not shown), wherein, first storage unit can be magnetic disk memory, and it is for storing each segmentation of this left matrix and this right matrix; This second storage unit can be distributed caching, and it is for storing the information of left matrix and right matrix segmentation.
Fig. 5 is the schematic diagram being stored in the left matrix of the second storage unit and the information of right matrix segmentation.Wherein, 1≤k≤r, r is the columns of left matrix, and r is the line number of right matrix.
Fig. 6 is the composition schematic diagram of the first copied cells 102 of the embodiment of the present invention 1.As shown in Figure 6, the first copied cells 102 comprises, the 4th determining unit 601 and the second copied cells 602.
Wherein, the 4th determining unit 601, for according to this first information, determines the number to participating in the node hierarchy copied; Second copied cells 602 has the node participating in copying, and for according to the number to node hierarchy, copies left matrix and right matrix.
In the present embodiment, the 4th determining unit 601 according to the maximal value in each row of left matrix and the every a line division number of right matrix, can determine the number that in the second copied cells 602, replica node is graded.Such as, the 4th determining unit 601 according to following formula (3), can determine the number N that replica node is graded
iteration:
Wherein, n is the ratio of the number of next stage replica node and the number of upper level replica node; N
iterationrepresent in the second copied cells 602, replica node has N
iterationlevel.
In the present embodiment, the number of the classification that the second copied cells 602 can be determined according to the 4th determining unit 601, arranges the total number of replica node.Such as, when the number of first order replica node is default N, when the ratio of the number of next stage replica node and the number of upper level replica node is n, be N in point number of stages of replica node
iterationwhen, N
iterationthe number of level replica node is
the number ading up to every one-level replica node of replica node at different levels is added.Here it should be noted that, due to from the first order to
the number summation of replica node and N
iterationthe number of level replica node is compared, and can ignore, therefore, in the second copied cells 602, the total number of replica node can equal N approx
iterationthe number of level replica node
When being provided with the total number of replica node, first order replica node according to the control information based on the above-mentioned first information, can copy.To the replica node of other grade, process of iteration can be adopted to send control information to every one-level replica node, and control this grade of replica node and copy, namely, next stage replica node can accept, from the control information of the upper level replica node be connected with this next stage replica node, to copy according to this control information.
Wherein, this control information can comprise data to be copied, and such as, data to be copied can be the data of each segmentation in left matrix represented by above-mentioned (1), (2) and right matrix.
In addition, this control information can also be distribute to the identification number copying the data obtained.Second copied cells can distribute this identification number in the following way:
According to the information of left matrix and right matrix segmentation, the identification number distributing to first order replica node can be determined.Due in follow-up matrix multiple computing, (division number is the i-th segment data needs of left matrix kth row with all segmentations in right matrix row k
) be multiplied successively, so need the content replication of kth row in left matrix i-th section
part; In like manner, in right matrix, the jth segment data of row k needs to copy
part.
Such as, above-mentioned (1) can be deformed into following formula (4), thus represent that the data of i-th segmentation of kth row in left matrix need the number copied, and determine the identification number copying the segment data obtained:
According to above-mentioned (4) formula, this segment data needs the number copied to be
the identification number copying the segment data obtained be 1 to
thus, when copying left matrix kth row the i-th segment data, the identification number distributing to each first order node is expressed as
Equally, can be deformed into following formula (5) by above-mentioned (2), thus represent the number that the data needs of a right matrix kth row jth segmentation copy, and determine the identification number copying the segment data obtained:
According to above-mentioned (5) formula, this segment data needs the number copied to be
the identification number copying the segment data obtained be 1 to
thus, when copying right matrix kth row jth segment data, the identification number distributing to each first order node is expressed as
The identification number distributing to self according to the number of the next stage node be attached thereto, can be distributed to the next stage node be attached thereto by each upper level replica node further.Such as, each first order replica node can distribute to the identification number of self
distribute to each second level replica node be connected with this first order replica node.
Fig. 7 is the schematic diagram that the downward first nodes of even higher level of node of the embodiment of the present invention 1 transmits control signal.As shown in Figure 7, suppose
so when i-th segmentation arranged left matrix kth copies, need to copy 10000 times, thus, the identification number distributing to each first order replica node 7100 is 1 to 10000, is expressed as 1#10000.
Next, each first order replica node 7100 sends identification number to 10 second level replica node 7201-7210 be connected with this node.As shown in Figure 7, the identification number of each second level replica node is sent to be respectively " 1#1000 ", " 1001#2000 " ..., " 8001#9000 ", " 9001#10000 ".
Equally, each second level replica node sends identification number to 10 third level replica nodes be connected with this node.As shown in Figure 7, the identification number of the third level replica node 7300-7310 be attached thereto is sent to be respectively " 2001#2100 ", " 2101#2200 " by second level replica node 7203 ..., " 2801#2900 ", " 2901#3000 ".The rest may be inferred, sent to the identification number of the fourth stage replica node 7400-7410 be attached thereto to be respectively " 2101#2110 ", " 2111#2120 " by third level replica node 7302 ..., " 22191#2200 ".
In the second copied cells, each replica node according to the control information received, can copy.In a concrete embodiment, N
iterationlevel replica node according to the identification number received, can also identify for node at the corresponding levels copies the segment data obtained.
Such as, fourth stage replica node 7410 in Fig. 7 is copied to the segment data of the left matrix kth row i-th section obtained, following record (I) can be formed as:
（I）
k-i-2191 element_list
k-i-2192 element_list
……
k-i-2199 element_list
k-i-2200 element_list
Wherein, element_list represents the element in this segment data;
Fourth stage replica node 7410 in Fig. 7 is copied to the segment data of the right matrix row k jth section obtained, following record (II) can be formed as:
（II）
k-2191-j element_list
k-2192-j element_list
……
k-2199-j element_list
k-2200-j element_list
Wherein, element_list represents the element in this segment data.
According to the first copied cells of the embodiment of the present invention, multistage replica node can be made to participate in copying left matrix and right matrix, thus, save the time of copying, improve the efficiency of data processing.
Fig. 8 is the composition schematic diagram of the first computing unit of the embodiment of the present invention 1.As shown in Figure 8, the first computing unit 103 comprises: the second computing unit 801 and the 3rd computing unit 802.
Wherein, the second computing unit 801 is for by each element copying each row in the left matrix that obtains and each element multiplication copying row corresponding in the right matrix that obtains, and to obtain the first result, wherein, the row of these row are number identical with the line number of this row; 3rd computing unit 802 is for suing for peace to this first result, to obtain the second result.
In the present embodiment, the second computing unit 801 can according to afterbody replica node (i.e. N
iterationlevel replica node) copy the segment data of left matrix and the right matrix obtained, the segment data that left matrix kth arranges is carried out gathering and being multiplied with the segment data of right matrix row k, to calculate the first result.
Such as, the second computing unit 801 can converge above-mentioned record (I) and record (II), and the result converged is expressed as following formula (6):
k-i-j element_list_(L，k，i)#element_list_(R，k，j) (6)
Wherein, element_list_ (L, k, i) represents the element of left matrix kth row i-th segmentation, and element_list_ (R, k, j) represents the element of right matrix row k jth segmentation; The result converged that second computing unit 801 can represent according to above-mentioned (6), by certain element L in element_list_ (L, k, i)
v,kwith certain element R in element_list_ (R, k, j)
k,ube multiplied, obtaining the first result, is " v-u value " form by this first outcome record, wherein, v represents the line number of each element in left matrix, and 1≤v≤m, m is the line number of left matrix, and u represents the row number of each element in right matrix, and 1≤u≤n, u is right matrix column number, and v-u represents index (key) value of the first result, and value represents the numerical value of this first result.
In addition, in the second computing unit 801, can comprise multiple computing node, these computing nodes are for completing the multiplication operation between segment data; Certainly, the second computing unit 801 also can complete computing of coordinating between segment data by calls copy node.
In the present embodiment, the 3rd computing unit 802 can have the data summation of the first result of same index (key) value, to obtain the second result.Such as, the 3rd computing unit 802 pairs index value is the numerical value summation of first result of v-u, obtains the second result Tv, u, and wherein, Tv, u represent in the result of left matrix and right matrix multiple the value of the element being positioned at the capable u row of v.
Fig. 9 is the left matrix of the embodiment of the present invention 1 and the schematic diagram of right matrix multiple.As shown in Figure 9, left matrix L has the capable r row of m, and right matrix R has the capable n row of r, the element multiplication that the element of left matrix v capable kth row and right matrix row k u arrange, the kth component of v capable u column element Tv, u in matrix T as a result, by Tv, the important addition of u, namely obtains Tv, the value of u.
Visible, in the embodiment of the present invention 1, left matrix and right matrix are when being multiplied, without the need to each column element one_to_one corresponding of each row element and right matrix of making left matrix, therefore, can be the element of 0 without the need to inputting value in left matrix and right matrix, and avoid the calculating to 0 element, thus reduction calculated amount, especially significantly can reduce calculated amount when sparse matrix is multiplied.
According to embodiments of the invention 1, left matrix and right matrix carried out segmentation and copied, and carries out matrix multiple calculating according to the result copied, thus, improve the extensibility of matrix multiple, and improve the efficiency of matrix multiple; In addition, the embodiment of the present invention 1 can make multistage node participate in copying left matrix and right matrix, thus, saves the time of copying, improves the efficiency of data processing; In addition, the account form that the embodiment of the present invention 1 adopts the row of left matrix to be multiplied with the corresponding row of right matrix, significantly can reduce calculated amount when matrix of coefficients is multiplied; In addition, store to the information of the row of left matrix and the row segmentation of right matrix, the information that each node obtains this segmentation can be convenient to.
Embodiment 2
Inventive embodiments provides a kind of data handling system, and this data handling system comprises data processing equipment as described in Example 1.
Figure 10 is the formation schematic block diagram of the data handling system 1000 of the embodiment of the present invention.As shown in Figure 10, this data processing is 1000 can comprise central processing unit 100 and storer 120; Storer 140 is coupled to central processing unit 100.It should be noted that this figure is exemplary; The structure of other types can also be used, supplement or replace this structure, to realize telecommunications functions or other functions.
In one embodiment, the function of data processing equipment can be integrated in processor 100.Wherein, processor 100 can be distributed processors, and this distributed processors can be the set of the processor of multiple replica node and computing node.In the present embodiment, processor 100 can be configured to:
According to the position of element and the quantity of element in left matrix and right matrix, to described left matrix and described right matrix segmentation, to determine the segment information of described segmentation; According to described segment information, described left matrix and described right matrix are copied; According to the described result copied, described left matrix to be multiplied with described right matrix calculating;
According to the row number of element in described left matrix, and the line number of element in described right matrix, pre-service is carried out to described left matrix and right matrix; According to described pretreated result, determine the quantity of described each column element of left matrix and the quantity of described right each row element of matrix; According to the quantity of described each column element of left matrix and the quantity of described right each row element of matrix, to each row of described left matrix and every a line segmentation of described right matrix, to determine the second information of the first information to each row segmentation of described left matrix and the every a line segmentation to described right matrix, wherein, the described first information comprises the first division number of each the row segmentation to described left matrix, and described second information comprises the second division number of the every a line segmentation to described right matrix;
According to described segment information, determine the number to the node hierarchy copied described in participation; According to the number to described node hierarchy, copy described left matrix and right matrix;
Node described in next stage, according to the control information of node described in from the upper level be connected with described next stage node, copy, and node described in the first order, according to the control information based on described segment information, copies described in carrying out described in carrying out;
By each element copying each row in the left matrix that obtains and each element multiplication copying row corresponding in the right matrix that obtains, to obtain the first result, wherein, the row of described row are number identical with the line number of described row; To described first result summation, to obtain the second result;
Store described segment information.
In another embodiment, data processing equipment can with central processing unit 100 separate configuration, such as data processing equipment can be configured to the computer cluster be connected with central processing unit 100, be realized the function of data processing equipment by the control of central processing unit.
As shown in Figure 10, this subscriber equipment 1000 can also comprise: input block 110, power supply 120.It should be noted that subscriber equipment 1000 is also not necessary to all parts comprised shown in Figure 10; In addition, subscriber equipment 1000 can also comprise the parts do not illustrated in Figure 10, can with reference to prior art.
As shown in Figure 10, central processing unit 100, sometimes also referred to as controller or operational controls, can comprise microprocessor or other processor devices and/or logical unit, and this central processing unit 100 receives input and the operation of all parts of control data disposal system.
Wherein, storer 140 can be such as one or more of in flash memory, hard disk driver, removable medium, volatile memory, nonvolatile memory, distributed caching or other appropriate device.The data of above-mentioned segment information and left matrix and right each segmentation of matrix can be stored, execution program for information about can be stored in addition.And central processing unit 100 can perform this program that this storer 140 stores, to realize information storage or data processing etc.The function of miscellaneous part and existing similar, repeats no more herein.Each parts of data handling system 1000 can be realized by specialized hardware, firmware, software or its combination, and do not depart from scope of the present invention.
Embodiment 3
The embodiment of the present invention 3 provides a kind of data processing method, and corresponding to the data processing equipment that embodiment 1 is somebody's turn to do, the content identical with embodiment 1 repeats no more.
Figure 11 is the process flow diagram of the data processing method of the embodiment of the present invention 3, and as shown in figure 11, the method comprises:
Step 111, according to the position of element and the quantity of element in left matrix and right matrix, to this left matrix and this right matrix segmentation, to determine the segment information of this segmentation;
Step 112, according to this segment information, copies this left matrix and this right matrix;
Step 113, according to the result that this copies, to be multiplied calculating to the right matrix of this left matrix and this.
In addition, in the method, this segment information can also be stored.
Figure 12 is the method flow diagram of the determination segment information of the embodiment of the present invention 3, and as shown in figure 12, the method comprises:
Step 121, according to the row number of element in this left matrix, and the line number of element in this right matrix, pre-service is carried out to this left matrix and right matrix;
Step 122, according to this pretreated result, determines the quantity of this each column element of left matrix and the quantity of this right each row element of matrix;
Step 123, according to the quantity of this each column element of left matrix and the quantity of this right each row element of matrix, to each row of this left matrix and every a line segmentation of this right matrix, to determine the second information of the first information to each row segmentation of this left matrix and the every a line segmentation to this right matrix.
In step 123, this first information comprises the first division number of each the row segmentation to left matrix, and this second information comprises the second division number of the every a line segmentation to right matrix.
Figure 13 is the method flow diagram copied matrix of the embodiment of the present invention 3, and as shown in figure 13, the method comprises:
Step 131, according to this segment information, determines the number to participating in this node hierarchy copied;
Step 132, according to the number to this node hierarchy, copies this left matrix and right matrix.
Wherein, in step 132, first order node, according to the control information based on segment information, copies; Next stage node, according to the control information from the even higher level of node be connected with this next stage node, copies.
Figure 14 is the method flow diagram be multiplied to matrix of the embodiment of the present invention 3, and as shown in figure 14, the method comprises:
Step 141, by each element copying each row in the left matrix that obtains and each element multiplication copying row corresponding in the right matrix that obtains, to obtain the first result, wherein, the row of these row are number identical with the line number of this row;
Step 142, to the summation of this first result, to obtain the second result.
In the present embodiment, the specific works mode of each step 9 please refer to the specific works mode of corresponding units in embodiment 1, repeats no more herein.
According to embodiments of the invention 3, left matrix and right matrix carried out segmentation and copied, and carries out matrix multiple calculating according to the result copied, thus, improve the extensibility of matrix multiple, and improve the efficiency of matrix multiple; In addition, the embodiment of the present invention 3 can make multistage node participate in copying left matrix and right matrix, thus, saves the time of copying, improves the efficiency of data processing; In addition, the account form that the embodiment of the present invention 1 adopts the row of left matrix to be multiplied with the corresponding row of right matrix, significantly can reduce calculated amount when matrix of coefficients is multiplied; In addition, store to the information of the row of left matrix and the row segmentation of right matrix, the information that each node obtains this segmentation can be convenient to.
The embodiment of the present invention also provides a kind of computer-readable program, wherein when performing described program in signal conditioning package or subscriber equipment, described program makes computing machine in described signal conditioning package or subscriber equipment, perform data processing method described in embodiment 3.
The embodiment of the present invention also provides a kind of storage medium storing computer-readable program, and wherein said computer-readable program makes computing machine in signal conditioning package or subscriber equipment, perform data method described in embodiment 3.
The embodiment of the present invention also provides a kind of computer-readable program, and wherein when performing described program in signal conditioning package or base station, described program makes computing machine in described signal conditioning package or base station, perform data processing method described in embodiment 3.
The embodiment of the present invention also provides a kind of storage medium storing computer-readable program, and wherein said computer-readable program makes computing machine in signal conditioning package or base station, perform data processing method described in embodiment 3.
Apparatus and method more than the present invention can by hardware implementing, also can by combination of hardware software simulating.The present invention relates to such computer-readable program, when this program is performed by logical block, this logical block can be made to realize device mentioned above or component parts, or make this logical block realize various method mentioned above or step.The invention still further relates to the storage medium for storing above program, as hard disk, disk, CD, DVD, flash storer etc.
More than in conjunction with concrete embodiment, invention has been described, but it will be apparent to those skilled in the art that these descriptions are all exemplary, is not limiting the scope of the invention.Those skilled in the art can make various variants and modifications according to spirit of the present invention and principle to the present invention, and these variants and modifications also within the scope of the invention.
About the embodiment comprising above embodiment, following remarks is also disclosed:
Remarks 1, a kind of data processing equipment, is characterized in that, described device comprises:
First determining unit, it is for according to the position of element and the quantity of element in left matrix and right matrix, to described left matrix and described right matrix segmentation, to determine the segment information of described segmentation;
First copied cells, it, for according to described segment information, copies described left matrix and described right matrix;
First computing unit, its result for copying described in basis, to be multiplied with described right matrix calculating to described left matrix.
Remarks 2, data processing equipment as described in remarks 1, the first determining unit comprises:
Pretreatment unit, it is for the row number according to element in described left matrix, and the line number of element in described right matrix, carries out pre-service to described left matrix and right matrix;
Second determining unit, it is according to the result of described pretreatment unit, determines the quantity of described each column element of left matrix and the quantity of described right each row element of matrix;
3rd determining unit, it is for the determination result according to described second determining unit, to each row of described left matrix and every a line segmentation of described right matrix, to determine the second information of the first information to each row segmentation of described left matrix and the every a line segmentation to described right matrix.
Remarks 3, data processing equipment as described in remarks 2, wherein, the described first information comprises the hop count of each the row segmentation to described left matrix, and described second information comprises the hop count of the every a line segmentation to described right matrix.
Remarks 4, data processing equipment as described in remarks 1, wherein, described first copied cells comprises:
4th determining unit, it, for according to described segment information, determines the number to the node hierarchy copied described in participation;
Second copied cells, it has the described node copied described in participation, for according to the number to described node hierarchy, copies described left matrix and right matrix.
Remarks 5, data processing equipment as described in remarks 4, wherein, in described second copied cells, node described in next stage, according to the control information of node described in from the upper level be connected with described next stage node, copies described in carrying out.
Remarks 6, data processing equipment as described in remarks 4, wherein, in described second copied cells, node described in the first order, according to the control information based on described segment information, copies described in carrying out.
Remarks 7, data processing equipment as described in remarks 1, wherein, described first computing unit comprises:
Second computing unit, it is for by each element copying each row in the left matrix that obtains and each element multiplication copying row corresponding in the right matrix that obtains, and to obtain the first result, wherein, the row of described row are number identical with the line number of described row;
3rd computing unit, it is for described first result summation, to obtain the second result.
Remarks 8, data processing equipment as described in remarks 1, wherein, described device also comprises:
Storage unit, it is for storing described segment information.
Remarks 9, a kind of data handling system, it has the data processing equipment as described in remarks 1-8.
Remarks 10, a kind of data processing method, is characterized in that, described method comprises:
According to the position of element and the quantity of element in left matrix and right matrix, to described left matrix and described right matrix segmentation, to determine the segment information of described segmentation;
According to described segment information, described left matrix and described right matrix are copied;
According to the described result copied, described left matrix to be multiplied with described right matrix calculating.
Remarks 11, data processing method as described in remarks 10, wherein, according to the position of element and the quantity of element in left matrix and right matrix, to described left matrix and described right matrix segmentation, to determine that the segment information of described segmentation comprises:
According to the row number of element in described left matrix, and the line number of element in described right matrix, pre-service is carried out to described left matrix and right matrix;
According to described pretreated result, determine the quantity of described each column element of left matrix and the quantity of described right each row element of matrix;
According to the quantity of described each column element of left matrix and the quantity of described right each row element of matrix, to each row of described left matrix and every a line segmentation of described right matrix, to determine the second information of the first information to each row segmentation of described left matrix and the every a line segmentation to described right matrix.
Remarks 12, data processing method as described in remarks 11, wherein, the described first information comprises the hop count of each the row segmentation to described left matrix, and described second information comprises the hop count of the every a line segmentation to described right matrix.
Remarks 13, data processing method as described in remarks 10, wherein, according to described segment information, copy described left matrix and described right matrix and comprise:
According to described segment information, determine the number to the node hierarchy copied described in participation;
According to the number to described node hierarchy, copy described left matrix and right matrix.
Remarks 14, data processing method as described in remarks 13, wherein, according to the result of described classification, copy described in carrying out and comprise:
Node described in next stage, according to the control information of node described in from the upper level be connected with described next stage node, copies described in carrying out.
Remarks 15, data processing method as described in remarks 13, wherein, according to the result of described classification, copy described in carrying out and comprise:
Node described in the first order, according to the control information based on described segment information, copies described in carrying out.
Remarks 16, data processing method as described in remarks 10, wherein, according to the described result copied, comprise the calculating that to be multiplied with described right matrix of described left matrix:
By each element copying each row in the left matrix that obtains and each element multiplication copying row corresponding in the right matrix that obtains, to obtain the first result, wherein, the row of described row are number identical with the line number of described row;
To described first result summation, to obtain the second result.
Remarks 17, data processing method as described in remarks 10, it is characterized in that, described method also comprises:
Store described segment information.
Claims (10)
1. a data processing equipment, is characterized in that, described device comprises:
First determining unit, it is for according to the position of element and the quantity of element in left matrix and right matrix, to described left matrix and described right matrix segmentation, to determine the segment information of described segmentation;
First copied cells, it, for according to described segment information, copies described left matrix and described right matrix;
First computing unit, its result for copying described in basis, to be multiplied with described right matrix calculating to described left matrix.
2. data processing equipment as claimed in claim 1, the first determining unit comprises:
Pretreatment unit, it is for the row number according to element in described left matrix, and the line number of element in described right matrix, carries out pre-service to described left matrix and right matrix;
Second determining unit, it is according to the result of described pretreatment unit, determines the quantity of described each column element of left matrix and the quantity of described right each row element of matrix;
3rd determining unit, it is for the determination result according to described second determining unit, to each row of described left matrix and every a line segmentation of described right matrix, to determine the second information of the first information to each row segmentation of described left matrix and the every a line segmentation to described right matrix.
3. data processing equipment as claimed in claim 2, wherein, the described first information comprises the hop count of each the row segmentation to described left matrix, and described second information comprises the hop count of the every a line segmentation to described right matrix.
4. data processing equipment as claimed in claim 1, wherein, described first copied cells comprises:
4th determining unit, it, for according to described segment information, determines the number to the node hierarchy copied described in participation;
Second copied cells, it has the described node copied described in participation, for according to the number to described node hierarchy, copies described left matrix and right matrix.
5. data processing equipment as claimed in claim 4, wherein, in described second copied cells, node described in next stage, according to the control information of node described in from the upper level be connected with described next stage node, copies described in carrying out.
6. data processing equipment as claimed in claim 4, wherein, in described second copied cells, node described in the first order, according to the control information based on described segment information, copies described in carrying out.
7. data processing equipment as claimed in claim 1, wherein, described first computing unit comprises:
Second computing unit, it is for by each element copying each row in the left matrix that obtains and each element multiplication copying row corresponding in the right matrix that obtains, and to obtain the first result, wherein, the row of described row are number identical with the line number of described row;
3rd computing unit, it is for described first result summation, to obtain the second result.
8. data processing equipment as claimed in claim 1, wherein, described device also comprises:
Storage unit, it is for storing described segment information.
9. a data handling system, it has the data processing equipment as described in claim 1-8.
10. a data processing method, is characterized in that, described method comprises:
According to the position of element and the quantity of element in left matrix and right matrix, to described left matrix and described right matrix segmentation, to determine the segment information of described segmentation;
According to described segment information, described left matrix and described right matrix are copied;
According to the described result copied, described left matrix to be multiplied with described right matrix calculating.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310685350.1A CN104699659A (en) | 2013-12-09 | 2013-12-09 | Data processing method, device and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310685350.1A CN104699659A (en) | 2013-12-09 | 2013-12-09 | Data processing method, device and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN104699659A true CN104699659A (en) | 2015-06-10 |
Family
ID=53346800
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310685350.1A Pending CN104699659A (en) | 2013-12-09 | 2013-12-09 | Data processing method, device and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104699659A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109003635A (en) * | 2017-06-07 | 2018-12-14 | 美光科技公司 | Data duplication |
| CN112667741A (en) * | 2020-04-13 | 2021-04-16 | 华控清交信息科技(北京)有限公司 | Data processing method and device and data processing device |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101719934A (en) * | 2009-12-11 | 2010-06-02 | 杭州华三通信技术有限公司 | Method, system and device for displaying uniform summary report on distributed data |
| CN102025756A (en) * | 2009-09-09 | 2011-04-20 | 中兴通讯股份有限公司 | Distributed system and data synchronization method thereof |
| CN103365828A (en) * | 2012-03-27 | 2013-10-23 | 卡西欧计算机株式会社 | Matrix calculation device and matrix calculation method |
-
2013
- 2013-12-09 CN CN201310685350.1A patent/CN104699659A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102025756A (en) * | 2009-09-09 | 2011-04-20 | 中兴通讯股份有限公司 | Distributed system and data synchronization method thereof |
| CN101719934A (en) * | 2009-12-11 | 2010-06-02 | 杭州华三通信技术有限公司 | Method, system and device for displaying uniform summary report on distributed data |
| CN103365828A (en) * | 2012-03-27 | 2013-10-23 | 卡西欧计算机株式会社 | Matrix calculation device and matrix calculation method |
Non-Patent Citations (3)
| Title |
|---|
| 吴志川等: "高度可伸缩的稀疏矩阵乘法", 《CNKI网络出版》 * |
| 孙远帅等: "基于 Hadoop的大矩阵乘法处理方法", 《计算机应用》 * |
| 张骏: "一种基于 MapReduce并行框架的大规模矩阵乘法运算的实现", 《计算机应用与软件》 * |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109003635A (en) * | 2017-06-07 | 2018-12-14 | 美光科技公司 | Data duplication |
| CN109003635B (en) * | 2017-06-07 | 2022-08-02 | 美光科技公司 | Apparatus and method for data replication |
| CN112667741A (en) * | 2020-04-13 | 2021-04-16 | 华控清交信息科技(北京)有限公司 | Data processing method and device and data processing device |
| CN112667741B (en) * | 2020-04-13 | 2022-07-08 | 华控清交信息科技(北京)有限公司 | Data processing method and device and data processing device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10402427B2 (en) | System and method for analyzing result of clustering massive data | |
| CN106970929B (en) | Data import method and device | |
| CN105550225A (en) | Index construction method and query method and apparatus | |
| CN112395293B (en) | Database and table dividing method, database and table dividing device, database and table dividing equipment and storage medium | |
| CN111512283B (en) | Radix estimation in a database | |
| CN104112008A (en) | Multi-table data association inquiry optimizing method and device | |
| EP2786221A2 (en) | Classifying attribute data intervals | |
| US9189489B1 (en) | Inverse distribution function operations in a parallel relational database | |
| CN114297258B (en) | Method and equipment for acquiring comprehensive arrangement data of multi-column data | |
| CN103177414A (en) | Structure-based dependency graph node similarity concurrent computation method | |
| CN107357522A (en) | Data processing method and device | |
| CN103500224A (en) | Data writing method and device and data reading method and device | |
| CN105677645B (en) | A kind of tables of data comparison method and device | |
| CN105224577A (en) | Multi-label text classification method and system | |
| CN106776757B (en) | Method and device for indicating user to complete online banking operation | |
| CN104699659A (en) | Data processing method, device and system | |
| CN110751400B (en) | Risk assessment method and device | |
| WO2024036709A1 (en) | Anomalous data detection method and apparatus | |
| CN116561135A (en) | Multi-feature data cross-query method, device and computer readable storage medium | |
| CN111784246A (en) | Logistics path estimation method | |
| CN105740210B (en) | Information similarity determination method and device | |
| US20210192296A1 (en) | Data de-identification method and apparatus | |
| KR102142767B1 (en) | Method and system for dara clustering using relative distance rate and distance between cluster's medoids | |
| CN114564501A (en) | Database data storage and query methods, devices, equipment and medium | |
| CN105761119A (en) | On-line number of people distribution calculation method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20150610 |
|
| WD01 | Invention patent application deemed withdrawn after publication |