CN105205107A - Internet of Things data similarity processing method - Google Patents

Abstract

The present invention provides a method for processing data similarity of the Internet of Things, comprising the following steps: obtaining multiple product records, selecting a first product record and a second product record with multiple identical attributes; storing the attributes of the first product record in In the first array, the attributes of the second product record are stored in the second array; each attribute of the first product record and the second product record is respectively calculated according to the corresponding attribute function; according to the first product Record and the second product record the importance of each attribute, and calculate the weight value of each attribute through the weight function; combine the third array of attribute similarity values and the fourth array of weight values, and calculate the first product record through the overall similarity function The overall similarity to the second product record. This application calculates the overall similarity of two product records with the same attribute according to their respective attribute similarity and attribute weight value, which has a fast processing speed and can save a lot of time and cost.

Description

A method for processing similarity of Internet of Things data

technical field

The invention relates to the field of data processing, in particular to a method for processing data similarity of the Internet of Things.

Background technique

Since the emergence of the Internet, the number of WEB pages on the Internet has grown rapidly, and it is precisely because of this growth rate that the world's largest information resource library has been formed. WEB information integration technology is to effectively process this information resource library, integrate relevant information, and provide data support for data mining, so that it can be better applied to information services in professional fields. In the current rapidly developing network age, information resources are increasingly abundant, and WEB information integration has become an important content in the information age, and WEB information integration has applications in many fields.

For example, in the field of the Internet of Things, product suppliers can publish product information through multiple WEB trading platforms, and buyers can obtain information from WEB trading platforms, and can contact product suppliers through the information released by product suppliers. In this process, it involves the processing of a large amount of data. However, because each WEB trading platform expresses information in different ways, it brings certain difficulties to information integration. In addition, when the same product supplier releases the same product on different WEB trading platforms, it may appear in different forms, which will cause the use of data crawlers to obtain data on these WEB product trading platforms, which will generate a lot of duplicate data. Therefore, It is very necessary to clean the duplicate data of product data from different WEB data sources and with different expressions, which is an important guarantee for judging whether there is duplicate data through the machine.

In the cleaning process of product data, the most important thing is to clear similar duplicate records in multiple product records, so as to ensure the establishment of a comprehensive, accurate, professional, and data quality-compliant product database; at this time, multiple records need to be Similarity calculation. At present, the calculation of data similarity is mainly realized by one-to-one comparison, which is very slow and consumes a lot of time.

Contents of the invention

In view of the defects and various deficiencies of the prior art described above, the technical problem to be solved by the present invention is to provide a method for processing similarity of IoT data that can save a lot of time and cost.

In order to achieve the above object, the present invention provides a method for processing similarity of Internet of Things data, comprising the following steps:

S1. Obtain multiple product records from the WEB trading platform, and select two product records with multiple identical attributes, namely the first product record and the second product record;

S2. Store the attributes of the first product record in the first array, and store the attributes of the second product record in the second array;

S3. Calculate corresponding attribute similarity values for each attribute of the first product record and the second product record according to the corresponding attribute function, and store the attribute similarity values of multiple attributes in the third array;

S4. According to the importance of each attribute of the first product record and the second product record, and calculate the weight value of each attribute through a weight function, and store the weight values of multiple attributes in the fourth array;

S5. Combining the third array of attribute similarity values and the fourth array of weight values, calculate the overall similarity between the first product record and the second product record through the overall similarity function.

Further, in the step S3, the attribute functions include a product nickname matching strategy function, a product price conversion matching strategy function, a normalized date matching strategy function, a normalized origin matching strategy function and an edit distance algorithm function.

Preferably, in the step S2, the attributes of the first product record are successively put into multiple first attribute arrays in the order of product name, price, production date, and place of production, and multiple first attribute arrays constitute the first array .

Preferably, in the step S2, the attributes of the second product record are successively put into multiple second attribute arrays in the order of product name, price, production date, and place of production, and multiple second attribute arrays constitute the second array .

A kind of Internet of things data similarity processing method that the present invention relates to has following beneficial effect:

This application calculates the overall similarity of two product records with the same attribute according to their respective attribute similarity and attribute weight value. The processing speed is fast and the calculation accuracy is high, thereby saving a lot of time and cost.

The above description is only an overview of the technical solution of the present invention. In order to understand the technical means of the present invention more clearly and implement it according to the contents of the specification, the patent will be described in detail below with preferred embodiments of the present invention and accompanying drawings.

Description of drawings

Fig. 1 is the flow chart of this application.

FIG. 2 is a flow chart of the product alias matching strategy function in this application.

Fig. 3 is a flow chart of the product price conversion matching strategy function in this application.

Fig. 4 is a flow chart of the normalized date matching strategy function in this application.

Fig. 5 is a flow chart of the standardized origin matching strategy function in this application.

detailed description

Preferred embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings.

As shown in Figure 1, the present invention provides a kind of data similarity processing method, comprises the following steps:

S1. Obtain multiple product records from the WEB trading platform, and select two product records with multiple identical attributes, namely the first product record A and the second product record B.

S2. Store the attributes of the first product record A in the first array a[], and store the attributes of the second product record B in the second array b[].

Both the first product record A and the second product record B have n attributes, so the first array a[] consists of n first attribute arrays a[0], a[1], a[2], a[3] , a[4]～a[n], the second array b] consists of n second attribute arrays b[0], b[1], b[2], b[3], b[4]～b [n] constitute. At the same time, multiple attributes of the first product record A are successively stored in the first attribute array a[0], a[1], a[2], a[3] according to the order of product name, price, production date and place of origin ], and the first attribute array a[4]～a[n] is used to save other secondary attributes of the first product record A; similarly, multiple attributes of the second product record B are classified according to product name, price, production The order of date and place of origin is successively stored in b[0], b[1], b[2], b[3] in the first attribute array, while the second attribute array b[4]~b[n] is used Other secondary attributes for saving the second product record B.

S3. For each attribute of the first product record A and the second product record B, calculate the corresponding attribute similarity value according to the corresponding attribute function, and save the attribute similarity values of multiple attributes in the third array c[] , the third array c[] is an array of double type.

In the step S3, the attribute function includes a product nickname matching strategy function Strategy_Name(), a product price conversion matching strategy function Strategy_Price(), a standardized date matching strategy function Strategy_Date(), a standardized origin matching strategy function Strategy_Origin() and an edit distance algorithm function Edit_Distance().

S4, according to the importance of each attribute of the first product record A and the second product record B, and calculate the weight value of each attribute through the weight function Weight (), and save the weight values of multiple attributes in the fourth array w[] Among them, the fourth array w[] is an array of double type.

S5. Combining the third array c[] of the attribute similarity value and the fourth array w[] of the weight value, calculate the overall similarity Sim of the first product record A and the second product record B through the overall similarity function Sim() (A, B).

This application calculates the overall similarity of two product records with the same attribute according to their respective attribute similarity and attribute weight value. The processing speed is fast and the calculation accuracy is high, thereby saving a lot of time and cost. Therefore, the present invention effectively overcomes various shortcomings in the prior art and has high industrial application value.

Further, as shown in Figure 2, the product alias matching strategy function Strategy_Name () includes the following steps:

N1. Select a set of data from the document and put it into the set S;

N2. Starting from the first element of the collection, each element is stored in a map container in C++STL to form a mapping with the first element;

N3. For the attribute value of the agricultural product name of records A and B, find the corresponding mapping value in the map container, and replace them;

N4. Comparing the replaced agricultural product names, if they are completely equal, the similarity Sim(Ak, Bk)=1, otherwise Sim(Ak,Bk)=0.

Preferably, as shown in Figure 3, the product price conversion matching strategy function Strategy_Price() includes the following steps:

P1. First define a map entity: map<string,double>price;

P2. Execute the following statement to match the unit with the converted value:

price["yuan/kg"]=1;

price["yuan/jin"]=2;

price["yuan/kg"]=1;

price["yuan/1000g"]=1;

price["yuan/500g"]=2;

price["yuan/100g"]=10;

price["Yuan/gram"]=1000;

price["yuan/ton"]＝0.001.

The meaning is that if "x yuan/kg" is to be converted into the unit "yuan/kg", x must be multiplied by 1, and if "x yuan/jin" is to be converted into the unit "yuan/kg", it must be Multiply x by 2, and so on;

P3. For the price attribute value Ak of record A, first separate the value and unit of the price. The method of segmentation is to start from the first character of the string and search backwards until the first one that does not belong to '0' is found. Between '9' and not the character p[i] of '.', at this time, the part from p[0] to p[i] is the value of the price, save them in the string a, and the rest One part is the unit, save them in the string b;

P4. Use the atof() function to convert the string a into a double value, and store it in the double variable c1;

P5. Execute the c1*=price[b] statement to multiply c1 by the conversion value of unit b and save it in c1, at this time c1 is the value of the converted input price;

P6. Use the same method for record B to obtain the final converted value c2 of the price attribute value Bk;

P7. Determine whether the value of c1-c2<=0.000001 is true to determine whether the two input prices are the same. Sim(Ak,Bk)=1 if true, Sim(Ak,Bk)=0 otherwise.

Further, as shown in Figure 4, the normalized date matching strategy function Strategy_Date () includes the following steps:

D1. Search from the first character of r1 to the back, if you find a character r1[i] that does not belong to '0'-'9', then it is the first separator, convert it to' /', that is, r1[i]='/', at this time, the year is from the first character to the i-1th character;

D2. If r1[i+1] is not the character '0', then go directly to step 3; if r1[i+1] is the character '0', then start from the i+2 position to the end of the string and put them all Move forward one bit, that is, r1[i+1, i+2...]=r1[i+2, i+3....];

D3. Save the value of i+1 into j, start from the i+1th character, and search backwards one by one until a character r1[i] that does not belong to '0'-'9' is found, then It is the second separator, convert it to '/', that is, r1[i]='/', at this time, the month from the jth character to the i-1th character is the month, and the prefix 0 has been removed the month of

D4, repeat step D2 to eliminate the prefix 0 in the date number; here, the first input date string r1 has completed the transformation of the separator and the elimination of the prefix 0;

D5. For the input date string r2, then use the same method above to complete the processing of r2; after processing, use the formula $Sim(rik,rjk)=\left\{\begin{array}{c}\frac{10-\left|C(rik,rjk)\right|}{10}\left(\right|C\left(rik,rjk\right)|\&le;10)\\ 0\left(\right|C\left(rik,rjk\right)|>10)\end{array}\right.$ Calculate the similarity between r1 and r2.

Further, as shown in Figure 5, the standardized origin matching strategy function Strategy_Origin() includes the following steps:

O1. Create collections Sprov, Scity, and Scoun to save all provincial administrative divisions, city administrative divisions and county administrative divisions respectively;

O2. Segment the origin attribute value of record A into Chinese words, put the divided words into the sets Sprov, Scity, and Scoun to retrieve which level of administrative division it belongs to, to distinguish provinces, cities, and counties, and then let the records of A Provinces, cities, and counties are stored in Aprov, Acity, and Acoun respectively, and NULL is assigned to the missing administrative division level. Do the same process for the origin attribute value of record B, so that the province, city, and county of B are stored in Bprov, Bcity, and Bcoun respectively;

O3. Complete the missing administrative division levels. Use the characteristics of administrative divisions to complete the missing administrative division levels from bottom to top, and do not deal with the parts that cannot be completed.

The data similarity processing method provided by the embodiment of the present invention has been introduced in detail above. For those of ordinary skill in the art, according to the idea of the embodiment of the present invention, there will be changes in the specific implementation and application scope. In summary, the contents of this specification should not be construed as limiting the present invention, and any changes made according to the design concept of the present invention are within the scope of protection of the present invention.

Claims (4)

1. an Internet of Things data similarity disposal route, is characterized in that: comprise the following steps:

S1, from WEB transaction platform, obtain many product records, select two product records with multiple same alike result, be respectively the first product record and the second product record;

S2, the attribute of the first product record is kept in the first array, the attribute of the second product record is kept in the second array;

S3, by corresponding attribute function, corresponding attribute similarity angle value is calculated respectively to each attribute of the first product record and the second product record, and the attribute similarity angle value of multiple attribute is kept in the 3rd array;

S4, calculated the weighted value of each attribute by weighting function according to the significance level of the first product record and each attribute of the second product record, and the weighted value of multiple attribute is kept in the 4th array;

S5, in conjunction with the 3rd array of attribute similarity angle value and the 4th array of weighted value, calculated the overall similarity of the first product record and the second product record by overall similarity function.

2. a kind of Internet of Things data similarity disposal route according to claim 1, it is characterized in that: in described step S3, attribute function comprises product another name matching strategy function, product price conversion matching strategy function, standardization date match policy function, standardization place of production matching strategy function and editing distance algorithmic function.

3. a kind of Internet of Things data similarity disposal route according to claim 1, it is characterized in that: in described step S2, the attribute of the first product record successively puts into multiple first attribute array according to the order in name of product, price, date of manufacture, the place of production, and multiple first attribute array forms described first array.

4. a kind of Internet of Things data similarity disposal route according to claim 1, it is characterized in that: in described step S2, the attribute of the second product record successively puts into multiple second attribute array according to the order in name of product, price, date of manufacture, the place of production, and multiple second attribute array forms described second array.