CN103646343A - Text based commodity classification treatment method and system - Google Patents

Abstract

The invention provides a text based commodity classification treatment method and system. The text based commodity classification treatment method comprises extracting text format of commodity data from a commodity database; grouping the commodity data and performing feature extraction; constructing out a classifier through a training document according to a probability model; performing classification on the commodity data. According to the text based commodity classification treatment method and system, the automatic classification on the commodity data is achieved and the classification speed is high and the text based commodity classification treatment method and system is suitable for the commodity classification treatment of high-volume electronic commerce websites due to the fact that the machine learning mode is adopted.

Description

A kind of text based commodity classification disposal route and system thereof

Technical field

The present invention relates to internet electronic business field, relate in particular to computer realization text based commodity classification disposal route and the system thereof utilized.

Background technology

Internet development is swift and violent, and increasing commodity are sold by the mode of ecommerce.The division of fast and accurately sold commodity being carried out to classification seems more important.Commodity classification is the needs in order to facilitate consumer to buy, and by the suitable group indication of commodity selection, systematically divides step by step large class, middle class, and group is down to the process of kind, pattern, specification etc.Classification is accurately conducive to more Ordering management commodity of e-commerce venture.And specific to e-commerce website, commodity classification be exactly some newly upper products incorporate into as the existing process of classification under certain.

In current internet electronic business website, still there are a lot of picking systems still by web editor or network seller self, to carry out manual sort.There is following drawback in such mode: 1, a large amount of commodity classification consumption in operation too much human cost.2, along with the continuous expansion of website scale, because artificial selection classification speed is slow, cause a lot of commodity can not upload in time website, thereby missed Transaction apparatus meeting.3, thus manual sort can be because different people causes the inconsistent of classification results to the different understanding of the feature of commodity.

Summary of the invention

For solving the existing problem of above-mentioned prior art, the present invention proposes a kind of text based commodity classification disposal route, and be based upon the system in the method.Proposed further by the existing goods related data in e-commerce website station, use Words partition system and Naive Bayes Classification Algorithm to classify, then indirect labor has proofreaded the picking system being optimized.Method and system of the present invention uses the mode of machine learning, and classification speed is fast, and the commodity classification that is applicable to large capacity e-commerce website is processed.

The present invention adopts following technical scheme: from merchandising database, extract text formatting commodity data, commodity data is divided into groups and carries out feature extraction, utilize training file to go out sorter according to Construction of probability model, commodity data is classified.

Preferably, wherein commodity classification data comprise: classification information and merchandise news.

Wherein classification information comprises: classification ID, category name, parent order ID.

Merchandise news comprises: classification number under commodity ID, descriptive labelling, commodity.

Preferably, the present invention is further comprising the steps: before commodity data is divided into groups, commodity data is carried out to data check.

Preferably, wherein commodity data is carried out to feature extraction and comprise: descriptive labelling is carried out to machine word segmentation processing, form to be sorted that comprises effective vocabulary.

Preferably, wherein to commodity data divide into groups to comprise by commodity data in proportion random packet be training file and test file.

Preferably, wherein probability model is naive Bayesian conversion, and naive Bayesian transformation calculations formula is:

Bayes's total probability formula $p\left(C\right|F_1,...,F_n)=\frac{p\left(C\right)p(F_1,...,F_n|C)}{p(F_1,...,F_n)}.,$ Wherein C represents classification set, F _irepresent a lexical item in descriptive labelling; P (C|F1 ..., Fn) be posterior probability; P (F1 ..., Fn|C) be likelihood function; P (F1 ..., Fn) be evidence; Utilize conditional independence assumption F _ifor condition independently obtains following probability model

$p\left(C\right|F_1,...,F_n)=\frac{1}{Z}p\left(C\right)\underset{i=1}{\overset{n}{\mathrm{\Π}}}p\left(F_i\right|C),$ Thereby utilize maximum a posteriori probability decision rule to go out final sorter from Construction of probability model $\mathrm{classify}(f_1,...,f_n)=\underset{c}{\arg \max }p(C=c)\underset{i=1}{\overset{n}{\mathrm{\Π}}}p(F_i=f_i|C=c).$

P (C=c) is the ratio of this classification commodity amount and entire service quantity in training file, and p (Fi=fi|C=c) is F for training the key word that in file, this classification comprises _iquantity and this classification in whole quantity ratio of key words, by calculating, most probable classification is the classification of sorter probable value maximum.

Preferably, the present invention also further comprises to commodity classification result is carried out error correction, renewal and sorter is carried out to timing training.

The invention provides a kind of text based commodity classification disposal system, it is characterized in that comprising: data extraction module, for extracting text formatting commodity data; Packet module, for dividing into groups to commodity data; Characteristic extracting module, for carrying out feature extraction to commodity data; Sorter, utilizes training file according to probability model, commodity data to be classified.

Preferably, wherein commodity classification data comprise: classification information and merchandise news.

Wherein classification information comprises: classification ID, category name, parent order ID.

Merchandise news comprises: classification number under commodity ID, descriptive labelling, commodity.

Preferably, above-mentioned commodity classification disposal system, further comprises data preprocessing module, before commodity data is divided into groups, commodity data is carried out to data check.

Preferably, in above-mentioned commodity classification disposal system, commodity data is carried out to feature extraction and comprise: descriptive labelling is carried out to machine word segmentation processing, form to be sorted that comprises effective vocabulary.

Preferably, above-mentioned commodity classification disposal system, wherein to commodity data divide into groups to comprise by commodity data in proportion random packet be training file and test file.

Preferably, above-mentioned commodity classification disposal system, further comprises to commodity classification result is carried out error correction, renewal and sorter is carried out to timing training.

Preferably, above-mentioned commodity classification disposal system, further comprises sort interface module, is used to other classification application that interface interchange is provided.

The present invention has realized commodity classification computer processing method and has been based upon the system in the method.Than prior art, technical scheme of the present invention do not need artificial a large amount of participations, utilize Construction of probability model sorter, take full advantage of machine learning ability, can to commodity data, classify quickly and accurately.

Accompanying drawing explanation

The present invention is further illustrated in conjunction with the embodiments with reference to the accompanying drawings.

Fig. 1 shows the commodity classification process flow figure of the preferred embodiment according to the present invention.

Fig. 2 shows the commodity classification disposal system block diagram of the preferred embodiment according to the present invention.

Embodiment

Below with diagram the principle of the invention accompanying drawing together with the detailed description to one or more embodiment of the present invention is provided.In conjunction with such embodiment, describe the present invention, but the invention is not restricted to any embodiment.Scope of the present invention is only defined by the claims, and the present invention contain manyly substitute, modification and equivalent.Set forth in the following description many details to provide thorough understanding of the present invention.These details are provided for exemplary purposes, and also can realize the present invention according to claims without some or all details in these details.

The basis of the technical program is: e-commerce website itself has accumulated a large amount of commodity, and has the institutional framework of commodity classification.With reference to figure 1, specific embodiment of the invention scheme is as follows:

1.1 merchandise newss are collected

From merchandising database, extract classification information and merchandise news generation text file format, the operation of convenient program afterwards, directly from local file reading out data, is accelerated travelling speed.Classification packets of information contains 3 column information < classification ID, category name, and parent order ID>, wherein parent order ID is 0 expression root classification.For example items for information can be following form:

<014006002,Bras,014006>

Can be directly in a tree structure of internal memory structure according to this file.

Merchandise news comprises 3 column informations: < commodity ID, and descriptive labelling, classification > under commodity, for example merchandise news can be following form:

<136202000,Car?nail?clippers?Cartoon?nail?clipper?ABS+steel?Materials?OPP?packing,018009005>

1.2 raw information pre-service

In the information of collecting in 1.1, need to be to data check.If classification information gives up and can not find the forlorn classification of parent object and the infull classification of data, to guarantee to generate complete classification information.In merchandise news, carry out equally data check, give up the merchandise news entry that data are incomplete.

1.3 merchandise news groupings

Commodity data after 1.2 step process is assigned to two groups in proportion at random, 90% commodity data is generated to commodity data training file product.train.10% commodity data is generated to commodity data test file product.test.Wherein aforementioned proportion can according to circumstances be set arbitrarily.

1.4 feature extraction

Merchandise news in 1.3 is carried out to feature extraction: product description is carried out to machine word segmentation processing, and word segmentation processing can incorporate the affiliated industry dictionary of website accumulation.Filter the invalid vocabulary unit such as stop-word.Word segmentation processing is to be mainly severally can explain semantic word by continuous text message cutting, for example, descriptive labelling: " Car nail clippers Cartoon nail clipper ABS+steel Materials OPP packing " will provide following word by word segmentation processing and filtration: " Car; nail clipper; Cartoon; ABS+steel, OPP ".Finally form to be sorted of comprising each effective key word.

1.5 Naive Bayes Classification

As follows according to Bayesian formula and conversion thereof:

$p\left(C\right|F_1,...,F_n)=\frac{p\left(C\right)p(F_1,...,F_n|C)}{p(F_1,...,F_n)}.,$ .(formula one).Formula one is Bayes's total probability formula, and C represents classification set, F _ia feature of presentation class, is also a lexical item in descriptive labelling, and P (C|F1 ..., Fn) be posterior probability; P (F1 ..., Fn|C) be likelihood function; P (F1 ..., Fn) be evidence.In reality, we are only concerned about the molecular moiety in formula one, because denominator does not rely on C and F _ivalue be given.Utilize conditional independence assumption F _ibetween be condition independently.Obtain a probability model (formula two) $p\left(C\right|F_1,...,F_n)=\frac{1}{Z}p\left(C\right)\underset{i=1}{\overset{n}{\mathrm{\&Pi;}}}p\left(F_i\right|C)$ (formula two).Use maximum a posteriori probability decision rule, can go out final sorter from Construction of probability model, the formula three of classifying. $\mathrm{classify}(f_1,...,f_n)=\underset{c}{\arg \max }p(C=c)\underset{i=1}{\overset{n}{\mathrm{\&Pi;}}}p(F_i=f_i|C=c).$ (formula three), wherein p (C=c) is the ratio of this classification commodity amount and entire service quantity in training file, p (Fi=fi|C=c) is F for training the key word that in file, this classification comprises _iquantity and this classification in whole quantity ratio of key words, by calculating, most probable classification is that formula three the right are worth maximum classification.

1.6 sorter training

According to the commodity data product.train file of 1.3 generations, it is training sample, the naive Bayesian formula of application 1.5 can generate sorter, and groundwork is to calculate the frequency of occurrences and each characteristic attribute of each classification in training sample to divide the conditional probability estimation to each classification.Its input is characteristic attribute and training sample, and output is sorter.

1.7 sorter evaluation and tests

According to the commodity data product.test file of 1.3 generations, it is test sample book, suppose that commodity data place is categorized as correct classification, apply above-mentioned sorter data are carried out to verification, through test, on the commodity data of millions and the basis of three Chiba subcategories, get in most probable 5 classification situations, conventionally can reach more than 96% accuracy rate.

1.8 artificial error correction, regularly training

Along with the operation of website and the passing of time, website itself can produce the commodity of a part of misclassification, and the department that need to have similar products to patrol and examine, can repartition correct class now by the commodity of misclassification at any time.In addition, classification structure is transition at any time also, when the transition of classification information, need to record the corresponding relation of old classification and new classification, so that sorter can be transformed into new classification while pushing old classification in time.Sorter itself also will regularly be trained, and according to the data variation rate of website product, a generalized case training in month once.

1.9 produce utilization disaggregated model

According to the disaggregated model of above-mentioned generation, in Web App, add disaggregated model, be written into internal memory, provide and need the application of online classification business that interface interchange is provided to other.

The present invention also provides a kind of text based commodity classification disposal system in addition, as shown in Figure 2, and comprising data extraction module, for extracting text formatting commodity data; Packet module, for dividing into groups to commodity data; Characteristic extracting module, for carrying out feature extraction to commodity data; Sorter, utilizes training file according to probability model, commodity data to be classified.

Above-mentioned commodity classification disposal system also comprises data preprocessing module, before commodity data is divided into groups, commodity data is carried out to data check.And comprise sort interface module, be used to other classification application that interface interchange is provided.

Disclosed content is only preferably embodiment of the present invention above; but protection scope of the present invention is not limited to this; anyly be familiar with those skilled in the art in the technical scope that the present invention discloses; the variation that can expect easily or replacement, within all should being encompassed in protection scope of the present invention.Therefore, protection scope of the present invention should be as the criterion with the protection domain of claim.

Claims (21)

1. a text based commodity classification disposal route, is characterized in that,

From merchandising database, extract text formatting commodity data, commodity data is divided into groups and carries out feature extraction, utilize training file to go out sorter according to Construction of probability model, commodity data is classified.

2. method according to claim 1, wherein said text formatting commodity data comprises: classification information and merchandise news.

3. method according to claim 2, wherein said classification information comprises classification ID, category name and parent order ID.

4. method according to claim 2, wherein said merchandise news comprises classification number under commodity ID, descriptive labelling and commodity.

5. method according to claim 1, further comprising the steps: before commodity data is divided into groups, commodity data to be carried out to pre-service.

6. method according to claim 4, wherein carries out feature extraction to commodity data and comprises: descriptive labelling is carried out to machine word segmentation processing, form to be sorted that comprises effective vocabulary.

7. method according to claim 1, wherein to commodity data divide into groups to comprise by commodity data in proportion random packet be training file and test file.

8. method according to claim 1, wherein probability model is naive Bayesian conversion.

9. method according to claim 8, wherein naive Bayesian transformation calculations formula is:

Bayes's total probability formula $p\left(C\right|F_1,...,F_n)=\frac{p\left(C\right)p(F_1,...,F_n|C)}{p(F_1,...,F_n)}.,$ Wherein C represents classification set, F _irepresent a lexical item in descriptive labelling; P (C|F1 ..., Fn) be posterior probability; P (F1 ..., Fn|C) be likelihood function; P (F1 ..., Fn) be evidence; Utilize conditional independence assumption F _ifor condition independently obtains following probability model

$p\left(C\right|F_1,...,F_n)=\frac{1}{Z}p\left(C\right)\underset{i=1}{\overset{n}{\mathrm{\&Pi;}}}p\left(F_i\right|C),$ Thereby utilize maximum a posteriori probability decision rule to go out final sorter from Construction of probability model $\mathrm{classify}(f_1,...,f_n)=\underset{c}{\arg \max }p(C=c)\underset{i=1}{\overset{n}{\mathrm{\&Pi;}}}p(F_i=f_i|C=c).,$ Wherein p (C=c) is the ratio of this classification commodity amount and entire service quantity in training file, and p (Fi=fi|C=c) is F for training the key word that in file, this classification comprises _iquantity and this classification in whole quantity ratio of key words, by calculating, most probable classification is the classification of sorter probable value maximum.

10. method according to claim 1, further comprises to commodity classification result is carried out error correction, renewal and sorter is carried out to timing training.

11. 1 kinds of text based commodity classification disposal systems, is characterized in that comprising:

Data extraction module, for extracting text formatting commodity data;

Packet module, for dividing into groups to commodity data;

Characteristic extracting module, for carrying out feature extraction to commodity data;

Sorter, utilizes training file according to probability model, commodity data to be classified.

12. systems according to claim 11, described text formatting commodity data comprises: classification information and merchandise news.

13. systems according to claim 12, wherein said classification information comprises classification ID, category name and parent order ID.

14. systems according to claim 12, wherein said merchandise news comprises the affiliated classification number of commodity ID, descriptive labelling and commodity.

15. systems according to claim 11, further comprise data preprocessing module, before commodity data is divided into groups, commodity data are carried out to data check.

16. systems according to claim 14, wherein carry out feature extraction to commodity data and comprise: descriptive labelling is carried out to machine word segmentation processing, form to be sorted that comprises effective vocabulary.

17. systems according to claim 1, wherein to commodity data divide into groups to comprise by commodity data in proportion random packet be training file and test file.

18. systems according to claim 1, wherein probability model is naive Bayesian conversion.

19. systems according to claim 18, wherein naive Bayesian transformation calculations formula is:

Bayes's total probability formula $p\left(C\right|F_1,...,F_n)=\frac{p\left(C\right)p(F_1,...,F_n|C)}{p(F_1,...,F_n)}.,$ Wherein C represents classification set, F _irepresent a lexical item in descriptive labelling; P (C|F1 ..., Fn) be posterior probability; P (F1 ..., Fn|C) be likelihood function; P (F1 ..., Fn) be evidence; Utilize conditional independence assumption F _ifor condition independently obtains following probability model

$p\left(C\right|F_1,...,F_n)=\frac{1}{Z}p\left(C\right)\underset{i=1}{\overset{n}{\mathrm{\&Pi;}}}p\left(F_i\right|C),$ Thereby utilize maximum a posteriori probability decision rule to go out final sorter from Construction of probability model $\mathrm{classify}(f_1,...,f_n)=\underset{c}{\arg \max }p(C=c)\underset{i=1}{\overset{n}{\mathrm{\&Pi;}}}p(F_i=f_i|C=c).,$ Wherein p (C=c) is the ratio of this classification commodity amount and entire service quantity in training file, and p (Fi=fi|C=c) is F for training the key word that in file, this classification comprises _iquantity and this classification in whole quantity ratio of key words, by calculating, most probable classification is the classification of sorter probable value maximum.

20. systems according to claim 11, further comprise to commodity classification result are carried out error correction, renewal and sorter is carried out to timing training.

21. systems according to claim 11, further comprise sort interface module, are used to other classification application that interface interchange is provided.

Images