US20110213777A1 - Method and Apparatus of Text Classification - Google Patents

Method and Apparatus of Text Classification Download PDF

Info

Publication number
US20110213777A1
US20110213777A1 US12996658 US99665810A US20110213777A1 US 20110213777 A1 US20110213777 A1 US 20110213777A1 US 12996658 US12996658 US 12996658 US 99665810 A US99665810 A US 99665810A US 20110213777 A1 US20110213777 A1 US 20110213777A1
Authority
US
Grant status
Application
Patent type
Prior art keywords
word
vector
category
words
text
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
US12996658
Other versions
US9208220B2 (en )
Inventor
Xiang Sun
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Alibaba Group Holding Ltd
Original Assignee
Alibaba Group Holding Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F17/00Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/30Information retrieval; Database structures therefor ; File system structures therefor
    • G06F17/3061Information retrieval; Database structures therefor ; File system structures therefor of unstructured textual data
    • G06F17/30705Clustering or classification
    • G06F17/30707Clustering or classification into predefined classes

Abstract

The present disclosure provides a technique of text categorization to simplify and optimize the classification. In one aspect, a method parses a given text into one or more words; determines a word vector in a spherical space model for one of the one or more words, a number of dimensions of the spherical space being equal to a number of categories, each category corresponding to a spherical space category vector; for each category, determines a distance between a sum of word vectors of the one or more words and the respective category vector; and classifies the text into one or more categories with the shortest distance. The present disclosure also provides an apparatus used to implement the method.

Description

    CROSS REFERENCE TO RELATED PATENT APPLICATIONS
  • [0001]
    This application is a national stage application of an international patent application PCT/US10/47868, filed Sep. 3, 2010, which claims priority from Chinese Patent Application No. 201010104512.4 filed on Feb. 1, 2010, entitled “Method and Apparatus of Text Classification,” which applications are hereby incorporated in their entirety by reference.
  • TECHNICAL FIELD
  • [0002]
    The present disclosure relates to the field of computer and communications and, more particularly, to the method and apparatus of text classification.
  • BACKGROUND
  • [0003]
    Text classification is an important component of text mining. It is based on predefined subject categories, and files are each assigned to a category. This automatic text classification system can help people find needed information more effectively. In one aspect, classifying information is one of the most fundamental cognitive processes. Moreover, conventional classification researches have produced enriching results and practical uses. Nevertheless, with the rapid growth of text messaging, especially the proliferation of online text messages, text classification is considered the key technology in processing and organizing large quantities of data. At present, text classification is widely used in various fields. However, due to the general increase in web-based information, demands for higher accuracy, and better verification, the demand for text classification technology is also growing. Accordingly, constructing an effective text classification system is still one of the main areas of research in the field of text mining.
  • [0004]
    In the field of natural language processing, texts are mainly represented using the vector space model (VSM). This method considers that each text contains a working concept used to express its independent attribute, and each attribute can be regarded as a dimension of the concept. These independent attributes are called text features wherein the text can express a set of characteristics. Furthermore, vectors often use the cosine of the angle method to measure a degree of similarity. Then the degree of similarity between the text vector and the vector of the candidate category is used to categorize the text.
  • [0005]
    With current technologies, one has to calculate the degree of similarity between the text vector and the candidate category. Each computation is quite lengthy and uses the angle of cosine to arrive at a measurement. Furthermore, the semantics of current technologies do not have any specification, and the classification is not very accurate.
  • SUMMARY OF THE DISCLOSURE
  • [0006]
    The present disclosure provides an exemplary implementation of a method of text categorization used to simplify and optimize the classification.
  • [0007]
    In one aspect, a method of text classification comprises: parsing a given text into one or more words; determining a word vector in a spherical space model for one of the one or more words, a number of dimensions of the spherical space being equal to a number of categories, each category corresponding to a spherical space category vector; for each category, determining a distance between a sum of word vectors of the one or more words and the respective category vector; and classifying the text into one or more categories with the shortest distance.
  • [0008]
    The method may also filter the one or more words to provide one or more filtered words that meet filtering conditions.
  • [0009]
    The distance between the sum of word vectors of the one or more words and the respective category vector may be a straight line or a spherical arc.
  • [0010]
    A word vector of a word may include one or more normalized word frequency values of the word in respective one or more categories, where the spherical space model may include a multi-dimensional sphere model having a unit length as a radius. The unit length may be 1.
  • [0011]
    In determining a distance between a sum of word vectors of the one or more words and the respective category vector, the method may accumulate normalized word frequency values of the one or more words to provide a normalized word vector sum, and classify the text into categories corresponding to a largest component of the normalized word vector sum.
  • [0012]
    In another aspect, an apparatus for text classification may include: a parsing module that parses a given text into one or more words; a query module that determines a word vector in a spherical space model for one of the one or more words, a number of dimensions of the spherical space being equal to a number of categories, each category corresponding to a spherical space category vector; a calculation module that, for each category, determines a distance between a sum of word vectors of the one or more words and the respective category vector; and a classification module that classifies the text into one or more categories with the shortest distance.
  • [0013]
    The apparatus may further include a filter module that the one or more words to provide one or more filtered words that meet filtering conditions.
  • [0014]
    The distance between the sum of word vectors of the one or more words and the respective category vector may be a straight line or a spherical arc.
  • [0015]
    A word vector of a word may include one or more normalized word frequency values of the word in respective one or more categories, where the spherical space model may include a multi-dimensional sphere model having a unit length as a radius. The unit length may be 1.
  • [0016]
    The calculation module may accumulate normalized word frequency values of the one or more words to provide a normalized word vector sum. The classification module may classify the text into categories corresponding to a largest component of the normalized word vector sum.
  • [0017]
    In yet another aspect, one or more computer-readable storage media having stored thereupon computer-executable instructions that, when executed by one or more computers, perform acts comprising: parsing a given text into one or more words; determining a word vector in a spherical space model for one of the one or more words, a number of dimensions of the spherical space being equal to a number of categories, each category corresponding to a spherical space category vector; for each category, determining a distance between a sum of word vectors of the one or more words and the respective category vector; and classifying the text into one or more categories with the shortest distance.
  • [0018]
    The acts may also include filtering the one or more words to provide one or more filtered words that meet filtering conditions.
  • [0019]
    The distance between the sum of word vectors of the one or more words and the respective category vector may be a straight line or a spherical arc.
  • [0020]
    A word vector of a word may include one or more normalized word frequency values of the word in respective one or more categories, where the spherical space model may include a multi-dimensional sphere model having a unit length as a radius. The unit length may be 1.
  • [0021]
    In determining a distance between a sum of word vectors of the one or more words and the respective category vector, normalized word frequency values of the one or more words may be accumulated to provide a normalized word vector sum, and the text may be classified into categories corresponding to a largest component of the normalized word vector sum.
  • [0022]
    The present disclosure pre-constructs a spherical space model. Moreover, this model classifies texts into categories by calculating the distance between each word vector and each category vector. The present disclosure describes text classification where the amount of calculation is substantially lessened compared with the existing angle cosine algorithm technique. The spherical space model uses the unit length as the radius. The unit length is also sum of the square of the normalized word frequency value of the words in each category. The unit length is equivalent and limited to the semantic information quantity of the words. Accordingly, in relation to current technologies, the present disclosure can further optimize text classification.
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • [0023]
    FIG. 1 shows a system diagram of an apparatus according to an embodiment of the present disclosure.
  • [0024]
    FIG. 2 shows a system diagram of the apparatus of FIG. 1 according to another embodiment of the present disclosure.
  • [0025]
    FIG. 3 shows a spherical space diagram according to an embodiment of the present disclosure.
  • [0026]
    FIG. 4 shows a flow chart of a primary method for text classification according to an embodiment of the present disclosure.
  • [0027]
    FIG. 5 shows a flow chart of a distance method for text classification according to an embodiment of the present disclosure;
  • [0028]
    FIG. 6 shows a flow chart of a vector sum method for text classification according to an embodiment of the present disclosure.
  • DETAILED DESCRIPTION
  • [0029]
    The present disclosure pre-constructs a spherical space model. Moreover, this model classifies texts into categories by calculating a distance between each word vector and each category vector. The present disclosure describes text classification where the amount of calculation is substantially lessened compared with the existing angle cosine algorithm technique. The spherical space model uses the unit length as the radius. The unit length is also a sum of the square of the normalized word frequency value of the words in each category. The unit length is equivalent and limited to the semantic information quantity of the word. Accordingly, in relation to the current technologies, the present disclosure can improve the accuracy of text classification.
  • [0030]
    As shown in FIG. 1, an apparatus for text classification according to an embodiment of the present disclosure is provided. In one embodiment, the apparatus comprises one or more servers. For example, the apparatus may be implemented in a processor-based server that includes one or more computer-readable storage media, such as memories, and communication means to communicate to a network and other devices and apparatuses connected to the network. The apparatus comprises: a parsing module 101, a query module 102, a calculation module 103, and a classification module 104.
  • [0031]
    Parsing module 101 parses a text into various parsed words.
  • [0032]
    Query module 102 determines the vector of each parsed word, referred to as word vector, in the spherical space model. The word vector comprises the normalized word frequency value of this word in each category. The spherical space model uses the unit length as radius. Its dimensions are equal to the number of categories corresponding to the spherical space category vector. The unit length can be held constant in order to facilitate the calculation. In one embodiment, the radius of the spherical space model is 1. Here, the distance between the word vectors and category vectors may be that of a straight line or spherical arc.
  • [0033]
    Calculation module 103 determines the distance between the word vectors and category vectors for each category.
  • [0034]
    Classification module 104 classifies the text into one or more categories with the shortest distance.
  • [0035]
    Calculation module 103, when calculating the distance between the word vectors and category vectors, accumulates the normalized word frequency values of the word vectors in each category to obtain the total normalized word vectors.
  • [0036]
    Classification module 104 classifies the text into categories corresponding to the largest component of the normalized word vectors.
  • [0037]
    In other embodiments, the apparatus further comprises: an interface module 105, a filter module 106, a structure module 107, and a storage module 108, as shown in FIG. 2.
  • [0038]
    Interface module 105 acquires texts to be classified from one or more external sources.
  • [0039]
    Filter module 106 filters the various parsed words obtained from parsing to obtain words that meet the filtering conditions. There are many kinds of filtering conditions. For example, based on the coefficient of variation of a word that is calculated from the word frequency value of the word, the word will be filtered out if its coefficient of variation is greater than a coefficient of variation threshold (e.g., 0.5). Through the coefficient of variation, those words having respective frequency values that do not vary much (such as the words “you”, “I” that have basically unvarying word frequency values under different categories) can be filtered out, while those words with significant variation in word frequency values (such as professional jargons, which appear more in the corresponding professional categories than in others) can be retained. Words with varying frequency value occur mainly in one or several categories. These words contribute to the accuracy of text categorization. In one embodiment, these words are classified as “outstanding” words, which can be retained after filtering. There may also be other filtering conditions that are not listed here in the interest of brevity.
  • [0040]
    Structure module 107 constructs the spherical space model.
  • [0041]
    Storage module 108 stores the spherical space model, as well as the classification of every text.
  • [0042]
    The process by which the structure module 107 constructs the spherical space model is described below.
  • [0043]
    Given a multi-dimensional spherical space S, the dimensions of S and the total number of categories are the same. Category Ci is the sphere endpoint and at the same time corresponds to the spherical space category vector. Ci=(0 . . . 0, 1, 0 . . . 0) is equivalent to the core (also called origin) that moves towards the spherical endpoint. The category vector of the ith dimension is 1 and the rest are 0. Assuming that a word emerged from two categories Ci and Cj that are statistically independent, then Ci and Cj in S must be perpendicular to each other. Thus, in general, the rest of the category vectors (Ci) are orthogonal to each other.
  • [0044]
    The word vector Wm of the mth word is the vector of S in the present disclosure. In m=1 . . . M, M is the total number of words, while in Wm=(V1, V2 . . . VN), Vi is the normalized word frequency value of the category Ci. i=1 . . . N, where N is the total number of categories. The normalized word frequency values from the core towards the spherical endpoint can indicate the coordinates of the category Ci. The diagram of the word vector and the category vector is shown in FIG. 3. As shown in FIG. 3, Ci, Cj and Ck illustrate the three vectors of the category wherein O is the core or the origin with coordinates at {0, 0 . . . 0}.
  • [0045]
    The present disclosure establishes that any amount of semantic information of a word is constant. The amount of semantic information may depend on either the perception of the main subject or representation of the existing object. The unit length is defined as constant and the length of the word vector in S (i.e., the distance of the endpoint of word vector from the origin O) is also constant. For convenience, the constant is set to 1. The distance of the endpoint of word vector from the origin O can be expressed as: |Wm−O|=1 (Formula 1), then based on Wm=(V1, V2 . . . VN), there is ΣVi2=1 (Formula 2). It is because of Formula 1 that one can know that the word vector Wm's endpoints fall within the sphere. Since the endpoints of the word vector Wm and the category vector Ci fall within the sphere, any semantic of a word and category can use the distance of Wm and Ci to express the degree of similarity. The shorter the distance, the closer they are. The distance of Wm and Ci can be calculated by means of straight line or spherical arc.
  • [0046]
    Because any amount of semantic information of a word is constant, the value of the word frequency have been normalized to obtain √{square root over (Σ(Fi×k)2)}=1, subsequently there is Σ(Fi×k)2=1, where Fi is the word frequency value of the word in the category Ci and k is the default normalization factor. From Σ(Fi×k)2=1,
  • [0000]
    k = 1 Fi 2 ( Formula 3 )
  • [0000]
    can be derived. Consequently, from Vi=Fi×k (Formula 4), one can obtain the word vector and word frequency value of the conversion function (or quantitative function) Wm=δ(Fi)=(Fi)×k (Formula 5).
  • [0047]
    As discussed above, structure module 107 uses the origin of the spherical structure as the core. With unit length l as the radius of the spherical space, the endpoints of the word vector Wm and the category vector Ci fall within the sphere. The spherical space model can be directly applied to train and review samples. The sample training process and text categorization process can be implemented by other pattern recognition or artificial means.
  • [0048]
    With regards to text D, D=ΣWm, Wm is the word vector of the mth word. Calculation module 103 computes the distance of ΣWm and the category vector Ci. Texts are classified under one or more categories with the shortest distance. Since ΣWm does not necessarily fall on the sphere, in order to facilitate the calculation, calculation module 103 can normalize D by multiplying the normalization factor k, then calculate the distance to the category vector Ci.
  • [0049]
    The shorter the distance between the word vector Wm and the category vector Ci, the degree of similarity between the two is greater. In order to simplify the calculation, it can be set that P=(Pi)=(ΣVmi) (Formula 6), Pi being the ith component of the weight categories. The greater the Pi, the shorter the distance to the category vector Ci is. Similarly, the greater ΣVmi, the shorter the distance to the category vector Ci is. Therefore, calculation module 103 accumulates the normalized value of the multiple words in a category and obtains the weight value of such category. Classification module 104 will classify the text into categories corresponding to the maximum weight value.
  • [0050]
    The principle behind the correlation between larger Pi and shorter distance of category vector Ci is discussed below.
  • [0051]
    Since D=ΣWm and Wm=(V1, V2 . . . VN), it follows that D=(ΣVm1, ΣVm2, . . . , ΣVmi, . . . , ΣVmn) where ΣVmi is the sum of all the normalized word frequency values in the document in the ith category. Moreover, with Pi=ΣVmi, D=(Pi). The distance of D to Ci can be expressed as:
  • [0000]
    D - Ci = { P 1 , P 2 , Pi , , Pn } × k - { 0 , 0 , , 0 , 1 , 0 , , 0 } = k × { P 1 , P 2 , Pi , , Pn } - { 0 , 0 , , 0 , 1 / k , 0 , , 0 } = k × sqrt ( ( P 1 - 0 ) 2 + ( P 2 - 0 ) 2 + + ( Pi - 1 / k ) 2 + + ( Pn - 0 ) 2 ) = k × sqrt ( P 1 2 + P 2 2 + + ( Pi 2 - 2 Pi / k + 1 / k 2 ) + + Pn 2 ) = k × sqrt ( Σ ( Pi 2 ) - 2 Pi / k + 1 / k 2 ) = sqrt ( Σ ( ( Pi × k ) 2 ) - 2 K × Pi + 1 )
  • [0052]
    Since Σ((Pi×k)2)=1, it follows that sqrt(E ((Pi×k)2)−2K×Pi+1)=sqrt (1−2K×Pi+1)=sqrt(2*(1−K×Pi)). This proves that the distance from D to Ci and Pi are inversely proportional. For instance, the category with the largest Pi is the one which is most proximate to D.
  • [0053]
    The aforementioned apparatus can be implemented in a single computer or in multiple computers with one or more components of the aforementioned apparatus implemented in each of the multiple computers. The functionality of the aforementioned apparatus can be achieved by coordinating the one or multiple computers. Each component can be implemented either in the form of software, hardware, or both.
  • [0054]
    The internal structure and functionality of the text classification apparatus has been described above. An implementation process of text classification will now be described.
  • [0055]
    As shown in FIG. 4, the main method according to an embodiment of the present disclosure comprises:
  • [0056]
    At 401, a given text is parsed into one or more words.
  • [0057]
    At 402, a respective word vector in the spherical space model corresponding to each of the one or more words is determined Each of the word vectors comprises the normalized word frequency value of the respective word in each category. The spherical space model uses the unit length as radius of the multi-dimensional sphere model. Its number of dimensions is equal to the number of categories, and the categories correspond to the spherical spatial category vectors.
  • [0058]
    At 403, for each category, the distance between the sum of the one or more word vectors and the category vector is determined.
  • [0059]
    At 404, the text is classified into one or more categories with the shortest distance.
  • [0060]
    The present disclosure provides text classification using the aforementioned distance and word vectors. Each of these two methods will now be described.
  • [0061]
    As shown in FIG. 5, a method for text classification according to the distance comprises a number of actions as described below.
  • [0062]
    At 501, a given text is parsed into one or more words.
  • [0063]
    At 502, the one or more words are filtered to result in one or more filtered words that meet the filtering conditions. Filter module 103 can filter words based on the word frequency value of each word being filtered. There may be many filter conditions such as the retention of those with word frequency value greater than a default value in all categories. Another condition is when the normalized word vector in the largest component (i.e. the maximum normalized word frequency value) is greater than the word frequency threshold. There may also be other filter conditions that are not listed here.
  • [0064]
    At 503, for each of the one or more filtered words that meet the filtering conditions, its normalized word frequency value for each category is determined. In particular, the normalized word frequency value for a word in each category is pre-stored. In the event that a word cannot be found from a search, then its normalized word frequency value will be set to 0 for all categories. If the word frequency value, rather than the normalized word frequency value, of a word is pre-stored for each category, then the query module 102 can be used to search for the word frequency values in order to normalize them to provide corresponding normalized word frequency values. Formula 4 provides an example of a specific implementation. Additionally, interfering words (such as rare words and common words, etc.) can be filtered out as an attempt to leave more technical jargon words to optimize the text classification.
  • [0065]
    At 504, for each category, the distance between the word vector of each of the one or more filtered words that meet the filtering conditions and the category vector of each category is determined This distance may be a straight line or spherical arc.
  • [0066]
    Prior to 504, a sum of the word vectors can be normalized to allow the normalized sum of word vectors to fall within the spherical space. Then, at 504, a distance between the normalized sum of word vectors and the category vector of each category can be determined.
  • [0067]
    At 505, the text is classified into one or more categories with the shortest distance.
  • [0068]
    There can be further classification of categories in the database based on stored texts.
  • [0069]
    As shown in FIG. 6, a vector sum method for text classification comprises a number of actions as described below.
  • [0070]
    At 601, a given text is parsed into one or more words.
  • [0071]
    At 602, the one or more words are filtered to result in one or more filtered words that meet the filtering conditions.
  • [0072]
    At 603, for each of the one or more filtered words that meet the filtering conditions, its normalized word frequency value for each category is determined. In particular, the normalized word frequency value for a word in each category is pre-stored.
  • [0073]
    At 604, for each category, normalized word frequency values of the one or more words are accumulated to provide a normalized word vector sum. Formula 6 provides an example of a specific implementation.
  • [0074]
    At 605, the text is classified into categories corresponding to the largest component(s) of the normalized word vector sum.
  • [0075]
    When one or more of the embodiments of the present disclosure are implemented in the form of computer-executable software, code, or instructions, such software, code, or instructions can be stored in one or more computer-readable storage media such as a floppy disk, hard disk, CD-ROM, flash memory, or other non-transitory storage medium. When executed by one or more computers, the software, code, or instructions may cause the one or more computers to perform embodiments of the text classification method described above.
  • [0076]
    The present disclosure improves VSM by pre-constructing the spherical space model. Moreover, this process calculates the distance between resultant word vector and the category vector. It also determines the category of the text. The present disclosure also implements text classification. In addition, with the angle cosine algorithm, the amount of calculation is substantially lessened. Moreover, the unit length acts as the radius of the spherical space model. The square sum of the normalized word frequency value of the words in each category can also be the unit length. The unit length is equivalent and limited to the semantic information quantity of the words. Accordingly, this method can improve the text classification accuracy of existing technologies.
  • [0077]
    With more accurate text classification, efficiency in text classification as well as search and retrieval of text can be greatly improved.
  • [0078]
    A person of ordinary skill in the art can alter or modify the present disclosure in many different ways without departing from the spirit and the scope of this disclosure. Accordingly, it is intended that the present disclosure covers all modifications and variation which falls within the scope of the claims of the present disclosure and their equivalent.

Claims (18)

  1. 1. A method of text classification, the method comprises:
    parsing a given text into one or more words;
    determining a word vector in a spherical space model for one of the one or more words, a number of dimensions of the spherical space being equal to a number of categories, each category corresponding to a spherical space category vector;
    for each category, determining a distance between a sum of word vectors of the one or more words and the respective category vector; and
    classifying the text into one or more categories with the shortest distance.
  2. 2. The method as recited in claim 1, further comprises:
    filtering the one or more words to provide one or more filtered words that meet filtering conditions.
  3. 3. The method as recited in claim 1, wherein the distance between the sum of word vectors of the one or more words and the respective category vector comprises a straight line or a spherical arc.
  4. 4. The method as recited in claim 1, wherein a word vector of a word comprises one or more normalized word frequency values of the word in respective one or more categories, the spherical space model comprising a multi-dimensional sphere model having a unit length as a radius.
  5. 5. The method as recited in claim 4, wherein the unit length is 1.
  6. 6. The method as recited in claim 1, wherein determining a distance between a sum of word vectors of the one or more words and the respective category vector comprises:
    accumulating normalized word frequency values of the one or more words to provide a normalized word vector sum; and
    classifying the text into categories corresponding to a largest component of the normalized word vector sum.
  7. 7. An apparatus for text classification, the apparatus comprising:
    a parsing module that parses a given text into one or more words;
    a query module that determines a word vector in a spherical space model for one of the one or more words, a number of dimensions of the spherical space being equal to a number of categories, each category corresponding to a spherical space category vector;
    a calculation module that, for each category, determines a distance between a sum of word vectors of the one or more words and the respective category vector; and
    a classification module that classifies the text into one or more categories with the shortest distance.
  8. 8. The apparatus as recited in claim 7, further comprising:
    a filter module that the one or more words to provide one or more filtered words that meet filtering conditions.
  9. 9. The apparatus as recited in claim 7, wherein the distance between the sum of word vectors of the one or more words and the respective category vector comprises a straight line or a spherical arc.
  10. 10. The apparatus as recited in claim 7, wherein a word vector of a word comprises one or more normalized word frequency values of the word in respective one or more categories, the spherical space model comprising a multi-dimensional sphere model having a unit length as a radius.
  11. 11. The apparatus as recited in claim 10, wherein the unit length is 1.
  12. 12. The apparatus as recited in claim 7, wherein the calculation module accumulates normalized word frequency values of the one or more words to provide a normalized word vector sum, and wherein the classification module classifies the text into categories corresponding to a largest component of the normalized word vector sum.
  13. 13. One or more computer-readable storage media having stored thereupon computer-executable instructions that, when executed by one or more computers, perform acts comprising:
    parsing a given text into one or more words;
    determining a word vector in a spherical space model for one of the one or more words, a number of dimensions of the spherical space being equal to a number of categories, each category corresponding to a spherical space category vector;
    for each category, determining a distance between a sum of word vectors of the one or more words and the respective category vector; and
    classifying the text into one or more categories with the shortest distance.
  14. 14. The one or more computer-readable storage media as recited in claim 13, wherein the acts further comprise:
    filtering the one or more words to provide one or more filtered words that meet filtering conditions.
  15. 15. The one or more computer-readable storage media as recited in claim 13, wherein the distance between the sum of word vectors of the one or more words and the respective category vector comprises a straight line or a spherical arc.
  16. 16. The one or more computer-readable storage media as recited in claim 13, wherein a word vector of a word comprises one or more normalized word frequency values of the word in respective one or more categories, the spherical space model comprising a multi-dimensional sphere model having a unit length as a radius.
  17. 17. The one or more computer-readable storage media as recited in claim 16, wherein the unit length is 1.
  18. 18. The one or more computer-readable storage media as recited in claim 13, wherein determining a distance between a sum of word vectors of the one or more words and the respective category vector comprises:
    accumulating normalized word frequency values of the one or more words to provide a normalized word vector sum; and
    classifying the text into categories corresponding to a largest component of the normalized word vector sum.
US12996658 2010-02-01 2010-09-03 Method and apparatus of text classification Active 2031-09-10 US9208220B2 (en)

Priority Applications (4)

Application Number Priority Date Filing Date Title
CN201010104512 2010-02-01
CN 201010104512 CN102141977A (en) 2010-02-01 2010-02-01 Text classification method and device
CN201010104512.4 2010-02-01
PCT/US2010/047868 WO2011093925A1 (en) 2010-02-01 2010-09-03 Method and apparatus of text classification

Publications (2)

Publication Number Publication Date
US20110213777A1 true true US20110213777A1 (en) 2011-09-01
US9208220B2 US9208220B2 (en) 2015-12-08

Family

ID=44319654

Family Applications (1)

Application Number Title Priority Date Filing Date
US12996658 Active 2031-09-10 US9208220B2 (en) 2010-02-01 2010-09-03 Method and apparatus of text classification

Country Status (5)

Country Link
US (1) US9208220B2 (en)
EP (1) EP2531896A4 (en)
JP (1) JP5587434B2 (en)
CN (1) CN102141977A (en)
WO (1) WO2011093925A1 (en)

Cited By (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103049569A (en) * 2012-12-31 2013-04-17 武汉传神信息技术有限公司 Text similarity matching method on basis of vector space model
CN103064970A (en) * 2012-12-31 2013-04-24 武汉传神信息技术有限公司 Search method for optimizing translators
US20140180692A1 (en) * 2011-02-28 2014-06-26 Nuance Communications, Inc. Intent mining via analysis of utterances
CN104794187A (en) * 2015-04-13 2015-07-22 西安理工大学 Feature selection method based on entry distribution
CN105005589A (en) * 2015-06-26 2015-10-28 腾讯科技(深圳)有限公司 Text classification method and text classification device
CN105243118A (en) * 2015-09-29 2016-01-13 武汉传神信息技术有限公司 Manuscript data classification method
US20170060832A1 (en) * 2015-08-26 2017-03-02 International Business Machines Corporation Linguistic based determination of text location origin
US20170083508A1 (en) * 2015-09-18 2017-03-23 Mcafee, Inc. Systems and Methods for Multilingual Document Filtering
WO2017210379A1 (en) * 2016-06-03 2017-12-07 Schlumberger Technology Corporation Detecting events in well reports
US9984068B2 (en) * 2015-09-18 2018-05-29 Mcafee, Llc Systems and methods for multilingual document filtering

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2016177626A (en) * 2015-03-20 2016-10-06 ヤフー株式会社 Information processing device, information processing method and program
CN106874291A (en) * 2015-12-11 2017-06-20 北京国双科技有限公司 Text categorization processing method and text categorization processing device

Citations (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4658429A (en) * 1983-12-29 1987-04-14 Hitachi, Ltd. System and method for preparing a recognition dictionary
US6513027B1 (en) * 1999-03-16 2003-01-28 Oracle Corporation Automated category discovery for a terminological knowledge base
US6611825B1 (en) * 1999-06-09 2003-08-26 The Boeing Company Method and system for text mining using multidimensional subspaces
US20040019601A1 (en) * 2002-07-25 2004-01-29 International Business Machines Corporation Creating taxonomies and training data for document categorization
US20040083224A1 (en) * 2002-10-16 2004-04-29 International Business Machines Corporation Document automatic classification system, unnecessary word determination method and document automatic classification method
US6735578B2 (en) * 2001-05-10 2004-05-11 Honeywell International Inc. Indexing of knowledge base in multilayer self-organizing maps with hessian and perturbation induced fast learning
US6751621B1 (en) * 2000-01-27 2004-06-15 Manning & Napier Information Services, Llc. Construction of trainable semantic vectors and clustering, classification, and searching using trainable semantic vectors
US6868411B2 (en) * 2001-08-13 2005-03-15 Xerox Corporation Fuzzy text categorizer
US20050228783A1 (en) * 2004-04-12 2005-10-13 Shanahan James G Method and apparatus for adjusting the model threshold of a support vector machine for text classification and filtering
US7024408B2 (en) * 2002-07-03 2006-04-04 Word Data Corp. Text-classification code, system and method
US20060085405A1 (en) * 2004-10-18 2006-04-20 Fu-Chiang Hsu Method for analyzing and classifying electronic document
US20060212294A1 (en) * 2005-03-21 2006-09-21 At&T Corp. Apparatus and method for analysis of language model changes
US20080177736A1 (en) * 2006-11-01 2008-07-24 International Business Machines Corporation Document clustering based on cohesive terms
US20080183471A1 (en) * 2000-11-02 2008-07-31 At&T Corp. System and method of pattern recognition in very high dimensional space
US7509578B2 (en) * 1999-04-28 2009-03-24 Bdgb Enterprise Software S.A.R.L. Classification method and apparatus
US7779349B2 (en) * 2000-09-26 2010-08-17 International Business Machines Corporation Method for adapting a K-means text clustering to emerging data

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7366705B2 (en) * 2004-04-15 2008-04-29 Microsoft Corporation Clustering based text classification
JP2007041721A (en) * 2005-08-01 2007-02-15 Ntt Resonant Inc Information classifying method and program, device and recording medium
JP5008137B2 (en) * 2007-11-06 2012-08-22 日本電信電話株式会社 Word vector generation device, the word vector generation method, a recording medium recording the program and program
JP5290570B2 (en) * 2007-12-26 2013-09-18 京セラ株式会社 Pictograph input support device, emoticons input support method, and program

Patent Citations (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4658429A (en) * 1983-12-29 1987-04-14 Hitachi, Ltd. System and method for preparing a recognition dictionary
US6513027B1 (en) * 1999-03-16 2003-01-28 Oracle Corporation Automated category discovery for a terminological knowledge base
US7509578B2 (en) * 1999-04-28 2009-03-24 Bdgb Enterprise Software S.A.R.L. Classification method and apparatus
US6611825B1 (en) * 1999-06-09 2003-08-26 The Boeing Company Method and system for text mining using multidimensional subspaces
US6751621B1 (en) * 2000-01-27 2004-06-15 Manning & Napier Information Services, Llc. Construction of trainable semantic vectors and clustering, classification, and searching using trainable semantic vectors
US7779349B2 (en) * 2000-09-26 2010-08-17 International Business Machines Corporation Method for adapting a K-means text clustering to emerging data
US20080183471A1 (en) * 2000-11-02 2008-07-31 At&T Corp. System and method of pattern recognition in very high dimensional space
US6735578B2 (en) * 2001-05-10 2004-05-11 Honeywell International Inc. Indexing of knowledge base in multilayer self-organizing maps with hessian and perturbation induced fast learning
US6868411B2 (en) * 2001-08-13 2005-03-15 Xerox Corporation Fuzzy text categorizer
US7024408B2 (en) * 2002-07-03 2006-04-04 Word Data Corp. Text-classification code, system and method
US20040019601A1 (en) * 2002-07-25 2004-01-29 International Business Machines Corporation Creating taxonomies and training data for document categorization
US20040083224A1 (en) * 2002-10-16 2004-04-29 International Business Machines Corporation Document automatic classification system, unnecessary word determination method and document automatic classification method
US20050228783A1 (en) * 2004-04-12 2005-10-13 Shanahan James G Method and apparatus for adjusting the model threshold of a support vector machine for text classification and filtering
US20060085405A1 (en) * 2004-10-18 2006-04-20 Fu-Chiang Hsu Method for analyzing and classifying electronic document
US20060212294A1 (en) * 2005-03-21 2006-09-21 At&T Corp. Apparatus and method for analysis of language model changes
US20080177736A1 (en) * 2006-11-01 2008-07-24 International Business Machines Corporation Document clustering based on cohesive terms

Cited By (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140180692A1 (en) * 2011-02-28 2014-06-26 Nuance Communications, Inc. Intent mining via analysis of utterances
CN103049569A (en) * 2012-12-31 2013-04-17 武汉传神信息技术有限公司 Text similarity matching method on basis of vector space model
CN103064970A (en) * 2012-12-31 2013-04-24 武汉传神信息技术有限公司 Search method for optimizing translators
CN104794187A (en) * 2015-04-13 2015-07-22 西安理工大学 Feature selection method based on entry distribution
CN105005589A (en) * 2015-06-26 2015-10-28 腾讯科技(深圳)有限公司 Text classification method and text classification device
US20170060832A1 (en) * 2015-08-26 2017-03-02 International Business Machines Corporation Linguistic based determination of text location origin
US20170083508A1 (en) * 2015-09-18 2017-03-23 Mcafee, Inc. Systems and Methods for Multilingual Document Filtering
US9984068B2 (en) * 2015-09-18 2018-05-29 Mcafee, Llc Systems and methods for multilingual document filtering
CN105243118A (en) * 2015-09-29 2016-01-13 武汉传神信息技术有限公司 Manuscript data classification method
WO2017210379A1 (en) * 2016-06-03 2017-12-07 Schlumberger Technology Corporation Detecting events in well reports

Also Published As

Publication number Publication date Type
CN102141977A (en) 2011-08-03 application
US9208220B2 (en) 2015-12-08 grant
WO2011093925A1 (en) 2011-08-04 application
EP2531896A4 (en) 2015-02-25 application
JP5587434B2 (en) 2014-09-10 grant
JP2013519133A (en) 2013-05-23 application
EP2531896A1 (en) 2012-12-12 application

Similar Documents

Publication Publication Date Title
US7099819B2 (en) Text information analysis apparatus and method
Cheema et al. Probabilistic reverse nearest neighbor queries on uncertain data
US6247016B1 (en) Decision tree classifier with integrated building and pruning phases
US20110137919A1 (en) Apparatus and method for knowledge graph stabilization
US20050086045A1 (en) Question answering system and question answering processing method
Erk et al. Exemplar-based models for word meaning in context
Potter et al. Visualizing summary statistics and uncertainty
US20090271391A1 (en) Method and apparatus for rating user generated content in seach results
Schnitzer et al. Local and global scaling reduce hubs in space
Lebanon et al. The locally weighted bag of words framework for document representation
US20090319269A1 (en) Method of Trainable Speaker Diarization
US20140279773A1 (en) Scoring Concept Terms Using a Deep Network
US20120290577A1 (en) Identifying visual contextual synonyms
US20080159624A1 (en) Texture-based pornography detection
US20110170777A1 (en) Time-series analysis of keywords
Doquire et al. Mutual information-based feature selection for multilabel classification
Yi et al. Multi-valued attribute and multi-labeled data decision tree algorithm
Strobl et al. Accounting for individual differences in Bradley-Terry models by means of recursive partitioning
Kim et al. Convolutional matrix factorization for document context-aware recommendation
US20150161230A1 (en) Generating an Answer from Multiple Pipelines Using Clustering
Sutton Introduction to k nearest neighbour classification and condensed nearest neighbour data reduction
US20100079464A1 (en) Information processing apparatus capable of easily generating graph for comparing of a plurality of commercial products
Hotho et al. Text clustering based on good aggregations
US7890512B2 (en) Automatic image annotation using semantic distance learning
US20120254242A1 (en) Methods and systems for mining association rules

Legal Events

Date Code Title Description
AS Assignment

Owner name: ALIBABA GROUP HOLDING LIMITED, CAYMAN ISLANDS

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:SUN, XIANG;REEL/FRAME:025460/0655

Effective date: 20101206