KR101479040B1 - Method, apparatus, and computer storage medium for automatically adding tags to document - Google Patents

Abstract

Embodiments of the present invention provide a method and apparatus for automatically adding a tag to a document, the method comprising: determining a plurality of candidate tag words; Determining a corpus containing a plurality of texts; Selecting common words from the corpus as characteristic words; For each characteristic word and candidate tag word, if a characteristic word occurs, determine the coincidence probability of the candidate tag word occurring at the same time; Extracting characteristic words from the document, calculating weight of the characteristic cues for each extracted characteristic word; Within the corpus, for candidate tag words, count the weighted coincidence probabilities of both candidate tag words and characteristic words occurring in the document; And selecting the candidate tag word having the highest probability of weighted coincidence as the tag word to be added to the document. Embodiments of the present invention can realize intelligence for adding tags to a document, and tags are not limited to keywords generated in a document.

Description

METHOD, APPARATUS, AND COMPUTER STORAGE MEDIUM FOR AUTOMATICALLY ADDING TAGS TO DOCUMENT,

This application claims the priority of Chinese Patent Application No. 201210001611.9, filed on January 5, 2012 with the title "METHOD AND APPARATUS FOR AUTOMATICALLY ADDED TAG TO DOCUMENT", filed with the State Intellectual Property Office, Are incorporated herein by reference in their entirety.

The present invention relates to the art of Internet documents, and more particularly to a method and apparatus for automatically adding tags to a document.

The tags used to organize content on the Internet are key words that are highly related to the document. The contents of the document are briefly described and classified by the tags to facilitate searching and sharing.

Currently, there are mainly three ways to add tags to a document: 1) the manner of a passive tag, where a particular tag is manually assigned to the document; 2) a keyword tag method in which an important keyword, which is extracted automatically from a document by analyzing contents of the document, is taken as a tag; And 3) a socialized tag method in which the tag is added to the user's document by the user himself. There are problems with all three approaches, for example: 1) With regard to the passive tag approach, tags can not be automatically added to a large amount of documents; 2) With respect to the keyword tag method, only keywords occurring in the document can be selected as tags, while not all of the keywords are suitable for the tag; And 3) With respect to the methods of socialization tags, this requires the user to add tags to the document alone, resulting in tags not being aligned due to inconsistent standards of different users.

According to one embodiment of the present invention, a method and apparatus are provided for automatically adding tags to a document, whereby tags that are not limited to keywords in the document can be intelligently added to the document.

The solution to one embodiment of the present invention is implemented as follows.

To automatically add tags to your document:

Determining a plurality of candidate tag words corresponding to the document;

Determining a corpus comprising a plurality of texts; Selecting words that are commonly used from corpus as characteristic words; Determining, for each of the characteristic words and the candidate tag words, a probability that the candidate tag word coincides with the characteristic word;

Extracting characteristic words from a document, and calculating a weight for each of the extracted characteristic words; And

Calculating, in the corpus, a weighted probability that each of the candidate tag words will occur simultaneously with all of the characteristic words extracted from the document; Selecting a candidate tag word having a high probability of occurrence of weighted weighting as a tag word to be added to the document.

Devices that automatically add tags to a document are:

A candidate tag word determination module configured to determine a plurality of candidate tag words corresponding to a document;

To select a corpus containing a plurality of texts, to select commonly used words from the corpus as characteristic words, and for each of the characteristic words and each of the candidate tag words to determine the probability that the candidate tag word coincides with the characteristic word A concurrent occurrence probability determination module configured;

A weight calculation module configured to extract characteristic words from a document and calculate a weight for each of the extracted characteristic words;

A weighted coincidence probability calculation module configured to calculate, in a corpus, a weighted probability that each of the candidate tag words will occur simultaneously with all of the characteristic words extracted from the document; And

And a tag word addition module configured to select a candidate tag word having a high weighted coincidence probability as a tag word to be added to the document.

In a method and apparatus for automatically adding a tag to a document in accordance with an embodiment of the present invention, a tag that is not limited to a keyword in the document calculates the probability that the feature word coincides with the candidate tag word in the corpus, Can be intelligently added to a document by switching to a vote from a word to a candidate tag word and taking the candidate tag word that has acquired the most votes as the tag word to be added to the document.

1 is a flow diagram of a method for automatically adding a tag to a document in accordance with one embodiment of the present invention.
Figure 2 is a schematic diagram of the structure of an apparatus for automatically adding tags to a document in accordance with one embodiment of the present invention.

According to one embodiment of the present invention, a method of automatically adding a tag to a document is provided. 1 is a flow chart of a method including the following steps.

In step 101, a plurality of candidate tag words corresponding to the document are determined.

At this stage, a plurality of candidate tag words corresponding to a document may be determined by, but not limited to, the following three methods:

1) the manner of a passive tag, where a particular tag is manually specified in the document;

2) a keyword tag method in which an important keyword automatically extracted from a document is extracted as a tag by analyzing contents of the document; And

3) The social tag method in which the tag is added to the user's document by the user himself.

In the case where the candidate tag words are determined by the manual tag method or the social tag method, the candidate tag words are not limited to words occurring in the document.

In step 102, a corpus containing a plurality of texts is determined.

For example, if one million texts are obtained from the Internet, one million acquired texts are collectively referred to as corpus.

In step 103, the commonly used words are selected as characteristic words from the corpus, and the probability of simultaneous occurrence of the candidate tag word and the characteristic word for each word of the characteristic words and for each word of the candidate tag words is determined in the corpus.

In step 104, the characteristic words are extracted from the document, and a weight for each word of the characteristic words is calculated.

In step 105, for each word of the candidate tag words, a weighted probability that a candidate tag word occurs simultaneously with all of the characteristic words occurring in the document is calculated in the corpus; A candidate tag word with a high probability of weighted concurrency is selected as the tag word to be added to the document.

At step 103, the coincidence probability is denoted as P (X | Y), where X denotes one of the candidate tag words and Y denotes one of the characteristic words occurring in the corpus. P (X | Y) can be determined by various methods such as the following.

In the first scheme, P (X | Y) is equal to the result of dividing the number of times X coincides with Y in the same text included in the corpus by the number of times Y occurs in the corpus.

이고, 여기서 H(X, Y)는 X 및 Y의 결합 엔트로피(combination entropy)를 표시하고, I(X,Y)는 X 및 Y의 상호 정보를 표시하고, H(X)는 X의 정보 엔트로피(information entropy)를 표시하고, H(Y)는 Y의 정보 엔트로피를 표시한다.In the second scheme,

Where H (X, Y) represents the combination entropy of X and Y, I (X, Y) represents the mutual information of X and Y, H (X) represents the information entropy of X (information entropy), and H (Y) denotes the information entropy of Y. [

In the third scheme, P (X | Y) is determined by using a vocabulary database such as wordnet.

At step 104, for each word of the extracted characteristic words, the weight for the characteristic word may be calculated based on the number of times the characteristic word occurred in the document and the number of texts in the corpus in which the characteristic word occurred.

Weight for the characteristic word (Y) to be extracted from a document is represented by W _Y, W _Y is: W _{and Y} is as Y are same as the product of the number of the text in the generated number of times, and Y generated in the document corpus (product) Lt; / RTI >

로 표시되고, 여기서 Y_i는 문서로부터 추출되는 특성 단어들 중 하나를 표시하고,

는 Y_i에 대한 가중치를 표시하고, n은 문서로부터 추출되는 특성 단어들의 수를 표시한다.In step 105, the weighted coincidence probability is

, Where Y _i represents one of the characteristic words extracted from the document,

Denotes the weight for Y _i , and n denotes the number of characteristic words extracted from the document.

In step 105, the weighted coincidence probability P _X can be calculated for candidate tag words that coincide with one or more characteristic words extracted from the document, rather than for all candidate tag words.

Certain embodiments will be introduced in more detail below.

First Embodiment

In step 1, a tag word set is prepared.

A number of candidate tag words corresponding to the document are obtained to construct a tag word set as desired. For example, if a tag needs to be added to documents associated with a movie, the tag word set may include tag types such as movie types and celebrities.

In step 2, a corpus is prepared.

A number of related texts may be collected as a corpus to be used for statistics of concurrent relationships between words from the Internet.

In step 3, characteristic words are extracted from the corpus.

Word segmentation is performed on the texts in the corpus. Then the term frequency (TF) of each word is counted. High frequency words, unused words and low frequency words are removed and the remaining commonly used words are selected as characteristic words.

In step 4, the probability P (X | Y) that each of the characteristic words occurs simultaneously with each of the candidate tag words is calculated.

P (X | Y) is equivalent to the number of simultaneous X and Y occurrences in the same text included in the corpus divided by the number of occurrences of Y in the corpus.

Where X denotes one of the candidate tag words, and Y denotes one of the characteristic words.

In step 5, the tag words are automatically added to the document, the specific steps of which are:

In step I, word breaking is performed on the document;

In step II, all of the characteristic words occurring in the document are extracted according to the word classification result, and the weight (W _Y ) for each extracted characteristic word Y is calculated as W _Y = TF × IDF, Y indicates the number of occurrences in the document and IDF indicates the number of texts in the corpus in which Y occurs;

Extracting candidate tag words coincident with at least one characteristic word (i. E., The coincidence probability is not zero) based on the coincidence probabilities calculated in step 4;

을 계산하고, 여기서 Y_i는 문서로부터 추출되는 특성 단어들 중 하나를 표시하고,

는 Y_i에 대한 가중치를 표시하고, n은 문서로부터 추출되는 특성 단어들의 수를 표시하고; 그리고In step IV, for each of the candidate tag words to be extracted, the weighted coincidence probability of the extracted candidate tag words with all of the characteristic words extracted from the document

, Where Y _i represents one of the characteristic words extracted from the document,

Denotes a weight for Y _i , n denotes the number of characteristic words extracted from the document; And

In step V, all of the candidate tag words extracted in descending order of P _x values are ranked, and one or more candidate tag words having the highest P _X are selected as tag words to be added to the document.

At this stage, a few of the candidate tag words are first extracted in step III, and then a weighted coincidence probability is calculated for each of the extracted candidate tag words. This can speed computation and save system resources. According to other embodiments of the invention, the weighted coincidence probability can be calculated for all of the candidate tag words. For a candidate tag word that does not have a coincidence relationship with any of the characteristic words, the calculated weighted coincidence probability P _X = 0 and the candidate tag word is ranked at the end of the queue of candidate tag words in step V Will be.

로 계산될 수 있고, 여기서 H(X,Y)는 X 및 Y의 결합 엔트로피를 표시하고, I(X,Y)는 X 및 Y의 상호 정보를 표시하고, H(X)는 X의 정보 엔트로피를 표시하고, H(Y)는 Y의 정보 엔트로피를 표시한다. 대안으로, 특성 단어 및 후보 태그 단어 사이의 관계는 wordnet과 같은 어휘 데이터베이스를 사용함으로써 결정된다.In another embodiment of the present invention, the co-occurrence probability P (X | Y) of the characteristic word and the candidate tag word may be calculated in other manners. For example, P (X | Y)

Where H (X, Y) denotes the combined entropy of X and Y, I (X, Y) denotes the mutual information of X and Y, H (X) denotes the information entropy of X , And H (Y) indicates the information entropy of Y. Alternatively, the relationship between the characteristic word and the candidate tag word is determined by using a vocabulary database such as wordnet.

According to one embodiment of the present invention, there is further provided an apparatus for automatically adding a tag to a document. Figure 2 is a schematic view of the structure of the device,

A candidate tag word determination module (201) configured to determine a plurality of candidate tag words corresponding to a document;

Determining a corpus containing a plurality of texts, selecting commonly used words from the corpus as characteristic words, and for each word of the characteristic words and each word of the candidate tag words, A coincidence probability determination module (202) configured to determine a coincidence probability;

A weight calculation module (203) configured to extract characteristic words from a document and calculate a weight for each word of characteristic words;

Within a corpus, a weighted coincidence probability calculation module (204) configured to calculate a weighted probability that each word of candidate tag words will occur simultaneously with all of the characteristic words occurring in the document; And

And a tag word addition module (205) configured to select a candidate tag word having a high weighted coincidence probability as a tag word to be added to the document.

In the above-described apparatus, the coincidence probability can be expressed as P (X | Y), where X denotes one of the candidate tag words and Y denotes one of the characteristic words occurring in the corpus. The coincidence probability determination module 202 may calculate P (X | Y) as follows.

P (X | Y) is the same as the result of dividing the number of simultaneous X and Y occurrences in the corpus by the number of occurrences of Y in the corpus.

이고, 여기서 H(X,Y)는 X 및 Y의 결합 엔트로피를 표시하고, I(X,Y)는 X 및 Y의 상호 정보(mutual information)를 표시한다.As an alternative,

, Where H (X, Y) denotes the entropy of the combination of X and Y, and I (X, Y) denotes the mutual information of X and Y.

Alternatively, P (X | Y) is determined by using a lexical database.

In the above-described apparatus, the weight for the characteristic word Y extracted from the document is represented by W _Y , which is calculated by the weight calculation module 203: W _Y is the number of times Y occurs in the document, Is equal to the product of the number of texts.

로서 표시될 수 있고, 여기서 Y_i는 문서로부터 추출되는 특성 단어들 중 하나를 표시하고,

는 Y_i에 대한 가중치를 표시하고, n은 문서로부터 추출되는 특성 단어들의 수를 표시한다.In the above-described apparatus, the probability of weighted simultaneous occurrence is

, Where Y _i represents one of the characteristic words extracted from the document,

Denotes the weight for Y _i , and n denotes the number of characteristic words extracted from the document.

In the apparatus described above, the weighted coincidence probability calculation module 204 may calculate the weighted coincidence probability for candidate tag words coincident with one or more characteristic words extracted from the document only.

In conclusion, in a method and apparatus for automatically adding a tag to a document according to embodiments of the present invention, a tag that is not limited to a keyword occurring in a document calculates a probability that a characteristic word coincides with a candidate tag word in a corpus , The simultaneous occurrence probability can be added to the document intelligently by converting the candidate word tag from the characteristic word into a vote vote and taking the candidate tag word obtained from the maximum tags as a tag word to be added to the document. The relevance between the tag word and the document is improved based on statistics on the coincidence probability according to embodiments of the present invention.

According to one embodiment of the present invention, there is further provided a machine-readable storage medium storing instructions that enable a machine to perform a method of automatically adding a tag to a document, as described herein. A system or apparatus may be provided that includes a storage medium on which software program codes embodying the functions of any of the above embodiments are stored, and a computer (or CPU or MPU) Lt; RTI ID = 0.0 > and / or < / RTI >

In this case, the program codes read from the storage medium may implement any one of the above-described embodiments. Therefore, the storage medium storing the program codes and program codes constitutes a part of the present invention.

Examples of storage media that provide the program codes are a hard disk, a hard disk, a magnetic optical disk, an optical disk (CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD- , Magnetic tape, non-volatile memory, and ROM. Optionally, the program codes may be downloaded from the server computer via the communications network.

Furthermore, any one of the above-described embodiments may be implemented in a computer-readable medium, such as a computer-readable medium or a computer-readable recording medium, As will be appreciated by those skilled in the art.

Furthermore, any one of the above-described embodiments may be implemented by writing program codes read from a storage medium to a memory provided in an expansion board inserted in the computer, or writing the program codes into a memory provided in an expansion unit connected to the computer And then instructing the CPU or the like mounted on the expansion board or expansion unit based on the program codes that perform some or all of the actual operations.

The above-described preferred embodiments of the present invention are not intended to limit the scope of the invention. Any modifications, equivalents, and improvements that fall within the spirit and principles of the present invention are within the scope of the present invention.

Claims (15)

A method to automatically add a tag to a document:
Determining a plurality of candidate tag words corresponding to the document;
Determining a corpus comprising a plurality of texts; Selecting words commonly used from the corpus as characteristic words; Determining, for each of the characteristic words and the candidate tag words, a probability that the candidate tag word coincides with the characteristic word;
Extracting characteristic words from the document and calculating a weight for each of the extracted characteristic words; And
Calculating, in the corpus, a weighted probability that each of the candidate tag words will occur simultaneously with all of the characteristic words extracted from the document; Selecting a candidate tag word having a high probability of weighted occurrence as a tag word to be added to the document,
Weight for the characteristic word Y to be extracted from said document that is represented by W _Y, W _Y is Y is identical to the product of the text, the number of (product) in the corpus that the number and Y generated by the document generation tags To the document automatically.
The method according to claim 1,
Wherein the coincidence probability is represented as P (X | Y), where X represents one of the candidate tag words and Y represents one of the characteristic words occurring in the corpus;
A method of automatically adding a tag to a document determined as a result of dividing the number of simultaneous occurrences of X and Y in the same text included in the corpus by the number of occurrences of Y in the corpus.
The method according to claim 1,
Wherein the coincidence probability is denoted as P (X | Y), X denotes one of the candidate tag words and Y denotes one of the characteristic words occurring in the corpus;
P (X | Y)

I (X, Y) represents a mutual information of X and Y, and H (X, Y) represents a combination entropy of X and Y, .
The method according to claim 1,
Wherein the coincidence probability is denoted as P (X | Y), X denotes one of the candidate tag words and Y denotes one of the characteristic words occurring in the corpus;
P (X | Y) is a method of automatically adding tags to a document determined by using a vocabulary database.
delete The method according to claim 1,
The weighted probability of simultaneous occurrence is

Y _i represents one of the characteristic words extracted from the document,

Represents a weight for Y _i , and n is a number indicating the number of characteristic words extracted from the document.
The method according to claim 1,
In the corpus, calculating the weighted probability that each of the candidate tag words will occur simultaneously with all of the characteristic words extracted from the document:
Wherein in the corpus, calculating a weighted probability that each of the candidate tag words coincides with at least one characteristic word extracted from the document.
A device that automatically adds tags to a document:
A candidate tag word determination module configured to determine a plurality of candidate tag words corresponding to the document;
Determining a corpus containing a plurality of texts, selecting words commonly used from the corpus as characteristic words, and for each of the characteristic words and the candidate tag words, determining whether the candidate tag word is concurrent with the characteristic word A coincidence probability determination module configured to determine a probability of occurrence;
A weight calculation module configured to extract characteristic words from the document and to calculate a weight for each of the extracted characteristic words;
A weighted coincidence probability calculation module configured to calculate, in the corpus, a weighted probability that each of the candidate tag words will occur simultaneously with all of the characteristic words extracted from the document; And
And a tag word addition module configured to select a candidate tag word having a high weighted coincidence probability as a tag word to be added to the document,
The weight for the characteristic word Y extracted from the document is represented by W _Y and the weight calculation module is equal to the product of W _Y by the number of times Y occurs in the document and the number of texts in the corpus where Y occurs A device that automatically adds tags to a document that are configured to calculate.
9. The method of claim 8,
Wherein the coincidence probability is represented as P (X | Y), where X represents one of the candidate tag words and Y represents one of the characteristic words occurring in the corpus;
Wherein the simultaneous occurrence probability determination module calculates a tag configured to calculate P (X | Y) as a result of dividing the number of simultaneous occurrence of X and Y in the same text included in the corpus by the number of times Y occurs in the corpus A device that automatically adds to a document.
9. The method of claim 8,
Wherein the coincidence probability is denoted as P (X | Y), X denotes one of the candidate tag words and Y denotes one of the characteristic words occurring in the corpus;
The coincidence probability determination module determines P (X | Y) as

, Where H (X, Y) denotes the combined entropy of X and Y, and I (X, Y) automatically adds a tag to the document indicating the mutual information of X and Y.
9. The method of claim 8,
Wherein the coincidence probability is denoted as P (X | Y), X denotes one of the candidate tag words and Y denotes one of the characteristic words occurring in the corpus;
Wherein the coincidence probability determination module automatically adds a tag to a document that is configured to calculate P (X | Y) by using a lexical database.
delete The method according to any one of claims 8 to 11,
The weighted probability of simultaneous occurrence is

Y _i represents one of the characteristic words extracted from the document,

≪ / _RTI > wherein n represents the number of characteristic words extracted from the document, and n represents a number representing the number of characteristic words extracted from the document.
The method according to any one of claims 8 to 11,
Wherein the weighted coincidence probability calculation module automatically adds within the corpus a tag that is configured to calculate a weighted probability that each of the candidate tag words coincides with one or more characteristic words extracted from the document.
A computer storage medium storing a computer program for implementing the method according to any one of claims 1 to 4, 6, and 7.

Images