CN103955453B - A kind of method and device for finding neologisms automatic from document sets - Google Patents

Abstract

The invention discloses a method and device for automatically discovering new words from a document collection, wherein, a template acquiring unit acquires one or more templates; Words matched by each template; the candidate template set adding unit selects at least a part of templates from the one or more templates to add to the candidate template set; the candidate word set adding unit extracts from the one or more templates Select at least a part of the words matched by each template and add them to the candidate word set; the new word set adding unit sorts the candidate words in the candidate word set based on the templates in the candidate template set, and sorts a certain number of candidate words based on the sorting. Words are added to the set of new words. Compared with the prior art, the method and device provided by the invention can effectively discover new words.

Description

A method and device for automatically discovering new words from a document collection

technical field

The invention relates to natural language processing technology, in particular to a method and device for automatically discovering new words from a document collection.

Background technique

In social networks, netizens like to use their own personalized language to express their views on politics, society, culture, etc. Generally, the more people disseminate personalized language, the easier it is to become a new hot word on the Internet (referred to as "new word"). At present, new words have important applications in automatic summarization, text clustering/classification, information retrieval, etc. According to statistics, more than 1,000 new Chinese words appear on the Internet every year, and most of these new words are time-sensitive in various fields. Professional terms, since most of these new words do not exist in the dictionary, it is difficult for existing word segmentation algorithms to identify these new words from the document set. Taking the emotional new word "Give force (adjective)" and the document "The performance is very good" as an example, the existing word segmentation algorithm usually performs the following word segmentation: performance/noun very/adverb give/verb force/noun, so that the new The word "Gili" cannot be segmented as a complete word to affect the recognition of new words.

Contents of the invention

One of the technical problems solved by the present invention is to improve the accuracy of new word recognition.

According to an embodiment of an aspect of the present invention, a method for automatically discovering new words from a document collection is provided, including:

Get one or more templates;

extracting words matching each of the one or more templates from the document set;

Selecting at least a part of the templates from the one or more templates and adding them to the set of candidate templates;

Selecting at least a part of words from the extracted words that match each template in the one or more templates and adding them to the candidate word set;

Based on the templates in the candidate template set, the candidate words in the candidate word set are sorted, and a certain number of candidate words are added to the new word set based on the templates in the candidate template set are used to sort the candidate words in the candidate word set. .

According to an embodiment of the present invention, the one or more templates are acquired in any of the following ways:

pre-specifies said one or more templates, or

After the document set is acquired, word segmentation processing is performed on the document set, and the one or more templates matching the specific regular expression are extracted from the word-segmented document set.

According to an embodiment of the present invention, the step of selecting at least a part of templates from the one or more templates and adding them to the set of candidate templates includes any of the following:

adding all the one or more templates to the set of candidate templates;

Based on the number of occurrences of each template of the one or more templates in the document set, some templates are added to the set of candidate templates.

According to an embodiment of the present invention, the step of adding a part of templates to the set of candidate templates based on the number of occurrences of each template of the one or more templates in the document set includes:

Adding the template that appears in the top f number of times in the document set to the set of candidate templates, where f is a positive integer; or

Templates whose occurrence times in the document set exceed a specific threshold are added to the set of candidate templates.

According to an embodiment of the present invention, the step of selecting at least a part of words from the extracted words that match each template in the one or more templates and adding them to the candidate word set includes any of the following:

All the matched words are added to the candidate word set;

Based on the matching times of the matched words and each template, some words are added to the candidate word set.

According to one embodiment of the present invention, based on the matching times of the matched words and each template, the step of adding some words to the candidate word set includes:

In the matching words, the words whose number of matches with each template ranks first g are added to the candidate word set, and g is a positive integer; or

Among the matched words, words whose matching times with each template exceed a certain threshold are added to the candidate word set.

According to an embodiment of the present invention, the method further includes: prior to sorting the candidate words in the candidate word set based on the templates in the candidate template set, sorting the templates in the candidate template set with a pre-specified new word set , and filter the set of candidate templates based on the ordering of the templates in the set of candidate templates with the set of pre-specified new words.

According to an embodiment of the present invention, the method further includes: sorting the templates in the candidate template set by using the obtained new word set, and filtering candidate candidates based on the sorting of the templates in the candidate template set based on the obtained new word set. template set, and use the filtered candidate template set to sort the candidate words in the candidate word set again and reorder a certain number of candidate words in the candidate word set based on the filtered candidate template set. The candidate words are added to the new word set.

According to an embodiment of the present invention, the templates in the candidate template set are sorted by calculating the template weights in the candidate template set based on the following formula and sorting the templates in the candidate template set according to the calculated template weights:

n _1i =k _1i +k _3i , n _2i =k _2i +k _4i ,

W represents a new word set, P represents a candidate template set, w _i represents a word in the new word set W, p _j represents a template in the candidate template set P, and k _1i represents a template found from the document set that matches the one or the number of matchings of each template in the multiple templates containing both w _i and p _j , k _2i means that the matching of each template in the one or more templates found in the document set contains w _i but does not contain the number of matches of p _j , k _3i represents the number of matches that contain p _j but does not contain w _i in the matching of each template in the one or more templates found in the document set, k _4i represents the number of matches found in the document set that contain neither p _j nor w _i among the matches to each of the one or more templates.

According to an embodiment of the present invention, the templates in the candidate template set are used to sort the candidate words in the candidate word set, and the templates in the candidate template set are used to sort the candidate words in the candidate word set. A certain number of candidate words are added to the new word set including:

The number of times that the templates in the candidate template set are matched with the first m candidate words is added to the new word set, and m is a positive integer; or

The candidate words whose matching times with the templates in the candidate template set exceed a certain threshold are added to the new word set.

According to an embodiment of the present invention, in the step of sorting the candidate words in the candidate word set based on the templates in the candidate template set,

According to one of LLR( _wi ), E( _wi ), P( _wi ), EMI( _wi ), 1/NMED( _wi ), or the product of any of them to calculate the candidate word set the weights of the candidate words, and sort the candidate words in the candidate word set based on the calculated weight;

Among them, w _i represents a candidate word in the candidate word set W, LLR( _wi ) represents the closeness of the statistical connection between the candidate word w _i and the template in the candidate template set, E( _wi ) represents the candidate word w _i Left information entropy, P(w _i ) represents the probability that the words in the candidate word w _i are combined to form a word, EMI( _wi ) and NMED( _wi ) respectively represent different measures of the semantic composition of the candidate word w _i ;

Wherein, in the step of sorting the candidate words in the candidate word set based on the templates in the candidate template set, LLR( _wi ), E( _wi ), P( _wi ), EMI( _wi ), 1/ NMED(w _i ) are calculated as follows:

n _1j =k _1j +k _3j , n _2j =k _2j +k _4j ,

Wherein W represents the set of candidate words, P represents the set of candidate templates, w _i represents a candidate word in W, p _j represents a template in the set of candidate templates P, and k _1j represents a template found from the document set that matches the one or The matching number of each template in multiple templates contains both w _i and p _j , k _2j means that the matching of each template found in the document set and the one or more templates contains w _i but does not contain the number of matches of p _j , k _3j represents the number of matches that contain p _j but does not contain w _i in the matching of each template in the one or more templates found in the document set, k _4j represents the number of matches that neither p _j nor w _i are found in the matching of each template in the one or more templates found in the document set;

Among them, L represents the left word l _o set that has appeared in the document set on the left side of the candidate word w _i and matches any template in the candidate template set, and c(l _o ) represents the left word l _o and the candidate word w _i The number of times that the left collocation has appeared and matched any template in the candidate template set, and N represents the total number of times that the candidate word w _i appears together with the templates in the candidate template set;

Among them, t _h represents the hth word in the candidate word w _i in the candidate word set, and n represents the number of individual words contained in the candidate word w _i ;

all(t _h ) indicates the number of occurrences of the hth word in the candidate word w _i in the document set, s(t _h ) indicates that the hth word in the candidate word w _i and any word are used as a single word in the document set the number of occurrences;

Among them, S is the total number of segments in the document set M, n indicates the number of words contained in the candidate word w _i , F indicates the number of segments in the document set containing the candidate word w _i , and F _h indicates that the document set contains the candidate word w _i . The number of segments of h words;

Where S is the total number of segments in the document set M, μ(g) represents the number of segments in the document set M that contain all the words contained in the candidate word w _i , (g) indicates the number of segments in which all the words contained in the candidate word w _i in the document set M appear strictly consecutively in one segment.

According to an embodiment of the present invention, the templates in the candidate template set are sorted by the obtained new word set, and the candidate template set is filtered based on the sorting of the templates in the candidate template set by the used new word set Steps include:

Filter the candidate words whose number of matching times with the templates in the candidate template set ranks in the top r from the candidate template set to obtain a filtered candidate template set, where r is a positive integer; or

Candidate words whose matching times with templates in the candidate template set are higher than a specific threshold are filtered from the candidate template set to obtain a filtered candidate template set.

According to an embodiment of another aspect of the present invention, there is also provided a device for automatically discovering new words from a document collection, including:

a template obtaining unit configured to obtain one or more templates;

a word extraction unit configured to extract words that match each of the one or more templates from the document set;

A candidate template set adding unit configured to select at least a part of templates from the one or more templates and add them to the candidate template set;

The candidate word set adding unit is configured to select at least a part of words from the extracted words matching each template in the one or more templates to add to the candidate word set;

The new word set adding unit is configured to sort the candidate words in the candidate word set based on the templates in the candidate template set, and the sorting of the candidate words in the candidate word set based on the templates in the candidate template set will be A certain number of candidate words are added to the new word set.

According to an embodiment of the present invention, the template acquisition unit acquires the one or more templates in any of the following ways:

pre-specifies said one or more templates, or

After the document set is acquired, word segmentation processing is performed on the document set, and the one or more templates matching the specific regular expression are extracted from the word-segmented document set.

According to one embodiment of the present invention, the device also includes:

The candidate template set filtering unit is configured to sort the templates in the candidate template set with a pre-specified new word set before sorting the candidate words in the candidate word set based on the templates in the candidate template set, and based on the The method is to sort and filter the candidate template set by using the pre-specified new word set to sort the templates in the candidate template set.

According to an embodiment of the present invention, the new word set adding unit is configured to use the filtered candidate template set to sort the candidate words in the candidate word set again and add a certain number of candidate words based on the sorting. to a collection of new words.

Compared with the prior art, the method for automatically discovering new words from a document set provided by an embodiment of the present invention can effectively, more accurately, and unsupervised discover new words than the solutions of the prior art.

Moreover, in the method for automatically discovering new words from a document set provided by an embodiment of the present invention, after the document set is obtained, the document set is subjected to word segmentation processing, and a specific regular expression is extracted from the word-cutting document set. The one or more templates that match the formula, so that the acquired templates are more standardized, so that new words can be found more accurately.

Moreover, in the method for automatically discovering new words from a document set provided by an embodiment of the present invention, based on the number of times each template of the one or more templates appears in the document set, a part of the templates are added to the candidate template set, namely By screening the templates, the calculation efficiency of the present invention is improved.

Moreover, the method for automatically discovering new words from the document collection provided by one embodiment of the present invention is to add the words that rank the top g among the matching words with each template into the candidate word set, where g is a positive integer; or Among the matched words, the words whose matching times with each template exceed a certain threshold are added to the candidate word set, that is, by screening out specific and better candidate words, new words can be discovered more accurately.

Moreover, in the method for automatically discovering new words from a document set provided by an embodiment of the present invention, before sorting the candidate words in the candidate word set based on the templates in the candidate template set, the candidate words are sorted with a predetermined new word set. The templates in the template set are sorted, and the candidate template set is filtered based on the sorting of the templates in the candidate template set with the pre-specified new word set, that is, by screening the templates, a new word that matches the new word to be discovered can be obtained. better templates for more accurate discovery of new words.

Those of ordinary skill in the art will appreciate that although the following detailed description refers to the illustrated embodiments and accompanying drawings, the present invention is not limited to these embodiments. Rather, the scope of the invention is broad and it is intended that the scope of the invention be limited only by the appended claims.

Description of drawings

Other characteristics, objects and advantages of the present invention will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

FIG. 1 shows a flowchart of a method for automatically discovering new words from a document collection according to an embodiment of the present invention;

Fig. 2 shows a schematic block diagram of an apparatus for automatically discovering new words from a document collection according to another embodiment of the present invention;

The same or similar reference numerals in the drawings represent the same or similar components.

detailed description

The present invention will be described in further detail below in conjunction with the accompanying drawings.

Fig. 1 shows a flowchart of a method for automatically discovering new words from a document set according to an embodiment of the present invention. According to one embodiment of the invention, the method includes:

Step S101, acquiring one or more templates.

Step S102, extracting words matching each template in the one or more templates from the document set;

Step S103, selecting at least a part of the templates from the one or more templates and adding them to the set of candidate templates;

Step S104, selecting at least a part of words from the extracted words that match each template in the one or more templates and adding them to the candidate word set;

Step S105, sorting the candidate words in the candidate word set based on the templates in the candidate template set, and adding a certain number of candidate words to the collection of new words.

Wherein, the document set may refer to a single document or a collection of multiple documents. Of course, the document set here is only an example, and may also be other corpus resources, such as dictionaries, microblog databases, etc., which are also applicable to the present invention.

Specifically, in step S101, one or more templates may be acquired in any of the following ways:

Way 1: Predetermine the one or more templates, for example, define the templates by part of speech such as adverbs, particles, etc., so as to enumerate one or more templates, or the one or more templates can also be preset or given set; or,

Method 2: After the document set is acquired, word segmentation is performed on the document set, and the one or more templates matching the specific matching rule are extracted from the word-segmented document set. The word segmentation processing method here is specifically not limited, such as the word segmentation method based on string matching, the word segmentation method based on understanding, the word segmentation method based on statistics, etc., are all applicable to the present invention, and are included here by reference here. The specific matching rules include rules such as regular matching based on regular expressions, which are not limited here and may also include other matching rules. The regular expression here mainly refers to an expression composed of one or more specific parts of speech, such as "noun+adjective", "adverb+space/*+interjection" are two different regular expressions, here A space/* stands for any other content and part of speech. Preferably, in this embodiment, if one or more templates are obtained in the second manner, different regular expressions are defined according to the part of speech of the new word to be discovered. For example, the purpose of this embodiment is to find emotional words, then the main defined regular expressions include "adverb+space/*+interjection", "adverb+space/*" and so on.

Wherein, the template mainly refers to the combination of words and spaces/custom symbols, for example, "too +/*+" and "good ++" are two different templates, and the spaces/custom symbols here represent any combination of words. Take enumerating a template as an example, define a template through "adverb + space + interjection" as follows: "true++ah".

In step S102, words matching each of the one or more templates are extracted from the document set. Specifically, the document set is "too powerful, too cheating, so awesome, so cheating, so beautiful, and very cheating", multiple templates are "too +/*+", "good ++" For example, the words "Great" and "Crap" in the document set match the template "too +/*+", and the words "Great", "Crap" and "Pretty" in the document set match the template "Good++" Matching, thus, the words extracted from the document set include: "Great", "Crap", and "Pretty". Of course, this embodiment does not limit the manner in which words are extracted from the document set, which is incorporated herein by reference.

In step S103, at least a part of templates are selected from the one or more templates and added to the candidate template set. Specifically, a part of templates may be selected from the one or more templates and added to the candidate template set, or all of the one or more templates may be added to the candidate template set.

Optionally, based on the number of occurrences of each template of the one or more templates in the document set, some templates are added to the set of candidate templates. Specifically, first count the number of occurrences of each template of the one or more templates in the document set, then filter the one or more templates according to certain rules, and add the filtered templates to the candidate template set . Still take the document set "too awesome, too cheating, so awesome, so cheating, so beautiful, very cheating", multiple templates "too +/*+", "good ++" as an example , because the template "too +/*+" matches "too awesome, too bad" in the document set, and thus, the number of occurrences of the template "too +/*+" in the document set is 2 , similarly, the number of occurrences of the template "good++" in the document set is 3 according to the statistics, and the one or more templates are further screened according to certain rules, and the screened templates are added to the candidate template set.

Optionally, the step of adding a part of templates to the set of candidate templates based on the number of occurrences of each template of the one or more templates in the document set includes:

Adding the template that appears in the top f number of times in the document set to the set of candidate templates, where f is a positive integer; or

Templates whose occurrence times in the document set exceed a specific threshold are added to the set of candidate templates.

For example, the above-mentioned document set is still "too powerful, too cheating, so awesome, so cheating, so beautiful, very cheating", multiple templates "too +/*+", "good+ +ah” as an example, the number of occurrences of the template “too +/*+” in the document set is 2, and the number of occurrences of the template “ok++” in the document set is 3, if the template will be included in the document set The template with the number of occurrences ranked first is added to the candidate template set, then the template "too +/*+" is filtered, and the template "good++" is added to the candidate template set; if the template that will appear in the document set Templates with more than one times are added to the candidate template set, and the templates "too +/*+" and "good ++" are added to the candidate template set at the same time.

In step S104, at least a part of the extracted words matching each template in the one or more templates is selected and added to the candidate word set. Specifically, a part of the extracted words that match each template in the one or more templates can be selected and added to the candidate word set, or the extracted words that match the one or more templates All the words matched by each template are added to the candidate word set.

Optionally, based on the number of matches between the matched words and each template, some words are added to the candidate word set. Specifically, the matching times of the matched words and each template are counted first, and then the matched words are screened according to certain rules, and the screened words are added to the candidate word set. Still take the above document set as "too powerful, too cheating, so awesome, so cheating, so beautiful, very cheating", multiple templates "too +/*+", "good ++" as For example, the number of matches between the word "to force" and the template "too+/*+" and the template "good++" is 1 each, and the word "bad dad" is matched with the template "too+/*+" and the template " The matching times of "good ++ ah" are also 1, and the matching times of the word "beautiful" and the template "too +/*+" and the template "good ++ ah" are 0 and 1 respectively, and further according to certain rules The matching words are screened, and the filtered words are combined and added to the candidate word set.

Optionally, based on the matching times of the matched words and each template, the step of adding some words to the candidate word set includes:

In the matching words, the words whose number of matches with each template ranks first g are added to the candidate word set, and g is a positive integer; or

Among the matched words, words whose matching times with each template exceed a certain threshold are added to the candidate word set.

For example, the above-mentioned document set is still "too powerful, too cheating, so awesome, so cheating, so beautiful, very cheating", multiple templates "too +/*+", "good+ +ah” as an example, after counting the number of matches between each word and each template, if the word that matches the template “too +/*+” ranks first or matches the template “好++ah” If the number of matches ranks first and is added to the candidate word set, then "Great", "Crap" and "Pretty" are added to the candidate word set; if the matched words are matched with the template "too +/*+" or the template " Words with more than 1 matching number of "good ++ ah" are added to the candidate word set, and the words "giant", "cheating" and "beautiful" are combined and added to the candidate word set.

In step S105, the candidate words in the candidate word set are sorted based on the templates in the candidate template set, and a certain number of candidate words are sorted based on the templates in the candidate template set to the candidate words in the candidate word set. added to the collection of new words.

Specifically, taking the candidate template set {"too +/*+", "good ++"}, and the candidate word set {"力力", "坐父"} as examples, it can be based on the relationship between each word in the candidate word set and The number of times each template in the candidate template set matches each candidate word is sorted. For example, the number of times the candidate word "Gili" matches the template "太+/*+" is 5, and the number of times it matches the template "好++ah" The number of times is 4; the number of times that the candidate word "坑爹" matches the template "too+/*+" is 3, and the number of times that the template "good ++" matches is 5, then both the candidate words and the candidate template can be The total number of matching times of all templates in the set is sorted. For example, the total number of matching times of the candidate word "Geeli" is 9, and the total number of matching times of the candidate word "Kengbai" is 8; The matching times of the templates are respectively sorted. Of course, the candidate words can also be sorted based on other methods or algorithms, which are not limited here.

Optionally, sorting the candidate words in the candidate word set based on the templates in the candidate template set, sorting a certain number of candidate words based on the templates in the candidate template set to the candidate words in the candidate word set Words added to the new word set include:

The number of times that the templates in the candidate template set are matched with the first m candidate words is added to the new word set, and m is a positive integer; or

The candidate words whose matching times with the templates in the candidate template set exceed a certain threshold are added to the new word set.

Optionally, in the step of sorting the candidate words in the candidate word set based on the templates in the candidate template set,

According to one of LLR( _wi ), E( _wi ), P( _wi ), EMI( _wi ), 1/NMED( _wi ), or the product of any of them to calculate the candidate word set The weight weight( _wi ) of the candidate word, for example, according to LLR( _wi ), LLR( _wi )*E( _wi ), LLR( _wi )*E( _wi )*P( _wi ), LLR( _wi )*E( _wi )*EMI( _wi ) or LLR( _wi )*E( _wi )/NMED( _wi ) calculates the weight of the candidate words in the candidate word set, and based on the The calculated weight ranks the candidate words in the candidate word set; of course, the calculation of the weight weight(w _i ) is not limited to the above-mentioned calculation method, any word that can evaluate the closeness or/and The calculation methods of the cohesion measure of the candidate words are all applicable to the present invention, and are hereby incorporated by reference.

Here, w _i represents a candidate word in the candidate word set W, LLR( _wi ) represents the closeness of the candidate word w _i to the template in the candidate template set, E( _wi ) represents the left side of the candidate word w _i Information entropy, P(w _i ) represents the probability that the words in the candidate word w _i combine to form a word, and EMI( _wi ) and NMED( _wi ) represent different measures of the semantic synthesis of the candidate word w _i respectively.

Wherein, in the step of sorting the candidate words in the candidate word set based on the templates in the candidate template set, LLR( _wi ), E( _wi ), P( _wi ), EMI( _wi ), 1 /NMED(w _i ) can be calculated as follows:

n _1j =k _1j +k _3j , n ₂ j =k _2j +k _4j ,

Wherein W represents the set of candidate words, P represents the set of candidate templates, w _i represents a candidate word in W, p _j represents a template in the set of candidate templates P, and k _1j represents a template found from the document set that matches the one or The matching number of each template in multiple templates contains both w _i and p _j , k _2j means that the matching of each template found in the document set and the one or more templates contains w _i but does not contain the number of matches of p _j , k _3j represents the number of matches that contain p _j but does not contain w _i in the matching of each template in the one or more templates found in the document set, k _4j represents the number of matches that neither p _j nor w _i are found in the matching of each template in the one or more templates found in the document set; , so awesome, so cheating, so cheating", the candidate word "Great", the template p ₁ "too +/*+" in the candidate template set P "too +/*+" as an example, the calculation can be obtained : k ₁₁ =1, k ₂₁ =1, k ₃₁ =1, k ₄₁ =2;

Among them, the calculation of E(w _i ) is as follows:

Among them, L represents the left word l _o set that has appeared in the document set on the left side of the candidate word w _i and matches any template in the candidate template set, and c(l _o ) represents the left word l _o and the candidate word w _i The number of times that the left collocation has appeared and matched with any template in the candidate template set, and N represents the total number of times that the candidate word w _i appears together with the templates in the candidate template set; usually the left information entropy E(w _i ) is higher The larger the value, the higher the diversity of collocations between the candidate words and the left-hand words in the candidate template set, which means that the candidate words are more likely to become new words; La", "It's too powerful", "It's so awesome", "It's really awesome"}, the candidate word "Great", the candidate template collection {"Good ++", "Too ++", "Too ++ ", "Good ++", "True ++"} as an example, then the set of words l _o on the left side is {good, too, true}, c(l ₁ )=2, c(l ₂ )= 2, c(l ₃ )=1, the calculation of E(w _i ) is as follows:

E(w _i )=-2/5log(2/5)-2/5log(2/5)-1/5log(1/5);

Among them, for the candidate word w _i in the candidate word set, the calculation of its P(w _i ) is as follows:

Among them, t _h represents the hth word in the candidate word w _i in the candidate word set, and n represents the number of individual words contained in the candidate word w _i ;

all(t _h ) indicates the number of occurrences of the hth word in the candidate word w _i in the document set, s(t _h ) indicates that the hth word in the candidate word w _i and any word are used as a single word in the document set The number of occurrences; Take the word "love to say" as an example, "love" is the first word in the word "love to say", and "say" is the second word in the word "love to say", after statistics, get "Love" appears 100 times in the document set, "say" appears 200 times in the document set, "love" and other words in the document set appear 50 times as a single word, "say" and other words in the document set form a single word The word appeared 150 times, thus, the probability of calculating the first word "love" into a word is: P(love)=(100-50)/100, and the probability of the second word "say" into a word is: P (say)=(200-150)/200, the probability that the candidate word " love to say " becomes a word separately is: P (love to say)=(P (love)/(1-P (love)))*(P (say)/(1-P(say)));

Among them, for the candidate word w _i in the candidate word set, the calculation of its semantic composition measure EMI is as follows:

Where S is the total number of segments in the document set M, n indicates the number of words contained in the candidate word w _i , F indicates the number of segments in the document set containing the candidate word w _i , and F _h indicates that the document set contains the candidate word w _i in the first The number of segments of h words. The segment mentioned here can be defined as the natural segment of each document in the document set M, and can also be defined as each document in the document set M, which is not limited here. For example, the document set M consists of multiple microblogs composition, each microblog can be regarded as a segment. Taking the candidate word "Geili" as an example, the total number of segments in the document set M is 200, the number of segments for the candidate word "Geili" in the document set M is 10, and the number of segments for the word "Gei" in the document set M is 20 , the number of segments of the word "force" in the document set M is 30, then S=200, n=2, F=10, F ₁ =20, F ₂ =30, and the calculation is:

Among them, for the candidate word w _i in the candidate word set, its semantic composition measure NMED is calculated as follows:

Where S is the total number of segments in the document set M, μ(g) represents the number of segments in the document set M that contain all the words contained in the candidate word w _i , (g) indicates the number of segments in which all the words contained in the candidate word w _i in the document set M appear strictly consecutively in one segment. Usually, the smaller the value of NMED(w _i ), the greater the probability that the words in the candidate word w _i and the characters other than the candidate word w _i form words. As far as all the characters in the candidate word _w appear scattered, for example, taking the candidate word "Gili" as an example, the situation where it appears in a segment of the document collection can be: "I have been busy for a day and have no strength at all. I really knelt down. .”; a situation where it appears strictly consecutively in a sentence segment in the document collection can be: “Their performance is really awesome!”.

Optionally, before sorting the candidate words in the candidate word set based on the templates in the candidate template set, sort the templates in the candidate template set with a predefined new word set, and sort the templates in the candidate template set based on the predefined new word set The set of new words filters the set of candidate templates by sorting the templates in the set of candidate templates. Wherein, the words in the new word set can be any number of any words specified in advance. In order to improve the accuracy of new word discovery, words that have been considered newer or more popular by users before new word discovery are usually selected to be added to the new word set.

Optionally, the step of sorting the templates in the set of candidate templates with a pre-specified set of new words, and filtering the set of candidate templates based on the sorting includes:

Filtering the candidate templates whose number of times matching the new words in the new word set ranks the top r from the candidate template set to obtain the filtered candidate template set, r is a positive integer; or

The candidate templates whose matching times with the new words in the new word set are higher than a specific threshold are filtered from the candidate template set to obtain a filtered candidate template set.

Optionally, the method further includes: sorting the templates in the candidate template set with the obtained new word set, and filtering the candidate template set based on the sorting of the templates in the candidate template set based on the obtained new word set, and using The candidate template set after filtering sorts the candidate words in the candidate word set again and adds a certain number of candidate words based on the sorting of the candidate words in the candidate word set again based on the filtered candidate template set to a collection of new words. The "again" here does not limit the number of times, and the number of times can be adjusted according to the set conditions. It should be noted that the "obtained new word set" here refers to the "pre-specified new word set" obtained by processing the "pre-specified new word set" through the method for automatically discovering new words from a document set provided by the present invention.

Optionally, the step of using the obtained new word set to sort the templates in the candidate template set, and filtering the candidate template set based on the used new word set to sort the templates in the candidate template set includes:

Filtering the candidate templates whose number of times matching the new words in the new word set ranks the top r from the candidate template set to obtain the filtered candidate template set, r is a positive integer; or

The candidate templates whose matching times with the new words in the new word set are higher than a specific threshold are filtered from the candidate template set to obtain a filtered candidate template set.

Optionally, using the obtained new word set to sort the templates in the candidate template set by calculating the template weights in the candidate template set based on the following formula and sorting the templates in the candidate template set according to the calculated template weights of:

n _1i =k _1i +k _3i , n _2i =k _2i +k _4i ,

Where W represents a new word set, P represents a candidate template set, _wi represents a word in the new word set W, p _j represents a template in the candidate template set P, and k _1i represents a template found from the document set that is similar to the The matching number of each template in one or more templates contains both w _i and p _j , and k _2i represents the number of matches found from the document set and each template in the one or more templates Contains w _i but does not contain the number of matches of p _j , k _3i represents the number of matches that contain p _j but does not contain w _i found in the matching of each template in the one or more templates found in the document set , k _4i represents the number of matches found in the document set that contain neither p _j nor w _i among the matches to each of the one or more templates.

Fig. 2 shows a schematic block diagram of an apparatus for automatically discovering new words from a document collection according to another embodiment of the present invention. According to another embodiment of the present invention, the device for automatically discovering new words from a document collection includes:

a template acquiring unit 201 configured to acquire one or more templates;

a word extraction unit 202 configured to extract words matching each template in the one or more templates from the document set;

The candidate template set adding unit 203 is configured to select at least a part of templates from the one or more templates and add them to the candidate template set;

The candidate word set adding unit 204 is configured to select at least a part of words from the extracted words that match each template in the one or more templates and add them to the candidate word set;

The new word set adding unit 205 is configured to sort the candidate words in the candidate word set based on the templates in the candidate template set, and to sort the candidate words in the candidate word set based on the templates in the candidate template set Add a certain number of candidate words to the new word set.

It should be understood that the block diagram shown in FIG. 2 is for the purpose of illustration only, rather than limiting the scope of the present invention. In some cases, some units or devices may be added or subtracted according to the specific situation.

Optionally, the template acquiring unit 201 acquires the one or more templates in any of the following ways:

pre-specifies said one or more templates, or

After the document set is acquired, word segmentation processing is performed on the document set, and the one or more templates matching the specific regular expression are extracted from the word-segmented document set.

Optionally, the device for automatically discovering new words from the document collection also includes:

The candidate template set filtering unit 206 is configured to sort the templates in the candidate template set with a pre-specified new word set before sorting the candidate words in the candidate word set based on the templates in the candidate template set, and based on The sorting of the templates in the candidate template set by using the pre-specified new word set filters the candidate template set.

Optionally, the new word set adding unit 205 is configured to use the filtered candidate template set to sort the candidate words in the candidate word set again and add a certain number of candidate words to the new word based on the sorting. gather.

Those skilled in the art know that the present invention can be realized as a device, an apparatus, a method or a computer program product. Therefore, the present disclosure can be specifically implemented in the following forms, namely: it can be completely hardware, it can be completely software, and it can also be a combination of hardware and software.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, program segment, or part of code that includes one or more Executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks in succession may, in fact, be executed substantially concurrently, or they may sometimes be executed in the reverse order, depending upon the functionality involved. It should also be noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented by a dedicated hardware-based system that performs the specified functions or operations , or may be implemented by a combination of dedicated hardware and computer instructions.

It will be apparent to those skilled in the art that the invention is not limited to the details of the above-described exemplary embodiments, but that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Accordingly, the embodiments should be regarded in all points of view as exemplary and not restrictive, the scope of the invention being defined by the appended claims rather than the foregoing description, and it is therefore intended that the scope of the invention be defined by the appended claims rather than by the foregoing description. All changes within the meaning and range of equivalents of the elements are embraced in the present invention. Any reference sign in a claim should not be construed as limiting the claim concerned.

Claims (10)

1. A method of automatically discovering new words from a document collection, comprising:

acquiring one or more templates (S101), wherein the templates comprise words and spaces or/and custom symbols;

extracting words matched with each template in the one or more templates from the document set (S102), wherein the words matched with each template in the one or more templates extracted from the document set are words except for words included in the template;

selecting at least one part of templates from the one or more templates and adding the selected part of templates into a candidate template set (S103);

at least selecting a part of words from the extracted words matched with each template in the one or more templates and adding the selected words into a candidate word set (S104);

the candidate words in the set of candidate words are ranked based on the templates in the set of candidate templates, and a number of candidate words are added to the set of new words based on the ranking of the candidate words in the set of candidate words with the templates in the set of candidate templates (S105).

2. The method of claim 1, wherein the one or more templates are obtained by any one of:

predefining said one or more templates, or

And after a document set is obtained, performing word segmentation processing on the document set, and extracting the one or more templates matched with the specific regular expression from the document set subjected to word segmentation processing.

3. The method of claim 1, wherein the step of selecting at least a portion of the templates from the one or more templates for adding to the set of candidate templates comprises any one of:

adding all of the one or more templates into a candidate template set;

adding a portion of the templates to a set of candidate templates based on a number of times each of the one or more templates occurs in the set of documents.

4. The method of claim 3, wherein adding a portion of the templates to the set of candidate templates based on a number of times each of the one or more templates occurs in the set of documents comprises:

adding the template with the first f names in the frequency of occurrence in the document set into a candidate template set, wherein f is a positive integer; or

Adding templates which appear in the document set for more than a certain threshold number of times into a candidate template set.

5. The method of claim 1, wherein the step of selecting at least a portion of the extracted words that match each of the one or more templates to be added to the set of candidate words comprises any one of:

adding all the matched words into a candidate word set;

and adding a part of words into a candidate word set based on the matching times of the matched words and the templates.

6. The method of claim 5, wherein adding a portion of words to a set of candidate words based on the number of matches of the matched words to each template comprises:

adding the words with the first g matched times of each template in the matched words into a candidate word set, wherein g is a positive integer; or

And adding the words of which the matching times with the templates exceed a specific threshold in the matched words into the candidate word set.

7. The method of claim 1, further comprising: before the candidate words in the candidate word set are sorted based on the templates in the candidate template set, the templates in the candidate template set are sorted by a preset new word set, and the candidate template set is filtered based on the sorting of the templates in the candidate template set by the preset new word set.

8. The method of claim 1, further comprising: and ranking the templates in the candidate template set by using the obtained new word set, filtering the candidate template set based on the ranking of the templates in the candidate template set by using the obtained new word set, ranking the candidate words in the candidate word set again by using the filtered candidate template set, and adding a certain number of candidate words into the new word set again based on the ranking of the candidate words in the candidate word set again by using the filtered candidate template set.

9. The method of claim 1, wherein in the step of ordering candidate words in the set of candidate words based on templates in the set of candidate templates,

according to LLR (w)_i)、E(w_i)、P(w_i)、EMI(w_i)、1/NMED(w_i) Calculating weights of the candidate words in the candidate word set, and ordering the candidate words in the candidate word set based on the calculated weights;

wherein, w_iRepresents one candidate word in the set of candidate words, LLR (W)_i) Represents a candidate word w_iCloseness of contact with templates in the candidate template set, E (w)_i) Represents a candidate word w_iLeft entropy of P (w)_i) Represents a candidate word w_iProbability of word-to-word, EMI (w)_i) And 1/NMED (w)_i) Respectively represent the candidate words w_iOf semantic synthesis, wherein LLR (w)_i)、E(w_i)、P(w_i)、EMI(w_i)、1/NMED(w_i) Respectively obtained by the following calculation:

n_1j＝k_1j+k_3j，n_2j＝k_2j+k_4j，

wherein W represents a set of candidate words, P represents a set of candidate templates, W_iRepresenting a candidate word in W, p_jRepresenting one template, k, of the set of candidate templates P_1jIndicating that a match found from the document set to each of the one or more templates contains both w and w_iAnd also contains p_jK is the number of matches_2jIndicating that a match found from the document set with each of the one or more templates contains w_iBut does not contain p_jK is the number of matches_3jIndicating that a match found from the document set with each of the one or more templates contains p_jBut does not contain w_iK is the number of matches_4jIndicating that matches found from the document set to each of the one or more templates contain no p_jNor w_iThe matching number of (2);

where L represents the document set and the candidate word w_iMatching the left side with the left side word l which is already generated and matched with any template in the candidate template set_oSet, c (l)_o) Represents the left-hand word l_oAnd candidate word w_iThe frequency of matching occurring on the left side and matching with any template in the candidate template set, N represents the candidate word w_iA total number of co-occurrences with templates in the set of candidate templates;

wherein t is_hRepresenting a candidate word w in a set of candidate words_iH-th word in (1), n represents a candidate word w_iThe number of individual words contained therein;

all(t_h) Represents a candidate word w_iThe number of times the h-th word in (b) appears in the document set, s (t)_h) Represents a candidate word w_iThe number of times that the h-th word and any word in the document set appear as a single word;

wherein S is the total number of paragraphs in the document set M, and n represents a candidate word w_iThe number of contained characters, F represents that the document set contains candidate words w_iNumber of speech segments of, F_hIndicating that the document set contains a candidate word w_iThe number of the h-th word;

wherein S is the total number of paragraphs in the document set M, and μ (g) represents that the candidate word w is contained in the document set M_iThe number of tokens of all words contained,representing candidate words w contained in the document set M_iAll the contained words appear strictly consecutively in the number of segments of a single phrase.

10. An apparatus for automatically discovering new words from a document collection, comprising:

a template obtaining unit (201) configured to obtain one or more templates, the templates comprising words and spaces or/and custom symbols;

a word extraction unit (202) configured to extract words matching each of the one or more templates from the document set, the words matching each of the one or more templates being words other than the words included in the template;

a candidate template set adding unit (203) configured to select at least a part of the one or more templates from the one or more templates to add to the candidate template set;

a candidate word set adding unit (204) configured to select at least a part of words from the extracted words matched with each of the one or more templates and add the selected words to a candidate word set;

a new word set adding unit (205) configured to rank the candidate words in the candidate word set based on the templates in the candidate template set, and add a certain number of candidate words to the new word set based on the ranking of the candidate words in the candidate word set by the templates in the candidate template set.