CN113705227B - Method, system, medium and equipment for constructing Chinese word-segmentation-free word embedding model - Google Patents

Abstract

The present invention provides a method, system, medium and equipment for constructing a Chinese word-segmentation-free word embedding model. The method for building the Chinese word-segmentation-free word embedding model includes: counting candidate segments in a corpus and word frequency information corresponding to the candidate segments; Combining the word frequency information to determine the correlation strength of the candidate fragments, and generating a word embedding vocabulary according to the correlation strength; constructing a positive sampling set and a negative sampling set according to the vocabulary, and combining the positive sampling set and the negative sampling set The sampling set builds a word embedding model. The present invention aims at the problem of excessive noise n-grams in the current non-segmented word embedding model vocabulary, takes Chinese corpus as the research object, and utilizes the skip-gram model of negative sampling to provide an improved non-segmented word using unsupervised correlation metrics. A method for word embedding models.

Description

Construction method, system, medium and equipment of Chinese word-segmentation-free word embedding model

technical field

The invention belongs to the technical field of natural language processing, and relates to a design method of a word embedding model, in particular to a construction method, system, medium and equipment of a Chinese unsegmented word embedding model.

Background technique

As a basic task in the field of natural language processing, word embedding plays an important role in downstream tasks such as machine translation and part-of-speech tagging. Since there are no obvious separators between words in the Chinese corpus, the existing Chinese word embedding usually needs to perform Chinese word segmentation first, so as to obtain the word after word segmentation as the target of word embedding. However, there are still many problems in the current Chinese word segmentation, and these problems will seriously affect the quality of Chinese word embedding. Therefore, for languages like Chinese, in order to avoid the impact of word segmentation errors, a word-segmentation-free word embedding model was proposed and proved to be superior to traditional word embedding methods.

The current non-segmented word embedding model mainly collects n-gram fragments with the highest frequency of Top-K words as the object of model training. But only considering word frequency will lead to a large number of noisy n-gram fragments in the word embedding vocabulary, which will affect the quality of the final generated word embedding.

Therefore, how to provide a design method of a word embedding model without word segmentation, reduce the impact of a large number of noise n-gram fragments on the quality of the final generated word embedding model, and improve the quality of the word embedding model has become an urgent problem for those skilled in the art. technical problem.

Contents of the invention

In view of the shortcomings of the prior art described above, the object of the present invention is to provide a method, system, medium and equipment for building a Chinese word-segmentation-free word embedding model, which is used to solve the problem that the prior art cannot reduce the impact of a large number of noise n-gram fragments on the final The impact of the quality of the generated word embedding model, the problem of improving the quality of the word embedding model.

In order to achieve the above purpose and other related purposes, the present invention provides a method for constructing a Chinese word embedding model without word segmentation, the method for building the Chinese word embedding model includes: a candidate segment in a statistical corpus and the candidate segment Corresponding word frequency information; determine the correlation strength of the candidate segment in combination with the word frequency information, and generate a word embedding vocabulary according to the correlation strength; construct a positive sampling set and a negative sampling set according to the vocabulary, and combine the Positive sampling sets and negative sampling sets build word embedding models.

In one embodiment of the present invention, the candidate segments are Chinese language model segments, and the step of counting the candidate segments in the corpus and the word frequency information corresponding to the candidate segments includes: counting different fixed lengths in the corpus Value corresponding to the Chinese language model segment and its word frequency information.

In an embodiment of the present invention, the step of determining the association strength of the candidate segments in combination with the word frequency information, and generating a word embedding vocabulary according to the association strength includes: determining the candidate fragments in combination with the word frequency information. The unsupervised correlation metric of the segment, the unsupervised correlation metric characterizes the correlation strength of the candidate segment; the correlation strengths are arranged in order from large to small, and the first K candidate segments of the correlation strength are selected as the vocabulary of word embedding surface.

In an embodiment of the present invention, the step of determining the unsupervised association metric index of the candidate segment in combination with the word frequency information includes: calculating the mutual information value of the candidate segment, and determining when the mutual information value is the smallest corresponding to The segment combination; determine the first set and the second set according to the segment combination, and calculate the statistical relationship value between the segment combination and the first set or the second set; combine the word frequency information, mutual information value and statistical relationship value The product of the three is used as an unsupervised correlation measure.

In an embodiment of the present invention, the step of determining the first set and the second set according to the fragment combination, and calculating the statistical relationship value between the fragment combination and the first set or the second set includes: The maximum value in the ratio of word frequency information and the word frequency of the first set, and the ratio of the word frequency of the second set is used as the numerator, and the set with the smallest word frequency in the first set and the second set is selected, and the number of elements in the set is taken. The reciprocal is used as the denominator; the fractional calculation value formed by the numerator and the denominator is used as the relative importance calculation value of the fragment combination in the first set or the second set; the statistics are determined according to the relative importance calculation value relationship value.

In an embodiment of the present invention, the calculation of the correlation strength is performed on the candidate segments of each length; according to the candidate segments of different lengths, the correlation strengths are arranged in order from large to small under each length , select the top K candidate fragments of the correlation strength as the word embedding vocabulary, and select different numbers of candidate fragments according to different lengths as the word embedding vocabulary.

In an embodiment of the present invention, the step of constructing a positive sampling set and a negative sampling set according to the vocabulary, and combining the positive sampling set and the negative sampling set to construct a word embedding model includes: combining with a skip-gram model On the basis of negative sampling, the method of parameter optimization is adopted to maximize the probability of positive sampling and minimize the probability of negative sampling to construct the word embedding model.

Another aspect of the present invention provides a system for constructing a Chinese non-segmented word embedding model. The system for constructing a Chinese non-segmented word embedding model includes: a segment statistics module for counting candidate segments in a corpus and the corresponding candidate segments Word frequency information; Association measurement module, used to determine the association strength of the candidate segment in conjunction with the word frequency information, and generate a word embedding vocabulary according to the association strength; Model generation module, used to construct positive sampling according to the vocabulary set and negative sampling set, and combine the positive sampling set and negative sampling set to build a word embedding model.

Another aspect of the present invention provides a medium on which a computer program is stored, and when the computer program is executed by a processor, the method for constructing the Chinese word-segmentation-free word embedding model is realized.

The last aspect of the present invention provides a device, including: a processor and a memory; the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the device executes the Chinese A method for building word embedding models without segmentation.

As mentioned above, the construction method, system, medium and equipment of the Chinese word-segmentation-free word embedding model described in the present invention have the following beneficial effects:

A new unsupervised association metric is proposed for screening n-gram segments with strong associations. Combining this unsupervised correlation measure with the word embedding model, a new unsegmented Chinese word embedding model for Chinese corpus is constructed. The word embedding model obtained by the present invention can show better performance in downstream tasks.

Description of drawings

Fig. 1 shows the principle flow chart of the construction method of the Chinese word embedding model without word segmentation in one embodiment of the present invention.

FIG. 2 is a flow chart of correlation measurement in an embodiment of the construction method of the Chinese word-segmentation-free word embedding model of the present invention.

Fig. 3 is a flow chart showing the correlation strength calculation in an embodiment of the construction method of the Chinese word-segmentation-free word embedding model of the present invention.

Fig. 4 shows the effect diagram comparing the construction method of the Chinese word-segmentation-free word embedding model of the present invention with the basic dictionary.

Fig. 5 shows the effect diagram of the comparison between the construction method of the Chinese word-segmentation-free word embedding model of the present invention and the rich dictionary.

FIG. 6 is a structural schematic diagram of an embodiment of the construction system of the Chinese word-segmentation-free word embedding model of the present invention.

FIG. 7 is a schematic diagram showing the structural connection of the construction equipment of the Chinese word-segmentation-free word embedding model in an embodiment of the present invention.

Component designation description

6 Construction system of Chinese word-segmentation-free word embedding model

61 Fragment statistics module

62 Association measurement module

63 Model generation module

7 Equipment

71 processor

72 memory

73 Communication interface

74 System bus

S11～S13 Steps

Steps S121～S122

S121A～S121C steps

Detailed ways

Embodiments of the present invention are described below through specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific implementation modes, and various modifications or changes can be made to the details in this specification based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, in the case of no conflict, the following embodiments and features in the embodiments can be combined with each other.

It should be noted that the diagrams provided in the following embodiments are only schematically illustrating the basic ideas of the present invention, and only the components related to the present invention are shown in the diagrams rather than the number, shape and shape of the components in actual implementation. Dimensional drawing, the type, quantity and proportion of each component can be changed arbitrarily during actual implementation, and the component layout type may also be more complicated.

The method for constructing the Chinese non-segmented word embedding model provided by the present invention aims at the problem of excessive noise n-grams in the vocabulary of the current non-segmented word embedding model. Taking Chinese corpus as the research object and using the skip-gram model of negative sampling, it provides A method for improving unsegmented word embedding models using unsupervised association metrics.

The principles and implementations of a construction method, system, medium and equipment of a Chinese word-segmentation-free word embedding model of this embodiment will be described in detail below in conjunction with FIGS. 1 to 7, so that those skilled in the art can understand this method without creative work The construction method, system, medium and equipment of the Chinese word-segmentation-free word embedding model of the embodiment.

Please refer to FIG. 1 , which shows a schematic flow chart of an embodiment of a method for constructing a Chinese word-segmentation-free word embedding model of the present invention. As shown in Figure 1, the construction method of the Chinese word embedding model without word segmentation specifically includes the following steps:

S11. Count candidate segments in the corpus and word frequency information corresponding to the candidate segments.

In this embodiment, the candidate segment is a Chinese language model segment, for example, the candidate segment is an n-gram segment, and the n-gram segments corresponding to different fixed length values and their word frequency information are counted in the corpus.

Specifically, implement a simple tokenizer through the n-gram model to obtain n-gram fragments. The model is based on the assumption that the appearance of the nth word is only related to the previous n-1 words, but not to any other words, and the probability of the entire sentence is the product of the occurrence probabilities of each word. In general, we only calculate the probability of two words before and after a word, that is, take n as 2, and calculate the probability of n-2,.n-1, n+1, n+2. If n=3, the calculation effect will be better; if n=4, the calculation amount will become very large.

Specifically, the corpus is sorted, and all possible n-gram fragments and their word frequency information under a fixed length are counted. Table 1 is formed by managing the n-gram fragments of different lengths and the corresponding word frequency information in a list. For example, in Table 1, the word "一" with a length of 1 Chinese character has a word frequency of 529285.

Table 1 Candidate field table

S12. Determine the association strength of the candidate segments in combination with the word frequency information, and generate a word embedding vocabulary according to the association strength.

In this embodiment, the correlation strength is calculated for the candidate segments of each length.

Further, according to candidate fragments of different lengths, at each length, the association strengths are arranged in order from large to small, and the top K candidate fragments of association strength are selected as the word embedding vocabulary, so as to select according to different lengths Different numbers of candidate segments serve as vocabulary for word embeddings.

Please refer to FIG. 2 , which shows a flow chart of correlation measurement in an embodiment of the construction method of the Chinese word-segmentation-free word embedding model of the present invention. As shown in Figure 2, S12 includes:

S121. Determine an unsupervised association metric of the candidate segment in combination with the word frequency information, where the unsupervised association metric characterizes an association strength of the candidate segment. The PATI (Pointwise Association with Times Information, unsupervised association measurement index) of the present invention can explore more strongly associated n-gram segments by considering more statistical information.

Please refer to FIG. 3 , which shows a flow chart of correlation strength calculation in an embodiment of the construction method of the Chinese word-segmentation-free word embedding model of the present invention. As shown in Figure 3, S121 includes:

S121A. Calculate the mutual information value of the candidate segments, and determine a segment combination corresponding to the minimum mutual information value.

Specifically, the mutual information value is an MP value. For each n-gram segment g=w _i w _i+1 ... w _i+s (0≤i≤Ns) of length s, the left, The right two parts are respectively a=w _i ...w _k-1 , b=w _k ...w _i+s (i<k<i+s), that is, g=concat(a,b). f _a , f _b and f _g represent the word frequencies of strings a, b and n-gram segment g in the corpus, respectively.

For an n-gram segment g=concat(a,b), its corresponding MP is defined as:

For an n-gram segment g of fixed length s, there always exists a specific combination of left and right a, b (a _m , b _m ) that minimizes MP. Subsequently, the calculation of AT will also be based on this specific combination (a _m ,b _m ).

S121B. Determine a first set and a second set according to the fragment combination, and calculate a statistical relationship value between the fragment combination and the first set or the second set.

In this embodiment, S121B includes:

(1) with the maximum value in the ratio of the word frequency of described word frequency information and the first set, and the ratio of the word frequency of the second set as the numerator, select the set with the smallest word frequency in the first set and the second set, get the The reciprocal of the number of elements in the set is used as the denominator.

For a specific combination (a _m ,b _m ) of n-gram segments g, there are a batch of n-gram segments (a _m ,b _h ) and (a _j ,b _m ) with the same length, then the first set {a _m ,*} and the second set {*,b _m } are defined as follows:

{a _m ,*}＝{(a _m ,b ₁ ),(a _m ,b ₂ ),…,(a _m ,b _h )} formula (2)

{*,b _m }＝{(a ₁ ,b _m ),(a ₂ ,b _m ),…,(a _j ,b _m )} formula (3)

和

分别表示在集合{a_m,*}和{*,b_m}中的所有n-gram片段的词频的总和，则其定义为：make

and

represent the sum of the word frequencies of all n-gram segments in the sets {a _m ,*} and {*,b _m } respectively, then it is defined as:

比值以及f_g与

比值中的最大值，即rate的定义如下：For n-gram segment g and its specific combination (a _m , b _m ), the variable rate represents f _g and

Ratio and f _g vs.

The maximum value in the ratio, ie rate, is defined as follows:

和

令sizeof代表集合内n-gram元素的个数，则AC可定义为：For two sets {a _m ,*} and {*,b _m } and their corresponding

and

Let sizeof represent the number of n-gram elements in the set, then AC can be defined as:

(2) The calculated value of the numerator composed of the numerator and the denominator is used as the calculated value of the relative importance of the combination of the fragments in the first set or the second set.

Specifically, given the variable rate and the variable AC, the times value of the n-gram segment g is defined as follows:

(3) Determine the value of the statistical relationship according to the calculated value of the relative importance.

Specifically, for a specific combination (a _m ,b _m ) of an n-gram segment g of length s, there is a unique variable times, then the calculation formula of AT is:

AT＝1+|logtimes| Formula (9)

S121C, taking the product of the word frequency information, the mutual information value and the statistical relationship value as an unsupervised correlation measurement index.

Specifically, the formula of PATI (Pointwise Association with Times Information, unsupervised association metrics) is as follows:

PATI＝F×MP×AT Formula (10)

Among them, F=fg is the word frequency information, MP is the mutual information value, and AT is the statistical relationship value.

MP is an improved version of the mutual information PMI, which takes into account more of the marginal variables of each n-gram segment g=concat(a,b) in the calculation of the correlation strength, that is, the left and right sides of the n-gram The statistics of parts a and b can be more sensitive to the local information of n-gram.

和(a_m,b_m)的词频信息以及(a_m,b_m)的前、后邻接数，来评估(a_m,b_m)在集合中的相对重要程度，times值越高，通常说明(a_m,b_m)作为一个整体是合理的。一般来说，大部分关联强度较高的合理n-gram的times值要远大于那些不合理的n-gram片段的times值。AT further measures the association strength of n-grams by exploiting the statistics of each n-gram fragment for a specific combination ( _am , b _m ) in the set { _am ,*} or {*,b _m }. The variable times is considered by

and (a _m ,b _m ) word frequency information and (a _m ,b _m ) front and back adjacency numbers to evaluate the relative importance of (a _m ,b _m ) in the set, the higher the times value, usually indicates (a _m , b _m ) is reasonable as a whole. Generally speaking, the times value of most reasonable n-grams with high correlation strength is much larger than the times value of those unreasonable n-gram fragments.

S122. Arrange the correlation strengths in descending order, and select K candidate segments with the top K correlation strengths as a word embedding vocabulary.

Specifically, the proposed unsupervised association metric is used to calculate the association strength of candidate n-gram segments for each segment length. Then select the n-gram segment with the highest Top-K correlation strength as the vocabulary of the word embedding model. Among them, the problem of finding the first k largest or the first k smallest among a large number of numbers is referred to as the Top-K problem.

S13. Construct a positive sampling set and a negative sampling set according to the vocabulary, and combine the positive sampling set and the negative sampling set to construct a word embedding model.

In this embodiment, based on the skip-gram model combined with negative sampling, the maximum likelihood estimation method is used to maximize the positive sampling probability and minimize the negative sampling probability to construct the word embedding model. The present invention uses unsupervised correlation metrics to filter word embedding vocabulary, reconstructs positive sampling and negative sampling of the word embedding model, reduces the influence of noisy n-gram fragments on the model, and thus improves the efficiency of word embedding in downstream tasks. Performance.

It should be noted that the maximum likelihood estimation is only one embodiment of parameter estimation and optimization in the present invention, and other methods that can realize parameter estimation and optimization are also included in the protection scope of the present invention.

Specifically, PFNE learns word embedding based on the skip-gram model of negative sampling, thereby reducing the amount of calculation when the model is gradient-descent and speeding up model training. The positive sampling set N _p of the model is the "central word-context pair" (w _t , w _c ) generated by the combination of vocabulary and corpus, and the negative sampling set N _n is constructed by building a sufficiently large unary language model vocabulary. Each n-gram segment is indexed in the table, and negative sampling samples are randomly obtained according to the word frequency of the n-gram in the vocabulary. The objective function of the PFNE model is defined as follows:

和

分别是中心词w_t和其上下文w_c的向量表示，模型使用极大似然估计，根据中心词来预测上下文，最大化正样本的概率，同时最小化负采样的概率，以使目标函数生成的词嵌入模型最优。对该目标函数的优化采用的是基于正、负采样的随机梯度下降方法。in,

and

They are the vector representations of the central word w _t and its context w _c respectively. The model uses maximum likelihood estimation to predict the context according to the central word, maximize the probability of positive samples, and minimize the probability of negative samples, so that the objective function generates The best word embedding model. The optimization of this objective function adopts the stochastic gradient descent method based on positive and negative sampling.

Please refer to Fig. 4 and Fig. 5, which are respectively shown as the effect diagram of the Chinese non-segmented word embedding model of the present invention compared with the basic dictionary and the effect diagram of the Chinese non-segmented word embedding model of the present invention compared with the rich dictionary . In Figure 4 and Figure 5, PFNE represents the result of comparing the n-gram fragments screened by the PATI algorithm with the dictionary; sembei is the result of comparing the n-gram fragments screened by frequency (word frequency) with the dictionary; SGNS- PMI is the result of comparing the n-gram fragments screened by PMI (Pointwise Mutual Information, mutual information) with the dictionary. The vertical axis is the precision rate, and the horizontal axis is the recall rate. Among them, the expressions of precision and recall are as follows:

Further, the higher and longer the curve, the more reasonable n-gram fragments are screened out. As can be seen in Fig. 4 and Fig. 5, the curve of the solid line PFNE utilizing the PATI algorithm is the highest and longest among the three curves, thus illustrating that the construction method of the Chinese no-segmentation word embedding model of the present invention is different from that of the prior art Compared with the word embedding model (basic dictionary or rich dictionary), more reasonable n-gram fragments can be screened out.

The scope of protection of the construction method of the Chinese word-segmentation-free word embedding model described in the present invention is not limited to the execution sequence of the steps listed in this embodiment. All schemes are included in the protection scope of the present invention.

This embodiment provides a computer storage medium on which a computer program is stored, and when the computer program is executed by a processor, the method for constructing the Chinese word-segmentation-free word embedding model is realized.

Those of ordinary skill in the art can understand that all or part of the steps for implementing the above method embodiments can be completed by hardware related to computer programs. The aforementioned computer program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiments; and the aforementioned computer-readable storage medium includes: ROM, RAM, magnetic disk or optical disk and other computer storage media that can store program codes.

The system for constructing the Chinese word-segmentation-free word embedding model provided by this embodiment will be described in detail below with reference to diagrams. It should be noted that it should be understood that the division of the various modules of the following systems is only a division of logical functions, and may be fully or partially integrated into a physical entity or physically separated during actual implementation. Moreover, these modules can be implemented in the form of calling software through processing elements, or can be implemented in the form of hardware, or some modules can be implemented in the form of calling software through processing elements, and some modules can be implemented in the form of hardware. For example: a certain module may be a separate processing element, or it may be integrated into a certain chip of the following system. In addition, a certain module may also be stored in the memory of the system described below in the form of program code, and be called by a processing element of the system described below to execute the function of a certain module described below. The implementation of other modules is similar. All or part of these modules can be integrated together, and can also be implemented independently. The processing element mentioned here may be an integrated circuit with signal processing capability. In the implementation process, each step of the above method or the following modules can be completed by an integrated logic circuit of hardware in the processor element or an instruction in the form of software.

The following modules may be one or more integrated circuits configured to implement the above method, for example: one or more specific integrated circuits (Application Specific Integrated Circuit, referred to as ASIC), one or more digital signal processors (Digital Signal Processor , DSP for short), one or more Field Programmable Gate Arrays (Field Programmable Gate Array, FPGA for short), etc. When one of the following modules is implemented in the form of a processing element calling program code, the processing element may be a general-purpose processor, such as a central processing unit (Central Processing Unit, referred to as CPU) or other processors that can call program code. These modules can be integrated together and implemented in the form of a System-on-a-chip (SOC for short).

Please refer to FIG. 6 , which is a structural schematic diagram of an embodiment of the construction system of the Chinese word-segmentation-free word embedding model of the present invention. As shown in FIG. 6 , the construction system 6 of the Chinese word embedding model without word segmentation includes: a fragment statistics module 61 , an association measurement module 62 and a model generation module 63 .

The segment statistics module 61 is used to count the candidate segments in the corpus and the word frequency information corresponding to the candidate segments.

In this embodiment, the candidate segments are Chinese language model segments, and the segment statistics module 61 is specifically used to count the Chinese language model segments corresponding to different fixed length values and their word frequency information in the corpus.

The association measurement module 62 is used to determine the association strength of the candidate segments in combination with the word frequency information, and generate a word embedding vocabulary according to the association strength.

In this embodiment, the association measurement module 62 is specifically configured to determine the unsupervised association measurement index of the candidate segment in combination with the word frequency information, and the unsupervised association measurement index represents the association strength of the candidate segment; The above correlation strengths are arranged in order from large to small, and the top K candidate segments of the correlation strength are selected as the word embedding vocabulary.

The model generation module 63 is used to construct a positive sampling set and a negative sampling set according to the vocabulary, and combine the positive sampling set and the negative sampling set to construct a word embedding model.

In this embodiment, the model generation module 63 is specifically used to construct the word embedding model based on the skip-gram model combined with negative sampling, adopting a parameter optimization method to maximize the positive sampling probability and minimize the negative sampling probability .

The construction system of the Chinese non-segmented word embedding model of the present invention can realize the construction method of the Chinese non-segmented word embedding model of the present invention, but the realization device of the construction method of the Chinese non-segmented word embedding model of the present invention includes But it is not limited to the structure of the construction system of the Chinese word-segmentation-free word embedding model listed in this embodiment. Any structural deformation and replacement of the prior art based on the principles of the present invention are included in the scope of protection of the present invention.

Please refer to FIG. 7 , which is a schematic structural connection diagram of an embodiment of the construction equipment of the Chinese word-segmentation-free word embedding model of the present invention. As shown in Figure 7, the present embodiment provides a kind of equipment 7, and described equipment 7 comprises: processor 71, memory 72, communication interface 73 or/and system bus 74; Memory 72 and communication interface 73 communicate with processing through system bus 74 The device 71 is connected and completes mutual communication, the memory 72 is used to store computer programs, the communication interface 73 is used to communicate with other devices, and the processor 71 is used to run computer programs, so that the device 7 executes the Chinese word without participle Steps in the method of building the embedding model.

The system bus 74 mentioned above may be a Peripheral Component Interconnect (PCI for short) bus or an Extended Industry Standard Architecture (EISA for short) bus or the like. The system bus can be divided into address bus, data bus, control bus and so on. The communication interface 73 is used to realize the communication between the database access device and other devices (such as client, read-write library and read-only library). The memory 72 may include a random access memory (Random Access Memory, RAM for short), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

Above-mentioned processor 71 can be general-purpose processor, comprises central processing unit (Central Processing Unit, be called for short CPU), network processor (Network Processor, be called for short NP) etc.; Can also be digital signal processor (Digital Signal Processing, be called for short DSP) , Application Specific Integrated Circuit (ASIC for short), Field Programmable Gate Array (Field Programmable Gate Array, FPGA for short) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

To sum up, the construction method, system, medium and equipment of the Chinese non-segmented word embedding model of the present invention propose a new unsupervised correlation measurement index for screening n-gram segments with strong correlation. Combining this unsupervised correlation measure with the word embedding model, a new unsegmented Chinese word embedding model for Chinese corpus is constructed. The word embedding model obtained by the present invention can show better performance in downstream tasks. The invention effectively overcomes various shortcomings in the prior art and has high industrial application value.

The above-mentioned embodiments only illustrate the principles and effects of the present invention, but are not intended to limit the present invention. Anyone skilled in the art can modify or change the above-mentioned embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or changes made by those skilled in the art without departing from the spirit and technical ideas disclosed in the present invention should still be covered by the claims of the present invention.

Claims (7)

1. The method for constructing the Chinese word-segmentation-free word embedding model is characterized by comprising the following steps of:

counting candidate segments in a corpus and word frequency information corresponding to the candidate segments;

determining the association strength of the candidate segments by combining the word frequency information, and generating a word embedded vocabulary according to the association strength;

constructing a positive sampling set and a negative sampling set according to the vocabulary, and constructing a word embedding model by combining the positive sampling set and the negative sampling set;

determining the association strength of the candidate segments by combining the word frequency information, and generating a word embedded vocabulary according to the association strength, wherein the word frequency information comprises the following steps: determining an unsupervised association metric of the candidate segment in combination with the word frequency information, wherein the unsupervised association metric characterizes the association strength of the candidate segment; sequentially arranging the association strengths from large to small, and selecting K candidate fragments with the previous association strength as a word embedded vocabulary;

wherein determining the unsupervised association metric of the candidate segment in combination with the word frequency information comprises:

a, calculating the mutual information value of the candidate fragments, and determining the corresponding fragment combination when the mutual information value is minimum; the mutual information value is MP value, MP is defined as:

f _a ，f _b and f _g The word frequency of the character strings a, b and the n-gram fragment g in the corpus is represented respectively;

b, determining a first set and a second set according to the fragment combination, and calculating a statistical relation value between the fragment combination and the first set or the second set, wherein the method comprises the following steps:

(1) Taking the maximum value of the ratio of the word frequency information to the word frequency of the first set and the ratio of the word frequency information to the word frequency of the second set as a molecule, selecting the set with the minimum word frequency in the first set and the second set, and taking the reciprocal of the number of elements in the set as a denominator;

for a specific combination of n-gram fragments g (a _m ,b _m ) A batch of n-gram fragments (a _m ,b _h ) And (a) _j ,b _m ) Then the first set { a } _m (x) and (b) a second set _m Defined as: { a _m ,*}＝{(a _m ,b ₁ ),(a _m ,b ₂ ),…,(a _m ,b _h ) } and { b _m }＝{(a ₁ ,b _m ),(a ₂ ,b _m ),…,(a _j ,b _m ) -a }; order the

And->

Respectively expressed in the set { a } _m Sum }, b _m The sum of the word frequencies of all n-gram fragments in }, respectively defined as:

and +.>

For n-gram fragment gSpecific combinations (a) _m ,b _m ) The variable rate represents f _g And (3) with

Ratio f _g And->

Maximum in the ratio, rate is defined as: />

For two sets { a } _m Sum }, b _m ' and its corresponding

And->

Let sizeof represent the number of n-gram elements in the set, then AC is defined as: />

(2) Combining the calculated values of the numerator and the denominator as calculated values of the relative importance of the segment combination in the first set or the second set; given a variable rate as a numerator and a variable AC as a denominator, the calculated relative importance value time value of the n-gram fragment g is defined as:

(3) Determining the statistical relationship value AT according to the relative importance degree calculated value time; the calculation formula of the statistical relation value AT is as follows: at=1+|logtimes|;

c, making the word frequency information F equal to F _g Taking the product of the word frequency information F, the mutual information value MP and the statistical relation value AT as an unsupervised association measurement index; said non-monitoringThe formula of the governor-connection measurement index PATI is as follows: pati=f×mp×at.

2. The method for constructing a word-segmentation-free Chinese word embedding model according to claim 1, wherein the candidate segments are Chinese language model segments, and the step of counting the candidate segments in the corpus and word frequency information corresponding to the candidate segments comprises:

and counting the Chinese language model fragments and word frequency information thereof corresponding to different fixed length values in the corpus.

3. The method for constructing a Chinese word-segmentation-free word embedding model according to claim 1, wherein the method comprises the following steps of:

calculating the association strength of the candidate fragments of each length;

according to candidate fragments with different lengths, the association strength is sequentially arranged from large to small under each length, and K candidate fragments with the front association strength are selected as word embedded vocabularies, so that different numbers of candidate fragments are selected as word embedded vocabularies according to different lengths.

4. The method for constructing word-embedding models without word segmentation in chinese according to claim 1, wherein the step of constructing positive and negative sampling sets according to the vocabulary and constructing the word-embedding models in combination with the positive and negative sampling sets includes:

based on a skip-gram model combined with negative sampling, a parameter optimization method is adopted to maximize positive sampling probability and minimize negative sampling probability, and the word embedding model is constructed.

5. The system for constructing the Chinese word-segmentation-free word embedding model is characterized by comprising the following components:

the segment statistics module is used for counting candidate segments in the corpus and word frequency information corresponding to the candidate segments;

the association measurement module is used for determining association strength of the candidate fragments by combining the word frequency information and generating a word embedded vocabulary according to the association strength;

the model generation module is used for constructing a positive sampling set and a negative sampling set according to the vocabulary, and constructing a word embedding model by combining the positive sampling set and the negative sampling set;

determining the association strength of the candidate segments by combining the word frequency information, and generating a word embedded vocabulary according to the association strength, wherein the word frequency information comprises the following steps: determining an unsupervised association metric of the candidate segment in combination with the word frequency information, wherein the unsupervised association metric characterizes the association strength of the candidate segment; sequentially arranging the association strengths from large to small, and selecting K candidate fragments with the previous association strength as a word embedded vocabulary;

wherein determining the unsupervised association metric of the candidate segment in combination with the word frequency information comprises:

a, calculating the mutual information value of the candidate fragments, and determining the corresponding fragment combination when the mutual information value is minimum; the mutual information value is MP value, MP is defined as:

f _a ，f _b and f _g The word frequency of the character strings a, b and the n-gram fragment g in the corpus is represented respectively;

b, determining a first set and a second set according to the fragment combination, and calculating a statistical relation value between the fragment combination and the first set or the second set, wherein the method comprises the following steps:

(1) Taking the maximum value of the ratio of the word frequency information to the word frequency of the first set and the ratio of the word frequency information to the word frequency of the second set as a molecule, selecting the set with the minimum word frequency in the first set and the second set, and taking the reciprocal of the number of elements in the set as a denominator;

for a specific combination of n-gram fragments g (a _m ,b _m ) A batch of n-gram fragments (a _m ,b _h ) And (a) _j ,b _m ) Then the first set { a } _m Sum of }, andtwo sets {, b _m Defined as: { a _m ,*}＝{(a _m ,b ₁ ),(a _m ,b ₂ ),…,(a _m ,b _h ) } and { b _m }＝{(a ₁ ,b _m ),(a ₂ ,b _m ),…,(a _j ,b _m ) -a }; order the

And->

Respectively expressed in the set { a } _m Sum }, b _m The sum of the word frequencies of all n-gram fragments in }, respectively defined as: />

and

For n-gram fragment g and specific combinations thereof (a _m ,b _m ) The variable rate represents f _g And (3) with

Ratio f _g And->

Maximum in the ratio, rate is defined as: />

For two sets { a } _m Sum }, b _m ' and its corresponding

And->

Let sizeof represent the number of n-gram elements in the set, then AC is defined as: />

(2) Combining the calculated values of the numerator and the denominator as calculated values of the relative importance of the segment combination in the first set or the second set; given a variable rate as a numerator and a variable AC as a denominator, the calculated relative importance value time value of the n-gram fragment g is defined as:

(3) Determining the statistical relationship value AT according to the relative importance degree calculated value time; the calculation formula of the statistical relation value AT is as follows: at=1+|logtimes|;

c, making the word frequency information F equal to F _g Taking the product of the word frequency information F, the mutual information value MP and the statistical relation value AT as an unsupervised association measurement index; the formula of the unsupervised association metric PATIs is as follows: pati=f×mp×at.

6. A medium having stored thereon a computer program, which when executed by a processor, implements a method of constructing a chinese word-segmentation-free embedding model according to any one of claims 1 to 4.

7. An apparatus, comprising: a processor and a memory;

the memory is used for storing a computer program, and the processor is used for executing the computer program stored in the memory, so that the device executes the method for constructing the Chinese word-segmentation-free embedding model according to any one of claims 1 to 4.

Images