CN115983253A - Illegal word expansion method, device, equipment and storage medium - Google Patents

Abstract

The invention belongs to the technical field of computers, and discloses a method, a device, equipment and a storage medium for expanding illegal words. The method comprises the following steps: generating a plurality of expansion words respectively corresponding to preset word roots through a preset illegal word expansion model; determining similarity scores corresponding to the expansion words and preset roots; determining industry initial scores of the expansion words in multiple industries based on a preset industry classification model; determining corresponding weight according to the expansion type corresponding to the preset root word; determining an industry final score corresponding to each expansion word according to the similarity score, the industry initial score and the weight; and taking each expansion word as a violation word of the corresponding industry according to the final industry score. Through the method, the preset root words are automatically expanded, industries corresponding to the expanded words are distinguished, data support is provided for judging violation of the advertisement, filling efficiency of the violation word bank is improved, the phenomenon that the violation words are searched from a large amount of text data repeatedly by manpower is avoided, and labor cost is reduced.

Description

Violation word expansion method, device, equipment and storage medium

technical field

The present invention relates to the field of computer technology, in particular to a method, device, equipment and storage medium for expanding illegal words.

Background technique

In the existing illegal judgment of advertisements, the existing illegal words are usually used to match the advertisements, and the existing illegal words are manually and repeatedly searched for illegal words with high advertising illegality from a large amount of text data. This method is subjective The accuracy is high, the thesaurus filling efficiency is low, and it is labor-intensive.

The above content is only used to assist in understanding the technical solution of the present invention, and does not mean that the above content is admitted as prior art.

Contents of the invention

The main purpose of the present invention is to provide a method, device, equipment and storage medium for expanding illegal words, aiming to solve how to realize automatic expansion of advertising illegal words, avoid manual repetition of finding illegal words from a large amount of text data, thereby reducing manpower cost technical issues.

In order to achieve the above object, the present invention provides a method for expanding illegal words, said method comprising the following steps:

Generate a number of extended words corresponding to the preset root roots through the preset violation word expansion model;

Determine the similarity score corresponding to each expanded word and the preset word root;

Based on the preset industry classification model, determine the industry initial score of each expanded word in multiple industries;

Determine the corresponding weight according to the expansion type corresponding to the preset root;

Determine the industry final score corresponding to each expanded word according to the similarity score, the industry initial score and the weight;

According to the final score of the industry, each expansion word is regarded as the violation word of the corresponding industry.

Optionally, the said expansion model generates a number of expansion words corresponding to the default roots through the preset violation word expansion model, including:

Obtain a number of related words corresponding to the preset root through the preset violation word expansion model;

Deduplication processing is performed on the several related words according to the preset database to obtain several expanded words.

Optionally, performing deduplication processing on the several related words according to the preset database to obtain several expanded words, including:

Determining several current words from a preset database;

Respectively determine the edit distance between the several current words and the several related words;

Deduplication processing is performed on the several related words according to the edit distance to obtain several expanded words.

Optionally, according to the edit distance, the several related words are deduplicated to obtain several extended words, including:

When the target edit distance corresponding to the target related word is less than a preset distance threshold, delete the target related word;

Use the remaining related words as some expansion words.

Optionally, after the final score of the industry uses each expansion word as a violation word of the corresponding industry, the method also includes:

Store each expanded word and corresponding industry in the preset database.

Optionally, after the final score of the industry uses each expansion word as a violation word of the corresponding industry, the method also includes:

Obtain the deletion instruction input by the user, and delete the corresponding target expansion word according to the deletion instruction;

Store the remaining extended words and corresponding industries in the preset database.

Optionally, the determination of the similarity score corresponding to each expanded word and the preset root includes:

Determining the probability value corresponding to each expanded word output by the preset violation word expansion model and the preset root;

Respectively determine the current edit distance between each expanded word and the preset word root;

Determine the similarity score corresponding to each expanded word and the preset word root according to the probability value and the current edit distance.

Optionally, the preset violation word expansion model includes a preset Word2Vec model and a preset Bert model.

Optionally, the determining the similarity score corresponding to each expanded word and the preset word root according to the probability value and the current edit distance includes:

Determine corresponding control parameters according to the preset violation word expansion model that generates each expansion word;

The similarity score corresponding to each expanded word and the preset word root is determined according to the control parameter, the probability value, and the current edit distance.

Optionally, before generating a number of extended words corresponding to the preset roots through the preset violation word expansion model, the method also includes:

According to the text data, the initial Word2Vec model is trained to obtain the trained preset Word2Vec model;

The initial Bert model is trained according to the text data, and the trained Bert model is fine-tuned based on text classification to obtain a preset Bert model.

In addition, in order to achieve the above purpose, the present invention also proposes a device for expanding illegal words, which includes:

The generation module is used to generate a number of extended words corresponding to the preset root roots through the preset violation word expansion model;

A determining module, configured to determine the similarity score corresponding to each expanded word and the preset root;

The industry classification module is used to determine the industry initial score of each expanded word in multiple industries based on the preset industry classification model;

The determination module is also used to determine the corresponding weight according to the expansion type corresponding to the preset root;

The industry classification module is also used to determine the final industry score corresponding to each expanded word according to the similarity score, the industry initial score and the weight;

The expansion module is used to use each expansion word as a violation word of the corresponding industry according to the final score of the industry.

Optionally, the generation module is also used to obtain a number of related words corresponding to the preset root through the preset violation word expansion model; perform deduplication processing on the number of related words according to the preset database to obtain a number of expanded words.

Optionally, the generation module is also used to determine several current words from a preset database; respectively determine the edit distance between the several current words and the several related words; Relevant words are deduplicated, and several expanded words are obtained.

Optionally, the generation module is further configured to delete the target related word when the target edit distance corresponding to the target related word is less than a preset distance threshold; and use the remaining related words as several extended words.

Optionally, the device for expanding illegal words also includes a storage module;

The storage module is configured to store each expanded word and corresponding industry in the preset database.

Optionally, the device for expanding illegal words also includes a storage module;

The storage module is configured to obtain a deletion instruction input by the user, delete the corresponding target expansion words according to the deletion instruction, and store the remaining expansion words and corresponding industries in the preset database.

Optionally, the determination module is also used to determine the probability value corresponding to each expanded word output by the preset violation word expansion model and the preset root; respectively determine the relationship between each expanded word and the preset root The current edit distance; determine the similarity score corresponding to each expanded word and the preset root according to the probability value and the current edit distance.

Optionally, the preset violation word expansion model includes a preset Word2Vec model and a preset Bert model.

In addition, in order to achieve the above object, the present invention also proposes a violation word expansion device, the violation word expansion device includes: a memory, a processor, and a violation word expansion stored in the memory and operable on the processor program, and the violation word expansion program is configured to implement the violation word expansion method as described above.

In addition, in order to achieve the above purpose, the present invention also proposes a storage medium, on which a violation word expansion program is stored, and when the violation word expansion program is executed by a processor, the above-mentioned method for expanding violation words is realized.

The present invention generates a number of expanded words corresponding to the preset root through the preset violation word expansion model; determines the similarity score corresponding to each expanded word and the preset root; determines the value of each expanded word in multiple industries based on the preset industry classification model Industry initial score; determine the corresponding weight according to the expansion type corresponding to the preset root; determine the final industry score corresponding to each expansion word according to the similarity score, industry initial score and weight; determine each expansion word as a violation of the corresponding industry according to the industry final score word. Through the above method, the default word root is automatically expanded, and the industry corresponding to each expanded word is distinguished according to the similarity of the expanded word and the type of expansion, providing data support for the identification of advertising violations, improving the filling efficiency of the illegal lexicon, and avoiding artificial Repeatedly looking for illegal words from a large amount of text data reduces labor costs.

Description of drawings

Fig. 1 is the structural representation of the illegal word expanding device of the hardware operating environment that scheme of the embodiment of the present invention relates to;

Fig. 2 is a schematic flow chart of the first embodiment of the method for expanding illegal words of the present invention;

Fig. 3 is a schematic flow chart of determining the industry score in an embodiment of the method for expanding the illegal words of the present invention;

Fig. 4 is the schematic flow chart of the second embodiment of the method for expanding the illegal words of the present invention;

Fig. 5 is a schematic flow chart of the third embodiment of the method for expanding illegal words of the present invention;

Fig. 6 is a schematic diagram of the expansion process of an embodiment of the method for expanding violation words of the present invention;

Fig. 7 is a structural block diagram of the first embodiment of the device for expanding illegal words according to the present invention.

Detailed ways

It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention. The realization of the purpose of the present invention, functional characteristics and advantages will be further described in conjunction with the embodiments and with reference to the accompanying drawings.

Referring to FIG. 1 , FIG. 1 is a schematic structural diagram of a device for expanding illegal words in a hardware operating environment related to the solution of an embodiment of the present invention.

As shown in Figure 1, this violation word expanding device can comprise: processor 1001, such as central processing unit (Central Processing Unit, CPU), communication bus 1002, user interface 1003, network interface 1004, memory 1005. Wherein, the communication bus 1002 is used to realize connection and communication between these components. The user interface 1003 may include a display screen (Display) and an input unit (such as a keyboard (Keyboard)); optionally, the user interface 1003 may also include a standard wired interface and a wireless interface. Optionally, the network interface 1004 includes a standard wired interface and a wireless interface (such as a Wireless-Fidelity (Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (Random Access Memory, RAM), or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk memory. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001 .

Those skilled in the art can understand that the structure shown in FIG. 1 does not constitute a limitation on the device for expanding illegal words, and may include more or less components than those shown in the figure, or combine some components, or arrange different components.

As shown in FIG. 1 , the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a violation word expansion program.

In the device for expanding the illegal words shown in Figure 1, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001, memory 1005 may be set in the device for expanding illegal words, and the device for expanding illegal words calls the program for expanding illegal words stored in memory 1005 through the processor 1001, and executes the method for expanding illegal words provided by the embodiment of the present invention.

An embodiment of the present invention provides a method for expanding illegal words. Referring to FIG. 2 , FIG. 2 is a schematic flowchart of a first embodiment of the method for expanding illegal words in the present invention.

In the present embodiment, described violation word expansion method comprises the following steps:

Step S10: Generate a number of extended words corresponding to the preset word roots through the preset violation word expansion model.

It can be understood that the execution subject of this embodiment is the device for expanding illegal words, and the device for expanding illegal words can be a computer, a server, etc., or other devices with the same or similar functions. limit.

It should be noted that the default expansion model of illegal words can be a Word2Vec model or a Bert model, and a number of expanded words related to the preset root are recalled through the Word2Vec model and/or Bert model. In the specific implementation, in order to further improve the violation For the expansion efficiency of the vocabulary, there can be multiple preset roots, and multiple preset roots can be input into the preset violation word expansion model to generate a large number of expanded words. The preset roots can be obtained from the preset database, among which, the preset Assume that the database stores a large number of existing offending words.

Step S20: Determine the similarity score corresponding to each expanded word and the preset word root.

It can be understood that the similarity score can be determined based on the edit distance between each expanded word and the preset root, and the closer the edit distance between the expanded word and the preset root, the smaller the similarity score. In the specific implementation, the output probability of the preset violation word expansion model is also considered, and the similarity score is determined according to the output probability and the edit distance. The greater the output probability and the greater the edit distance, the higher the similarity score.

Step S30: Determine the industry initial score of each expanded word in multiple industries based on the preset industry classification model.

It should be noted that the preset industry classification model can be a lightweight fastText model, which classifies each expanded word by industry, and determines the industry initial score of each expanded word in each preset industry.

Step S40: Determine the corresponding weight according to the expansion type corresponding to the preset word root.

It can be understood that, in this embodiment, the offending words are divided into three categories, including A-category words, B-category words and C-category words, wherein, A-category words come from words with high illegality and high triggering properties; B-category words It is derived from words with high illegality and low triggering nature; C-type words are new words collected from the customer needs of the business department and provided by the business department. When expanding different types of illegal words, a differentiated expansion method is adopted. Among them, since the A-type words are highly illegal and high-triggering words, the A expansion type is to generalize the industry of the A-type words to other industries. In various industries, for example: buying a house, helping with account registration for free, the entry industry belongs to "real estate", and "help with account registration for free" is an illegal word in this industry, so if this word is generalized, then you can get "free help" Into key points", then it can be generalized from "real estate" to "education and training" industry, so as to get an illegal word in the "education and training" industry. Type B words may be more illegal, but they are usually less triggering in the industry. The B expansion type is to continue to expand the word of the entry in this industry. For example, in the above example, the illegal word industry belongs to "real estate" , if its triggering is low, then by continuing to expand in the "real estate" industry, the richness of illegal words in the "real estate" industry will be higher, and the triggering will be improved. Illegal words in the "real estate" industry. For category C words, the C expansion type is to expand the amount of illegal matching from the perspective of enriching the illegal lexicon. " and other words with positive connotations are illegal words.

It should be noted that in this embodiment, different industry weights are set according to the extension type of the preset root word. For example, when expanding through type A, the weight corresponding to this industry is much smaller than the weight corresponding to other industries. When expanding, the corresponding weight of this industry is much greater than that of other industries, and when expanding through C expansion type, the corresponding weight of each industry is similar.

Step S50: Determine the industry final score corresponding to each expanded word according to the similarity score, the industry initial score and the weight.

It should be understood that, assuming that the similarity score is expressed as sim_score, the industry initial score is expressed as F(q,q _i ,cate), and the weight is expressed as γ _i and δ _i , the final industry corresponding to each expanded word is determined according to formula (1). Score:

final_score＝γ _i *sim_score+δ _i *F(q,q _i ,cate) (1)

Step S60: According to the final score of the industry, each expanded word is regarded as a violation word of the corresponding industry.

It should be noted that this embodiment restricts the final industry scores according to the type of expansion and the similarity score, so as to determine the final industry scores corresponding to each expanded word in each industry, and sort the final scores of several industries corresponding to each expanded word. Determine the score with the largest value and the corresponding industry, bind each expanded word and the corresponding industry, and obtain the illegal words expanded in each industry.

Referring to Fig. 3 for illustration, Fig. 3 is a schematic flow diagram of determining industry scores in an embodiment of the method for expanding illegal words of the present invention. The preset root is the illegal root q selected from the preset database, and its corresponding word category is a type A word , belonging to industry 1, according to the expansion model of preset illegal words to generate a number of expansion words corresponding to the preset root, obtain the extension word candidate set, which contains a number of expansion words q1, q2, q3..., determine the expansion words and the default root The similarity scores s1, s2, s3... corresponding to q, based on the preset industry classification model, determine the industry initial scores f1, f2, f3... of each expanded word in industry 1-industry n, and determine the weight according to the expansion type corresponding to the A category word , that is, coefficients, according to the similarity scores s1, s2, s3..., coefficients and industry initial scores f1, f2, f3... and the corresponding industry final scores fs1, fs2, fs3... of each expanded word, based on the industry final score, each expanded word As the illegal words corresponding to the industry, for example, the expanded words corresponding to industry 1 are q3, q5, q1....

In this embodiment, a number of extended words corresponding to the preset root are generated through the preset violation word expansion model; the similarity score corresponding to each expanded word and the preset root is determined; based on the preset industry classification model, it is determined that each expanded word is in multiple industries Determine the corresponding weight according to the expansion type corresponding to the preset root; determine the final industry score corresponding to each expanded word according to the similarity score, industry initial score and weight; use each expanded word as the corresponding industry according to the final score of the industry offending word. Through the above method, the default word root is automatically expanded, and the industry corresponding to each expanded word is distinguished according to the similarity of the expanded word and the type of expansion, providing data support for the identification of advertising violations, improving the filling efficiency of the illegal lexicon, and avoiding artificial Repeatedly looking for illegal words from a large amount of text data reduces labor costs.

Referring to FIG. 4 , FIG. 4 is a schematic flowchart of a second embodiment of the method for expanding illegal words according to the present invention.

Based on the above-mentioned first embodiment, the step S10 of the method for expanding the violation word in this embodiment includes:

Step S101: Obtain a number of related words corresponding to the preset word roots through the preset violation word expansion model.

Step S102: Perform deduplication processing on the several related words according to the preset database to obtain several expanded words.

It can be understood that a large number of existing illegal words are stored in the preset database, and related words that are the same as the existing illegal words are deleted to obtain several expanded words.

Further, the step S102 includes: determining several current words from a preset database; respectively determining the edit distance between the several current words and the several related words; Perform deduplication processing to obtain a number of extended words.

Further, the deduplication processing is performed on the several related words according to the edit distance to obtain several extended words, including: when the target edit distance corresponding to the target related word is less than a preset distance threshold, delete the target related word ; Use the remaining related words as some expansion words.

It should be noted that, in order to further reduce the repeated related words and avoid the waste of computing resources caused by industry division of repeated related words, in this embodiment, by determining the edit distance between related words and existing illegal words, the relevant Words are deduplicated to get a number of extended words.

It is understandable that the related words whose edit distance is less than the preset distance threshold are deleted, and the preset distance threshold can be set independently according to business needs, which is used to distinguish whether related words and existing offending words are highly similar, and use edit distance Perform a deduplication operation to remove related words that are highly similar to the string, for example, expand "what is a potato" into "what is a potato", the two are highly similar, and remove the expanded related words.

Further, after the step S60, the method further includes: storing each expanded word and corresponding industry in the preset database.

It should be noted that the expanded words obtained from the expansion and the corresponding industries are stored in the preset database, so as to further supplement the existing illegal thesaurus. When expanding the illegal words, the preset root is obtained from the preset database. The specific process can be In order to select preset word roots from various industries according to business needs according to the user's choice, the user will mark the extension type of the selected preset word root, and expand the illegal words through the extension method corresponding to the extension type. In addition, you can also expand the When the words are stored, the auditors are prompted to mark each extended word, and determine the word category corresponding to each extended word, that is, to confirm whether each extended word belongs to a type A word, a B type word, or a C type word, and to mark each extended word, corresponding industry and The corresponding word categories are stored in the preset database. When expanding the illegal words, the preset roots and corresponding word categories are obtained from the preset database, and the illegal word expansion is performed according to the expansion type corresponding to the word categories. In a specific implementation, each expanded word can also be directly classified according to the expanded type of the preset root, that is, the expanded word obtained by expanding the A-type word is marked as the A-type word.

Further, after the step S60, the method further includes: obtaining a deletion instruction input by the user, and deleting the corresponding target expansion words according to the deletion instruction; storing the remaining expansion words and corresponding industries in the preset database middle.

It should be noted that in this embodiment, a manual review process is provided. After a number of expanded words are obtained, a manual input deletion command is received, which further avoids the repetition of illegal words and ensures the data accuracy of the illegal lexicon.

This embodiment obtains some related words corresponding to the preset root through the preset violation word expansion model; according to the preset database, some related words are deduplicated to obtain some expanded words; determine the similarity between each expanded word and the preset root Score; Determine the initial industry score of each expanded word in multiple industries based on the preset industry classification model; determine the corresponding weight according to the expansion type corresponding to the preset root; determine the corresponding value of each expanded word according to the similarity score, industry initial score and weight The final score of the industry; according to the final score of the industry, each expansion word is regarded as the violation word of the corresponding industry. Through the above method, the preset word root is automatically expanded, and the duplicate processing is carried out according to the preset database, which avoids the repetition of illegal words, further improves the filling efficiency of the illegal lexicon, and distinguishes each expansion according to the similarity of the expanded words and the type of expansion. The industry corresponding to the word provides data support for the identification of advertising violations, avoiding manual repetition of finding illegal words from a large amount of text data, and reducing labor costs.

Referring to FIG. 5 , FIG. 5 is a schematic flowchart of a third embodiment of a method for expanding illegal words according to the present invention.

Based on the above-mentioned first embodiment, the step S20 of the method for expanding the violation word in this embodiment includes:

Step S201: Determine the probability value corresponding to the preset word root for each expanded word output by the preset violation word expansion model.

Step S202: Determine the current edit distance between each expanded word and the preset root.

Step S203: Determine the similarity score corresponding to each expanded word and the preset word root according to the probability value and the current edit distance.

It is understandable that, assuming that the output probability value of the model is expressed as M(q,q _i ), the current edit distance is expressed as L(q,q _i ), where q is the preset root of the input, and q _i is the model recalled Expanded words, determine the similarity score corresponding to each expanded word and the preset word root according to formula (2):

sim_score=αM(q,q _i )+βL(q,q _i ) (2)

Further, the preset violation word expansion model includes a preset Word2Vec model and a preset Bert model.

Specifically, the step S203 includes: determining the corresponding control parameters according to the preset violation word expansion model for generating each expanded word; The similarity score corresponding to the preset root.

It should be noted that there are differences between the preset Word2Vec model and the preset Bert model. The control parameters α and β corresponding to the preset Word2Vec model and the preset Bert model are stored in the preset storage area in advance. The corresponding α and β are determined by the model, and the corresponding extended words are determined according to the formula (2) according to the control parameters α and β, the probability value M(q,q _i ) and the current edit distance L(q,q _i ). similarity score.

Further, before the step S10, the method also includes: training the initial Word2Vec model according to the text data to obtain a trained preset Word2Vec model; training the initial Bert model according to the text data, and classifying the initial Bert model based on the text Fine-tune the trained Bert model to obtain the preset Bert model.

It is understandable that before the expansion of illegal words, a large amount of text data is used to unsupervisedly train the Word2Vec model, and on the basis of the pre-trained model Bert, the Bert model is fine-tuned by text classification, and the preset Word2Vec model and the preset Bert model are obtained. .

Illustrate with reference to Fig. 6 by way of example, Fig. 6 is the expansion flow schematic diagram of an embodiment of the method for expanding the violation word of the present invention; A, B, and the C class violation root are input into the violation word expansion model obtained through a large amount of text data training, and several Approximate extended words, where the illegal word extended model includes the Word2Vec model and the Bert model. According to the existing illegal lexicon, several approximate extended words are deduplicated, and the fastText model is used to classify each extended word, and the industry initial corresponding to each extended word is determined. Score, and determine the industry corresponding to each expanded word based on the corresponding expanded category of the illegal word, manually check whether the expanded illegal word is illegal and mark the corresponding category, and store it in the existing illegal thesaurus.

The present embodiment generates a number of expanded words corresponding to the default root through the preset violation word expansion model; determines the probability value corresponding to each expansion word output by the default violation word expansion model and the preset root; determines respectively each expansion word and the preset The current edit distance between the roots; determine the similarity score between each expanded word and the preset root according to the probability value and the current edit distance; determine the industry initial score of each expanded word in multiple industries based on the preset industry classification model; The extension type corresponding to the preset word root determines the corresponding weight; the final industry score corresponding to each expanded word is determined according to the similarity score, the industry initial score, and the weight; and each expanded word is regarded as a violation word of the corresponding industry according to the final score of the industry. Through the above method, the preset word root is automatically expanded, the similarity is determined according to the model output probability value and the edit distance, and the industry corresponding to each expanded word is distinguished according to the expanded word similarity and expanded type, and the industry is scored through the similarity and expanded type Restrictions are made to improve the accuracy of industry division, provide data support for the identification of advertising violations, improve the efficiency of filling the illegal lexicon, avoid manual repetition of finding illegal words from a large amount of text data, and reduce labor costs.

In addition, the embodiment of the present invention also proposes a storage medium, on which a violation word expansion program is stored, and when the violation word expansion program is executed by a processor, the above-mentioned method for expanding violation words is implemented.

Since the storage medium adopts all the technical solutions of all the above-mentioned embodiments, it at least has all the beneficial effects brought by the technical solutions of the above-mentioned embodiments, which will not be repeated here.

Referring to FIG. 7, FIG. 7 is a structural block diagram of the first embodiment of the apparatus for expanding illegal words according to the present invention.

As shown in Figure 7, the device for expanding the illegal words proposed by the embodiment of the present invention includes:

The generating module 10 is configured to generate several expanded words respectively corresponding to preset root words through the preset violation word expansion model.

The determining module 20 is configured to determine the similarity score corresponding to each expanded word and the preset word root.

The industry classification module 30 is configured to determine the industry initial score of each expanded word in multiple industries based on a preset industry classification model.

The determination module 20 is further configured to determine the corresponding weight according to the extension type corresponding to the preset word root.

The industry classification module 30 is further configured to determine the industry final score corresponding to each expanded word according to the similarity score, the industry initial score and the weight.

The expansion module 40 is configured to use each expansion word as a violation word of the corresponding industry according to the final score of the industry.

It should be understood that the above is only an example, and does not constitute any limitation to the technical solution of the present invention. In specific applications, those skilled in the art can make settings according to needs, and the present invention is not limited thereto.

In this embodiment, a number of extended words corresponding to the preset root are generated through the preset violation word expansion model; the similarity score corresponding to each expanded word and the preset root is determined; based on the preset industry classification model, it is determined that each expanded word is in multiple industries Determine the corresponding weight according to the expansion type corresponding to the preset root; determine the final industry score corresponding to each expanded word according to the similarity score, industry initial score and weight; use each expanded word as the corresponding industry according to the final score of the industry offending word. Through the above method, the default word root is automatically expanded, and the industry corresponding to each expanded word is distinguished according to the similarity of the expanded word and the type of expansion, providing data support for the identification of advertising violations, improving the filling efficiency of the illegal lexicon, and avoiding artificial Repeatedly looking for illegal words from a large amount of text data reduces labor costs.

It should be noted that the workflow described above is only illustrative and does not limit the protection scope of the present invention. In practical applications, those skilled in the art can select part or all of them to implement according to actual needs. The purpose of the scheme of this embodiment is not limited here.

In addition, for technical details that are not described in detail in this embodiment, refer to the method for expanding illegal words provided in any embodiment of the present invention, which will not be repeated here.

In one embodiment, the generation module 10 is also used to obtain a number of related words corresponding to the preset root through the preset violation word expansion model; perform deduplication processing on the number of related words according to the preset database to obtain a number of extensions word.

In one embodiment, the generation module 10 is further configured to determine a number of current words from a preset database; respectively determine the edit distance between the number of current words and the number of related words; The several related words are deduplicated to obtain several expanded words.

In an embodiment, the generating module 10 is further configured to delete the target related word when the target edit distance corresponding to the target related word is less than a preset distance threshold; and use the remaining related words as several extended words.

In one embodiment, the device for expanding illegal words also includes a storage module;

The storage module is configured to store each expanded word and corresponding industry in the preset database.

In one embodiment, the device for expanding illegal words also includes a storage module;

The storage module is configured to obtain a deletion instruction input by the user, delete the corresponding target expansion words according to the deletion instruction, and store the remaining expansion words and corresponding industries in the preset database.

In one embodiment, the determination module 20 is also used to determine the probability value corresponding to each expanded word output by the preset violation word expansion model and the preset root; respectively determine each expanded word and the preset The current edit distance between word roots; determine the similarity score corresponding to each expanded word and the preset word root according to the probability value and the current edit distance.

In one embodiment, the preset violation word expansion model includes a preset Word2Vec model and a preset Bert model.

In one embodiment, the determination module 20 is further configured to determine corresponding control parameters according to the preset violation word expansion model for generating each expanded word; determine according to the control parameters, the probability value and the current edit distance The similarity score corresponding to each expanded word and the preset word root.

In one embodiment, the device for expanding illegal words also includes a storage module;

The storage module is used to train the initial Word2Vec model according to the text data to obtain the trained preset Word2Vec model; train the initial Bert model according to the text data, and fine-tune the trained Bert model based on text classification , to get the preset Bert model.

Furthermore, it should be noted that in this document, the term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article or system comprising a set of elements includes not only those elements, but also other elements not expressly listed, or elements inherent in such a process, method, article, or system. Without further limitations, an element defined by the phrase "comprising a..." does not preclude the presence of additional identical elements in the process, method, article or system comprising that element.

The serial numbers of the above embodiments of the present invention are for description only, and do not represent the advantages and disadvantages of the embodiments.

Through the description of the above embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus a necessary general-purpose hardware platform, and of course also by hardware, but in many cases the former is better implementation. Based on such an understanding, the essence of the technical solution of the present invention or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product is stored in a storage medium (such as a read-only memory (Read Only Memory) , ROM)/RAM, magnetic disk, optical disk), including several instructions to make a terminal device (which can be a mobile phone, computer, server, or network device, etc.) execute the methods described in various embodiments of the present invention.

The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process conversion made by using the description of the present invention and the contents of the accompanying drawings, or directly or indirectly used in other related technical fields , are all included in the scope of patent protection of the present invention in the same way.

The invention discloses A1, a method for expanding illegal words, and the method for expanding illegal words includes:

Generate a number of extended words corresponding to the preset root roots through the preset violation word expansion model;

Determine the similarity score corresponding to each expanded word and the preset word root;

Based on the preset industry classification model, determine the industry initial score of each expanded word in multiple industries;

Determine the corresponding weight according to the expansion type corresponding to the preset root;

Determine the industry final score corresponding to each expanded word according to the similarity score, the industry initial score and the weight;

According to the final score of the industry, each expansion word is regarded as the violation word of the corresponding industry.

A2, the method for expanding the illegal words as described in A1, the described expanded words corresponding to the default roots are generated by the preset illegal word expansion model, including:

Obtain a number of related words corresponding to the preset root through the preset violation word expansion model;

Deduplication processing is performed on the several related words according to the preset database to obtain several expanded words.

A3, the method for expanding the illegal words as described in A2, the described several related words are deduplicated according to the preset database, and some expanded words are obtained, including:

Determining several current words from a preset database;

Respectively determine the edit distance between the several current words and the several related words;

Deduplication processing is performed on the several related words according to the edit distance to obtain several expanded words.

A4, the method for expanding the illegal words as described in A3, the described several related words are deduplicated according to the edit distance, and some expanded words are obtained, including:

When the target edit distance corresponding to the target related word is less than a preset distance threshold, delete the target related word;

Use the remaining related words as some expansion words.

A5, the method for expanding the violation words as described in A2, after the described according to the final score of the industry, each expansion word is used as the violation words of the corresponding industry, the method also includes:

Store each expanded word and corresponding industry in the preset database.

A6, the method for expanding the violation words as described in A2, after the described according to the final score of the industry, each expansion word is used as the violation words of the corresponding industry, the method also includes:

Obtain the deletion instruction input by the user, and delete the corresponding target expansion word according to the deletion instruction;

Store the remaining extended words and corresponding industries in the preset database.

A7, the method for expanding the illegal words as described in A1, the determination of the similarity score corresponding to each expanded word and the preset word root includes:

Determining the probability value corresponding to each expanded word output by the preset violation word expansion model and the preset root;

Respectively determine the current edit distance between each expanded word and the preset word root;

Determine the similarity score corresponding to each expanded word and the preset word root according to the probability value and the current edit distance.

A8. The method for expanding illegal words as described in A7, wherein the preset expanded model for illegal words includes a preset Word2Vec model and a preset Bert model.

A9, the method for expanding the illegal word as described in A8, said determining the similarity score corresponding to each expanded word and the preset word root according to the probability value and the current edit distance, including:

Determine corresponding control parameters according to the preset violation word expansion model that generates each expansion word;

The similarity score corresponding to each expanded word and the preset word root is determined according to the control parameter, the probability value, and the current edit distance.

A10, as described in A8 Violation word expansion method, described by preset violation word expansion model before generating a number of expansion words corresponding to the preset root, said method also includes:

According to the text data, the initial Word2Vec model is trained to obtain the trained preset Word2Vec model;

The initial Bert model is trained according to the text data, and the trained Bert model is fine-tuned based on text classification to obtain a preset Bert model.

The present invention also discloses B11, a device for expanding illegal words. The device for expanding illegal words includes:

The generation module is used to generate a number of extended words corresponding to the preset root roots through the preset violation word expansion model;

A determining module, configured to determine the similarity score corresponding to each expanded word and the preset root;

The industry classification module is used to determine the industry initial score of each expanded word in multiple industries based on the preset industry classification model;

The determination module is also used to determine the corresponding weight according to the expansion type corresponding to the preset root;

The industry classification module is also used to determine the final industry score corresponding to each expanded word according to the similarity score, the industry initial score and the weight;

The expansion module is used to use each expansion word as a violation word of the corresponding industry according to the final score of the industry.

B12, the device for expanding the illegal words as described in B11, the generation module is also used to obtain some related words corresponding to the preset root through the preset illegal word expansion model; according to the preset database, the several related words are deduplicated Processing, get some expansion words.

B13, as described in B12 said violation word expanding device, described generating module is also used for determining some current words from preset database; Determine the editing distance between described several current words and described several related words respectively; According to The edit distance performs deduplication processing on the several related words to obtain several expanded words.

B14, as described in B13 Violation word expanding device, described generating module, is also used for when the target editing distance corresponding to target related word is less than preset distance threshold, deletes described target related word; Use remaining related word as several expand the word.

B15, the violation word expansion device as described in B12, the violation word expansion device also includes a storage module;

The storage module is configured to store each expanded word and corresponding industry in the preset database.

B16, the violation word expansion device as described in B12, the violation word expansion device also includes a storage module;

The storage module is configured to obtain a deletion instruction input by the user, delete the corresponding target expansion words according to the deletion instruction, and store the remaining expansion words and corresponding industries in the preset database.

B17, the violation word expansion device as described in B11, the determination module is also used to determine the probability value corresponding to each expansion word output by the preset violation word expansion model and the preset root; determine each expansion word respectively The current edit distance from the preset word root; determine the similarity score corresponding to each expanded word and the preset word root according to the probability value and the current edit distance.

B18. The device for expanding violation words as described in B17, wherein the preset violation word expansion model includes a preset Word2Vec model and a preset Bert model.

The invention also discloses C19, a device for expanding illegal words. The device includes: a memory, a processor, and a program for expanding illegal words that is stored in the memory and can run on the processor. The program is configured to implement the violating word expansion method described in any one of A1 to A10.

The present invention also discloses D20, a storage medium, on which a violation word expansion program is stored, and when the violation word expansion program is executed by a processor, the violation word expansion method described in any one of A1 to A10 is realized .

Claims (10)

1. A method for expanding illegal words is characterized by comprising the following steps:

generating a plurality of expansion words respectively corresponding to preset word roots through a preset illegal word expansion model;

determining similarity scores corresponding to the expansion words and the preset root words;

determining industry initial scores of the expansion words in multiple industries based on a preset industry classification model;

determining corresponding weight according to the expansion type corresponding to the preset root word;

determining an industry final score corresponding to each expansion word according to the similarity score, the industry initial score and the weight;

and taking each expansion word as a violation word of the corresponding industry according to the final industry score.

2. The method for expanding illegal words according to claim 1, wherein the generating of the plurality of expansion words respectively corresponding to the preset root of word by the preset illegal word expansion model comprises:

acquiring a plurality of related words corresponding to a preset root of a word through a preset illegal word expansion model;

and carrying out duplicate removal processing on the plurality of related words according to a preset database to obtain a plurality of expansion words.

3. The method for expanding illegal words according to claim 2, wherein the step of performing deduplication processing on the plurality of related words according to a preset database to obtain a plurality of expanded words comprises the steps of:

determining a plurality of current words from a preset database;

respectively determining editing distances between the current words and the related words;

and carrying out duplication elimination processing on the plurality of related words according to the editing distance to obtain a plurality of expansion words.

4. The method for expanding illegal words according to claim 3, wherein the removing the duplicate of the related words according to the edit distance to obtain expanded words comprises:

deleting the target related words when the target editing distance corresponding to the target related words is smaller than a preset distance threshold;

and taking the rest related words as a plurality of expansion words.

5. The method for expanding illegal words according to claim 2, wherein after each expansion word is used as the illegal word of the corresponding industry according to the industry final score, the method further comprises:

and storing each expansion word and the corresponding industry into the preset database.

6. The method for expanding illegal words according to claim 2, wherein after each expanded word is used as the illegal word of the corresponding industry according to the industry final score, the method further comprises:

acquiring a deleting instruction input by a user, and deleting the corresponding target expansion word according to the deleting instruction;

and storing the remaining expansion words and the corresponding industries in the preset database.

7. The method for extending illegal words according to claim 1, wherein the determining the similarity score of each extended word corresponding to the preset root word comprises:

determining probability values corresponding to the expansion words output by the preset illegal word expansion model and the preset root of word;

respectively determining the current editing distance between each expansion word and the preset root;

and determining the similarity score of each expansion word corresponding to the preset root word according to the probability value and the current editing distance.

8. An illegal word expansion device, characterized in that the illegal word expansion device comprises:

the generating module is used for generating a plurality of expansion words corresponding to the preset root respectively through the preset illegal word expansion model;

the determining module is used for determining similarity scores corresponding to the expansion words and the preset root;

the industry classification module is used for determining industry initial scores of the expansion words in multiple industries based on a preset industry classification model;

the determining module is further configured to determine a corresponding weight according to the expansion type corresponding to the preset root word;

the industry classification module is further used for determining industry final scores corresponding to the expansion words according to the similarity scores, the industry initial scores and the weights;

and the expansion module is used for taking each expansion word as the violation word of the corresponding industry according to the industry final score.

9. An illegal word expansion device, characterized in that the device comprises: a memory, a processor, and an offending word extension program stored on the memory and executable on the processor, the offending word extension program configured to implement the offending word extension method of any of claims 1-7.

10. A storage medium having stored thereon an illegal word expansion program that, when executed by a processor, implements an illegal word expansion method according to any one of claims 1 to 7.

Images