WO2021218012A1 - 中英双语语料库的构建方法、及其相关设备 - Google Patents

中英双语语料库的构建方法、及其相关设备 Download PDF

Info

Publication number
WO2021218012A1
WO2021218012A1 PCT/CN2020/117388 CN2020117388W WO2021218012A1 WO 2021218012 A1 WO2021218012 A1 WO 2021218012A1 CN 2020117388 W CN2020117388 W CN 2020117388W WO 2021218012 A1 WO2021218012 A1 WO 2021218012A1
Authority
WO
WIPO (PCT)
Prior art keywords
entity
sentence
chinese
english
preset
Prior art date
Application number
PCT/CN2020/117388
Other languages
English (en)
French (fr)
Inventor
邓悦
金戈
徐亮
Original Assignee
平安科技（深圳）有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 平安科技（深圳）有限公司 filed Critical 平安科技（深圳）有限公司
Publication of WO2021218012A1 publication Critical patent/WO2021218012A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/40—Processing or translation of natural language
    • G06F40/58—Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/31—Indexing; Data structures therefor; Storage structures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/33—Querying
    • G06F16/3331—Query processing
    • G06F16/3332—Query translation
    • G06F16/3337—Translation of the query language, e.g. Chinese to English
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/20—Natural language analysis
    • G06F40/205—Parsing
    • G06F40/211—Syntactic parsing, e.g. based on context-free grammar [CFG] or unification grammars
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/20—Natural language analysis
    • G06F40/279—Recognition of textual entities
    • G06F40/289—Phrasal analysis, e.g. finite state techniques or chunking
    • G06F40/295—Named entity recognition
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/30—Semantic analysis

Definitions

  • This application relates to the field of computer technology, and in particular to a method for constructing a Chinese-English bilingual corpus and related equipment.
  • the traditional Chinese-English bilingual corpus training methods based on the bilingual dialogue system are mainly divided into two categories: One method is to use the corpus of the corresponding field to match the Chinese-English bilingual corpus to be tested, and there is no need for parallel corpus, but the inventor found it
  • the training process is unstable and complex, so the method can only be limited to small-scale data, and the accuracy is not high;
  • another method is to use existing multilingual resources to automatically generate "pseudo bilingual documents", this method Although stable, the inventor found that this method is time-consuming and insufficient in accuracy due to the large amount of data and uncertainty in the training data.
  • the dialogue system uses the Chinese-English bilingual corpus for recognition, there is a situation of semantic recognition errors, which further affects the accuracy of the dialogue system.
  • the embodiment of the application provides a method for constructing a Chinese-English bilingual corpus and related equipment to solve the problem that the accuracy of traditional Chinese-English bilingual corpus training is not high, which further affects the low accuracy of the dialogue system using the Chinese-English bilingual corpus problem.
  • a method for constructing a Chinese-English bilingual corpus including:
  • the target estimate is compared with a preset threshold, and if the preset conditions are met, the bilingual entity word network and the comparable sentence network are used to combine into a Chinese-English bilingual corpus.
  • a device for constructing a Chinese-English bilingual corpus including:
  • the first acquisition module is used to acquire Chinese entities, English entities, and the mapping relationship and mutual translation relationship between the Chinese entities and the English entities from the preset entity library;
  • the construction module is used to construct a bilingual entity word network according to preset requirements according to the Chinese entity, the English entity, the mapping relationship, and the mutual translation relationship;
  • the second acquiring module is configured to acquire the context words corresponding to each of the Chinese entities and each of the English entities from a preset database;
  • the first calculation module is configured to calculate the single-language representation estimate and cross-language representation of the bilingual entity word network based on the Chinese entity, the English entity, the context word, the preset hyperlink set, and the preset sentence set Entity valuation
  • the second calculation module is used to obtain a comparable sentence network and a training sentence, and use the training sentence to calculate a cross-language sentence estimate corresponding to the comparable sentence network;
  • a summation module configured to perform a weighted summation of the single-language representation estimate, the cross-language entity estimate, and the cross-language sentence estimate to obtain a target estimate
  • the combination module is used to compare the target estimate with a preset threshold, and if the preset conditions are met, use the bilingual entity word network and the comparable sentence network to combine into a Chinese-English bilingual corpus.
  • a computer device includes a memory, a processor, and computer-readable instructions stored in the memory and capable of running on the processor.
  • the processor executes the computer-readable instructions to implement the following The steps of the construction method of the English bilingual corpus:
  • the target estimate is compared with a preset threshold, and if the preset conditions are met, the bilingual entity word network and the comparable sentence network are used to combine into a Chinese-English bilingual corpus.
  • a computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the method for constructing a Chinese-English bilingual corpus are realized as follows:
  • the target estimate is compared with a preset threshold, and if the preset conditions are met, the bilingual entity word network and the comparable sentence network are used to combine into a Chinese-English bilingual corpus.
  • the above-mentioned Chinese-English bilingual corpus construction method and related equipment build a bilingual entity word network based on the mapping relationship and mutual translation relationship, which can strengthen the association relationship between Chinese entities and English entities, and calculate single-language representation estimates and cross-language
  • entity valuation and cross-language sentence valuation can accurately determine whether the bilingual entity word network and the comparable sentence network meet the set requirements.
  • the target valuation meets the preset conditions, use the bilingual entity word network and comparable sentences
  • the network constitutes a Chinese-English bilingual corpus.
  • the Chinese-English bilingual corpus is composed of two types of networks, it can improve the correlation between different corpora in the Chinese-English bilingual corpus, further improve the accuracy of the corpus in the Chinese-English bilingual corpus, and further improve the application The accuracy of the dialogue system of the Chinese-English bilingual corpus.
  • Fig. 1 is a flowchart of a method for constructing a Chinese-English bilingual corpus provided by an embodiment of the present application
  • step S2 is a flowchart of step S2 in the method for constructing a Chinese-English bilingual corpus provided by an embodiment of the present application;
  • step S4 in the method for constructing a Chinese-English bilingual corpus provided by an embodiment of the present application
  • step S5 is a flowchart of step S5 in the method for constructing a Chinese-English bilingual corpus provided by an embodiment of the present application;
  • step S53 is a flowchart of step S53 in the method for constructing a Chinese-English bilingual corpus provided by an embodiment of the present application;
  • step S7 is a flowchart of step S7 in the method for constructing a Chinese-English bilingual corpus provided by an embodiment of the present application;
  • Fig. 7 is a schematic diagram of a device for constructing a Chinese-English bilingual corpus provided by an embodiment of the present application.
  • Fig. 8 is a basic structural block diagram of a computer device provided by an embodiment of the present application.
  • the construction method of the Chinese-English bilingual corpus provided in this application is applied to the server, and the server can be implemented by an independent server or a server cluster composed of multiple servers.
  • a method for constructing a Chinese-English bilingual corpus is provided, which includes the following steps:
  • S1 Obtain Chinese entities, English entities, and the mapping relationship and mutual translation relationship between Chinese entities and English entities from the preset entity library.
  • the mapping relationship between the Chinese entity and the English entity refers to setting the connection relationship between the Chinese entity and the English entity according to the actual needs of the user. For example, there is a mapping relationship between the Chinese entity "apple” and the English entity "fruit”.
  • the mutual translation relationship refers to the mutual translation relationship between Chinese entities and English entities. For example, if the Chinese entity is “apple” and the English entity is “apple”, the English corresponding to "apple” is “apple” , The Chinese entity and the English entity apple belong to a mutual translation relationship.
  • the Chinese entity, the English entity, and the mapping relationship and mutual translation relationship between the Chinese entity and the English entity are acquired from the preset entity library.
  • the preset entity database refers to a database specifically used to store Chinese entities, English entities, and the mapping relationship and mutual translation relationship between Chinese entities and English entities.
  • S2 Construct a bilingual entity word network according to preset requirements based on Chinese entities, English entities, mapping relationships and mutual translation relationships.
  • a bilingual entity word network is constructed according to preset requirements according to the Chinese entity, the English entity, the mapping relationship and the mutual translation relationship between the Chinese entity and the English entity obtained in step S1.
  • the preset requirements refer to the requirements for constructing a bilingual entity word network according to the actual needs of users.
  • contextual words refer to words that have an association relationship with a Chinese entity or an English entity in a dialogue scenario.
  • the context word corresponding to the legal word is regarded as the context word corresponding to the Chinese entity.
  • Reason get the context words corresponding to English entities.
  • the preset database refers to a database specially used for storing legal words and context words corresponding to legal words.
  • the preset database contains the legal words “apple” and “pear”, where the context words corresponding to "apple” are “apple” and “fruit”, and the context words corresponding to "pear” are “pear” and “fruit”. ; If the Chinese entity is “apple”, the Chinese entity is matched with the legal words “apple” and “pear”, and the legal word “apple” is the same as the Chinese entity, then the contextual words "apple” and "fruit” are used as the Context words for Chinese entities.
  • S4 Calculate the single-language representation valuation and cross-language entity valuation of the bilingual entity word network based on Chinese entities, English entities, context words, preset hyperlink sets, and preset sentence sets.
  • the single-language representation estimates and cross-language entity estimates corresponding to the bilingual entity word network are calculated according to the preset calculation formulas. value.
  • the preset calculation formula refers to a formula set by the user to calculate the corresponding single-language representation valuation and cross-language entity valuation.
  • the preset hyperlink collection refers to a collection composed of hyperlinks selected by the user.
  • the preset sentence set refers to the set of sentences selected by the user in advance from Baidu Baike.
  • S5 Obtain a comparable sentence network and a training sentence, and use the training sentence to calculate a cross-language sentence estimate corresponding to the comparable sentence network.
  • the comparable sentence network and the training sentence are obtained from the preset initial library, and the preset calculation method and the training sentence are used to calculate the cross-language sentence estimate corresponding to the comparable sentence network.
  • the preset initial database refers to a database specially used for storing comparable sentence networks and training sentences.
  • the preset calculation method refers to a calculation method that is preset to calculate the cross-language sentence estimate corresponding to the comparable sentence network based on the training sentence.
  • the comparable sentence network refers to a network composed of Chinese sentences, English sentences, and the association relationship between Chinese sentences and English sentences.
  • S6 Perform weighted summation on single-language representation valuation, cross-language entity valuation, and cross-language sentence valuation to obtain a target valuation.
  • the single-language representation valuation, cross-language entity valuation, and cross-language sentence valuation are respectively multiplied by their corresponding preset weights, and the multiplied results are summed, and the calculated result is obtained as Target valuation.
  • the preset weight refers to the proportion value preset by the user, and the specific value can be 0.3, or it can be set according to the actual needs of the user, and there is no limitation here.
  • the preset condition is a condition set by the user according to actual needs, which can be specifically set according to a comparison situation obtained by comparing a target estimate with a preset threshold.
  • the target estimate is compared with a preset threshold. If the comparison result is that the target estimate is less than or equal to the preset threshold, the current bilingual entity word network and the comparable sentence network are combined into a Chinese-English bilingual corpus; if the comparison result is the target If the estimate is greater than the preset threshold, the current bilingual entity word network and the comparable sentence network are iteratively updated until the target estimate is less than or equal to the threshold threshold.
  • the preset threshold refers to a value set according to the actual needs of the user, and there is no limitation here.
  • the bilingual entity word network is constructed based on the mapping relationship and the mutual translation relationship, which can strengthen the association relationship between Chinese entities and English entities, by calculating single-language representation valuation, cross-language entity valuation, and cross-language sentence valuation It can accurately determine whether the bilingual entity word network and the comparable sentence network meet the set requirements.
  • the bilingual entity word network and the comparable sentence network are used to form a Chinese-English bilingual corpus.
  • the English bilingual corpus is composed of two networks, which can improve the correlation between different corpora in the Chinese-English bilingual corpus, further improve the accuracy of the corpus in the Chinese-English bilingual corpus, and further improve the accuracy of the dialogue system using the Chinese-English bilingual corpus sex.
  • step S2 constructing a bilingual entity word network according to preset requirements according to Chinese entities, English entities, mapping relationships, and mutual translation relationships includes the following steps:
  • step S1 all the Chinese entities obtained in step S1 are taken as the first set, and all the English entities obtained are taken as the second set.
  • S22 Obtain from the mapping database the first mapping entity that has a mapping relationship with the Chinese entity as the third set, and the second mapping entity that has the mapping relationship with the English entity as the fourth set, where the mapping database includes the first mapping entity and The second mapping entity.
  • the mapping database contains different first entities and second entities, each first entity has its corresponding first mapping entity, and each second entity has its corresponding second mapping entity, and There is a mapping relationship between the first mapping entity and the first entity, and there is a mapping relationship between the second mapping entity and the second entity.
  • Match the Chinese entity with the first entity in the mapping database If the Chinese entity is the same as the first entity, obtain the first mapping entity corresponding to the first entity, and use all the first mapping entities as the third set; , The English entity is matched with the second entity, and if the English entity is the same as the second entity, the second mapping entity corresponding to the second entity is obtained, and all the second mapping entities are set as the fourth set.
  • S23 Acquire Chinese entities and English entities that have a mutual translation relationship as mutual translation entities, and combine all the mutual translation entities into a fifth set.
  • Chinese entities and English entities that have a mutual translation relationship are obtained from a preset entity library as the mutual translation entities, and all the mutual translation entities are combined into the fifth set.
  • E is a bilingual entity word network
  • ⁇ zh is the first set
  • ⁇ en is the second set
  • R zh is the third set
  • R en is the fourth set
  • R is the fifth set.
  • the five sets are combined into a new set according to formula (1), and the new set is regarded as a bilingual entity Word network.
  • the bilingual entity word network can be constructed quickly and accurately according to formula (1) to ensure the construction of the bilingual entity word network Accuracy, to further improve the accuracy of the subsequent construction of a Chinese-English bilingual corpus using a bilingual entity word network.
  • the context words include Chinese context words and English context words.
  • step S4 the calculation is performed based on the Chinese entity, the English entity, the context word, the preset hyperlink set, and the preset sentence set.
  • the single-language representation evaluation and cross-language entity evaluation of the bilingual entity word network include the following steps:
  • the preset processing port refers to a processing port specifically used to convert Chinese entities or English entities into word vector features, which specifically uses the word2vec model to perform vector feature conversion processing.
  • the converted word vector features are obtained, and the word vector features are used as training entities.
  • L is the single-language representation valuation
  • zh is the Chinese entity
  • en is the English entity
  • D is a set of preset sentences
  • A is a set of preset hyperlinks
  • G is a contextual word
  • Means (i) If Whether it is a context word; (ii) if connected to Entity; (iii) If q exists in A, judge Whether it is a context word of q, q is an element in D.
  • I is the cross-lingual entity valuation
  • Is the current entity Indicates context words. If the current entity is a Chinese entity, it means the English context word corresponding to the Chinese entity; if the current entity is an English entity, it means the Chinese context word corresponding to the English entity; Entities in other languages that are connected.
  • the single-language representation valuation and cross-language entity valuation corresponding to the bilingual entity word network can be calculated quickly and accurately, respectively, so as to ensure that subsequent evaluations based on single-language representation and The cross-lingual entity valuation determines the accuracy of the target valuation.
  • step S5 obtaining a comparable sentence network and a training sentence, and using the training sentence to calculate a cross-language sentence estimate corresponding to the comparable sentence network includes the following steps:
  • S51 Obtain a comparable sentence network from a preset initial library, where the comparable sentence network includes Chinese sentences and English sentences.
  • the comparable sentence network is directly obtained from the preset initial library.
  • the comparable sentence network is composed of Chinese sentences and English sentences, and there is a pre-set association relationship between them. However, as the amount of data increases, the association relationship is inaccurate, so it is necessary to compare the comparable sentence network. Perform optimization processing.
  • S52 Obtain a Chinese sentence and an English sentence containing two identical entities as a training sentence, where the training sentence contains a Chinese sentence vector corresponding to the Chinese sentence.
  • the same entity between the Chinese sentence and the English sentence refers to the Chinese entity and the English entity that have a mutual translation relationship.
  • the Chinese entity "apple” and the English entity "apple” belong to the same entity.
  • the training sentences are obtained directly from a preset training library, where the preset training library refers to a database specially used for storing training sentences.
  • the training sentence is converted into a comprehensive vector according to a preset vector conversion method.
  • the preset vector conversion conversion method may specifically be to convert the training sentence into a comprehensive vector through the word2vec model.
  • J is the cross-language sentence valuation
  • Is the integrated vector Is the Chinese sentence vector
  • K is the comparable sentence network.
  • the integrated vector and the Chinese sentence vector are brought into formula (4), and formula (4) is used to calculate the cross-language sentence estimate.
  • formula (4) can quickly and accurately calculate the cross-language sentence estimate corresponding to the comparable sentence network, ensuring the accuracy of the subsequent use of the cross-language sentence estimate to determine the target estimate.
  • converting the training sentence into a comprehensive vector includes the following steps:
  • S531 Perform semantic accuracy judgment on the training sentence, determine the first weight value of the training sentence according to the judgment result, and use the training sentence whose first weight value is determined as the first target sentence.
  • the training sentence is imported into the preset semantic port to determine the semantic accuracy, output the accuracy, and obtain the weight value corresponding to the accuracy from the preset weight table as the first weight value, and finally The training sentence with the first weight value is used as the first target sentence.
  • the preset semantic port refers to a processing port that is pre-trained to perform semantic accuracy judgments for training sentences and output accuracy according to the judgment results.
  • the preset weight table refers to a data table used to store weight values corresponding to different accuracy and accuracy.
  • S532 Perform sentence vector conversion on the first target sentence to obtain the first vector.
  • the first target sentence is imported into the preset vector conversion port for sentence vector conversion processing, and the converted first vector is obtained.
  • the preset vector conversion port refers to a processing port specially used for sentence vector conversion processing.
  • S533 Determine the second weight value of each vocabulary contained in the Chinese sentence and the English sentence in the training sentence by means of sentence meaning matching for the training sentence, and use the training sentence whose second weight value is determined as the second target sentence.
  • sentence meaning matching refers to a method of judging whether the translation corresponding to the English sentence in the training sentence matches the Chinese sentence based on the Chinese sentence in the training sentence.
  • the training sentence is imported to the preset matching port for sentence meaning matching processing, and after the sentence meaning matching processing, each group of words in the English sentence is assigned a second weight value, and the training sentence with the second weight value is used as The second target sentence.
  • the preset matching port refers to a processing port that is pre-trained to perform sentence meaning matching on the training sentence, and assigns a second weight value corresponding to each group of words in the English sentence included in the training sentence according to the matching result.
  • the specific processing process of the preset matching port is: the Chinese sentence and English sentence in the training sentence are converted into Chinese sentence vector and English sentence vector through the word2vec model, and the vector of each dimension in the Chinese sentence vector and each of the English sentence vectors are The vectors of dimensions are compared. If the vector of the Chinese sentence vector in the same dimension is different from the English sentence vector, the second weight value corresponding to the pre-trained vector ratio is assigned to the second weight corresponding to the vector in the English sentence vector value.
  • each vector has its corresponding vocabulary, and the vectors of Chinese vocabulary and English vocabulary that have a mutual translation relationship are the same.
  • the training sentence contains the Chinese sentence: "Xiaoming used to be a student of Peking University", the English sentence is: "Xiaoming spent 7 years in Peking University.”
  • the Chinese sentence is converted into a Chinese sentence vector through the word2vec model: (1,2,3,4,5), the English sentence is converted into an English sentence vector through the word2vec model: (1,0,3,4,9), where the second dimension vector 0 in the English sentence vector corresponds to The vocabulary in the English sentence is spent, and the vocabulary in the English sentence corresponding to the fifth dimension vector 9 is 7 years.
  • the second dimension 2 is different from 0, and the fifth dimension 5 is different from 9, if the pre-trained vector ratio is 2:0, the corresponding second weight value
  • the second weight value corresponding to the vector ratio of 5:9 is 1%, and the second weight value corresponding to the same ratio is 100%;
  • the second weight value corresponding to the second dimension vector in the English sentence vector is 50%, That is, the second weight value corresponding to spent is 50%;
  • the second weight value corresponding to the fifth dimension vector is 1%, that is, the second weight value of the vocabulary of 7 years is 1%, and other vectors with the same ratio correspond to the second weight value.
  • the value is 100%.
  • S534 Perform sentence vector conversion on the English sentence in the second target sentence to obtain a second vector.
  • the second target sentence is imported into the preset vector conversion port for sentence vector conversion processing, and the second vector after the conversion processing is obtained.
  • S535 Calculate a comprehensive vector corresponding to the training sentence based on the first vector and the second vector.
  • the comprehensive vector corresponding to the training sentence is calculated according to formula (5):
  • Is the integrated vector Is the first vector
  • Is the second vector Is the training sentence
  • the semantic accuracy of the training sentence by judging the semantic accuracy of the training sentence, it is possible to identify whether there is a semantic error in the training sentence, and determine the first weight value of the training sentence according to the judgment to determine the first target sentence, thereby improving
  • the semantic accuracy of the first target sentence according to the sentence meaning matching method, the similarity between the sentence meaning of the Chinese sentence and the English sentence in the training sentence can be recognized, and then the first word of each vocabulary in the English sentence can be determined according to the similarity.
  • the two weight values determine the second target sentence, improve the accuracy of the second target sentence, and finally calculate the comprehensive vector corresponding to the training sentence according to the first target sentence and the second target sentence, which can ensure the accuracy of the comprehensive vector.
  • step S7 the target estimate is compared with a preset threshold, and if the preset conditions are reached, the bilingual entity word network and the comparable sentence network are used to form a Chinese-English bilingual corpus including the following step:
  • the target estimate is compared with a preset threshold.
  • step S71 if the target estimate is less than or equal to the preset threshold, the bilingual entity word network and the comparable sentence network corresponding to the target estimate are combined into a Chinese-English bilingual corpus.
  • step S71 if the target estimate is greater than the preset threshold, the bilingual entity word network and the comparable sentence network are iteratively updated according to the preset parameter update method until the target estimate is less than or equal to the preset threshold.
  • the iteratively updated bilingual entity word network and comparable sentence network are combined into a Chinese-English bilingual corpus.
  • the preset parameter update method refers to a method for setting the parameters in the bilingual entity word network and the comparable sentence network to update according to the actual needs of the user.
  • the Chinese-English bilingual corpus is determined when the target estimate is less than or equal to the preset threshold, and the target estimate is greater than the preset threshold.
  • the bilingual entity word network and the comparable sentence network are updated iteratively, and the Chinese-English bilingual corpus is determined when the target estimate is less than or equal to the preset threshold.
  • the method of comparing the target estimate with the preset threshold to determine the Chinese-English bilingual corpus can ensure that the Chinese-English bilingual corpus is determined when the set conditions are met, thereby improving the accuracy of the Chinese-English bilingual corpus.
  • the construction method of the Chinese-English bilingual corpus further includes: storing the Chinese-English bilingual corpus in the blockchain.
  • the above-mentioned Chinese-English bilingual corpus can also be stored in a node of a blockchain.
  • a device for constructing a Chinese-English bilingual corpus corresponds to the method for constructing a Chinese-English bilingual corpus in the above-mentioned embodiment.
  • the device for constructing a Chinese-English bilingual corpus includes a first acquisition module 71, a construction module 72, a second acquisition module 73, a first calculation module 74, a second calculation module 75, a summation module 76 and a combination module 77.
  • the detailed description of each functional module is as follows:
  • the first obtaining module 71 is used to obtain Chinese entities, English entities, and the mapping relationship and mutual translation relationship between Chinese entities and English entities from the preset entity library;
  • the building module 72 is used to obtain Chinese entities, English entities, The mapping relationship and mutual translation relationship are used to construct a bilingual entity word network according to preset requirements;
  • the second acquisition module 73 is used to acquire the context words corresponding to each Chinese entity and each English entity from the preset database; the first calculation module 74.
  • the second calculation module 75 uses To obtain the comparable sentence network and training sentences, and use the training sentences to calculate the cross-language sentence estimation corresponding to the comparable sentence network;
  • the summation module 76 is used for single-language representation estimation, cross-language entity estimation and cross-language sentence estimation Perform weighted summation to obtain the target estimate;
  • the combination module 77 is used to compare the target estimate with a preset threshold, and if the preset conditions are met, use the bilingual entity word network and the comparable sentence network to combine into a Chinese-English bilingual corpus.
  • the building module 72 includes:
  • the third acquisition sub-module is used to acquire all Chinese entities as the first set, and all English entities as the second set;
  • the fourth acquisition sub-module is used to acquire the first mapping entity that has a mapping relationship with the Chinese entity as the third set, and the second mapping entity that has a mapping relationship with the English entity as the fourth set from the mapping database, wherein the mapping database Contains a first mapping entity and a second mapping entity;
  • the fifth acquisition sub-module is used to acquire Chinese entities and English entities that have a mutual translation relationship as mutual translation entities, and combine all the mutual translation entities into the fifth set;
  • the network construction sub-module is used to construct a bilingual entity word network based on the first set, the second set, the third set, the fourth set and the fifth set according to formula (1):
  • E is a bilingual entity word network
  • ⁇ zh is the first set
  • ⁇ en is the second set
  • R zh is the third set
  • R en is the fourth set
  • R is the fifth set.
  • the first calculation module 74 includes:
  • the conversion sub-module is used to import Chinese entities and English entities into the preset processing port for vector feature conversion to obtain training entities;
  • the third calculation sub-module is used to calculate the single language representation estimate based on the training entity, context words, preset hyperlink set and preset sentence set according to formula (2):
  • L is the single-language representation valuation
  • zh is the Chinese entity
  • en is the English entity
  • D is a set of preset sentences
  • A is a set of preset hyperlinks
  • G is a contextual word
  • Means (i) If Whether it is a context word; (ii) if connected to Entity; (iii) If q exists in A, judge Whether it is a context word of q, where q is an element in D;
  • the fourth calculation sub-module is used to calculate cross-language entity estimates based on context words according to formula (3):
  • I is the cross-lingual entity valuation
  • Is the current entity Indicates context words. If the current entity is a Chinese entity, it means the English context word corresponding to the Chinese entity; if the current entity is an English entity, it means the Chinese context word corresponding to the English entity; Entities in other languages that are connected.
  • the second calculation module 75 includes:
  • the sixth acquisition sub-module is used to acquire a comparable sentence network from the preset initial library, where the comparable sentence network includes Chinese sentences and English sentences;
  • the seventh acquisition sub-module is used to acquire a Chinese sentence and an English sentence containing two identical entities as training sentences, where the training sentence contains a Chinese sentence vector corresponding to the Chinese sentence;
  • the integrated vector conversion sub-module is used to convert training sentences into integrated vectors according to the preset vector conversion mode
  • the fifth calculation sub-module is used to calculate the cross-language sentence estimate according to formula (4) according to the integrated vector and the Chinese sentence vector:
  • J is the cross-language sentence valuation
  • Is the integrated vector Is the Chinese sentence vector
  • K is the comparable sentence network.
  • the integrated vector conversion sub-module includes:
  • the judgment unit is used to judge the semantic accuracy of the training sentence, determine the first weight value of the training sentence according to the judgment result, and use the training sentence with the first weight value as the first target sentence;
  • the first vector obtaining unit is configured to perform sentence vector conversion on the first target sentence to obtain the first vector
  • the matching unit is used to determine the second weight value of each vocabulary contained in the Chinese sentence and English sentence in the training sentence by matching the sentence meaning of the training sentence, and use the training sentence with the second weight value as the second Target sentence
  • the second vector obtaining unit is used to perform sentence vector conversion on the English sentence in the second target sentence to obtain the second vector;
  • the integrated vector calculation unit is used to calculate the integrated vector corresponding to the training sentence based on the first vector and the second vector.
  • combination module 77 includes:
  • the comparison sub-module is used to compare the target estimate with the preset threshold
  • the first comparison sub-module is used to combine the bilingual entity word network and the comparable sentence network into a Chinese-English bilingual corpus if the target estimate is less than or equal to the preset threshold;
  • the second comparison sub-module is used to iteratively update the bilingual entity word network and the comparable sentence network according to the preset parameter update method if the target estimate is greater than the preset threshold, until the target estimate is less than or equal to the preset threshold, iteratively
  • the updated bilingual entity word network and comparable sentence network are combined into a Chinese-English bilingual corpus.
  • FIG. 8 is a block diagram of the basic structure of the computer device 80 in an embodiment of the present application.
  • the computer device 80 includes a memory 81, a processor 82, and a network interface 83 that are communicatively connected to each other through a system bus. It should be pointed out that FIG. 8 only shows a computer device 80 with components 81-83, but it should be understood that it is not required to implement all of the illustrated components, and more or fewer components may be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculation and/or information processing in accordance with pre-set or stored instructions.
  • Its hardware includes, but is not limited to, a microprocessor, a dedicated Integrated Circuit (Application Specific Integrated Circuit, ASIC), Programmable Gate Array (Field-Programmable Gate Array, FPGA), Digital Processor (Digital Signal Processor, DSP), embedded equipment, etc.
  • ASIC Application Specific Integrated Circuit
  • ASIC Application Specific Integrated Circuit
  • FPGA Field-Programmable Gate Array
  • DSP Digital Processor
  • the computer device may be a computing device such as a desktop computer, a notebook, a palmtop computer, and a cloud server.
  • the computer device can interact with the user through a keyboard, a mouse, a remote control, a touch panel, or a voice control device.
  • the memory 81 includes at least one type of readable storage medium, the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static Random access memory (SRAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), programmable read only memory (PROM), magnetic memory, magnetic disk, optical disk, etc.
  • the memory 81 may be an internal storage unit of the computer device 80, such as a hard disk or memory of the computer device 80.
  • the memory 81 may also be an external storage device of the computer device 80, such as a plug-in hard disk, a smart media card (SMC), and a secure digital device equipped on the computer device 80. (Secure Digital, SD) card, Flash Card, etc.
  • the memory 81 may also include both an internal storage unit of the computer device 80 and an external storage device thereof.
  • the memory 81 is generally used to store an operating system and various application software installed in the computer device 80, such as computer-readable instructions for the construction method of the Chinese-English bilingual corpus, etc.
  • the memory 81 can also be used to temporarily store various types of data that have been output or will be output.
  • the processor 82 may be a central processing unit (Central Processing Unit, CPU), a controller, a microcontroller, a microprocessor, or other data processing chips.
  • the processor 82 is generally used to control the overall operation of the computer device 80.
  • the processor 82 is configured to run computer-readable instructions or processed data stored in the memory 81, for example, run computer-readable instructions of the method for constructing the Chinese-English bilingual corpus.
  • the network interface 83 may include a wireless network interface or a wired network interface, and the network interface 83 is generally used to establish a communication connection between the computer device 80 and other electronic devices.
  • the computer-readable storage medium may be non-volatile or volatile.
  • the computer-readable storage medium stores a process of entity information in Chinese and English, and the process of entity information in Chinese and English can be executed by at least one processor, so that the at least one processor executes any of the above-mentioned Chinese-English bilingual corpus Steps of the construction method.
  • the above-mentioned Chinese-English bilingual corpus can also be stored in a node of a blockchain.
  • the technical solution of this application essentially or the part that contributes to the existing technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium (such as ROM/RAM, magnetic disk, The optical disc) includes a number of instructions to enable a computer device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in the various embodiments of the present application.
  • a computer device which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.
  • the blockchain referred to in this application is a new application mode of computer technology such as distributed data storage, point-to-point transmission, consensus mechanism, and encryption algorithm.
  • Blockchain essentially a decentralized database, is a series of data blocks associated with cryptographic methods. Each data block contains a batch of network transaction information for verification. The validity of the information (anti-counterfeiting) and the generation of the next block.
  • the blockchain can include the underlying platform of the blockchain, the platform product service layer, and the application service layer.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Computational Linguistics (AREA)
  • Artificial Intelligence (AREA)
  • Health & Medical Sciences (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • General Health & Medical Sciences (AREA)
  • Databases & Information Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Software Systems (AREA)
  • Machine Translation (AREA)

Abstract

一种中英双语语料库的构建方法及相关设备，涉及计算机技术领域，应用于智慧城市中，具体应用于智慧生活中，所述中英双语语料库的构建方法包括：获取中文实体、英文实体，以及中文实体与英文实体之间的映射关系和互译关系，并按照预设要求构建双语实体词语网络；根据中文实体、英文实体、上下文词语、预设超链接集合和预设语句集，计算双语实体词语网络的单语言表征估值和跨语言实体估值；利用训练语句计算获取到的可比语句网络对应的跨语言语句估值；根据三个估值计算目标估值；并根据目标估值，利用双语实体词语网络和可比语句网络，组合成中英双语语料库，并将之存储于区块链上。通过利用两种网络之间的关联性以提高中英双语语料库中语料的准确性。

Description

中英双语语料库的构建方法、及其相关设备
本申请要求于2020年4月29日提交中国专利局、申请号为202010356769.2，发明名称为“中英双语语料库的构建方法、及其相关设备”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及计算机技术领域，尤其涉及一种中英双语语料库的构建方法、及其相关设备。
背景技术
目前，基于双语对话系统的中英双语语料库传统的训练方法主要分为两类：一类方法是使用对应领域的语料库以匹配待测试的中英双语语料库，不需要并行语料库，但是发明人发现其训练过程不稳定，且复杂性高，只能将方法限制在小规模数据上，且准确性不高；另一类方法是使用已有的多语言资源自动生成“伪双语文档”，这种方法虽然稳定，但发明人发现这种方法由于训练数据存在数据量大且不确定性，导致训练耗时且准确性不足。从而导致对话系统在使用中英双语语料库进行识别时，存在语义识别错误的情况，进一步影响对话系统的准确性。
发明内容
本申请实施例提供一种中英双语语料库的构建方法、及其相关设备，以解决传统中英双语语料库训练的准确度不高，进一步影响运用中英双语语料库的对话系统的准确性不高的问题。
一种中英双语语料库的构建方法，包括：
从预设实体库中获取中文实体、英文实体，以及中文实体与英文实体之间的映射关系和互译关系；
根据所述中文实体、所述英文实体、所述映射关系和所述互译关系，按照预设要求构建双语实体词语网络；
从预设数据库中获取每个所述中文实体和每个所述英文实体分别对应的上下文词语；
基于所述中文实体、所述英文实体、所述上下文词语、预设超链接集合和预设语句集，计算所述双语实体词语网络的单语言表征估值和跨语言实体估值；
获取可比语句网络和训练语句，并利用所述训练语句计算所述可比语句网络对应的跨语言语句估值；
对所述单语言表征估值、所述跨语言实体估值和所述跨语言语句估值进行加权求和，得到目标估值；
将所述目标估值与预设阈值进行比较，若达到预设条件利用所述双语实体词语网络和所述可比语句网络，组合成中英双语语料库。
一种中英双语语料库的构建装置，包括：
第一获取模块，用于从预设实体库中获取中文实体、英文实体，以及中文实体与英文实体之间的映射关系和互译关系；
构建模块，用于根据所述中文实体、所述英文实体、所述映射关系和所述互译关系，按照预设要求构建双语实体词语网络；
第二获取模块，用于从预设数据库中获取每个所述中文实体和每个所述英文实体分别 对应的上下文词语；
第一计算模块，用于基于所述中文实体、所述英文实体、所述上下文词语、预设超链接集合和预设语句集，计算所述双语实体词语网络的单语言表征估值和跨语言实体估值；
第二计算模块，用于获取可比语句网络和训练语句，并利用所述训练语句计算所述可比语句网络对应的跨语言语句估值；
求和模块，用于对所述单语言表征估值、所述跨语言实体估值和所述跨语言语句估值进行加权求和，得到目标估值；
组合模块，用于将所述目标估值与预设阈值进行比较，若达到预设条件利用所述双语实体词语网络和所述可比语句网络，组合成中英双语语料库。
一种计算机设备，包括存储器、处理器以及存储在所述存储器中并可在所述处理器上运行的计算机可读指令，所述处理器执行所述计算机可读指令时实现如下所述的中英双语语料库的构建方法的步骤：
从预设实体库中获取中文实体、英文实体，以及中文实体与英文实体之间的映射关系和互译关系；
根据所述中文实体、所述英文实体、所述映射关系和所述互译关系，按照预设要求构建双语实体词语网络；
从预设数据库中获取每个所述中文实体和每个所述英文实体分别对应的上下文词语；
基于所述中文实体、所述英文实体、所述上下文词语、预设超链接集合和预设语句集，计算所述双语实体词语网络的单语言表征估值和跨语言实体估值；
获取可比语句网络和训练语句，并利用所述训练语句计算所述可比语句网络对应的跨语言语句估值；
对所述单语言表征估值、所述跨语言实体估值和所述跨语言语句估值进行加权求和，得到目标估值；
将所述目标估值与预设阈值进行比较，若达到预设条件利用所述双语实体词语网络和所述可比语句网络，组合成中英双语语料库。
一种计算机可读存储介质，所述计算机可读存储介质存储有计算机可读指令，所述计算机可读指令被处理器执行时实现如下所述中英双语语料库的构建方法的步骤：
从预设实体库中获取中文实体、英文实体，以及中文实体与英文实体之间的映射关系和互译关系；
根据所述中文实体、所述英文实体、所述映射关系和所述互译关系，按照预设要求构建双语实体词语网络；
从预设数据库中获取每个所述中文实体和每个所述英文实体分别对应的上下文词语；
基于所述中文实体、所述英文实体、所述上下文词语、预设超链接集合和预设语句集，计算所述双语实体词语网络的单语言表征估值和跨语言实体估值；
获取可比语句网络和训练语句，并利用所述训练语句计算所述可比语句网络对应的跨语言语句估值；
对所述单语言表征估值、所述跨语言实体估值和所述跨语言语句估值进行加权求和，得到目标估值；
将所述目标估值与预设阈值进行比较，若达到预设条件利用所述双语实体词语网络和所述可比语句网络，组合成中英双语语料库。
上述中英双语语料库的构建方法、及其相关设备，基于映射关系、互译关系构建双语实体词语网络，能够加强中文实体与英文实体之间的关联关系，通过计算单语言表征估值、跨语言实体估值和跨语言语句估值的方式，能够准确判断双语实体词语网络和可比语句网络是否达到设定要求，最后在目标估值达到预设条件的情况下，利用双语实体词语网络和可比语句网络构成中英双语语料库，由于中英双语语料库是由2种网络构成，能够提高中 英双语语料库中不同语料之间的关联性，进一步能够提高中英双语语料库中语料的准确性，进一步提高运用中英双语语料库的对话系统的准确性。
附图说明
为了更清楚地说明本申请实施例的技术方案，下面将对本申请实施例的描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动性的前提下，还可以根据这些附图获得其他的附图。
图1是本申请实施例提供的中英双语语料库的构建方法的流程图；
图2是本申请实施例提供的中英双语语料库的构建方法中步骤S2的流程图；
图3是本申请实施例提供的中英双语语料库的构建方法中步骤S4的流程图；
图4是本申请实施例提供的中英双语语料库的构建方法中步骤S5的流程图；
图5是本申请实施例提供的中英双语语料库的构建方法中步骤S53的流程图；
图6是本申请实施例提供的中英双语语料库的构建方法中步骤S7的流程图；
图7是本申请实施例提供的中英双语语料库的构建装置的示意图；
图8是本申请实施例提供的计算机设备的基本机构框图。
具体实施方式
下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其他实施例，都属于本申请保护的范围。
本申请提供的中英双语语料库的构建方法应用于服务端，服务端具体可以用独立的服务器或者多个服务器组成的服务器集群实现。在一实施例中，如图1所示，提供一种中英双语语料库的构建方法，包括如下步骤：
S1：从预设实体库中获取中文实体、英文实体，以及中文实体与英文实体之间的映射关系和互译关系。
在本申请实施例中，中文实体与英文实体之间的映射关系是指根据用户的实际需求对中文实体与英文实体之间设定连接关系。例如，中文实体“苹果”与英文实体“fruit”之间存在映射关系。
需要说明的是，互译关系是指中文实体与英文实体之间互为翻译关系，例如，若中文实体为“苹果”，英文实体为“apple”，由于“苹果”对应的英文为“apple”，则中文实体与英文实体apple之间属于互译关系。
具体地，通过从预设实体库中获取中文实体、英文实体，以及中文实体与英文实体之间的映射关系和互译关系。其中，预设实体库是指专门用于存储中文实体、英文实体、以及中文实体与英文实体之间的映射关系和互译关系的数据库。
S2：根据中文实体、英文实体、映射关系和互译关系，按照预设要求构建双语实体词语网络。
在本申请实施例中，根据步骤S1获取到的中文实体、英文实体、中文实体与英文实体之间的映射关系和互译关系，按照预设要求构建双语实体词语网络。
其中，预设要求是指根据用户实际需求设定用于构建双语实体词语网络的要求。
S3：从预设数据库中获取每个中文实体和每个英文实体分别对应的上下文词语。
在本申请实施例中，上下文词语是指在对话场景下与中文实体或英文实体存在关联关系的词语。通过将中文实体和英文实体分别与预设数据库中的每个合法词语进行匹配，当匹配到中文实体和合法词语相同时，将该合法词语对应的上下文词语作为该中文实体对应 的上下文词语，同理，得到英文实体对应的上下文词语。
其中，预设数据库是指专门用于存储合法词语及合法词语对应的上下文词语的数据库。
例如，预设数据库中包含合法词语“苹果”、“pear”，其中，“苹果”对应的上下文词语为“apple”和“水果”，“pear”对应的上下文词语为“梨”和“水果”；若中文实体为“苹果”，将中文实体分别与合法词语“苹果”、“pear”进行匹配，得到合法词语“苹果”与中文实体相同，则将上下文词语“apple”和“水果”作为该中文实体的上下文词语。
S4：基于中文实体、英文实体、上下文词语、预设超链接集合和预设语句集，计算双语实体词语网络的单语言表征估值和跨语言实体估值。
在本申请实施例中，基于中文实体、英文实体、上下文词语、预设超链接集合和预设语句集，根据预设计算公式计算双语实体词语网络对应的单语言表征估值和跨语言实体估值。其中，预设计算公式是指用户设定用于计算单语言表征估值和跨语言实体估值对应的公式。
预设超链接集合是指由用户选定超链接构成的集合。
预设语句集是指用户预先从百度百科选定的语句构成的集合。
S5：获取可比语句网络和训练语句，并利用训练语句计算可比语句网络对应的跨语言语句估值。
具体地，从预设初始库中获取可比语句网络和训练语句，并利用预设计算方式和训练语句，计算可比语句网络对应的跨语言语句估值。
其中，预设初始库是指专门用于存储可比语句网络与训练语句的数据库。
预设计算方式是指预先设定根据训练语句计算可比语句网络对应的跨语言语句估值的计算方法。
需要说明的是，可比语句网络是指中文语句、英文语句，以及中文语句与英文语句之间的关联关系共同构成的网络。
S6：对单语言表征估值、跨语言实体估值和跨语言语句估值进行加权求和，得到目标估值。
具体地，将单语言表征估值、跨语言实体估值和跨语言语句估值分别于其对应的预设权值进行相乘，并将相乘结果进行求和计算，得到计算后的结果作为目标估值。
其中，预设权值是指用户预先设定的占比值，其具体的取值可以是0.3，也可根据用户的实际需要进行设置，此处不做限制。
S7：将目标估值与预设阈值进行比较，若达到预设条件利用双语实体词语网络和可比语句网络，组合成中英双语语料库。
在本申请实施例中，预设条件是用户根据实际需求设定的条件，其具体可以是根据目标估值与预设阈值进行比较得到的比较情况进行设定。
具体地，将目标估值于预设阈值进行比较，若比较结果为目标估值小于等于预设阈值，则将当前双语实体词语网络和可比语句网络组合成中英双语语料库；若比较结果为目标估值大于预设阈值，则对当前双语实体词语网络和可比语句网络进行迭代更新，直到目标估值小于等于阈值阈值为止。
其中，预设阈值是指根据用户的实际需求设定的值，此处不做限制。
本实施例中，基于映射关系、互译关系构建双语实体词语网络，能够加强中文实体与英文实体之间的关联关系，通过计算单语言表征估值、跨语言实体估值和跨语言语句估值的方式，能够准确判断双语实体词语网络和可比语句网络是否达到设定要求，最后在目标估值达到预设条件的情况下，利用双语实体词语网络和可比语句网络构成中英双语语料库，由于中英双语语料库是由2种网络构成，能够提高中英双语语料库中不同语料之间的关联性，进一步能够提高中英双语语料库中语料的准确性，进一步提高运用中英双语语料库的 对话系统的准确性。
在一实施例中，如图2所示，步骤S2中，根据中文实体、英文实体、映射关系和互译关系，按照预设要求构建双语实体词语网络包括如下步骤：
S21：获取所有中文实体作为第一集合，所有英文实体作为第二集合。
具体地，将步骤S1获取到的所有中文实体作为第一集合，将获取到的所有英文实体作为第二集合。
S22：从映射数据库中获取与中文实体存在映射关系的第一映射实体作为第三集合，以及与英文实体存在映射关系的第二映射实体作为第四集合，其中，映射数据库包含第一映射实体和第二映射实体。
在本申请实施例中，映射数据库包含不同的第一实体和第二实体，每个第一实体有其对应的第一映射实体，每个第二实体都有其对应的第二映射实体，且第一映射实体与第一实体之间存在映射关系，第二映射实体与第二实体之间存在映射关系。
将中文实体与映射数据库中的第一实体进行匹配，若中文实体与第一实体相同，则获取该第一实体对应的第一映射实体，并将所有第一映射实体作为第三集合；同理，将英文实体与第二实体进行匹配，若英文实体与第二实体相同，则获取该第二实体对应的第二映射实体，并将所有第二映射实体作为第四集合。
S23：获取存在互译关系的中文实体和英文实体作为互译实体，并将所有互译实体组合成第五集合。
在本申请实施例中，从预设实体库中获取存在互译关系的中文实体和英文实体作为互译实体，并将所有互译实体组合成第五集合。
S24：基于第一集合、第二集合、第三集合、第四集合和第五集合，按照公式(1)构建双语实体词语网络：
E＝(ε zh∪ε en,R zh∪R en∪R)    公式(1)
其中，E为双语实体词语网络，ε zh为第一集合，ε en为第二集合，R zh为第三集合，R en为第四集合，R为第五集合。
具体地，根据第一集合、第二集合、第三集合、第四集合和第五集合，按照公式(1)将这五个集合组合成一个新的集合，并将该新的集合作为双语实体词语网络。
本实施例中，通过分别获取第一集合、第二集合、第三集合、第四集合和第五集合，根据公式(1)能够快速准确地构建双语实体词语网络，保证双语实体词语网络构建的准确性，进一步提高后续利用双语实体词语网络构建中英双语语料库的准确性。
在一实施例中，如图3所示，上下文词语包含中文上下文词语和英文上下文词语，步骤S4中，即基于中文实体、英文实体、上下文词语、预设超链接集合和预设语句集，计算双语实体词语网络的单语言表征估值和跨语言实体估值包括如下步骤：
S41：将中文实体和英文实体导入到预设处理端口进行向量特征转换，得到训练实体。
在本申请实施例中，预设处理端口是指专门用于将中文实体或英文实体转换成词向量特征的处理端口，其具体是运用word2vec模型进行向量特征转换处理。
具体地，通过直接将中文实体和英文实体导入到预设处理端口中进行向量特征转换，得到转换处理后的词向量特征，并将词向量特征作为训练实体。
需要说明的是，在将中文实体和英文实体转换成词向量特征的同时，存在部分词向量特征包含超链接的形式。
S42：基于训练实体、上下文词语、预设超链接集合和预设语句集，根据公式(2)计算单语言表征估值：
Figure PCTCN2020117388-appb-000001
其中，L为单语言表征估值，zh为中文实体，en为英文实体，
Figure PCTCN2020117388-appb-000002
为训练实体，D为预设语句集，A为预设超链接集合，G为上下文词语，
Figure PCTCN2020117388-appb-000003
表示：(i)如果
Figure PCTCN2020117388-appb-000004
Figure PCTCN2020117388-appb-000005
是否是上下文词语；(ii)如果
Figure PCTCN2020117388-appb-000006
链接到
Figure PCTCN2020117388-appb-000007
的实体；(iii)如果A中存在q，判断
Figure PCTCN2020117388-appb-000008
是否为q的上下文词语，q为D中的一个元素。
S43：基于上下文词语，根据公式(3)计算跨语言实体估值：
Figure PCTCN2020117388-appb-000009
其中，I为跨语言实体估值，
Figure PCTCN2020117388-appb-000010
为当前实体，
Figure PCTCN2020117388-appb-000011
表示上下文词语，若当前实体为中文实体，则表示中文实体对应的英文上下文词语；若当前实体为英文实体，则表示英文实体对应的中文上下文词语；即表示与
Figure PCTCN2020117388-appb-000012
连接的其他语言的实体。
本实施例中，根据公式(2)和公式(3)，能够分别快速准确地计算出双语实体词语网络对应的单语言表征估值和跨语言实体估值，保证后续根据单语言表征估值和跨语言实体估值确定目标估值的准确性。
在一实施例中，如图4所示，步骤S5中，即获取可比语句网络和训练语句，并利用训练语句计算可比语句网络对应的跨语言语句估值包括如下步骤：
S51：从预设初始库中获取可比语句网络，其中，可比语句网络包含中文语句和英文语句。
在本申请实施例中，通过直接从预设初始库中获取可比语句网络。
需要说明的是，可比语句网络由中文语句和英文语句构成，其之间存在预先设定好的关联关系，但由于随着数据量的增多存在关联关系不准确的情况，故需要对可比语句网络进行优化处理。
S52：获取包含2个相同实体的中文语句和英文语句作为训练语句，其中，训练语句包含中文语句对应的中文句向量。
在本申请实施例中，中文语句和英文语句之间的相同实体是指存在互译关系的中文实体和英文实体，例如：中文实体“苹果”和英文实体“apple”属于相同实体。
具体地，通过直接从预设训练库中获取训练语句，其中，预设训练库是指专门用于存储训练语句的数据库。
S53：根据预设向量转换方式，将训练语句转换成综合向量。
具体地，根据预设向量转换方式，将训练语句转换成综合向量。
其中，预设向量转换转换方式具体可以是通过word2vec模型将训练语句转换为综合向量。
S54：根据综合向量和中文句向量，按照公式(4)计算跨语言语句估值：
Figure PCTCN2020117388-appb-000013
其中，J为跨语言语句估值，
Figure PCTCN2020117388-appb-000014
为综合向量，
Figure PCTCN2020117388-appb-000015
为中文句向量，K为可比语句网络。
具体地，将综合向量和中文句向量带入到公式(4)中，利用公式(4)计算跨语言语句估值。
本实例中，通过获取可比语句网络和训练语句，利用公式(4)能够快速准确地计算可比语句网络对应的跨语言语句估值，保证后续利用跨语言语句估值确定目标估值的准确性。
在一实施例中，如图5所示，S53中，即根据预设向量转换方式，将训练语句转换成综合向量包括如下步骤：
S531：对训练语句进行语义准确性判断，根据判断结果确定训练语句的第一权重值，并将确定第一权重值的训练语句作为第一目标语句。
在本申请实施例中，将训练语句导入到预设语义端口中进行语义准确性判断，输出准确度，并从预设权重表中获取该准确度对应的权重值作为第一权重值，最后将带有第一权重值的训练语句作为第一目标语句。
其中，预设语义端口是指预先训练好用于针对训练语句进行语义准确性判断，并根据判断结果输出准确度的处理端口。
预设权重表是指用于存储不同的准确度及准确度对应的权重值的数据表。
例如：将“温哥华是加拿大的首都。”和“温哥华是加拿大的重要城市。”这两句话导入到预设语义端口中，预设语义端口经过语义准确性判断得到前者表述的关系是错误的，后者表述的信息才是正确的。则其输出前者的准确度为0％，后者的准确度为100％。
S532：对第一目标语句进行句向量转换，得到第一向量。
具体地，将第一目标语句导入到预设向量转换端口中进行句向量转换处理，得到转换处理后的第一向量。其中，预设向量转换端口是指专门用于进行句向量转换处理的处理端口。
S533：通过对训练语句进行句意匹配的方式，确定训练语句中的中文语句和英文语句中包含每个词汇的第二权重值，并将确定第二权重值的训练语句作为第二目标语句。
在本申请实施例中，句意匹配是指以训练语句中的中文语句为基础，判断训练语句中的英文语句对应的翻译是否与中文语句相匹配的方式。具体地，将训练语句导入到预设匹配端口进行句意匹配处理，并在句意匹配处理后对英文语句中的每组词汇赋予第二权重值，将带有第二权重值的训练语句作为第二目标语句。
其中，预设匹配端口是指预先训练好用于对训练语句进行句意匹配，并根据匹配结果赋予训练语句包含的英文语句中每组词汇对应的第二权重值的处理端口。
预设匹配端口的具体处理过程为：将训练语句中的中文语句和英文语句通过word2vec模型转换成中文句向量和英文句向量，并将中文句向量中每个维度的向量与英文句向量中每个维度的向量进行比较，若同个维度下中文句向量的向量与英文句向量不相同，则根据预先训练好的向量比值对应的第二权重值，赋予英文句向量中该向量对应的第二权重值。
需要说明的是，每个向量都有其对应的词汇，且存在互译关系的中文词汇和英文词汇的向量相同。
例如，训练语句包含的中文语句为：“小明曾经是北京大学的一名学生”，英文语句为：“Xiaoming spent 7 years in Peking University.”，将中文语句通过word2vec模型转换成中文句向量为：(1,2,3,4,5)，将英文语句通过word2vec模型转换成英文句向量为：(1,0,3,4,9)，其中，英文句向量中第二维度向量0对应的英文语句中的词汇为spent，第五维度向量9对应的英文语句中的词汇为7years。
通过对中文句向量和英文句向量中每个维度的向量进行比较，得到第二维度2与0不同，第五维度5与9不同，若预先训练好的向量比值2:0对应的第二权重值为50％，向量比值5:9对应的第二权重值为1％，相同比值的第二权重值为100％；则英文句向量中第二维度的向量对应的第二权重值为50％，即spent对应的第二权重值为50％；第五维度的向量对应的第二权重值为1％，即7years的词汇的第二权重值为1％，其他相同比值的向量对应的第二权重值为100％。
S534：对第二目标语句中的英文语句进行句向量转换，得到第二向量。
具体地，将第二目标语句导入到预设向量转换端口中进行句向量转换处理，得到转换处理后的第二向量。
S535：基于第一向量和第二向量，计算训练语句对应的综合向量。
在本申请实施例中，根据第一向量和第二向量，按照公式(5)计算训练语句对应的综合向量：
Figure PCTCN2020117388-appb-000016
其中，
Figure PCTCN2020117388-appb-000017
为综合向量，
Figure PCTCN2020117388-appb-000018
为第一向量，
Figure PCTCN2020117388-appb-000019
为第二向量，
Figure PCTCN2020117388-appb-000020
为训练语句，
Figure PCTCN2020117388-appb-000021
为预设句向量。
本实施例中，通过对训练语句进行语义准确性判断的方式，能够识别训练语句是否存在语义错误的情况，并根据判断情况确定训练语句的第一权重值以确定第一目标语句，从而能够提高对第一目标语句的语义精确度，再根据句意匹配的方式，能够识别训练语句中的中文语句与英文语句句意之间的相似度，进而根据相似度确定英文语句中每个词汇的第二权重值以确定第二目标语句，提高第二目标语句的准确性，最后根据第一目标语句和第二目标语句计算训练语句对应的综合向量，能够保证综合向量的精确度。
在一实施例中，如图6所示，步骤S7中，即将目标估值与预设阈值进行比较，若达到预设条件利用双语实体词语网络和可比语句网络，组合成中英双语语料库包括如下步骤：
S71：将目标估值与预设阈值进行比较。
具体地，将目标估值与预设阈值进行比较。
S72：若目标估值小于等于预设阈值，则将双语实体词语网络和可比语句网络组合成中英双语语料库。
具体地，根据步骤S71的比较方式，若目标估值小于等于预设阈值，则将该目标估值对应的双语实体词语网络和可比语句网络组合成中英双语语料库。
S73：若目标估值大于预设阈值，则按照预设参数更新方式，对双语实体词语网络和可比语句网络进行迭代更新，直到目标估值小于等于预设阈值，将迭代更新后的双语实体词语网络和可比语句网络组合成中英双语语料库。
具体地，根据步骤S71的比较方式，若目标估值大于预设阈值，则按照预设参数更新方式，对双语实体词语网络和可比语句网络进行迭代更新，直到目标估值小于等于预设阈值，将迭代更新后的双语实体词语网络和可比语句网络组合成中英双语语料库。
其中，预设参数更新方式是指根据用户实际需求，设定对双语实体词语网络和可比语句网络中的参数进行更新的方式。
本实施例中，通过将目标估值与预设阈值进行比较的方式，在目标估值小于等于预设阈值的情况下确定中英双语语料库，在目标估值大于预设阈值的情况下重新对双语实体词语网络和可比语句网络进行迭代更新，并在目标估值小于等于预设阈值的情况下确定中英双语语料库。利用目标估值与预设阈值进行比较以确定中英双语语料库的方式，能够保证在达到设定条件的情况下确定中英双语语料库，进而提高中英双语语料库的准确性。
在一实施例中，步骤S7之后，该中英双语语料库的构建方法还包括：将中英双语语料库存储于区块链中。
需要强调的是，为进一步保证上述中英双语语料库的私密和安全性，上述中英双语语料库还可以存储于一区块链的节点中。
应理解，上述实施例中各步骤的序号的大小并不意味着执行顺序的先后，各过程的执行顺序应以其功能和内在逻辑确定，而不应对本申请实施例的实施过程构成任何限定。
在一实施例中，提供一种中英双语语料库的构建装置，该中英双语语料库的构建装置与上述实施例中中英双语语料库的构建方法一一对应。如图7所示，该中英双语语料库的构建装置包括第一获取模块71，构建模块72，第二获取模块73，第一计算模块74，第二计算模块75，求和模块76和组合模块77。各功能模块详细说明如下：
第一获取模块71，用于从预设实体库中获取中文实体、英文实体，以及中文实体与英文实体之间的映射关系和互译关系；构建模块72，用于根据中文实体、英文实体、映射关系和互译关系，按照预设要求构建双语实体词语网络；第二获取模块73，用于从预设数据库中获取每个中文实体和每个英文实体分别对应的上下文词语；第一计算模块74，用于基于中文实体、英文实体、上下文词语、预设超链接集合和预设语句集，计算双语实体词语网络的单语言表征估值和跨语言实体估值；第二计算模块75，用于获取可比语句网络和训练语句，并利用训练语句计算可比语句网络对应的跨语言语句估值；求和模块76，用于对单语言表征估值、跨语言实体估值和跨语言语句估值进行加权求和，得到目标估值；组合模块77，用于将目标估值与预设阈值进行比较，若达到预设条件利用双语实体词语网络和可比语句网络，组合成中英双语语料库。
进一步地，构建模块72包括：
第三获取子模块，用于获取所有中文实体作为第一集合，所有英文实体作为第二集合；
第四获取子模块，用于从映射数据库中获取与中文实体存在映射关系的第一映射实体作为第三集合，以及与英文实体存在映射关系的第二映射实体作为第四集合，其中，映射数据库包含第一映射实体和第二映射实体；
第五获取子模块，用于获取存在互译关系的中文实体和英文实体作为互译实体，并将所有互译实体组合成第五集合；
网络构建子模块，用于基于第一集合、第二集合、第三集合、第四集合和第五集合，按照公式(1)构建双语实体词语网络：
E＝(ε zh∪ε en,R zh∪R en∪R)     公式(1)
其中，E为双语实体词语网络，ε zh为第一集合，ε en为第二集合，R zh为第三集合，R en为第四集合，R为第五集合。
进一步地，第一计算模块74包括：
转换子模块，用于将中文实体和英文实体导入到预设处理端口进行向量特征转换，得到训练实体；
第三计算子模块，用于基于训练实体、上下文词语、预设超链接集合和预设语句集，根据公式(2)计算单语言表征估值：
Figure PCTCN2020117388-appb-000022
其中，L为单语言表征估值，zh为中文实体，en为英文实体，
Figure PCTCN2020117388-appb-000023
为训练实体，D为预设语句集，A为预设超链接集合，G为上下文词语，
Figure PCTCN2020117388-appb-000024
表示：(i)如果
Figure PCTCN2020117388-appb-000025
Figure PCTCN2020117388-appb-000026
是否是上下文词语；(ii)如果
Figure PCTCN2020117388-appb-000027
链接到
Figure PCTCN2020117388-appb-000028
的实体；(iii)如果A中存在q，判断
Figure PCTCN2020117388-appb-000029
是否为q的上下文词语，q为D中的一个元素；
第四计算子模块，用于基于上下文词语，根据公式(3)计算跨语言实体估值：
Figure PCTCN2020117388-appb-000030
其中，I为跨语言实体估值，
Figure PCTCN2020117388-appb-000031
为当前实体，
Figure PCTCN2020117388-appb-000032
表示上下文词语，若当前实体为中文实体，则表示中文实体对应的英文上下文词语；若当前实体为英文实体，则表示英文实体对应的中文上下文词语；即表示与
Figure PCTCN2020117388-appb-000033
连接的其他语言的实体。
进一步地，第二计算模块75包括：
第六获取子模块，用于从预设初始库中获取可比语句网络，其中，可比语句网络包含中文语句和英文语句；
第七获取子模块，用于获取包含2个相同实体的中文语句和英文语句作为训练语句，其中，训练语句包含中文语句对应的中文句向量；
综合向量转换子模块，用于根据预设向量转换方式，将训练语句转换成综合向量；
第五计算子模块，用于根据综合向量和中文句向量，按照公式(4)计算跨语言语句估值：
Figure PCTCN2020117388-appb-000034
其中，J为跨语言语句估值，
Figure PCTCN2020117388-appb-000035
为综合向量，
Figure PCTCN2020117388-appb-000036
为中文句向量，K为可比语句网络。
进一步地，综合向量转换子模块包括：
判断单元，用于对训练语句进行语义准确性判断，根据判断结果确定训练语句的第一权重值，并将确定第一权重值的训练语句作为第一目标语句；
第一向量获取单元，用于对第一目标语句进行句向量转换，得到第一向量；
匹配单元，用于通过对训练语句进行句意匹配的方式，确定训练语句中的中文语句和英文语句中包含每个词汇的第二权重值，并将确定第二权重值的训练语句作为第二目标语句；
第二向量获取单元，用于对第二目标语句中的英文语句进行句向量转换，得到第二向量；
综合向量计算单元，用于基于第一向量和第二向量，计算训练语句对应的综合向量。
进一步地，组合模块77包括：
比较子模块，用于将目标估值与预设阈值进行比较；
第一比较子模块，用于若目标估值小于等于预设阈值，则将双语实体词语网络和可比语句网络组合成中英双语语料库；
第二比较子模块，用于若目标估值大于预设阈值，则按照预设参数更新方式，对双语实体词语网络和可比语句网络进行迭代更新，直到目标估值小于等于预设阈值，将迭代更新后的双语实体词语网络和可比语句网络组合成中英双语语料库。
本申请的一些实施例公开了计算机设备。具体请参阅图8，为本申请的一实施例中计算机设备80基本结构框图。
如图8中所示意的，所述计算机设备80包括通过系统总线相互通信连接存储器81、处理器82、网络接口83。需要指出的是，图8中仅示出了具有组件81-83的计算机设备80，但是应理解的是，并不要求实施所有示出的组件，可以替代的实施更多或者更少的组件。其中，本技术领域技术人员可以理解，这里的计算机设备是一种能够按照事先设定或存储的指令，自动进行数值计算和/或信息处理的设备，其硬件包括但不限于微处理器、专用集成电路(Application Specific Integrated Circuit，ASIC)、可编程门阵列(Field－Programmable Gate Array，FPGA)、数字处理器(Digital Signal Processor，DSP)、嵌入式设备等。
所述计算机设备可以是桌上型计算机、笔记本、掌上电脑及云端服务器等计算设备。所述计算机设备可以与用户通过键盘、鼠标、遥控器、触摸板或声控设备等方式进行人机交互。
所述存储器81至少包括一种类型的可读存储介质，所述可读存储介质包括闪存、硬盘、多媒体卡、卡型存储器(例如，SD或DX存储器等)、随机访问存储器(RAM)、静态随机访问存储器(SRAM)、只读存储器(ROM)、电可擦除可编程只读存储器(EEPROM)、可编程只读存储器(PROM)、磁性存储器、磁盘、光盘等。在一些实施例中，所述存储器81可以是所述计算机设备80的内部存储单元，例如该计算机设备80的硬盘或内存。在另一些实施例中，所述存储器81也可以是所述计算机设备80的外部存储设备，例如该计算机设备80上配备的插接式硬盘，智能存储卡(Smart Media Card,SMC)，安全数字(Secure Digital,SD)卡，闪存卡(Flash Card)等。当然，所述存储器81还可以既包括所述计算机设备80的内部存储单元也包括其外部存储设备。本实施例中，所述存储器81通常用于存储安装于所述计算机设备80的操作系统和各类应用软件，例如所述中英双语语料库的构建方法的计算机可读指令等。此外，所述存储器81还可以用于暂时地存储已经输出或者将要输出的各类数据。
所述处理器82在一些实施例中可以是中央处理器(Central Processing Unit，CPU)、控制器、微控制器、微处理器、或其他数据处理芯片。该处理器82通常用于控制所述计算机设备80的总体操作。本实施例中，所述处理器82用于运行所述存储器81中存储的计算机可读指令或者处理数据，例如运行所述中英双语语料库的构建方法的计算机可读指令。
所述网络接口83可包括无线网络接口或有线网络接口，该网络接口83通常用于在所述计算机设备80与其他电子设备之间建立通信连接。
本申请还提供了另一种实施方式，即提供一种计算机可读存储介质，所述计算机可读存储介质可以是非易失性，也可以是易失性。所述计算机可读存储介质存储有中英文实体信息的流程，所述中英文实体信息的流程可被至少一个处理器执行，以使所述至少一个处理器执行上述任意一种中英双语语料库的构建方法的步骤。
需要强调的是，为进一步保证上述中英双语语料库的私密和安全性，上述中英双语语料库还可以存储于一区块链的节点中。
通过以上的实施方式的描述，本领域的技术人员可以清楚地了解到上述实施例方法可借助软件加必需的通用硬件平台的方式来实现，当然也可以通过硬件，但很多情况下前者是更佳的实施方式。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质(如ROM/RAM、磁碟、光盘)中，包括若干指令用以使得一台计算机设备(可以是手机，计算机，服务器，空调器，或者网络设备等)执行本申请各个实施例所述的方法。
本申请所指区块链是分布式数据存储、点对点传输、共识机制、加密算法等计算机技术的新型应用模式。区块链(Blockchain)，本质上是一个去中心化的数据库，是一串使用密码学方法相关联产生的数据块，每一个数据块中包含了一批次网络交易的信息，用于验证其信息的有效性(防伪)和生成下一个区块。区块链可以包括区块链底层平台、平台产品服务层以及应用服务层等。
最后应说明的是，显然以上所描述的实施例仅仅是本申请一部分实施例，而不是全部的实施例，附图中给出了本申请的较佳实施例，但并不限制本申请的专利范围。本申请可以以许多不同的形式来实现，相反地，提供这些实施例的目的是使对本申请的公开内容的理解更加透彻全面。尽管参照前述实施例对本申请进行了详细的说明，对于本领域的技术人员来而言，其依然可以对前述各具体实施方式所记载的技术方案进行修改，或者对其中部分技术特征进行等效替换。凡是利用本申请说明书及附图内容所做的等效结构，直接或间接运用在其他相关的技术领域，均同理在本申请专利保护范围之内。

Claims (20)

  1. 一种中英双语语料库的构建方法，所述中英双语语料库的构建方法包括：
    从预设实体库中获取中文实体、英文实体，以及中文实体与英文实体之间的映射关系和互译关系；
    根据所述中文实体、所述英文实体、所述映射关系和所述互译关系，按照预设要求构建双语实体词语网络；
    从预设数据库中获取每个所述中文实体和每个所述英文实体分别对应的上下文词语；
    基于所述中文实体、所述英文实体、所述上下文词语、预设超链接集合和预设语句集，计算所述双语实体词语网络的单语言表征估值和跨语言实体估值；
    获取可比语句网络和训练语句，并利用所述训练语句计算所述可比语句网络对应的跨语言语句估值；
    对所述单语言表征估值、所述跨语言实体估值和所述跨语言语句估值进行加权求和，得到目标估值；
    将所述目标估值与预设阈值进行比较，若达到预设条件利用所述双语实体词语网络和所述可比语句网络，组合成中英双语语料库。
  2. 如权利要求1所述的中英双语语料库的构建方法，其中，所述根据所述中文实体、所述英文实体、所述映射关系和所述互译关系，按照预设要求构建双语实体词语网络的步骤包括：
    获取所有所述中文实体作为第一集合，所有所述英文实体作为第二集合；
    从映射数据库中获取与所述中文实体存在所述映射关系的第一映射实体作为第三集合，以及与所述英文实体存在所述映射关系的第二映射实体作为第四集合，其中，所述映射数据库包含所述第一映射实体和所述第二映射实体；
    获取存在所述互译关系的所述中文实体和所述英文实体作为互译实体，并将所有所述互译实体组合成第五集合；
    基于所述第一集合、所述第二集合、所述第三集合、所述第四集合和所述第五集合，按照如下公式构建所述双语实体词语网络：
    E＝(ε zh∪ε en,R zh∪R en∪R)
    其中，E为所述双语实体词语网络，ε zh为所述第一集合，ε en为所述第二集合，R zh为所述第三集合，R en为所述第四集合，R为所述第五集合。
  3. 如权利要求1所述的中英双语语料库的构建方法，其中，所述上下文词语包含中文上下文词语和英文上下文词语，基于根据所述中文实体、所述英文实体、所述上下文词语、预设超链接集合和预设语句集，计算所述双语实体词语网络的单语言表征估值和跨语言实体估值的步骤包括：
    将所述中文实体和所述英文实体导入到预设处理端口进行向量特征转换，得到训练实体；
    基于所述训练实体、所述上下文词语、预设超链接集合和预设语句集，根据如下公式计算所述单语言表征估值：
    Figure PCTCN2020117388-appb-100001
    其中，L为所述单语言表征估值，zh为所述中文实体，en为所述英文实体，
    Figure PCTCN2020117388-appb-100002
    为所述 训练实体，D为预设语句集，A为预设超链接集合，G为所述上下文词语，
    Figure PCTCN2020117388-appb-100003
    表示：(i)如果
    Figure PCTCN2020117388-appb-100004
    是否是所述上下文词语；(ii)如果
    Figure PCTCN2020117388-appb-100005
    链接到
    Figure PCTCN2020117388-appb-100006
    的实体；(iii)如果A中存在q，判断
    Figure PCTCN2020117388-appb-100007
    是否为q的上下文词语，q为D中的一个元素；
    基于所述上下文词语，根据如下公式计算所述跨语言实体估值：
    Figure PCTCN2020117388-appb-100008
    其中，I为所述跨语言实体估值，
    Figure PCTCN2020117388-appb-100009
    为当前实体，
    Figure PCTCN2020117388-appb-100010
    表示所述上下文词语，若所述当前实体为所述中文实体，则表示所述中文实体对应的所述英文上下文词语；若当前实体为所述英文实体，则表示所述英文实体对应的所述中文上下文词语；即表示与
    Figure PCTCN2020117388-appb-100011
    连接的其他语言的实体。
  4. 如权利要求1所述的中英双语语料库的构建方法，其中，所述获取可比语句网络和训练语句，并利用所述训练语句计算所述可比语句网络对应的跨语言语句估值的步骤包括：
    从预设初始库中获取所述可比语句网络，其中，所述可比语句网络包含中文语句和英文语句；
    获取包含2个相同实体的所述中文语句和所述英文语句作为所述训练语句，其中，所述训练语句包含所述中文语句对应的中文句向量；
    根据预设向量转换方式，将所述训练语句转换成综合向量；
    根据所述综合向量和所述中文句向量，按照如下公式计算所述跨语言语句估值：
    Figure PCTCN2020117388-appb-100012
    其中，J为所述跨语言语句估值，
    Figure PCTCN2020117388-appb-100013
    为所述综合向量，
    Figure PCTCN2020117388-appb-100014
    为所述中文句向量，K为所述可比语句网络。
  5. 如权利要求4所述的中英双语语料库的构建方法，其中，所述根据预设向量转换方式，将所述训练语句转换成综合向量的步骤包括：
    对所述训练语句进行语义准确性判断，根据判断结果确定所述训练语句的第一权重值，并将确定所述第一权重值的所述训练语句作为第一目标语句；
    对所述第一目标语句进行句向量转换，得到第一向量；
    通过对所述训练语句进行句意匹配的方式，确定所述训练语句中的所述中文语句和所述英文语句中包含每个词汇的第二权重值，并将确定所述第二权重值的所述训练语句作为第二目标语句；
    对所述第二目标语句中的所述英文语句进行句向量转换，得到第二向量；
    基于所述第一向量和所述第二向量，计算所述训练语句对应的综合向量。
  6. 如权利要求1所述的中英双语语料库的构建方法，其中，所述将所述目标估值与预设阈值进行比较，若达到预设条件利用所述双语实体词语网络和所述可比语句网络，组合成中英双语语料库的步骤包括：
    将目标估值与预设阈值进行比较；
    若所述目标估值小于等于预设阈值，则将所述双语实体词语网络和所述可比语句网络组合成所述中英双语语料库；
    若所述目标估值大于预设阈值，则按照预设参数更新方式，对所述双语实体词语网络和所述可比语句网络进行迭代更新，直到所述目标估值小于等于预设阈值，将迭代更新后的双语实体词语网络和可比语句网络组合成所述中英双语语料库。
  7. 如权利要求1所述的中英双语语料库的构建方法，其中，在所述组合成中英双语语料库之后还包括：将所述中英双语语料库存储于区块链中。
  8. 一种中英双语语料库的构建装置，所述中英双语语料库的构建装置包括：
    第一获取模块，用于从预设实体库中获取中文实体、英文实体，以及中文实体与英文实体之间的映射关系和互译关系；
    构建模块，用于根据所述中文实体、所述英文实体、所述映射关系和所述互译关系，按照预设要求构建双语实体词语网络；
    第二获取模块，用于从预设数据库中获取每个所述中文实体和每个所述英文实体分别对应的上下文词语；
    第一计算模块，用于基于所述中文实体、所述英文实体、所述上下文词语、预设超链接集合和预设语句集，计算所述双语实体词语网络的单语言表征估值和跨语言实体估值；
    第二计算模块，用于获取可比语句网络和训练语句，并利用所述训练语句计算所述可比语句网络对应的跨语言语句估值；
    求和模块，用于对所述单语言表征估值、所述跨语言实体估值和所述跨语言语句估值进行加权求和，得到目标估值；
    组合模块，用于将所述目标估值与预设阈值进行比较，若达到预设条件利用所述双语实体词语网络和所述可比语句网络，组合成中英双语语料库。
  9. 一种计算机设备，包括存储器、处理器以及存储在所述存储器中并可在所述处理器上运行的计算机可读指令，所述处理器执行所述计算机可读指令时实现如下所述的中英双语语料库的构建方法的步骤：
    从预设实体库中获取中文实体、英文实体，以及中文实体与英文实体之间的映射关系和互译关系；
    根据所述中文实体、所述英文实体、所述映射关系和所述互译关系，按照预设要求构建双语实体词语网络；
    从预设数据库中获取每个所述中文实体和每个所述英文实体分别对应的上下文词语；
    基于所述中文实体、所述英文实体、所述上下文词语、预设超链接集合和预设语句集，计算所述双语实体词语网络的单语言表征估值和跨语言实体估值；
    获取可比语句网络和训练语句，并利用所述训练语句计算所述可比语句网络对应的跨语言语句估值；
    对所述单语言表征估值、所述跨语言实体估值和所述跨语言语句估值进行加权求和，得到目标估值；
    将所述目标估值与预设阈值进行比较，若达到预设条件利用所述双语实体词语网络和所述可比语句网络，组合成中英双语语料库。
  10. 如权利要求9所述的计算机设备，其中，所述根据所述中文实体、所述英文实体、所述映射关系和所述互译关系，按照预设要求构建双语实体词语网络的步骤包括：
    获取所有所述中文实体作为第一集合，所有所述英文实体作为第二集合；
    从映射数据库中获取与所述中文实体存在所述映射关系的第一映射实体作为第三集合，以及与所述英文实体存在所述映射关系的第二映射实体作为第四集合，其中，所述映射数据库包含所述第一映射实体和所述第二映射实体；
    获取存在所述互译关系的所述中文实体和所述英文实体作为互译实体，并将所有所述互译实体组合成第五集合；
    基于所述第一集合、所述第二集合、所述第三集合、所述第四集合和所述第五集合， 按照如下公式构建所述双语实体词语网络：
    E＝(ε zh∪ε en,R zh∪R en∪R)
    其中，E为所述双语实体词语网络，ε zh为所述第一集合，ε en为所述第二集合，R zh为所述第三集合，R en为所述第四集合，R为所述第五集合。
  11. 如权利要求9所述的计算机设备，其中，所述上下文词语包含中文上下文词语和英文上下文词语，基于根据所述中文实体、所述英文实体、所述上下文词语、预设超链接集合和预设语句集，计算所述双语实体词语网络的单语言表征估值和跨语言实体估值的步骤包括：
    将所述中文实体和所述英文实体导入到预设处理端口进行向量特征转换，得到训练实体；
    基于所述训练实体、所述上下文词语、预设超链接集合和预设语句集，根据如下公式计算所述单语言表征估值：
    Figure PCTCN2020117388-appb-100015
    其中，L为所述单语言表征估值，zh为所述中文实体，en为所述英文实体，
    Figure PCTCN2020117388-appb-100016
    为所述训练实体，D为预设语句集，A为预设超链接集合，G为所述上下文词语，
    Figure PCTCN2020117388-appb-100017
    表示：(i)如果
    Figure PCTCN2020117388-appb-100018
    是否是所述上下文词语；(ii)如果
    Figure PCTCN2020117388-appb-100019
    链接到
    Figure PCTCN2020117388-appb-100020
    的实体；(iii)如果A中存在q，判断
    Figure PCTCN2020117388-appb-100021
    是否为q的上下文词语，q为D中的一个元素；
    基于所述上下文词语，根据如下公式计算所述跨语言实体估值：
    Figure PCTCN2020117388-appb-100022
    其中，I为所述跨语言实体估值，
    Figure PCTCN2020117388-appb-100023
    为当前实体，
    Figure PCTCN2020117388-appb-100024
    表示所述上下文词语，若所述当前实体为所述中文实体，则表示所述中文实体对应的所述英文上下文词语；若当前实体为所述英文实体，则表示所述英文实体对应的所述中文上下文词语；即表示与
    Figure PCTCN2020117388-appb-100025
    连接的其他语言的实体。
  12. 如权利要求9所述的计算机设备，其中，所述获取可比语句网络和训练语句，并利用所述训练语句计算所述可比语句网络对应的跨语言语句估值的步骤包括：
    从预设初始库中获取所述可比语句网络，其中，所述可比语句网络包含中文语句和英文语句；
    获取包含2个相同实体的所述中文语句和所述英文语句作为所述训练语句，其中，所述训练语句包含所述中文语句对应的中文句向量；
    根据预设向量转换方式，将所述训练语句转换成综合向量；
    根据所述综合向量和所述中文句向量，按照如下公式计算所述跨语言语句估值：
    Figure PCTCN2020117388-appb-100026
    其中，J为所述跨语言语句估值，
    Figure PCTCN2020117388-appb-100027
    为所述综合向量，
    Figure PCTCN2020117388-appb-100028
    为所述中文句向量，K为所 述可比语句网络。
  13. 如权利要求12所述的计算机设备，其中，所述根据预设向量转换方式，将所述训练语句转换成综合向量的步骤包括：
    对所述训练语句进行语义准确性判断，根据判断结果确定所述训练语句的第一权重值，并将确定所述第一权重值的所述训练语句作为第一目标语句；
    对所述第一目标语句进行句向量转换，得到第一向量；
    通过对所述训练语句进行句意匹配的方式，确定所述训练语句中的所述中文语句和所述英文语句中包含每个词汇的第二权重值，并将确定所述第二权重值的所述训练语句作为第二目标语句；
    对所述第二目标语句中的所述英文语句进行句向量转换，得到第二向量；
    基于所述第一向量和所述第二向量，计算所述训练语句对应的综合向量。
  14. 如权利要求9所述的计算机设备，其中，所述将所述目标估值与预设阈值进行比较，若达到预设条件利用所述双语实体词语网络和所述可比语句网络，组合成中英双语语料库的步骤包括：
    将目标估值与预设阈值进行比较；
    若所述目标估值小于等于预设阈值，则将所述双语实体词语网络和所述可比语句网络组合成所述中英双语语料库；
    若所述目标估值大于预设阈值，则按照预设参数更新方式，对所述双语实体词语网络和所述可比语句网络进行迭代更新，直到所述目标估值小于等于预设阈值，将迭代更新后的双语实体词语网络和可比语句网络组合成所述中英双语语料库。
  15. 一种计算机可读存储介质，所述计算机可读存储介质存储有计算机可读指令，所述计算机可读指令被处理器执行时实现如下所述中英双语语料库的构建方法的步骤：
    从预设实体库中获取中文实体、英文实体，以及中文实体与英文实体之间的映射关系和互译关系；
    根据所述中文实体、所述英文实体、所述映射关系和所述互译关系，按照预设要求构建双语实体词语网络；
    从预设数据库中获取每个所述中文实体和每个所述英文实体分别对应的上下文词语；
    基于所述中文实体、所述英文实体、所述上下文词语、预设超链接集合和预设语句集，计算所述双语实体词语网络的单语言表征估值和跨语言实体估值；
    获取可比语句网络和训练语句，并利用所述训练语句计算所述可比语句网络对应的跨语言语句估值；
    对所述单语言表征估值、所述跨语言实体估值和所述跨语言语句估值进行加权求和，得到目标估值；
    将所述目标估值与预设阈值进行比较，若达到预设条件利用所述双语实体词语网络和所述可比语句网络，组合成中英双语语料库。
  16. 如权利要求15所述的计算机可读存储介质，其中，所述根据所述中文实体、所述英文实体、所述映射关系和所述互译关系，按照预设要求构建双语实体词语网络的步骤包括：
    获取所有所述中文实体作为第一集合，所有所述英文实体作为第二集合；
    从映射数据库中获取与所述中文实体存在所述映射关系的第一映射实体作为第三集合，以及与所述英文实体存在所述映射关系的第二映射实体作为第四集合，其中，所述映射数据库包含所述第一映射实体和所述第二映射实体；
    获取存在所述互译关系的所述中文实体和所述英文实体作为互译实体，并将所有所述互译实体组合成第五集合；
    基于所述第一集合、所述第二集合、所述第三集合、所述第四集合和所述第五集合， 按照如下公式构建所述双语实体词语网络：
    E＝(ε zh∪ε en,R zh∪R en∪R)
    其中，E为所述双语实体词语网络，ε zh为所述第一集合，ε en为所述第二集合，R zh为所述第三集合，R en为所述第四集合，R为所述第五集合。
  17. 如权利要求15所述的计算机可读存储介质，其中，所述上下文词语包含中文上下文词语和英文上下文词语，基于根据所述中文实体、所述英文实体、所述上下文词语、预设超链接集合和预设语句集，计算所述双语实体词语网络的单语言表征估值和跨语言实体估值的步骤包括：
    将所述中文实体和所述英文实体导入到预设处理端口进行向量特征转换，得到训练实体；
    基于所述训练实体、所述上下文词语、预设超链接集合和预设语句集，根据如下公式计算所述单语言表征估值：
    Figure PCTCN2020117388-appb-100029
    其中，L为所述单语言表征估值，zh为所述中文实体，en为所述英文实体，
    Figure PCTCN2020117388-appb-100030
    为所述训练实体，D为预设语句集，A为预设超链接集合，G为所述上下文词语，
    Figure PCTCN2020117388-appb-100031
    表示：(i)如果
    Figure PCTCN2020117388-appb-100032
    是否是所述上下文词语；(ii)如果
    Figure PCTCN2020117388-appb-100033
    链接到
    Figure PCTCN2020117388-appb-100034
    的实体；(iii)如果A中存在q，判断
    Figure PCTCN2020117388-appb-100035
    是否为q的上下文词语，q为D中的一个元素；
    基于所述上下文词语，根据如下公式计算所述跨语言实体估值：
    Figure PCTCN2020117388-appb-100036
    其中，I为所述跨语言实体估值，
    Figure PCTCN2020117388-appb-100037
    为当前实体，
    Figure PCTCN2020117388-appb-100038
    表示所述上下文词语，若所述当前实体为所述中文实体，则表示所述中文实体对应的所述英文上下文词语；若当前实体为所述英文实体，则表示所述英文实体对应的所述中文上下文词语；即表示与
    Figure PCTCN2020117388-appb-100039
    连接的其他语言的实体。
  18. 如权利要求15所述的计算机可读存储介质，其中，所述获取可比语句网络和训练语句，并利用所述训练语句计算所述可比语句网络对应的跨语言语句估值的步骤包括：
    从预设初始库中获取所述可比语句网络，其中，所述可比语句网络包含中文语句和英文语句；
    获取包含2个相同实体的所述中文语句和所述英文语句作为所述训练语句，其中，所述训练语句包含所述中文语句对应的中文句向量；
    根据预设向量转换方式，将所述训练语句转换成综合向量；
    根据所述综合向量和所述中文句向量，按照如下公式计算所述跨语言语句估值：
    Figure PCTCN2020117388-appb-100040
    其中，J为所述跨语言语句估值，
    Figure PCTCN2020117388-appb-100041
    为所述综合向量，
    Figure PCTCN2020117388-appb-100042
    为所述中文句向量，K为所 述可比语句网络。
  19. 如权利要求18所述的计算机可读存储介质，其中，所述根据预设向量转换方式，将所述训练语句转换成综合向量的步骤包括：
    对所述训练语句进行语义准确性判断，根据判断结果确定所述训练语句的第一权重值，并将确定所述第一权重值的所述训练语句作为第一目标语句；
    对所述第一目标语句进行句向量转换，得到第一向量；
    通过对所述训练语句进行句意匹配的方式，确定所述训练语句中的所述中文语句和所述英文语句中包含每个词汇的第二权重值，并将确定所述第二权重值的所述训练语句作为第二目标语句；
    对所述第二目标语句中的所述英文语句进行句向量转换，得到第二向量；
    基于所述第一向量和所述第二向量，计算所述训练语句对应的综合向量。
  20. 如权利要求15所述的计算机可读存储介质，其中，所述将所述目标估值与预设阈值进行比较，若达到预设条件利用所述双语实体词语网络和所述可比语句网络，组合成中英双语语料库的步骤包括：
    将目标估值与预设阈值进行比较；
    若所述目标估值小于等于预设阈值，则将所述双语实体词语网络和所述可比语句网络组合成所述中英双语语料库；
    若所述目标估值大于预设阈值，则按照预设参数更新方式，对所述双语实体词语网络和所述可比语句网络进行迭代更新，直到所述目标估值小于等于预设阈值，将迭代更新后的双语实体词语网络和可比语句网络组合成所述中英双语语料库。
PCT/CN2020/117388 2020-04-29 2020-09-24 中英双语语料库的构建方法、及其相关设备 WO2021218012A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010356769.2A CN111695361A (zh) 2020-04-29 2020-04-29 中英双语语料库的构建方法、及其相关设备
CN202010356769.2 2020-04-29

Publications (1)

Publication Number Publication Date
WO2021218012A1 true WO2021218012A1 (zh) 2021-11-04

Family

ID=72476871

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/117388 WO2021218012A1 (zh) 2020-04-29 2020-09-24 中英双语语料库的构建方法、及其相关设备

Country Status (2)

Country Link
CN (1) CN111695361A (zh)
WO (1) WO2021218012A1 (zh)

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111695361A (zh) * 2020-04-29 2020-09-22 平安科技（深圳）有限公司 中英双语语料库的构建方法、及其相关设备
US11741318B2 (en) 2021-03-25 2023-08-29 Nec Corporation Open information extraction from low resource languages
CN114997162A (zh) * 2022-05-26 2022-09-02 中国工商银行股份有限公司 一种训练数据提取方法和装置
CN115269627A (zh) * 2022-05-27 2022-11-01 上海迈伺通健康科技有限公司 使用中文指令操作数据库的方法、系统、设备及存储介质

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP2199925A1 (en) * 2008-12-03 2010-06-23 Xerox Corporation Dynamic translation memory using statistical machine translation
CN109033320A (zh) * 2018-07-18 2018-12-18 无码科技（杭州）有限公司 一种双语新闻聚合方法及系统
CN110765785A (zh) * 2019-09-19 2020-02-07 平安科技（深圳）有限公司 基于神经网络的中英翻译方法、及其相关设备
CN110874537A (zh) * 2018-08-31 2020-03-10 阿里巴巴集团控股有限公司 多语言翻译模型的生成方法、翻译方法及设备
CN110941964A (zh) * 2019-12-11 2020-03-31 北京小米移动软件有限公司 双语语料筛选方法、装置及存储介质
CN111695361A (zh) * 2020-04-29 2020-09-22 平安科技（深圳）有限公司 中英双语语料库的构建方法、及其相关设备

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP2199925A1 (en) * 2008-12-03 2010-06-23 Xerox Corporation Dynamic translation memory using statistical machine translation
CN109033320A (zh) * 2018-07-18 2018-12-18 无码科技（杭州）有限公司 一种双语新闻聚合方法及系统
CN110874537A (zh) * 2018-08-31 2020-03-10 阿里巴巴集团控股有限公司 多语言翻译模型的生成方法、翻译方法及设备
CN110765785A (zh) * 2019-09-19 2020-02-07 平安科技（深圳）有限公司 基于神经网络的中英翻译方法、及其相关设备
CN110941964A (zh) * 2019-12-11 2020-03-31 北京小米移动软件有限公司 双语语料筛选方法、装置及存储介质
CN111695361A (zh) * 2020-04-29 2020-09-22 平安科技（深圳）有限公司 中英双语语料库的构建方法、及其相关设备

Also Published As

Publication number Publication date
CN111695361A (zh) 2020-09-22

Similar Documents

Publication Publication Date Title
WO2021218012A1 (zh) 中英双语语料库的构建方法、及其相关设备
WO2021135910A1 (zh) 基于机器阅读理解的信息抽取方法、及其相关设备
JP4945086B2 (ja) 論理形式のための統計的言語モデル
CN110765763A (zh) 语音识别文本的纠错方法、装置、计算机设备和存储介质
US20060015322A1 (en) Method and apparatus for improving statistical word alignment models using smoothing
WO2021051513A1 (zh) 基于神经网络的中英翻译方法、及其相关设备
US20110288852A1 (en) Dynamic bi-phrases for statistical machine translation
WO2021051517A1 (zh) 基于卷积神经网络的信息检索方法、及其相关设备
US20080208566A1 (en) Automated word-form transformation and part of speech tag assignment
WO2021208727A1 (zh) 基于人工智能的文本错误检测方法、装置、计算机设备
WO2020082673A1 (zh) 发票检验方法、装置、计算设备和存储介质
WO2021174871A1 (zh) 数据查询方法、系统、计算机设备及存储介质
WO2021068563A1 (zh) 样本数据处理方法、装置、计算机设备及存储介质
CN110134780B (zh) 文档摘要的生成方法、装置、设备、计算机可读存储介质
WO2021072863A1 (zh) 文本相似度计算方法、装置、电子设备及计算机可读存储介质
CN111259262A (zh) 一种信息检索方法、装置、设备及介质
CN112559895A (zh) 一种数据处理方法、装置、电子设备及存储介质
CN110489423B (zh) 一种信息抽取的方法、装置、存储介质及电子设备
KR102608867B1 (ko) 업계 텍스트를 증분하는 방법, 관련 장치 및 매체에 저장된 컴퓨터 프로그램
CN109034199B (zh) 数据处理方法及装置、存储介质和电子设备
CN111460117B (zh) 对话机器人意图语料生成方法、装置、介质及电子设备
CN113177407A (zh) 数据字典的构建方法、装置、计算机设备及存储介质
CN111858966B (zh) 知识图谱的更新方法、装置、终端设备及可读存储介质
JP6687944B2 (ja) 自動翻訳システム、自動翻訳方法、およびプログラム
WO2021072864A1 (zh) 文本相似度获取方法、装置、电子设备及计算机可读存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20932953

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20932953

Country of ref document: EP

Kind code of ref document: A1