WO2022102364A1 - 文生成モデル生成装置、文生成モデル及び文生成装置 - Google Patents

文生成モデル生成装置、文生成モデル及び文生成装置 Download PDF

Info

Publication number
WO2022102364A1
WO2022102364A1 PCT/JP2021/038829 JP2021038829W WO2022102364A1 WO 2022102364 A1 WO2022102364 A1 WO 2022102364A1 JP 2021038829 W JP2021038829 W JP 2021038829W WO 2022102364 A1 WO2022102364 A1 WO 2022102364A1
Authority
WO
WIPO (PCT)
Prior art keywords
sentence
input
output
data
decoder
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2021/038829
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
保静 松岡
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
NTT Docomo Inc
Original Assignee
NTT Docomo Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by NTT Docomo Inc filed Critical NTT Docomo Inc
Priority to JP2022561365A priority Critical patent/JP7805309B2/ja
Priority to US18/252,140 priority patent/US12333267B2/en
Publication of WO2022102364A1 publication Critical patent/WO2022102364A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/40—Processing or translation of natural language
    • G06F40/58—Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/30—Semantic analysis
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/40—Processing or translation of natural language
    • G06F40/42—Data-driven translation
    • G06F40/44—Statistical methods, e.g. probability models
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/40—Processing or translation of natural language
    • G06F40/55—Rule-based translation
    • G06F40/56—Natural language generation

Definitions

  • the present invention relates to a sentence generation model generator, a sentence generation model, and a sentence generator.
  • Patent Document 1 discloses a technique for generating a document corresponding to an input document by using a machine learning model.
  • the present invention has been made in view of the above problems, and an object thereof is to obtain an output sentence in a second language according to a specific condition according to an input sentence in the first language.
  • the sentence generation model generator generates a sentence in a second language different from the first language in response to the input of the input sentence in the first language. It is a sentence generation model generator that generates a generation model by machine learning, and the sentence generation model is an encoder-decoder model composed of an encoder and a decoder including a neural network, and is for learning used for machine learning of the sentence generation model.
  • the data includes the first data, the context and the second data, the first data includes an array of a plurality of words constituting the input sentence, and the second data includes a plurality of output sentences corresponding to the input sentence.
  • the context contains one or more second language words related to the second data
  • the sentence generation model generator inputs the first data to the encoder according to the order of the word arrangement.
  • the decoder input unit that inputs the context, the start symbol that is a predetermined symbol that means the start of output of the output sentence, and the words that make up the second data to the decoder according to the order of the arrangement, and after the input of the start symbol.
  • An update unit that updates the weight coefficient constituting the encoder and the decoder based on the error between the word array output from the decoder and the word array included in the second data in the latter stage, and the update unit. It includes a model output unit that outputs a statement generation model with updated weight coefficients.
  • the sentence generation model is composed of an encoder decoder model including an encoder and a decoder.
  • the second data that is, the context including the word related to the output sentence is used. It is input to the decoder together with the second data. Therefore, since the sentence generation model learns the relationship between the context and the second data, it is possible to obtain a sentence generation model that outputs an output sentence according to the conditions of the output sentence set in the context.
  • FIG. 13 is a diagram showing the configuration of a sentence generation model generation program.
  • FIG. 14 is a diagram showing the configuration of a sentence generation program.
  • the sentence generation model of the present embodiment is constructed by machine learning for operating a computer and generating an output sentence in a second language different from the first language in response to an input sentence in the first language. It is a model.
  • the statement generation model includes a neural network and is composed of an encoder-decoder model including an encoder and a decoder.
  • the sentence generation model generation device of this embodiment is a device that generates a sentence generation model by machine learning.
  • the sentence generation device is a device that generates an output sentence of a second language in response to an input of an input sentence of the first language by using a sentence generation model constructed by machine learning.
  • English composition may be written using specific expressions (for example, gerunds, infinitives, etc.).
  • specific expressions for example, gerunds, infinitives, etc.
  • the Japanese sentence “We study English to be able to speak with foreigners (Watashitachiha gaikokujinto hanaseruyoninarutameni eigowo benkyosuru)” is replaced with “We study English so that we”. It is required to translate it into English as "can talk to foreigners.”
  • the English translation "We study English to become able to speak with foreigners.” Is output or highly evaluated.
  • FIG. 1 is a diagram showing a functional configuration of a sentence generation model generation device according to the present embodiment.
  • the sentence generation model generation device 10 is a device that generates a sentence generation model by machine learning that generates an output sentence in a second language different from the first language in response to an input sentence in the first language.
  • the sentence generation model generation device 10 functionally includes a context generation unit 11, an encoder input unit 12, a decoder input unit 13, an update unit 14, and a model output unit 15.
  • Each of these functional units 11 to 15 may be configured in one device, or may be distributed and configured in a plurality of devices.
  • the sentence generation model generation device 10 is configured to be accessible to storage means such as the model storage unit 30 and the corpus storage unit 40.
  • the model storage unit 30 and the corpus storage unit 40 may be configured in the sentence generation model generation device 10, or as shown in FIG. 1, outside the sentence generation model generation device 10 from the sentence generation model generation device. It may be configured as another accessible device.
  • the model storage unit 30 is a storage means that stores a sentence generation model that has been learned or is a learning process, and can be configured by a storage, a memory, or the like.
  • the corpus storage unit 40 is a storage means for storing learning data used for machine learning of a sentence generation model, a corpus for generating learning data, and the like, and may be composed of storage, memory, and the like. can.
  • FIG. 2 is a diagram showing a functional configuration of the sentence generator according to the present embodiment.
  • the sentence generation device 20 is a device that generates an output sentence in a second language different from the first language in response to an input of an input sentence in the first language by using a sentence generation model constructed by machine learning.
  • the sentence generation device 20 functionally includes an input unit 21, a context input unit 22, a word input unit 23, and an output unit 24.
  • the sentence generation device 20 may further include a created sentence acquisition unit 25, a created sentence input unit 26, and a created sentence evaluation unit 27.
  • Each of these functional units 21 to 27 may be configured in one device, or may be distributed and configured in a plurality of devices.
  • the sentence generation device 20 is configured to be accessible to the model storage unit 30 that stores the trained sentence generation model.
  • the model storage unit 30 may be configured in the sentence generation device 20 or may be configured in another external device.
  • sentence generation model generation device 10 and the sentence generation device 20 are configured in different devices (computers) is shown, but these may be configured integrally.
  • each functional block may be realized using one physically or logically coupled device, or two or more physically or logically separated devices can be directly or indirectly (eg, for example). , Wired, wireless, etc.) and may be realized using these plurality of devices.
  • the functional block may be realized by combining the software with the one device or the plurality of devices.
  • Functions include judgment, decision, judgment, calculation, calculation, processing, derivation, investigation, search, confirmation, reception, transmission, output, access, solution, selection, selection, establishment, comparison, assumption, expectation, and assumption. Broadcasting, notifying, communicating, forwarding, configuring, reconfiguring, allocating, mapping, assigning, etc., but limited to these I can't.
  • a functional block (configuration unit) that makes transmission function is called a transmitting unit (transmitting unit) or a transmitter (transmitter).
  • the realization method is not particularly limited.
  • the sentence generation model generation device 10 and the sentence generation device 20 in the embodiment of the present invention may function as a computer.
  • FIG. 3 is a diagram showing an example of the hardware configuration of the sentence generation model generation device 10 and the sentence generation device 20 according to the present embodiment.
  • the sentence generation model generation device 10 and the sentence generation device 20 are each physically configured as a computer device including a processor 1001, a memory 1002, a storage 1003, a communication device 1004, an input device 1005, an output device 1006, a bus 1007, and the like. You may.
  • the word "device” can be read as a circuit, device, unit, etc.
  • the hardware configuration of the sentence generation model generation device 10 and the sentence generation device 20 may be configured to include one or more of the devices shown in the figure, or may be configured not to include some of the devices. good.
  • Each function in the sentence generation model generation device 10 and the sentence generation device 20 is performed by the processor 1001 by loading predetermined software (program) on the hardware such as the processor 1001 and the memory 1002, and the communication device 1004 performs the calculation. It is realized by controlling communication and reading and / or writing of data in the memory 1002 and the storage 1003.
  • the processor 1001 operates, for example, an operating system to control the entire computer.
  • the processor 1001 may be configured by a central processing unit (CPU: Central Processing Unit) including an interface with a peripheral device, a control device, an arithmetic unit, a register, and the like.
  • CPU Central Processing Unit
  • the functional units 11 to 15, 21 to 27 shown in FIGS. 1 and 2 may be realized by the processor 1001.
  • the processor 1001 reads a program (program code), a software module and data from the storage 1003 and / or the communication device 1004 into the memory 1002, and executes various processes according to these.
  • a program program code
  • a program that causes a computer to execute at least a part of the operations described in the above-described embodiment is used.
  • the functional units 11 to 15, 21 to 27 of the sentence generation model generation device 10 and the sentence generation device 20 may be stored in the memory 1002 and realized by a control program operated by the processor 1001.
  • Processor 1001 may be mounted on one or more chips.
  • the program may be transmitted from the network via a telecommunication line.
  • the memory 1002 is a computer-readable recording medium, and is composed of at least one such as a ROM (Read Only Memory), an EPROM (Erasable Programmable ROM), an EEPROM (Electrically Erasable Programmable ROM), and a RAM (Random Access Memory). May be done.
  • the memory 1002 may be referred to as a register, a cache, a main memory (main storage device), or the like.
  • the memory 1002 can store a statement generation model generation method and a program (program code), a software module, and the like that can be executed to implement the statement generation model generation method and the statement generation method according to the embodiment of the present invention.
  • the storage 1003 is a computer-readable recording medium, and is, for example, an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, an optical magnetic disk (for example, a compact disk, a digital versatile disk, a Blu-ray). It may consist of at least one (registered trademark) disk), smart card, flash memory (eg, card, stick, key drive), floppy (registered trademark) disk, magnetic strip, and the like.
  • the storage 1003 may be referred to as an auxiliary storage device.
  • the storage medium described above may be, for example, a database, server or other suitable medium containing memory 1002 and / or storage 1003.
  • the communication device 1004 is hardware (transmission / reception device) for communicating between computers via a wired and / or wireless network, and is also referred to as, for example, a network device, a network controller, a network card, a communication module, or the like.
  • the input device 1005 is an input device (for example, a keyboard, a mouse, a microphone, a switch, a button, a sensor, etc.) that accepts an input from the outside.
  • the output device 1006 is an output device (for example, a display, a speaker, an LED lamp, etc.) that outputs to the outside.
  • the input device 1005 and the output device 1006 may have an integrated configuration (for example, a touch panel).
  • each device such as the processor 1001 and the memory 1002 is connected by the bus 1007 for communicating information.
  • the bus 1007 may be composed of a single bus or may be composed of different buses between the devices.
  • the sentence generation model generation device 10 and the sentence generation device 20 include a microprocessor, a digital signal processor (DSP: Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), and an FPGA (Field Programmable Gate Array). ) And the like, and a part or all of each functional block may be realized by the hardware.
  • the processor 1001 may be implemented on at least one of these hardware.
  • FIG. 4 is a diagram showing the configuration of the sentence generation model of the present embodiment.
  • the sentence generation model MD is an encoder-decoder model including a neural network and composed of an encoder en and a decoder de.
  • the neural network constituting the encoder-decoder model is not limited, but is, for example, a recurrent neural network (RNN).
  • the sentence generation model MD may be a neural network called a transformer.
  • the learning data used for machine learning of the sentence generation model MD of the present embodiment includes the first data a, the second data b, and the context c.
  • the first data a includes an array of a plurality of words constituting an input sentence of the first language.
  • the second data b includes an array of a plurality of words constituting the output sentence of the second language corresponding to the input sentence.
  • the output sentence is, for example, a parallel translation of the input sentence.
  • the context c includes one or more second language words associated with the second data b.
  • the first data a constituting the input sentence of the first language is input to the encoder en.
  • the first data a is divided into words by, for example, morphological analysis.
  • Each of the divided words is embedded into a corresponding word vector and input to the encoder en according to the arrangement order in the first data a (input sentence).
  • the encoder en outputs a vector (for example, output of an intermediate layer, source target attention, etc.) indicating a calculation result based on the first data a to the decoder de.
  • the decoder sequentially outputs a sequence of words based on the input of a vector from the encoder and a predetermined start symbol (vector) indicating the start of output.
  • the context c is input to the decoder de of the sentence generation model MD of the present embodiment in the stage before the input of the start symbol ss.
  • the decoder de outputs a sequence of words (vectors) in the output sentence t based on the output from the encoder en and the input of the context c and the start symbol ss.
  • the output sentence t is composed of the sequence of words output so far.
  • the second data b corresponding to the output sentence (translation of the input sentence in the second language) corresponding to the first data a (input sentence) is obtained word by word in the subsequent stage of inputting the start symbol ss. It is input to the decoder de according to the order of arrangement.
  • the context c includes one or more second language words related to the second data b.
  • the generation of the context c and the like will be described in detail later, but the context c may be configured to include one or more words constituting a part of the second data b. Further, the context c may be a second language question sentence whose answer sentence is an output sentence composed of words included in the second data b.
  • the functional part of the sentence generation model generation device 10 will be described with reference to FIG. 1 again.
  • the context generation unit 11 generates a context based on the corpus. With reference to FIGS. 5 to 7, context generation and learning data including the context will be described.
  • FIG. 5 is a diagram showing an example of generation of the first data, the second data, and the context based on the corpus.
  • the context generation unit 11 acquires the corpus cp0 from, for example, the corpus storage unit 40.
  • the corpus cp0 consists of a first sentence cp01 composed of a first language and a second sentence cp02 composed of a second language.
  • the first sentence cp01 is a Japanese sentence
  • the second sentence cp02 is an English translation of the first sentence cp01.
  • the context generation unit 11 extracts the word cx that constitutes a part of the second sentence cp02 as the context.
  • the context generation unit 11 may randomly extract the word cx from a plurality of words constituting the second sentence cp02. Further, the context generation unit 11 may extract the word cx based on the designated input indicating the characteristic part of the second sentence cp02.
  • the context generation unit 11 generates the context c01 based on the extracted word cx. Further, the context generation unit 11 generates the first data a01 and the second data b01 in the training data based on the first sentence cp01 and the second sentence cp02, respectively, and the first data a01, the context c01, and the context generation unit 11 generate the first data a01 and the second data b01.
  • the training data consisting of the start symbol ss and the second data b01 is generated.
  • the context generation unit 11 may include information indicating the relationship with the second data b01 in the context.
  • the context generation unit 11 includes the symbol cl01 indicating that the word cx is a word to be used in the syntax of the second data b01 (output sentence) in the context c01.
  • FIG. 6 is a diagram showing an example of the generation of the first data, the second data, and the context based on the corpus.
  • the context generation unit 11 acquires the corpus cp1 from, for example, the corpus storage unit 40.
  • Corpus cp1 is the first sentence cp11 composed in the first language "What is a man trying to do? I'm going to take a picture of a bird. (Otokonohito ha nani wo shiyotoshiteimasuka? Tori no shashin wo torutsumoridesu)"
  • a second sentence cp12 "What is the man going to do? He is going to take pictures of birds." Composed of the second language.
  • FIG. 1 is the first sentence cp11 composed in the first language "What is a man trying to do? I'm going to take a picture of a bird. (Otokonohito ha nani wo shiyotoshiteimasuka? Tori no shas
  • the first sentence cp11 is a Japanese sentence
  • the second sentence cp12 is an English translation of the first sentence cp11.
  • the first sentence cp11 is the question sentence cpq1 "What is a man trying to do? (Otokonohito ha nani wo shiyotoshiteimasuka?)" And the answer sentence cpa1 "I'm going to take a picture of a bird. (Tori no shashin wo torutsumoridesu). ) ”.
  • the second sentence cp12 consists of a question sentence cpq2 “What is the man going to do?” And an answer sentence cpa2 “He is going to take pictures of birds.”.
  • the context generation unit 11 extracts from the corpus the answer sentence composed of the first language and the answer sentence of the second language which is the translation thereof as the first data and the second data.
  • the context generation unit 11 intends to take a picture of the first data a02cpa1 “birds” for each of the answer sentence cpa1 in the first language and the answer sentence cpa2 in the second language. torutsumoridesu) ”and the second data b02“ He is going to take pictures of birds. ”.
  • the context generation unit 11 extracts the question sentence as a context based on the corpus including the question sentence composed in the second language and the answer sentence to the question sentence.
  • the context generation unit 11 extracts the question sentence cpq2 from the corpus cp12 composed of the question sentence cpq2 and the answer sentence cpa2 composed of the second language, and uses it as the context c02. Then, the context generation unit 11 generates learning data including the first data a02, the context c02, the start symbol ss, and the second data b02.
  • the context generation unit 11 may include information indicating the relationship with the second data b02 in the context c02.
  • the context generation unit 11 includes the symbol cl02 indicating that the context c02 is a question sentence having the second data b02 as the answer sentence in the context c02.
  • FIG. 7 is a diagram showing an example of the first data, the second data, and the context used for learning the sentence generation model.
  • the learning data of the sentence generation model MD may include an arbitrary symbol, which is a predetermined symbol having no linguistic meaning and content, as the first data instead of the array of a plurality of words constituting the input sentence.
  • the learning data is composed of the first data consisting of an arbitrary symbol a03 having no meaning and the context c03 consisting of a second language question sentence and b03 consisting of a second language answer sentence. May be done.
  • the context generation unit 11 extracts the question sentence as the context c03, extracts the answer sentence as the second data, and further adds the arbitrary symbol a03, based on the corpus consisting of the question sentence and the answer sentence in the second language. You may generate training data.
  • the context generation unit 11 includes the symbol cl03 indicating that the context c03 is a question sentence having the second data b03 as the answer sentence in the context c03. Further, the context generation unit 11 includes the start symbol ss03 at the beginning of the second data b03. The start symbol ss03 means the start of the answer sentence and may mean that the answer sentence is an answer to the question sentence constituting the context.
  • the relationship between the context and the second data can be learned by the decoder. can. Therefore, it is possible to expand the learning data at low cost, and it is possible to improve the accuracy of the output sentence output by the decoder with respect to the desired output.
  • the encoder input unit 12 inputs the first data a to the encoder en according to the order of word arrangement.
  • the decoder input unit 13 inputs the context c, the start symbol ss which is a predetermined symbol meaning the start of output of the output sentence, and the second data b into the decoder de word by word according to the arrangement order.
  • the update unit 14 sets the encoder en and the decoder de based on the error between the word sequence output from the decoder de and the word sequence included in the second data b in the subsequent stage after the input of the start symbol ss. Update the weighting factors that make up.
  • the encoder input unit 12 puts the word vectors of the words constituting the first data a into the input layer of the RNN constituting the encoder en in word order. Enter in sequence according to.
  • the output of the intermediate layer of the encoder en based on the input of the last word vector of the first data a is output to the decoder de.
  • the decoder input unit 13 sequentially inputs the word vectors of the words constituting the context c (including the symbol cl indicating the relationship with the second data) into the input layer of the RNN constituting the decoder de in order of words. Further, the decoder input unit 13 sequentially inputs the start symbol ss and the second data b to the decoder de according to the word order. When the start symbol ss is input to the decoder de, the decoder de sequentially outputs a sequence of word vectors of the output sentence t together with the likelihood (for example, by the softmax function).
  • the update unit 14 calculates the error between the word sequence output from the decoder de and the word sequence of the second data b for each word, and constitutes a neural network of the encoder en and the decorator de by, for example, the error back propagation method. Update the weighting factor.
  • FIG. 8 is a diagram for explaining a schematic configuration of a transformer which is an example of an encoder / decoder model.
  • the encoder input unit 12 is the word constituting the first data a1 “Yes, Ari masu”.
  • the word vectors aw11 to aw14 are input to the input layer ila of the encoder en1 according to the order of word arrangement.
  • Transformers allow parallel processing of input data rather than sequential word input as in RNNs.
  • the self-attention sa1 for the intermediate layer mla is calculated from the input layer ila, and the word vector is converted into a vector corresponding to the self-attention sa1.
  • the self-attention sa2 for the output layer ola is calculated from the intermediate layer mla, and the word vector is further converted.
  • the source target attention ta for the input layer ilb of the decoder de1 is calculated from the output layer ola of the encoder en1.
  • the decoder input unit 13 uses the word vectors cw11 to cw12 of the words constituting the context c1, the start symbol ss, and the word vectors bw11 to bw13 of the words constituting the second data b1 "Yes There is” according to the order of the word arrangement.
  • data is input to the input layer ilb of the decoder de1 in parallel.
  • the self-attention sa3 for the intermediate layer mlb is calculated from the input layer ilb, and the vector is converted according to the self-attention sa3.
  • the self-attention sa4 for the output layer olb is calculated from the intermediate layer mlb, and the vector conversion according to the self-attention sa4 is performed.
  • the update unit 14 makes an error between the word sequences t11 to t13 based on the word vector wv output in the subsequent stage after the input of the start symbol ss and the word sequences bw11 to bw13 of the words constituting the second data b1 for each word. Calculate and update the weighting factors for calculating self-attention and source-target attention by the error backpropagation method.
  • the model output unit 15 outputs the sentence generation model MD obtained after machine learning based on the required amount of learning data.
  • the model output unit 15 may store the sentence generation model MD in the model storage unit 30.
  • FIG. 9 is a diagram schematically showing a sentence generation process by the sentence generation model.
  • the sentence generation model MD2 is a model learned and constructed by the sentence generation model generation device 10.
  • the statement generation model MD2 includes an encoder en2 and a decoder de2.
  • the sentence generation model MD (MD1, MD2), which is a model including a trained neural network, is read or referred to by a computer, and is regarded as a program that causes the computer to perform a predetermined process and realizes a predetermined function. Can be done.
  • the trained sentence generation model MD (MD1, MD2) of the present embodiment is used in a computer including a CPU and a memory.
  • the CPU of the computer corresponds to each layer with respect to the input data input to the input layer of the neural network according to the command from the learned sentence generation model MD (MD1, MD2) stored in the memory. It operates to output the result (probability) from the output layer by performing an operation based on the learned weight coefficient (parameter) and a function.
  • the input unit 21 inputs the words aw21 to aw24 constituting the input data a2 "Yes, Ari masu" constituting the input sentence into the encoder en2 according to the arrangement order.
  • the encoder en2 outputs the calculation result to the decoder de2.
  • the context input unit 22 inputs the words ct2, cw21 to cw26 constituting the input context c2 composed of one or more second language words related to the output sentence into the decoder de2 in the order of arrangement.
  • the word ct2 is a symbol indicating the relationship with the output sentence of the context.
  • the input context c2 "Is he interesting in history?" Is information for specifying the conditions and context of the output sentence, for example, a question sentence in which the word to be included in the output sentence and the output sentence are the answers. There may be.
  • the word input unit 23 inputs the start symbol ss to the decoder de2 after the input of the input context c2.
  • the decoder de2 outputs the word tw21 at the beginning of the output sentence t2 according to the start symbol ss.
  • the word input unit 23 sequentially inputs the words output from the decoder de2 in the previous stage to the decoder de2.
  • the decoder de2 sequentially outputs a series of words tw21 to tw24 constituting the output sentence t2 "Yes, he is" according to the words sequentially input.
  • the output unit 24 generates the output sentence t2 by arranging the words tw21 to tw24 sequentially output in each stage of the decoder de2 when the terminal symbol es indicating the end of the output of the output sentence is output. Then, the output unit 24 outputs the generated output sentence t2.
  • the mode of output of the output sentence t2 is not limited, but may be, for example, storage in a predetermined storage means, display in a display means, output by voice, or the like.
  • FIG. 10 is a diagram showing an evaluation process of a created sentence in an evaluation system configured by the sentence generation device 20.
  • the decoder de3 shown in FIG. 10 outputs, for each word, the likelihood indicating the plausibility as a word constituting the output sentence t3 for each of the words tw31 to tw34 to be output in each stage after the input of the start symbol ss. ..
  • the evaluation system configured by the sentence generation device 20 evaluates the created sentence created and input by the user with the output sentence t3 as the correct answer.
  • the user inputs, for example, a bilingual translation of the input sentence in a second language as a creation sentence in response to the presentation of the input sentence corresponding to the output sentence t3. In this embodiment, it is assumed that the created sentence input by the user is evaluated, but the created sentence created and input by a person other than the user, a device, or the like may be evaluated.
  • the created sentence acquisition unit 25 acquires the created sentence r3 created by the user in the second language and input to the evaluation system.
  • the created sentence r3 consists of an array of words rw31 to rw34.
  • the created sentence input unit 26 replaces the words tw31 to tw34 output from the decoder de3 in the previous stage with the words constituting the created sentence r3 created in the second language.
  • the word vectors of rw31 to rw34 are sequentially input to the decoder de3.
  • the created sentence evaluation unit 27 is based on the input of the start symbol ss and the sequential input of the words rw31 to rw34 constituting the created sentence r3, and the created sentence evaluation unit 27 outputs each word rw31 from the decoder de3 in each step after the input of the start symbol ss.
  • the created sentence r3 is evaluated based on the comparison between the likelihood of rw34 and the likelihood of each word tw31 to tw34 constituting the output sentence t3.
  • the decoder de3 outputs the likelihood of each word of the entire vocabulary handled by the sentence generation model generation device 10 and the sentence generation device 20 at each output stage.
  • the output sentence t3 is configured by arranging the words having the highest likelihood in each output stage.
  • the created sentence evaluation unit 27 determines each of the created sentences r3 from the likelihood of each vocabulary output in response to the input of the start symbol ss and the words (rw31 to rw34) output in the previous stage. The likelihood associated with the words rw31 to rw34 and the terminator es is acquired.
  • the created sentence evaluation unit 27 calculates the evaluation value of the created sentence r3 by comparing the likelihood of each word tw31 to tw34 constituting the output sentence t3 with the likelihood of each word rw31 to rw34 of the created sentence r3. And output.
  • the method for calculating the evaluation value is not limited, but may be based on, for example, the ratio of the likelihoods for each word in each sentence t3, r3, and the total or average of the likelihoods for each sentence t3, r3.
  • the likelihood of each word constituting the output sentence and each word obtained by sequentially inputting each word constituting the created and input created sentence into the decoder is evaluated based on the comparison with the likelihood of. This makes it possible to configure an evaluation system that evaluates the plausibility of the created sentence as a parallel translation corresponding to the input sentence.
  • FIG. 11 is a flowchart showing the processing contents of the sentence generation model generation method in the sentence generation model generation device 10.
  • step S1 the sentence generation model generation device 10 acquires learning data including the first data a, the second data b, and the context c.
  • the learning data may be data generated in advance based on the corpus and stored in the corpus storage unit 40, or may be data generated based on the corpus by the context generation unit 11.
  • step S2 the first data a is input to the encoder en according to the order of word arrangement.
  • step S3 the decoder input unit 13 inputs the context c to the decoder de. Subsequently, in step S4, the decoder input unit 13 inputs the start symbol ss to the decoder de. Further, in step S5, the decoder input unit 13 inputs the second data b to the decoder de word by word according to the arrangement order.
  • step S6 the update unit 14 calculates an error for each word between the sequence of words output from the decoder de and the sequence of words included in the second data b in the subsequent stage after the input of the start symbol ss, and the error is calculated.
  • the weighting coefficient constituting the encoder en and the decoder de is updated by the back propagation method.
  • step S7 the update unit 14 determines whether or not machine learning based on the required amount of learning data has been completed. If it is determined that the learning is completed, the process proceeds to step S8. On the other hand, if it is not determined that the learning is completed, the processes of steps S1 to S6 are repeated.
  • step S8 the model output unit 15 outputs the trained sentence generation model MD.
  • FIG. 12 is a flowchart showing the processing contents of the sentence generation method using the trained sentence generation model MD in the sentence generation device 20.
  • step S11 the input unit 21 inputs the words of the input data constituting the input sentence to the encoder of the sentence generation model according to the arrangement order for each word.
  • the encoder outputs the calculation result to the decoder.
  • step S12 the context input unit 22 inputs an input context for designating the conditions of the output sentence to the decoder according to the sequence order for each word. Subsequently, in step S13, the word input unit 23 inputs the start symbol ss to the decoder after the input of the input context.
  • step S14 the output unit 24 acquires a word (or symbol) output from the output layer of the decoder.
  • step S15 the output unit 24 determines whether or not the output from the decoder is a terminal symbol indicating the end of the output of the output statement. If it is determined that the output from the decoder is a terminal symbol, the process proceeds to step S17. On the other hand, if the output from the decoder is not determined to be a terminal symbol, the process proceeds to step S16.
  • step S16 the word input unit 23 inputs the word output from the output layer in the previous stage of the decoder to the input layer in the current stage of the decoder. Then, the process returns to step S14.
  • step S17 the output unit 24 arranges the words sequentially output from the output layer at each stage of the decoder to generate an output sentence. Then, in step S18, the output unit 24 outputs an output statement.
  • FIG. 13 is a diagram showing the configuration of a sentence generation model generation program.
  • the sentence generation model generation program P1 includes a main module m10, a context generation module m11, an encoder input module m12, a decoder input module m13, an update module m14, and a model that collectively control the sentence generation model generation process in the sentence generation model generation device 10. It is configured to include an output module m15. Then, each module m11 to m15 realizes each function for the context generation unit 11, the encoder input unit 12, the decoder input unit 13, the update unit 14, and the model output unit 15.
  • the sentence generation model generation program P1 may be transmitted via a transmission medium such as a communication line, or may be stored in the recording medium M1 as shown in FIG. good.
  • FIG. 14 is a diagram showing the structure of the sentence generation program.
  • the sentence generation program P2 includes a main module m20, an input module m21, a context input module m22, a word input module m23, and an output module m24 that collectively control the sentence generation process in the sentence generation device 20. Further, the sentence generation program P2 may further include a created sentence acquisition module m25, a created sentence input module m26, and a created sentence evaluation module m27. Is configured with. Then, each module m21 to m27 provides functions for the input unit 21, the context input unit 22, the word input unit 23, the output unit 24, the created sentence acquisition unit 25, the created sentence input unit 26, and the created sentence evaluation unit 27. It will be realized.
  • the sentence generation program P2 may be transmitted via a transmission medium such as a communication line, or may be stored in the recording medium M2 as shown in FIG.
  • the sentence generation model is composed of an encoder decoder model including an encoder and a decoder.
  • the second data that is, the context including the word related to the output sentence is used. It is input to the decoder together with the second data. Therefore, since the sentence generation model learns the relationship between the context and the second data, it is possible to obtain a sentence generation model that outputs an output sentence according to the conditions of the output sentence set in the context.
  • the context may include one or more words constituting a part of the second data.
  • the context includes the words to be used in the output sentence, it is possible to generate a sentence generation model capable of outputting the output sentence including the words to be used.
  • the sentence generation model generator is a corpus composed of a first sentence composed of a first language and a second sentence which is a bilingual translation of the first sentence composed of a second language.
  • a context generator which extracts words constituting a part of the second sentence as a context, may be further included.
  • a context for designating a word to be used in an output sentence as a condition can be obtained as learning data based on the corpus.
  • the context may be a second language question sentence whose answer sentence is an output sentence composed of words included in the second data.
  • a sentence generation model that can output an answer sentence that matches the question sentence as an output sentence is generated. can.
  • the sentence generation model generation device has a context generation unit that extracts the question sentence as a context based on the question sentence composed of the second language and the corpus including the answer sentence to the question sentence. It may be further included.
  • a context for designating a context to be followed in the output sentence as a condition can be obtained as learning data.
  • the first data is an arbitrary symbol which is a predetermined symbol having no linguistic meaning and content instead of an array of a plurality of words constituting the input sentence. It may be.
  • the relationship between the context and the second data can be learned in the decoder even if the first data corresponding to the input sentence which is the translation of the output sentence does not exist. Therefore, it is possible to expand the learning data at low cost, and it is possible to improve the accuracy of the output sentence output by the decoder with respect to the desired output.
  • the context may include information indicating the relationship with the second data.
  • the decoder can learn how to use the conditions specified in the context. Therefore, it is possible to improve the accuracy of the output statement output by the decoder with respect to the desired output.
  • the sentence generation model causes a computer to function, and in response to the input of the input sentence of the first language, the output sentence of the second language different from the first language is output. It is a sentence generation model trained by machine learning for generation, and the learning data used for machine learning of the sentence generation model is the first data including an array of a plurality of words constituting the input sentence, the input sentence. A second data containing an array of words constituting the output sentence corresponding to the second data, and a context containing one or more second language words related to the second data, and the sentence generation model includes a neural network.
  • the two data are input to the decoder in the order of the context, the start symbol, the words of the second data and the arrangement of the symbols, and are included in the sequence of words output from the decoder in the subsequent stage after the input of the start symbol and the second data. It is constructed by machine learning that updates the weighting coefficients that make up the encoder and decoder based on the word-by-word error with the word sequence.
  • the sentence generation model is composed of an encoder decoder model including an encoder and a decoder.
  • the first data corresponding to the input sentence is input to the encoder
  • the second data corresponding to the output sentence is input to the decoder
  • the second data that is, the word related to the output sentence is included.
  • the context is input to the decoder along with the second data. Therefore, since the relationship between the context and the second data is learned, the sentence generation model can output an output sentence according to the condition of the output sentence set in the context.
  • the sentence generation device uses a sentence generation model constructed by machine learning, and responds to the input of the input sentence of the first language, and is different from the first language.
  • a sentence generator that generates output sentences in different second languages, and the learning data used for machine learning of the sentence generation model is the first data including an array of a plurality of words corresponding to the input sentences, and the input sentence.
  • the sentence generation model includes a neural network and an encoder, including a second data containing an array of multiple words corresponding to the corresponding output sentences, and a context containing one or more second language words associated with the second data.
  • the data is input to the decoder according to the context, the start symbol, the word of the second data, and the arrangement order of the symbols, and the sequence of words output from the decoder in the subsequent stage after the input of the start symbol and the word included in the second data.
  • the context input unit to input the input context consisting of one or more second language words related to the output sentence to the decoder, and the start symbol to be input to the decoder after the input of the input context, and the start symbol.
  • the word input unit that sequentially inputs the words output from the decoder in the previous stage to the decoder and the words sequentially output in each stage of the decoder are arranged to generate an output sentence and generate it. It is provided with an output unit for outputting the output text.
  • the sentence generation model is composed of an encoder decoder model including an encoder and a decoder.
  • the first data corresponding to the input sentence is input to the encoder
  • the second data corresponding to the output sentence is input to the decoder
  • the second data that is, the word related to the output sentence is included.
  • the context is input to the decoder along with the second data.
  • the trained sentence generation model learns the relationship between the context and the second data. Therefore, by inputting the input data constituting the input sentence to the encoder and inputting the input context for specifying the condition of the output sentence to the decoder, the output sentence according to the desired condition can be output.
  • the decoder outputs, for each word, the plausibility indicating the plausibility as a word constituting the output sentence for each of the words to be output in each stage after the input of the start symbol. Then, in each stage after the input of the start symbol, the sentence generator sequentially inputs the words constituting the created sentence created in the second language into the decoder in place of the words output from the decoder in the previous stage. Based on the composition sentence input unit and the input of the start symbol and the sequential input of each word constituting the composition sentence, the likelihood of each word constituting the composition sentence output from the decoder at each stage after the input of the start symbol. , A created sentence evaluation unit that evaluates the created sentence based on the comparison with the likelihood of each word constituting the output sentence is further provided.
  • the likelihood of each word constituting the output sentence is compared with the likelihood of each word obtained by sequentially inputting each word constituting the created and input created sentence into the decoder. Based on this, the composition is evaluated. This makes it possible to configure an evaluation system that evaluates the plausibility of the created sentence as a parallel translation corresponding to the input sentence.
  • Each aspect / embodiment described in the present specification includes LTE (Long Term Evolution), LTE-A (LTE-Advanced), SUPER 3G, IMT-Advanced, 4G, 5G, FRA (Future Radio Access), W-CDMA. (Registered Trademark), GSM (Registered Trademark), CDMA2000, UMB (Ultra Mobile Broadband), IEEE 802.11 (Wi-Fi), IEEE 802.16 (WiMAX), IEEE 802.20, UWB (Ultra-WideBand), It may be applied to Bluetooth®, other systems that utilize suitable systems and / or next-generation systems that are extended based on them.
  • the input / output information and the like may be saved in a specific location (for example, memory) or may be managed by a management table. Information to be input / output may be overwritten, updated, or added. The output information and the like may be deleted. The input information or the like may be transmitted to another device.
  • the determination may be made by a value represented by 1 bit (0 or 1), by a boolean value (Boolean: true or false), or by comparing numerical values (for example, a predetermined value). It may be done by comparison with the value).
  • the notification of predetermined information (for example, the notification of "being X") is not limited to the explicit one, but is performed implicitly (for example, the notification of the predetermined information is not performed). May be good.
  • software, instructions, etc. may be transmitted and received via a transmission medium.
  • the software may use wired technology such as coaxial cable, fiber optic cable, twist pair and digital subscriber line (DSL) and / or wireless technology such as infrared, wireless and microwave to website, server, or other.
  • wired technology such as coaxial cable, fiber optic cable, twist pair and digital subscriber line (DSL)
  • DSL digital subscriber line
  • wireless technology such as infrared, wireless and microwave to website, server, or other.
  • the information, signals, etc. described in this disclosure may be represented using any of a variety of different techniques.
  • data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description are voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, light fields or photons, or any of these. It may be represented by a combination of.
  • system and "network” used herein are used interchangeably.
  • information, parameters, etc. described in the present specification may be represented by an absolute value, a relative value from a predetermined value, or another corresponding information. ..
  • determining and “determining” used in this disclosure may include a wide variety of actions.
  • “Judgment” and “decision” are, for example, judgment (judging), calculation (calculating), calculation (computing), processing (processing), derivation (deriving), investigation (investigating), search (looking up, search, inquiry). It may include (eg, searching in a table, database or another data structure), ascertaining as “judgment” or “decision”.
  • judgment and “decision” are receiving (for example, receiving information), transmitting (for example, transmitting information), input (input), output (output), and access. It may include (for example, accessing data in memory) to be regarded as “judgment” or “decision”.
  • judgment and “decision” are considered to be “judgment” and “decision” when the things such as solving, selecting, choosing, establishing, and comparing are regarded as “judgment” and “decision”. Can include. That is, “judgment” and “decision” may include considering some action as “judgment” and “decision”. Further, “judgment (decision)” may be read as “assuming", “expecting”, “considering” and the like.
  • any reference to that element does not generally limit the quantity or order of those elements. These designations can be used herein as a convenient way to distinguish between two or more elements. Thus, references to the first and second elements do not mean that only two elements can be adopted there, or that the first element must somehow precede the second element.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Computational Linguistics (AREA)
  • General Health & Medical Sciences (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Probability & Statistics with Applications (AREA)
  • Machine Translation (AREA)
PCT/JP2021/038829 2020-11-13 2021-10-20 文生成モデル生成装置、文生成モデル及び文生成装置 Ceased WO2022102364A1 (ja)

Priority Applications (2)

Application Number Priority Date Filing Date Title
JP2022561365A JP7805309B2 (ja) 2020-11-13 2021-10-20 文生成モデル生成装置、文生成モデル及び文生成装置
US18/252,140 US12333267B2 (en) 2020-11-13 2021-10-20 Text generation model generating device, text generation model, and text generating device

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2020189333 2020-11-13
JP2020-189333 2020-11-13

Publications (1)

Publication Number Publication Date
WO2022102364A1 true WO2022102364A1 (ja) 2022-05-19

Family

ID=81601010

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2021/038829 Ceased WO2022102364A1 (ja) 2020-11-13 2021-10-20 文生成モデル生成装置、文生成モデル及び文生成装置

Country Status (3)

Country Link
US (1) US12333267B2 (https=)
JP (1) JP7805309B2 (https=)
WO (1) WO2022102364A1 (https=)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2019036093A (ja) * 2017-08-14 2019-03-07 日本電信電話株式会社 モデル学習装置、変換装置、方法、及びプログラム
CN111160049A (zh) * 2019-12-06 2020-05-15 华为技术有限公司 文本翻译方法、装置、机器翻译系统和存储介质
JP2020528625A (ja) * 2017-07-25 2020-09-24 ▲騰▼▲訊▼科技（深▲セン▼）有限公司 翻訳方法、ターゲット情報決定方法および関連装置、記憶媒体

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP6312942B2 (ja) * 2015-10-09 2018-04-18 三菱電機株式会社 言語モデル生成装置、言語モデル生成方法とそのプログラム
US11182435B2 (en) * 2016-11-25 2021-11-23 Nippon Telegraph And Telephone Corporation Model generation device, text search device, model generation method, text search method, data structure, and program
JP7103264B2 (ja) 2019-02-20 2022-07-20 日本電信電話株式会社 生成装置、学習装置、生成方法及びプログラム

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2020528625A (ja) * 2017-07-25 2020-09-24 ▲騰▼▲訊▼科技（深▲セン▼）有限公司 翻訳方法、ターゲット情報決定方法および関連装置、記憶媒体
JP2019036093A (ja) * 2017-08-14 2019-03-07 日本電信電話株式会社 モデル学習装置、変換装置、方法、及びプログラム
CN111160049A (zh) * 2019-12-06 2020-05-15 华为技术有限公司 文本翻译方法、装置、机器翻译系统和存储介质

Also Published As

Publication number Publication date
US20240303445A1 (en) 2024-09-12
US12333267B2 (en) 2025-06-17
JPWO2022102364A1 (https=) 2022-05-19
JP7805309B2 (ja) 2026-01-23

Similar Documents

Publication Publication Date Title
JP7073437B2 (ja) ハイパーネットワークトレーニングの方法及び装置、電子機器、記憶媒体
EP3792789A1 (en) Translation model training method, sentence translation method and apparatus, and storage medium
CN109711548B (zh) 超参数的选取方法、使用方法、装置及电子设备
JP7222082B2 (ja) 認識誤り訂正装置及び訂正モデル
CN110673748B (zh) 输入法中候选长句的提供方法及装置
JP7335300B2 (ja) 知識事前訓練モデルの訓練方法、装置及び電子機器
JP7062056B2 (ja) 作成文章評価装置
JP2020112915A (ja) データ生成装置
WO2021070819A1 (ja) 採点モデル学習装置、採点モデル及び判定装置
WO2021020299A1 (ja) 人気度推定システム及び地理的特徴生成モデル
JP2022029273A (ja) 文類似度算出装置、学習済モデル生成装置及び分散表現モデル
JP7229347B2 (ja) 内部状態変更装置
JP7682862B2 (ja) 句点削除モデル学習装置、句点削除モデル及び判定装置
WO2023079911A1 (ja) 文生成モデル生成装置、文生成モデル及び文生成装置
WO2022102364A1 (ja) 文生成モデル生成装置、文生成モデル及び文生成装置
JP7194759B2 (ja) 翻訳用データ生成システム
JP7691411B2 (ja) 翻訳文章算出装置
JP2020177387A (ja) 文出力装置
JP7575894B2 (ja) 作成文章評価装置
JP2024077792A (ja) 文生成装置
WO2019098185A1 (ja) 発話文生成システム及び発話文生成プログラム
JP2024168531A (ja) 文生成モデル生成装置、文生成モデル及び文生成装置
CN114676319B (zh) 获取商家名称的方法、装置及可读存储介质
JP2022025917A (ja) 対話装置
JP7696296B2 (ja) 翻訳装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21891611

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2022561365

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21891611

Country of ref document: EP

Kind code of ref document: A1

WWG Wipo information: grant in national office

Ref document number: 18252140

Country of ref document: US