JP2023509031A - Translation method, device, device and computer program based on multimodal machine learning - Google Patents

Abstract

It discloses a translation method based on multimodal machine learning and relates to the technical field of artificial intelligence. The method performs semantic associations on n modal source statements to build a semantic association diagram. In the semantic relationship diagram, a first connecting edge is used to connect semantic nodes of the same modal, and a second connecting edge is used to connect semantic nodes of different modals, and the semantic relation diagram connects between multiple modal source statements. Express semantic associations well. Then perform sufficient semantic fusion on the feature vectors in the semantic relationship diagram to obtain a coded feature vector after coding, and then obtain a more accurate target statement after decoding the coded feature vector. . The goal statement more closely approximates the content, emotion, language environment, etc. that the multimodal source statements collectively represent.

Description

TECHNICAL FIELD The present application relates to the technical field of artificial intelligence, and more particularly to a translation method, device, device and storage medium based on multimodal machine learning.

This application takes precedence over a Chinese patent application with application number 2020104325972 filed on May 20, 2020 and titled "Translation method, device, apparatus and storage medium based on multimodal machine learning" All rights reserved, the entire contents of which are hereby incorporated by reference.

Machine translation is the process of converting one type of natural language into another type of natural language using a computer.

In some application scenarios, the machine translation model can translate a plurality of different expressions of the source language into the target language, that is, translate the multimodal source language into the target language. Illustratively, obtain pictures and corresponding English annotations, perform feature extraction on the pictures and English annotations respectively by a machine translation model, then fuse the extracted features, and further based on the fused features: Translate and get French annotations corresponding to pictures and English annotations.

Embodiments of the present application provide a translation method, apparatus, apparatus and storage medium based on multimodal machine learning, which can perform sufficient semantic fusion for multiple modal source languages in the process of feature encoding, It makes the target statement decoded by the encoded vector more closely match the content, emotion, etc. expressed by the source language. The technical means are as follows.

According to one aspect of the present application, there is provided a multimodal machine learning-based translation method performed by a computing device, the method comprising:
constructing a semantic relationship diagram based on n source statements belonging to different modals, wherein the semantic relationship diagram is used to combine n different modal semantic nodes with the same modal semantic nodes; and a second connecting edge used to connect semantic nodes of different modals, wherein the semantic node is used to indicate one semantic unit of the source statement in one kind of modal. , where n is a positive integer greater than 1;
extracting a plurality of first word vectors from the semantic association diagram;
encoding the plurality of first word vectors to obtain n encoded feature vectors;
and decoding the n encoded feature vectors to obtain a translated target statement.

According to another aspect of the present application, there is provided a translation device based on multimodal machine learning, the device comprising:
A semantic association module for building a semantic association diagram based on n source statements belonging to different modals, wherein the semantic association diagram includes n different modal semantic nodes and the same modal semantic nodes. and a second connecting edge used to connect semantic nodes of different modals, wherein the semantic node is one semantic unit of the source statement in one kind of modal a semantic association module, wherein n is a positive integer greater than 1;
a feature extraction module used to extract a plurality of first word vectors from the semantic association diagram;
a vector encoding module used to encode the plurality of first word vectors to obtain n encoded feature vectors;
a vector decoding module used to decode the n encoded feature vectors to obtain a translated target statement.

According to another aspect of the present application, a computer apparatus is provided, the computer apparatus comprising:
memory;
a processor coupled to the memory;
The processor is configured to load and execute executable instructions to implement the multimodal machine learning based translation method described in the above one aspect and alternative embodiments thereof.

According to another aspect of the present application, there is provided a computer-readable storage medium having stored thereon at least one instruction, at least one segment of a program, code set or set of instructions, wherein the at least one instruction, at least A segment of program, code set or instruction set is loaded and executed by a processor to implement the multimodal machine learning based translation method described in the above one aspect and its alternative embodiments.

In order to more clearly describe the technical means in the embodiments of the present application, the following briefly introduces the drawings that need to be used in the description of the embodiments. Obviously, the drawings described below are simply These are just some examples of the present application. Persons skilled in the art can further derive other drawings based on these drawings without creative effort.

FIG. 1 is a structural schematic diagram of a multimodal machine translation model provided by one exemplary embodiment of the present application. FIG. 2 is a structural schematic diagram of a computer system provided by an exemplary embodiment of the present application. FIG. 3 is a flowchart of a multimodal machine learning based translation method provided by one exemplary embodiment of the present application. FIG. 4 is a flowchart for constructing a semantic relationship diagram provided by one exemplary embodiment of the present application. FIG. 5 is a flow chart of a translation method based on multimodal machine learning provided by another exemplary embodiment of the present application. FIG. 6 is a flowchart of a translation method based on multimodal machine learning provided by another exemplary embodiment of the present application. FIG. 7 is a structural schematic diagram of a multimodal machine translation model provided by another exemplary embodiment of the present application. FIG. 8 is a curve diagram of model test results provided by one exemplary embodiment of the present application. FIG. 9 is a curve diagram of model test results provided by another exemplary embodiment of the present application. FIG. 10 is a curve diagram of model test results provided by another exemplary embodiment of the present application. FIG. 11 is a block diagram of a translation device based on multimodal machine learning provided by one exemplary embodiment of the present application. FIG. 12 is a structural schematic diagram of a server provided by an exemplary embodiment of the present application.

In order to make the purpose, technical means and advantages of the present application clearer, the embodiments of the present application are described in more detail below with reference to the drawings.

The nouns involved in this application are interpreted as follows.

Artificial Intelligence (AI): The use of digital computers or machines controlled by digital computers to imitate, augment and augment human intelligence, sense the environment, or acquire and use knowledge It is the technical science of theories, methods, techniques and application systems for obtaining optimum results. In other words, artificial intelligence is a synthetic technology of computer science that is intended to understand the nature of intelligence and produce new intelligent machines that can react in a manner similar to human intelligence. . Artificial intelligence is the study of the design principles and implementation methods of various intelligent devices, enabling the devices to have the functions of sensing, reasoning and decision-making.

Artificial intelligence technology is a comprehensive discipline with a wide range of related fields, including hardware-level technology and software-level technology. The underlying technologies of artificial intelligence generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operating/interactive system, and mechatronics. Artificial intelligence software technology mainly includes several major directions such as computer vision technology, speech processing technology, natural language processing technology and machine learning/deep learning.

Here, Natural Language Processing (NLP) is one important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can realize effective communication between humans and computers through natural language. Natural language processing is a science that brings together linguistics, computer science, and mathematics. Therefore, research in this area concerns natural language, the language that people use on a daily basis. Therefore, it is closely related to the study of linguistics. Natural language processing techniques generally include techniques such as text processing, semantic understanding, machine translation, robotic dialogue, and knowledge graphs.

Machine Learning (ML) is a discipline that intersects multiple disciplines and involves multiple disciplines such as probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. His research focuses on how computers acquire new knowledge or skills by mimicking or implementing human learning behaviors, and reorganizing existing knowledge structures to continually improve their performance. Machine learning is the core of artificial intelligence, the fundamental way to make computers intelligent, and its application spans various fields of artificial intelligence. Machine learning and deep learning generally include techniques such as artificial neural networks, trust networks, reinforcement learning, transfer learning, inductive learning, and analogy learning.

We provide a multimodal machine translation model, which can accurately translate n different modal source statements into target statements. Here, modal refers to a form of language expression, and for example, a statement may use a system such as graph expression or character expression. A source statement refers to a statement to be translated, which includes a sentence to be translated in the first language class in text form and a language to be translated in non-text form. A target statement refers to a translated sentence in a second language class in text form, where the second language class is different from the first language class. Exemplarily, the source statement includes an English statement and an illustration of the English statement, and the Chinese statement corresponding to the English statement and the illustration can be obtained by translation through a multimodal machine translation model.

As shown in FIG. 1, it shows a structural schematic diagram of a multimodal machine translation model 100 provided by one exemplary embodiment of the present application. The multimodal machine translation model 100 includes a multimodal graph representation layer 101, a first word vector layer 102, a multimodal fusion encoder 103 and a decoder 104,
The multimodal graph representation layer 101 is used to perform semantic association for n modal source languages to obtain a semantic association diagram. The semantic relationship diagram includes n kinds of different modal semantic nodes, a first connecting edge used to connect the same modal semantic nodes, and a second connecting edge used to connect different modal semantic nodes. and n is a positive integer greater than 1. Here, one semantic node is used to indicate one semantic unit of source statements in one kind of modal. Taking English as an example, one semantic node corresponds to one word, and taking Chinese as an example, one semantic node corresponds to one Chinese character.

the first word vector layer 102 is used to extract a plurality of first word vectors from the semantic association diagram;
a multimodal fusion encoder 103 is used to encode the plurality of first word vectors to obtain n encoded feature vectors;
A decoder 104 is used to decode the n encoded feature vectors to obtain a post-translational target statement.

In some optional embodiments, the multimodal graph representation layer 101 obtains n sets of semantic nodes, one set of semantic nodes corresponding to one modal source statement; By adding the first connecting edge between any two of the semantic nodes of the same modal and adding the second connecting edge between any two of the semantic nodes of the different modals, the semantic relationship diagram is: used for obtaining and

In some optional embodiments, the multimodal graph representation layer 101 extracts the semantic nodes from each modal's source language to obtain n sets of semantic nodes corresponding to the n modal source languages. used for
The multimodal graph representation layer 101 connects between semantic nodes in the same modal for n sets of semantic nodes using a first connecting edge, and connects n sets of semantic nodes using a second connecting edge. It is used to make connections between semantic nodes between different modals to obtain a semantic relationship diagram.

In some alternative embodiments, the n modal source statements include a first textual source statement and a second non-textual source statement, and the n sets of semantic nodes are the first semantic node and the second non-textual source statement. including a second semantic node;
The multimodal graph representation layer 101 obtains the first semantic node, which is obtained by processing the first source statement, and obtains candidate semantic nodes. wherein the candidate semantic nodes are obtained by processing the second source statement; and obtaining a first probability distribution of the candidate semantic nodes, wherein the first probability the distribution is obtained by computing according to the semantic association between the first semantic node and the candidate semantic nodes; and determining the second semantic node from the candidate semantic nodes. , the second semantic node is determined by the multimodal graph representation layer based on the first probability distribution.

In some alternative embodiments, multimodal graph representation layer 101 extracts a first semantic node from a first source statement and candidate semantic nodes from a second source statement; calculating a first probability distribution of the candidate semantic nodes according to the semantic association between and the candidate semantic nodes; and determining a second semantic node from the candidate semantic nodes based on the first probability distribution. .

In some optional embodiments, the multimodal graph representation layer 101 adds the i-th kind of first connecting edge between any two semantic nodes in the same modal in the i-th set of semantic nodes. , the i-th kind of first connecting edge corresponds to the i-th modal, and i is a positive integer less than or equal to n.

That is, the multimodal graph representation layer 101 determines the i-th kind of first connecting edge corresponding to the i-th modal, and uses the i-th kind of first connecting edge to the i-th set of semantic nodes. Used to connect semantic nodes within the same modal, i is a positive integer less than or equal to n.

In some optional embodiments, n encoded feature vectors are subjected to intra-modal fusion and inter-modal fusion e times on the plurality of first word vectors to obtain the encoded feature vector. obtained by the process of Here, the intra-modal fusion refers to semantic fusion between the first word vectors in the same modal, and the inter-modal fusion is the semantic fusion between the first word vectors of different modals. point to where e is a positive integer.

In some alternative embodiments, the multimodal fusion encoder 103 includes e encoding modules 1031 connected in series, each encoding module 1031 corresponding one-to-one to n modals. n intra-modal fusion layers 11 and n inter-modal fusion layers 12, e being a positive integer,
The first encoding module 1031 inputs the first word vector to each of the n intra-modal fusion layers 11 in the first encoding module, and the n intra-modal fusion layers 11 respectively convert the first word vector into is used to perform semantic fusion inside the same modal to obtain n first hidden layer vectors, one said first hidden layer vector corresponds to one modal, that is, for n modals Obtain n first hidden layer vectors corresponding one-to-one,
The first encoding module 1031 inputs n first hidden layer vectors to each inter-modal fusion layer 12 in the first encoding module, and each inter-modal fusion layer 12 converts the n first hidden layer vectors. The hidden layer vector is used to perform semantic fusion between different modals to obtain n first intermediate vectors, one intermediate vector corresponds to one modal, that is, n modals Obtain n first intermediate vectors corresponding one-to-one to
The j-th encoding module 1031 performs the j-th encoding process on the n first intermediate vectors, and continues until the last encoding module outputs n encoded feature vectors. , one coded feature vector corresponds to one modal, that is, the last one coding module outputs n coded feature vectors corresponding one-to-one to n modals and j is a positive integer greater than 1 and less than or equal to e.

In some alternative embodiments, each encoding module 1031 further includes n first vector transformation layers 13, said one vector transformation layer corresponding to one modal, i.e. n modal are n first vector transformation layers 13 corresponding one-to-one to
The encoding module 1031 further inputs the n first intermediate vectors to the n first vector transformation layers 13 corresponding to the respective modals to which they belong, and non-linearly transforms them into n first intermediate vectors after the non-linear transformation. Used to get intermediate vectors.

In some alternative embodiments, the hierarchical structure in each encoding module 1031 of the e serially connected encoding modules 1031 is the same.

In some alternative embodiments, different intra-modal fusion layers are assigned different or the same self-attention functions, and different inter-modal fusion layers are assigned different or the same feature fusion functions.

In some alternative embodiments, the multimodal machine translation model 100 further comprises a second word vector layer 105 and a classifier 106, and the decoder 104 comprises d serially connected decoding modules 1042, d is a positive integer;
The second word vector layer 105 is to obtain a first target phrase, where the first target phrase is a translated phrase in the target statement, and perform feature extraction on the first target phrase. to obtain a second word vector; and
The decoder 104 is used for extracting features by combining the second word vector and the encoded feature vector by d decoding modules 1042 connected in series to obtain a decoded feature vector,
Classifier 106 is used to determine a probability distribution corresponding to the decoded feature vector and to determine a second target phrase after the first target phrase based on the probability distribution.

In some alternative embodiments, each decoding module 1042 of the d serially connected decoding modules 1042 both includes a first self-attention layer 21 and a second self-attention layer 22;
The first decoding module 1042 inputs the second word vector to the first self-attention layer 21 in the first decoding module 1042, performs feature extraction on the second word vector by the first self-attention layer 21, used to obtain the second hidden layer vector,
The first decoding module 1042 inputs the second hidden layer vector and the encoded feature vector to the second self-attention layer 22 in the first decoding module 1042, and the second self-attention layer 22 decodes the second hidden layer vector and the encoded feature vector. is used to obtain a second intermediate vector by performing feature extraction in combination with the modified feature vector,
The k-th decoding module inputs the second intermediate vector to the k-th decoding module 1042 to perform the k-th decoding process, and continues until the last decoding module outputs a decoded feature vector. used, k is a positive integer greater than 1 and less than or equal to d.

In some optional embodiments, each decoding module 1042 further includes a second vector transformation layer 23,
The decoding module 1042 is used to input the second intermediate vector to the second vector transformation layer 23 and perform nonlinear transformation to obtain the second intermediate vector after nonlinear transformation.

As described above, the multimodal machine translation model provided by this embodiment performs semantic association for n modal source languages through the multimodal graph representation layer to obtain a semantic association diagram. In the semantic relationship diagram, a first connecting edge is used to connect semantic nodes of the same modal, and a second connecting edge is used to connect semantic nodes of different modals, and the semantic relation diagram is used to connect multiple modal source languages. Express semantic associations well. Then a multimodal fusion encoder performs sufficient semantic fusion on the feature vectors in the semantic association diagram to obtain a coded feature vector after coding. After decoding the encoded feature vector, a more accurate target statement is obtained. The goal statement more closely approximates the content, emotion and language environment, etc. that the multimodal source statements collectively represent.

Referring to FIG. 2 , it shows a structural schematic diagram of a computer system provided by one exemplary embodiment of the present application, which includes a terminal 220 and a server 240 .

An operating system is installed in the terminal 220, and an application program is installed in the operating system, and the application program supports a multimodal source language translation function. Illustratively, the application programs may include instant messaging software, financial software, game software, shopping software, video playback software, community service software, audio software, education software, payment software and translation software, etc. is integrated with the above multimodal source language translation function.

Terminal 220 and server 240 are coupled to each other via a wired or wireless network. Server 240 includes at least one of a server, multiple servers, a cloud computing platform, and a virtualization center. Illustratively, server 240 includes a processor and memory. Here, a computer program is stored in the memory, and the processor can read and execute the computer program to realize a multimodal source language translation function.

Alternatively, the server 240 is responsible for the main computational tasks and the terminal 220 is responsible for the secondary computational tasks. Alternatively, the server 240 is responsible for the secondary computational work and the terminal 220 is responsible for the main computational work. Alternatively, both the server 240 and the terminal 220 perform cooperative computation using a distributed computing architecture.

In some alternative embodiments, server 240 provides background services to application programs on terminal 220 in the process of implementing the multimodal language translation function. Illustratively, the terminal 220 collects n modal source statements, transmits the n modal source statements to the server 240, and the server 240 executes the translation method based on multimodal machine learning provided by the present application. and n is a positive integer greater than one.

Illustratively, the terminal 220 includes a data transmission control member, and the terminal 220 transmits the two different modal source statements of the translatable statement and the image matching the translatable statement to the server 240 by the data transmission control member. Upload. The server 240 executes the translation method based on multimodal machine learning provided by the present application to translate the two modal source statements into target statements.

In some alternative embodiments, the source statements may include audio signals. If the n modal source statements include audio signals, the terminal 220 or server 240 first converts the audio signals into text before translating the n modal source statements. Illustratively, terminal 220 collects audio signals via a microphone, or terminal 220 receives audio signals transmitted from other terminals.

The above multimodal machine learning based translation method can be applied to the multimedia news translation scene. Illustratively, the terminal 220 uploads multimedia news including text and images to the server 240, and the server 240 executes the multimodal machine learning-based translation method provided by the present application to translate the first language in the multimedia news. Translate a class of characters into a second language class of characters.

The above multimodal machine learning based translation method can be applied to the foreign language document translation scene. Exemplarily, the terminal 220 uploads characters and illustrations corresponding to the characters in the foreign language document to the server 240, and the server 240 executes the translation method based on multimodal machine learning provided by the present application, Translating characters of one language class into characters of a second language class.

The above multimodal machine learning based translation method can be applied to the foreign language website translation scene. Illustratively, the terminal 220 collects characters and character illustrations on foreign language websites, uploads the characters and character illustrations to the server 240, and the server 240 executes the translation method based on multimodal machine learning provided by the present application. and translate the characters of the first language class on the foreign language website into the characters of the second language class, and further realize the translation for the foreign language website.

In some alternative embodiments, the manner in which terminal 220 presents the translated text includes phonetic or textual formats.

As should be explained, in some alternative embodiments, the terminal 220 executes the multimodal machine learning-based translation method provided by the present application, and also translates the n modal source statements.

Terminal 220 may generally refer to one of a plurality of terminals, and this embodiment describes only terminal 220 as an example. The terminal 220 can be a smart phone, tablet computer, e-book reader, MPEG Audio Layer 3 (Moving Picture Experts Group Audio Layer III, MP3) player, MPEG Audio Layer 4 (Moving Picture Experts Group Audio Layer IV, MP4) player, laptop pocket. It may include at least one of a computer, a desktop computer, and a laptop. In the following embodiments, terminals 220 include smart phones and personal computer devices as examples.

As will be appreciated by those skilled in the art, the number of terminals 220 may be greater or less. For example, there may be only one terminal, or there may be tens or hundreds of terminals, or more. Embodiments of the present application do not limit the number and device types of terminals 220 .

Referring to FIG. 3, it shows a flowchart of a multimodal machine learning based translation method provided by one exemplary embodiment of the present application. The method is applied to the computer equipment shown in FIG. 2, the computer equipment includes a terminal or server, the method includes: a.

Step 301: The computing device makes semantic associations for the n modal source statements to build a semantic association diagram.

The above semantic relationship diagram includes n types of different modal semantic nodes, a first connecting edge used to connect the same modal semantic nodes, and a second connecting edge used to connect different modal semantic nodes. and n is a positive integer greater than 1.

Taking a modal source statement as an example, the source statement corresponds to a set of semantic nodes, and the set of semantic nodes includes at least one semantic node used to indicate a semantic unit in the source statement.

The computing device is configured with a multimodal fusion encoder and decoder, and the computing device extracts semantic nodes from each modal source statement by a multimodal graph representation layer to generate n-tuples corresponding to the n modal source statements. A semantic node is obtained, and a connection between semantic nodes in the same modal is performed for n sets of semantic nodes using the first connecting edge by the multimodal graph representation layer. That is, add a first connecting edge between any two semantic nodes of the same modal, and use the second connecting edge to connect between semantic nodes between different modals for n sets of semantic nodes. . That is, a second connecting edge is added between different modal semantic nodes to obtain a semantic association diagram.

Optionally, the n modal source statements include a textual first source statement and a non-textual second source statement. The n-tuple of semantic nodes includes a first semantic node and a second semantic node. A computing device extracts a first semantic node from a first source statement and a candidate semantic node from a second source statement with a multimodal graph representation layer, invokes the multimodal graph representation layer, and extracts the first semantic node and the candidate Compute a first probability distribution of the candidate semantic nodes according to semantic associations with the semantic nodes, and invoke a multimodal graph representation layer to determine a second semantic node from the candidate semantic nodes based on the first probability distribution.

Here, regarding the extraction of semantic nodes in the first source statement in text format, the computer device performs word segmentation processing on the first source statement, obtains m terms after word segmentation, and obtains m words and phrases after word segmentation. the phrase corresponds to the first semantic node in the first source statement, m is a positive integer,
For extracting semantic nodes in the non-textual second source statement, the computer device extracts a target semantically corresponding to at least one of the m terms from the second source statement, the target being the second source statement. It is the second semantic node in the source statement.

Illustratively, as shown in FIG. 4, the two modal source statements include a translatable image 31 and a translatable statement 32, and the content of the translatable statement 32 includes "Two boys are playing with a toy car." . Each English word corresponds to one first semantic node, Vx1, Vx2, Vx3, Vx4, Vx5, Vx6, Vx7 and Vx8 respectively. The computer device cuts a candidate image from the translation target image 31 based on the semantic of the semantic node, calculates a first probability distribution based on the semantic association between the semantic node and the candidate image, and calculates Vx1 and Vx1 from the candidate image based on the first probability distribution. Determine target image 1 and target image 2 corresponding to the semantics of Vx2, and target image 3 corresponding to the semantics of Vx6, Vx7 and Vx8. Vo1, Vo2 and Vo3 corresponding to target image 1, target image 2 and target image 3 respectively are three second semantic nodes in the image 31 to be translated. The computing device performs intra-modal semantic coupling using the first coupling edge (solid line) between every two of Vx1, Vx2, Vx3, Vx4, Vx5, Vx6, Vx7 and Vx8, and two of Vo1, Vo2 and Vo3. Intra-modal semantic coupling is performed using the first connecting edge between each node, and inter-modal semantic coupling is performed using the second connecting edge (dashed line) between the first semantic node and the second semantic node.

As an option, different modals are associated with different first connecting edges. When the computer device performs intra-modal connection to the semantic node, the multimodal graph representation layer determines the i-th type of first connection edge corresponding to the i-th modal, and determines the i-th type of first connection edge. is used to perform a connection between semantic nodes within the same modal for the i-th set of semantic nodes. That is, add the i-th kind of first connecting edge between any two semantic nodes in the i-th set of semantic nodes, where i is a positive integer less than or equal to n.

Alternatively, in translating two modal source statements, if the two modal source statements are text and an image, respectively, the computer device uses a visual grounding tool to highlight the text between the two modal source statements. Establish semantic associations and build semantic association diagrams.

Step 302: The computing device extracts a plurality of first word vectors from the semantic association diagram.

Illustratively, the computing device processes the semantic relationship diagram using a word embedding scheme to obtain a plurality of first word vectors. Word embedding refers to mapping words to word vectors, and alternative word embedding methods include:
performing word embedding with a neural network model;
performing word embedding by performing dimensionality reduction on the term co-occurrence matrix;
It includes at least one of four types: word embedding according to a probabilistic model; and word embedding for words according to the semantics of the context in which the word is located.

For example, One-Hot Encoding is used to represent words in a textual source statement, followed by word embeddings using an embedding matrix.

Step 303: The computing device encodes the plurality of first word vectors to obtain n encoded feature vectors.

The computing device performs intramodal feature extraction on the first word vector with a multimodal fusion encoder, and then performs intermodal feature fusion on the vector obtained by feature extraction.

As an illustrative example, the value of n is 3. The multimodal fusion encoder includes a first feature extraction function corresponding to the first modal, a second feature extraction function corresponding to the second modal, and a third feature extraction function corresponding to the third modal. The computing device performs feature extraction within a first modal on the first word vector with a first feature extraction function, performs feature extraction within a second modal on the first word vector with a second feature extraction function, and performs feature extraction on the first modal with a second feature extraction function. Perform feature extraction in a third modal on the first word vector by three feature extraction functions to finally obtain three hidden layer functions. The multimodal fusion encoder further includes a first feature fusion function corresponding to the first modal, a second feature fusion function corresponding to the second modal, and a third feature fusion function corresponding to the third modal. The computing device performs inter-modal feature fusion on the three hidden layer functions by a first feature fusion function, performs inter-modal feature fusion on the three hidden layer functions by a second feature fusion function, and Inter-modal feature fusion is performed on the above three hidden layer functions by a three-feature fusion function to obtain three hidden layer vectors after feature fusion, that is, encoded feature vectors.

Step 304: The computing device decodes the n encoded feature vectors to obtain a translated target statement.

The computer device calls a decoder to decode the n encoded feature vectors to obtain a post-translation target statement. The target statement is a statement obtained by translating n modal source statements into a designated language class.

As described above, the translation method based on multimodal machine learning provided by the present embodiment performs semantic association for n modal source statements by the multimodal graph representation layer, constructs a semantic association diagram, In the semantic relationship diagram, the first connecting edge is used to connect the semantic nodes of the same modal and the second connecting edge is used to connect the semantic nodes of the different modals, and the semantic relation diagram connects the source statements of the multiple modals. Express semantic associations well. Then a multimodal fusion encoder performs sufficient semantic fusion on the feature vectors in the semantic association diagram to obtain a coded feature vector after coding, and a more accurate target after decoding the coded feature vector. Get statement. The goal statement more closely approximates the content, emotion and language environment, etc. that the multimodal source statements collectively represent.

Based on FIG. 3, the multimodal fusion encoder includes e encoding modules connected in series, each encoding module having n intra-modal fusions corresponding one-to-one to the n modals. layer and n inter-modal fusion layers, where e is a positive integer. Accordingly, step 303 may include step 3031, and as in FIG. 5, the steps are as follows.
Step 3031: The computing device performs e intra-modal fusion and inter-modal fusion on the plurality of first word vectors by e encoding modules connected in series to obtain n encoded feature vectors. .

Here, intra-modal fusion means performing semantic fusion between first word vectors in the same modal, and inter-modal fusion means performing semantic fusion between first word vectors of different modals. .

Illustratively, intramodal and intermodal fusion of the above encoded feature vectors can be achieved by the following steps.

1) input the first word vector into n intra-modal fusion layers in the first encoding module respectively, and perform the same intra-modal semantic fusion on the first word vector by the n intra-modal fusion layers respectively; to obtain n first hidden layer vectors. One above-mentioned first hidden layer vector corresponds to one modal, that is, obtain n first hidden layer vectors corresponding to n modals one-to-one.

Illustratively, the computing device inputs a first word vector to a first intra-modal fusion layer in a first encoding module, and the first intra-modal fusion layer applies an intra-modal semantic to the first word vector. Fusion is performed to obtain the first hidden layer vector, the first word vector is input to the second intra-modal fusion layer in the first encoding module, and the second intra-modal fusion layer synthesizes the first word vector Perform intramodal semantic fusion on the vector to obtain the second first hidden layer vector, . . . , input the first word vector into the nth intramodal fusion layer in the first encoding module, n Perform intra-modal semantic fusion on the first word vector by the th intra-modal fusion layer to obtain the n-th first hidden layer vector.

A feature extraction function is set within the intra-modal fusion layer, and optionally the feature extraction function includes a self-attention function. Optionally, different or the same self-attention functions are set in different intra-modal fusion layers. As should be explained, different self-attention functions refer to different parameters in the function, and different self-attention functions corresponding to different modals have different parameters in the function corresponding to different modals.

2) input the n first hidden layer vectors into each inter-modal fusion layer in the first encoding module, and each inter-modal fusion layer performs different inter-modal fusions for the n first hidden layer vectors; Perform semantic fusion to obtain n first intermediate vectors. One intermediate vector corresponds to one modal, that is, obtain n first intermediate vectors corresponding to n modals one-to-one.

Illustratively, the computer device inputs n first hidden layer vectors into a first inter-modal fusion layer in a first encoding module, and generates n first hidden layers by the first inter-modal fusion layer. Perform cross-modal semantic fusion on the layer vectors to obtain the first intermediate vector corresponding to the first modal, and convert the n first hidden layer vectors to the second input to the inter-modal fusion layer, and perform inter-modal semantic fusion on the n first hidden layer vectors by the second inter-modal fusion layer to obtain the second first intermediate vector corresponding to the second modal , input the n first hidden layer vectors into the nth intermodal fusion layer in the first encoding module, and for the n first hidden layer vectors by the nth intermodal fusion layer to perform semantic fusion between modals to obtain the nth first intermediate vector corresponding to the nth modal.

Feature fusion functions are set in the inter-modal fusion layers, and optionally the feature fusion functions set in different inter-modal fusion layers are different or the same. As should be explained, different feature fusion functions refer to different parameters in the functions, or different calculation methods of the functions.

Optionally, each encoding module further comprises n first vector transformation layers corresponding one-to-one to the n modals. After obtaining the n first intermediate vectors, the computer device further inputs the n first intermediate vectors to the n first vector transformation layers corresponding to the respective modals to perform nonlinear transformation, Obtain n first intermediate vectors after transformation.

3) The n first intermediate vectors are input to the j-th encoding module to perform the j-th encoding process, and the last encoding module outputs n encoded feature vectors. continue until One coded feature vector corresponds to one modal, ie until the last one coding module outputs n coded feature vectors corresponding one-to-one to n modals.

The computer device inputs the n intermediate vectors to the second encoding module to perform the second encoding process, obtains the re-encoded n first intermediate vectors, . . . input the encoded n first intermediate vectors to the j-th encoding module and perform the j-th encoding process to obtain the n encoded first intermediate vectors, . . . The re-encoded n first intermediate vectors are input to the e-th encoding module and subjected to the e-th encoding process to obtain n encoded feature vectors. Here, j is a positive integer greater than 1 and less than or equal to e. Optionally, the hierarchical structure in each of the e encoding modules connected in series is the same. That is, the jth encoding module follows the steps of the first encoding module encoding the first intermediate vector, and so on until the last one encoding module outputs an encoded feature vector.

Illustratively, the self-attention mechanism is used in this embodiment to model semantic information within the same modal. Then the j-th encoding module computes the first hidden layer vector [equation 1] corresponding to the text statement, and the formula is
[Equation 2],
Here, [Formula 3] refers to the first word vector corresponding to the text statement or the first intermediate vector output by the (j−1)-th encoding module, x is the semantic node of the text statement, and MultiHead(Q, K, V) is a multi-attention mechanism modeling function, with triplet (Queries, Key, Values) as input, Q is query , K is the key matrix, and V is the value matrix, where Q, K and V are computed and obtained from [Equation 4] and the parameter vector.

The j-th multimodal fusion encoder computes the first hidden layer vector [equation 5] corresponding to the image, and the equation is
[Equation 6],

Here, [Formula 7] refers to the first word vector corresponding to the image or the first intermediate vector output by the (j−1)-th encoding module,

In this embodiment, we further use a cross-modal fusion mechanism based on a gating mechanism to model the semantic fusion between multimodals, so that the j-th coding module generates the first intermediate vector or coding feature corresponding to the text statement Compute the vector [Equation 8], the formula is
[Number 9],
[Equation 10],

where A denotes a set. Correspondingly, [Equation 11] is the set of neighboring nodes in the semantic association diagram of the first semantic node [Equation 12]. [13] denotes the u-th semantic node of the text statement, where u is a positive integer. [Equation 14] is the semantic representation vector of the sth semantic node of the image in the jth encoding module, [Equation 15] is the semantic representation vector of the uth semantic node of the text statement in the jth encoding module is a representation vector. [Equation 16] and [Equation 17] are parameter matrices, [Equation 18] indicates a negative exclusive OR operation, and Sigmoid( ) is an s-curve function. o is used to mark the semantic nodes of the image and the vectors computed by the semantic nodes of the image. Further, the same calculation method is used to calculate the first intermediate vector or encoded feature vector [Eq.

After going through multimodal interfusion, this embodiment further uses a FeedForward Neural Network (FFN) to generate the final encoded feature vector, which corresponds to the encoded feature vector corresponding to the text statement and the image. Each encoded feature vector for
[number 20],
[Equation 21],

where { } indicates a set, [23] indicates the encoding feature vector corresponding to the u-th semantic node of the text statement in the j-th encoding module, and [number 24] shows the encoded feature vector corresponding to the sth semantic node of the image in the jth encoding module.

As described above, the multimodal machine learning-based translation method provided by the present embodiment performs semantic association for n modal source languages through the multimodal graph representation layer to construct a semantic association diagram. In the semantic relationship diagram, a first connecting edge is used to connect semantic nodes of the same modal, and a second connecting edge is used to connect semantic nodes of different modals, and the semantic relation diagram is used to connect multiple modal source languages. Express semantic associations well. Then, a multimodal fusion encoder performs sufficient semantic fusion on the feature vectors in the semantic relationship diagram to obtain a coded feature vector after coding, and after decoding the coded feature vector, a more accurate Get a goal statement. The goal statement more closely approximates the content, emotion and linguistic environment, etc. that the multimodal source language collectively represents.

In the method, the multimodal fusion encoder includes e encoding modules connected in series. Each encoding module includes an intra-modal fusion layer and an inter-modal fusion layer, wherein multiple alternating intra-modal and inter-modal feature fusions are performed to obtain a more complete encoding feature vector with semantic fusion; In addition, more accurate target statements corresponding to n modal source languages can be decoded.

Based on FIG. 3, the decoder further includes d serially connected decoding modules, where d is a positive integer. Accordingly, step 304 may include steps 3041-3044, which, as shown in FIG. 6, are as follows.

Step 3041: The computing device obtains the first target phrase through the second word vector layer.

Here, the first target phrase is the translated phrase in the target statement. The computer device translates the words in the target statement one by one, and after translating the rth word in the target statement, the rth word is taken as the first target word and used to translate the r+1th word. In other words, the computer device inputs the rth phrase into the second word vector layer, where r is a non-negative integer.

Step 3042: The computing device performs feature extraction on the first target phrase through the second word vector layer to obtain a second word vector.

Illustratively, the computing device performs word embedding on the first target phrase with a second vector layer to obtain a second word vector. Word embedding is a technique of representing words as real vectors in a vector space, and in this embodiment, word embedding refers to mapping words to word vectors. For example, if you map "watashi" to get the word vector (0.1,0.5,5), i.e. (0.1,0.5,5) performs word embedding on "watashi". is the word vector after

Step 3043: The computing device performs feature extraction by combining the second word vector and the encoded feature vector with d serially connected decoding modules to obtain a decoded feature vector.

The computing device calls d serially connected decoding modules to process the encoded feature vector and the second word vector according to the attention mechanism to extract the decoded feature vector.

Optionally, each decoding module of the d series-connected decoding modules includes one first self-attention layer, one second self-attention layer and one second vector transformation layer. For decoding feature vector extraction, the computing device inputs the second word vector into the first self-attention layer in the first decoding module, performs feature extraction on the second word vector by the first self-attention layer, Obtaining a second hidden layer vector, inputting the second hidden layer vector and the encoded feature vector into the second self-attention layer in the first decoding module, and obtaining the second hidden layer vector and the encoded feature vector by the second self-attention layer and the vector to perform feature extraction to obtain a second intermediate vector; outputs a decoded feature vector, where k is a positive integer greater than 1 and less than or equal to d.

wherein the first self-attention layer is used to process the second word vector based on the self-attention mechanism to extract the second hidden layer vector, and the second self-attention layer is used to extract the goal statement based on the attention mechanism. is used to process the second hidden layer vector and the encoded feature vector to obtain a second intermediate vector. The first self-attention layer includes a first self-attention function, the second self-attention layer includes a second self-attention function, and the parameters of the first self-attention function and the second self-attention function are different.

Optionally, each decoding module further includes a second vector transformation layer, and after calculating and obtaining the second intermediate vector, the computer device further inputs the second intermediate vector to the second vector transformation layer to perform the non-linear transformation. to obtain a second intermediate vector after nonlinear transformation.

Step 3044: The computer device inputs the decoded feature vector into the classifier, calculates the probability distribution corresponding to the decoded feature vector by the classifier, and determines the second target phrase after the first target phrase based on the probability distribution. .

Optionally, the classifier includes a normalization (softmax) function, the computing device calculates a probability distribution corresponding to the decoded feature vector by the softmax function, and determines the first target phrase based on the probability distribution corresponding to the decoded feature vector. Determine a later second target phrase.

As described above, the multimodal machine learning-based translation method provided by the present embodiment performs semantic association for n modal source languages through the multimodal graph representation layer to construct a semantic association diagram. In the semantic relationship diagram, a first connecting edge is used to connect semantic nodes of the same modal, and a second connecting edge is used to connect semantic nodes of different modals, and the semantic relation diagram is used to connect multiple modal source languages. Express semantic associations well. Then a multimodal fusion encoder performs sufficient semantic fusion on the feature vectors in the semantic association diagram to obtain a coded feature vector after coding, and a more accurate target after decoding the coded feature vector. Get statement. The goal statement more closely approximates the content, emotion and linguistic environment, etc. that the multimodal source language collectively represents.

The method further iteratively pays attention to the encoded feature vector and the second hidden layer vector using the linguistic class of the target statement by d decoding modules to decode a more accurate target statement.

As should be further explained, in test comparisons against the multimodal machine translation model provided by the present application and previous multimodal neural machine translation (NMT), the multimodal machine translation provided by the present application was found to be It became clear that the translation effect of the machine translation model was the highest. By way of example, taking the input data as an example of two kinds of source languages, image and text, the above test comparison is detailed as follows.

The multimodal machine translation model provided by the present application is built on the codec framework of attention, with the goal function of maximizing the log-likelihood of the training data. In essence, the multimodal fusion encoder provided by the present application may be viewed as one multimodal augmented graph neural network (GNN). To construct a multimodal fusion encoder, the input image and text are represented as a correspondence as a multimodal graph (i.e., semantic association diagram), and then multiple multimodal fusion layers based on the multimodal graph. to learn node (ie, semantic node) representations to provide decoders with attention-based context vectors.

First, regarding the construction of the multimodal graph, formally the multimodal graph is undirected and can be formalized as G=(V,E). Here, in the node set V, each node represents a textual phrase or visual object. Here nodes corresponding to text are called semantic nodes, nodes corresponding to visual objects are called visual nodes, and the following policies are used to build semantic associations between nodes.

1. Node Extraction (1) To make full use of the text information, make every word in the text a separate text node. For example, in FIG. 4 the multimodal graph contains a total of 8 text nodes, each corresponding to one word in the input statement (ie, the translatable statement). (2) using a Stanford parser to identify all noun phrases in the input statement, then applying a visual grounding toolkit to match individual noun phrases in the input image (i.e., the image to be translated); Identifies the bounding box (visual object) that All detected visual objects are then made into independent visual nodes. For example, in FIG. 4 text nodes Vx1 and Vx2 correspond to visual nodes Vo1 and Vo2, and text nodes Vx6, Vx7 and Vx8 correspond to visual node Vo3.

2. To capture various semantic associations between multimodal semantic units, two types of edges (ie, connecting edges) are used to connect semantic nodes. The two types of edges in the edge set E are: (1) any two semantic nodes in the same modal are connected by one intra-modal edge (first connecting edge); (2) any text node and All corresponding visual nodes are connected by one inter-modal edge (second connecting edge). Illustratively, Vo1 and Vo2 are connected using an intra-modal edge (solid line), and Vo1 and Vx1 are connected using an inter-modal edge (solid line), as shown in FIG.

Second, for the embedding layer, it is necessary to introduce a word embedding layer to initialize the state of the node before inputting the multimodal fusion layer stacked with multimodal graphs. For each text node Vxu, define its initial state Hxu as the sum of the word embedding and the position embedding. For the initial state Hos of the visual node Vos, the visual features are extracted by a fully-connected layer of the Region Of Interest pooling (ROI pool) layer in Faster-RCNN, and then a linear rectification function ( A multi-layer perceptron with a Rectified Linear Unit (ReLU) as the activation function should be used to project the visual features into the same space as the textual representation.

Here, RCCN is a Rich feature hierarchy for accurate object detection and semantic segmentation.

3. As in FIG. 7, the left part shows the encoder, and on top of the embedding layer 402 is stacked a multimodal fusion layer based on the e-layer graph, thereby encoding the above multimodal graph. In the multimodal fusion layer, intramodal and intermodal fusions are performed sequentially to update all node states. Thus, the final node state is a simultaneous encoding of co-modal contextual and cross-modal semantic information. In particular, since visual nodes and text nodes are two types of semantic units that contain different modal information, we use functions with similar operations but different parameters to model the node state update process.

Illustratively, in j multimodal fusion layers, updating text node states [eq.25] and visual node states [eq.26] mainly involves the following steps.

Step 1: Intra-modal fusion. In this step, self-attention is used to perform information fusion between adjacent nodes within the same modal to generate contextual representations of individual nodes. Formally, the formula for computing the context representation [eq.27] of all text nodes is
[Equation 28],

where MultiHead(Q, K, V) is the multi-attention mechanism modeling function (also called multi-head self-attention function), with query matrix Q, key matrix K and value matrix V as inputs. Similarly, the formula for the context representation [Equation 29] of all visual nodes is
[Equation 30].

In particular, the initial states of visual objects are extracted by deep learning algorithms (deep CNNs), and thus one simplified multi-headed self-attention is applied to represent the initial states of visual objects. Here we delete the obtained linear term values and the final output.

Step 2: Inter-modal fusion. A kind of cross-modal gating control mechanism with element manipulation properties is used to learn the semantic information of the cross-modal neighborhood of individual nodes when performing feature fusion during multimodal. Specifically, the method for generating the state representation [equation 31] of the text node Vxu is
[number 32],
[Equation 33],

Here, [Formula 34] is a set of neighboring nodes in the multimodal graph of node Vxu, and [Formula 35] and [Formula 36] are parameter matrices. Similarly, the scheme for generating the state representation of the text node Vos [Equation 37] is
[Number 38],
[Equation 39],

Here, [Formula 40] is a set of neighboring nodes in the multimodal graph of node Vos, and [Formula 41] and [Formula 42] are parameter matrices.

After going through the above multimodal fusion process, a feedforward neural network is used to generate the final Dell hidden layer representation. The computation process for text node state [eq.43] and image node state [eq.44] is:
[Number 45],
[Equation 46],

Here, [Equation 47] indicates that all text node states and image node states have been updated.

Fourth, the decoder is similar to a conventional transformer decoder. Since visual information is already fused to all text nodes by multiple graph-based multimodal fusion layers, it is permissible for decoders to focus only on text node states and exploit multimodal context dynamically. , i.e., input only the text node state to the decoder.

As shown in the right part of FIG. 7, d identical layers are superimposed to generate the target hidden state. Here, each layer is composed of three sub-layers. Specifically, the top two sub-layers are masking self-attentive Ej and codec-attentive Tj, respectively, thereby integrating the target and source language side contexts,
[Number 48],
[Equation 49],

Here, S(j-1) denotes the target-side hidden state in the j-1th layer. In particular, S(0) is the embedding vector of the input target phrase, and [Equation 50] is the top layer hidden state in the decoder. Then, a fully connected feedforward neural network in one position direction is used to generate S(j), and the equation is
[Equation 51],

Finally, a softmax layer is used to define the probability distribution that generates the target statement, which takes as input the hidden state of the top layer [Equation 52], and
[Equation 53],

where X is the input translatable statement, I is the input translatable image, Y is the target statement (ie, translation statement), and W and b are parameters of the softmax layer.

In the experimental process, the translation task is to translate English into French and German, and the Multi30K dataset is used as the dataset. Here, each image in the dataset is paired with corresponding English descriptions and human-translated German and French. The training, validation and test sets contain 29000, 1014 and 1000 examples respectively. Besides this, we also evaluate various models in the WMT17 test set and the fuzzy MSCOCO test set, which contain 1000 and 461 examples respectively. In this experiment, the preprocessed statements are used directly to split words into subwords by byte-pair encoding and 10000 union operations.

Visual Features: First, a Stanford parser is used to identify noun phrases from individual source statements, then a visual grounding toolkit is used to detect relevant visual objects for the identified noun phrases. For individual phrases, the negative effects of rich visual objects are mitigated by keeping the highest predicted probability of its corresponding visual object. In individual sentences, the average number of objects and words is on the order of 3.5 and 15.0, respectively. Finally, we compute 2048-dimensional features of these objects using a pre-trained ResNet-100 Faster RCNN.

Settings: Use Transformers as a basis. Because the training corpus is relatively small, the post-training model tends to be overfitting, and we first perform one small grid search to obtain the hyperparameters in a set of English-to-German translation validation sets. Specifically, the word embedding dimensionality and hidden size are 128 and 256, respectively. The decoder has four layers and the number of heads of attention is four. Set the dropout rate as 0.5. Each lot consists of approximately 2000 source code symbols and target tokens. We apply the Adam optimizer with a given learning rate to optimize various models and use the same other settings. Finally, the Bilingual Evaluation Understudy (BLEU) index and the METEOR index are used to assess translation quality. All models were run in triplicate in individual experiments, and average results were reported, as should be noted.

Basic model: Besides the text-based transformer (TransformFormer, TF), it also uses visual features, uses several kinds of effective methods to transform, and compares the model provided by the examples of the present application with the transformer. bottom.

1, ObjectAsToken (TF). This is a variation of the Transformer where all visual objects are treated as additional source code symbols and placed before the input statement.

2, Enc-att (TF). It uses an encoder-based image attention mechanism in the transformer, adding visual feature vectors based on individual source annotations and attention.

3, Doubly-att (TF). This is one double attention transformer. In each decoding layer, we insert one cross-modal multi-head attention sub-layer before the fully connected feed-follow layer, thereby generating a visual context vector based on the visual features.

Correspondingly, the performance of several major types of multimodal neural machine translation (NMT) models, such as Doubly-att (RNN), Soft-att (RNN), Stochastlc-att (RNN), Fusion-conv (RNN), Trg-mul (RNN), VMM T (RNN) and Dellberation Network (TF) are deployed. Here, RNN is a recurrent neural network (Recurrent Neural Network).

The number of multimodal fusion layers, e, is one important hyperparameter and directly determines the degree of fine-grained semantic fusion in the encoder. Therefore, we first examine the effect it has on the English to German translation validation set. Experimental results are shown in FIG. 8, where the model reaches the optimal p-form when e is 3. Therefore, e=3 was used in all subsequent experiments.

Table 1 shows the main results of the English to German translation task. Compared to Fusion-conv (RNN) and Trg-mul (RNN) in METEOR, the performance of the model provided by the examples of the present application is superior to most previous models. Two sets of results were determined by the system state on the WMT2017 test set, which was selected based on METEOR. Comparing with the basic model, the following conclusions can be drawn.

First, the model provided by the embodiments of the present application is superior to ObjectAsToken (TF). The model combines regional visual features and text together to form attentionable sequences and utilizes self-attention mechanisms for multimodal fusion. The basic reasons for this include two points: first, we modeled the semantic correspondence between semantic units of different modals, and second, we distinguished the model parameters of different modals. be.

Next, the model provided by the examples of the present application is also significantly superior to Enc-att(TF). Here, Enc-att(TF) may be viewed as a single-layer semantic fusion encoder. Besides the advantages of modeling semantic correspondences, it is further speculated that multi-layered multimodal semantic interactions are also advantageous for NMT.

Third, compared to Doubly-att (TF), which exploits only attentional mechanisms to extract visual information, the models provided by the embodiments of the present application are significantly improved because they provide sufficient multimodal fusion in the encoder. It is

We also divide the test set into different groups based on the length of the source sentence and the number of noun phrases, and then compare the performance of different models in the test set of each group. Figures 9 and 10 show the BLEU scores for the above groups. In summary, the models provided by the examples of the present application still always reach optimal performance in all groups. Therefore, it can be said that the effectiveness and versatility of the model provided by the examples of the present application have been verified again. It should be noted that sentences with more phrases generally have longer sentences, and the model provided by the examples of the present application has a deeper meaning than the refinement of the basic model. It is presumed that long sentences often contain a relatively large number of ambiguous words. Therefore, long sentences may need to make better use of visual information as supplemental information than short sentences, and this can be achieved by the model multimodal semantic interaction provided by the embodiments herein.

Furthermore, [Table 4] further shows the training and decoding speed of the model and the base model provided by the embodiments of the present application. In the training process, the model provided by the embodiments of the present application can process approximately 1.1K tokens per second, which is comparable to other multimodal models. As for the decoding process, the model provided by the examples of the present application translated approximately 16.7 phrases per second, slightly slower than the transformer. Other than that, the model provided by the examples of the present application only introduces a small amount of additional parameters to obtain better performance.

To study the effectiveness of different ingredients, further experiments were performed comparing the model provided by the Examples of the present application with the following variations in [Table 2].

(1) Inter-modal fusion. In this variation, two independent Transformer Encoders are used to learn the semantic representations of words and visual objects respectively, and then a double attention decoder is used to merge the text and visual context into the decoder. The results in row 3 in Table 2 show that eliminating inter-modal fusion leads to a significant decrease in performance. This indicates that semantic interactions between multimodal semantic units are useful for multimodal representation learning.

(2) visual grounding to total connectivity; It completely binds words and visual objects together and establishes correspondences between modals. The results in row 4 in Table 2 demonstrate that this change results in a noticeable drop in performance. The underlying reason is that a fully coupled semantic correspondence introduces a great deal of noise into the models provided by the embodiments of the present application.

(3) From different parameters to unified parameters. When building this variation, we assign unified parameters to update node states in different modes. As can be seen, the performance degradation reported by row 5 in [Table 2] also proved the effectiveness of the method using different parameters.

(4) Visual node attention. Unlike the model that considers only text nodes, it is allowed for this variant decoder to consider these two types of nodes using a double attention decoder. As can be observed from the results of row 6 in Table 2, considering all nodes does not yield any further improvement. The above results confirm the original assumption, ie, it can be said that the visual information is already fully incorporated into the text nodes in the encoder.

(5) Text node attention and visual node attention. However, when only visual nodes are considered, the performance of the model drops sharply, which is shown in row 7 in Table 2. This is because the number of visual nodes is much lower than text nodes, but text nodes cannot generate enough translation context.

Exemplarily, further experiments are performed on the English to French translation dataset. As can be seen from Table 3, compared to all previous models, the model provided by the examples of the present application still obtains better performance. This again proves that the model provided by the present embodiment in multimodal NMT is valid and universal for different languages.

In Table 2, a comparison is made with the machine translation models provided in the relevant multimodal NMT system and the multimodal NMT system provided by the examples of this application. As is evident from the BLEU and METEOR indices, the machine translation model provided by the present application also achieves better performance for translation between English and French, with three of the four index values being the highest. values (numbers in bold).

Referring to FIG. 11, there is shown a translation device based on multimodal machine learning provided by one exemplary embodiment of the present application. The device may be part or all of a computer device in software, hardware, or a combination thereof, and the device may include a semantic association module 501, a feature extraction module 502, a vector encoding module 503, and a vector decoding module 504. include.

The semantic association module 501 is used to obtain a semantic association diagram based on n source statements belonging to different modals. The above semantic relationship diagram includes n types of different modal semantic nodes, a first connecting edge used to connect the same modal semantic nodes, and a second connecting edge used to connect different modal semantic nodes. , where the semantic node is used to indicate one semantic unit of the source statement in one kind of modal, and n is a positive integer greater than one.

Alternatively, the multi-modal graph representation layer is used to make semantic associations for n modal source languages to build a semantic association diagram, which semantic association diagram consists of n different modal semantic nodes and , including a first connecting edge used to connect semantic nodes of the same modal and a second connecting edge used to connect semantic nodes of different modals, where n is a positive integer greater than 1 ,
the feature extraction module 502 is used to extract a plurality of first word vectors from the semantic association diagram, optionally extracting the first word vectors from the semantic association diagram by a first word vector layer;
The vector encoding module 503 is used to encode the plurality of first word vectors to obtain n encoded feature vectors, optionally encoding the first word vectors with a multimodal fusion encoder. , to get the encoded feature vector, and
The vector decoding module 504 is used to decode the n encoded feature vectors to obtain the post-translational target statement, optionally calling the decoder to decode the encoded feature vectors to process the translation. Get later goal statements.

In some optional embodiments, the semantic association module 501 obtains n sets of semantic nodes, one set of semantic nodes corresponding to one modal source statement, and the same modal and add the second connecting edge between any two of the semantic nodes of different modals to obtain the semantic relationship diagram Used for things and things. Optionally, the semantic association module 501 extracts semantic nodes from each modal's source language by the multimodal graph representation layer to obtain n sets of semantic nodes corresponding to the n modal source languages; The multimodal graph representation layer uses a first connecting edge to connect between semantic nodes in the same modal for n sets of semantic nodes, and uses a second connecting edge to connect n sets of semantic nodes. and making connections between semantic nodes across different modals to obtain a semantic relationship diagram.

In some alternative embodiments, the n modal source languages include a textual first source language and a non-textual second source language, and the n sets of semantic nodes are the first semantic nodes and the including a second semantic node;
The semantic association module 501 is to obtain the first semantic node, the first semantic node is obtained by the multimodal graph representation layer processing the first source statement; obtaining a node, wherein the candidate semantic node is obtained by a multimodal graph representation layer processing the second source statement; and obtaining a first probability distribution of the candidate semantic node. wherein the first probability distribution is obtained by the multimodal graph representation layer computing according to semantic associations between the first semantic nodes and the candidate semantic nodes; Determining the second semantic node from a node, the second semantic node being determined by the multimodal graph representation layer based on the first probability distribution.

Alternatively, the semantic association module 501 extracts the first semantic nodes from the first source statement and the candidate semantic nodes from the second source language with the multimodal graph representation layer and invokes the multimodal graph representation layer. calculating a first probability distribution of the candidate semantic nodes according to the semantic association between the first semantic node and the candidate semantic nodes using and determining a second semantic node.

In some optional embodiments, the semantic association module 501 is used to add a first connecting edge of the i-th type between any two semantic nodes within the same modal in the i-th set of semantic nodes. , the i-th kind of first connecting edge corresponds to the i-th modal, and i is a positive integer less than or equal to n.

Optionally, the semantic association module 501 determines the i-th kind of first connecting edge corresponding to the i-th modal through the multimodal graph representation layer, and uses the i-th kind of first connecting edge to construct the i-th set of It is used to connect semantic nodes within the same modal to semantic nodes, and i is a positive integer less than or equal to n.

In some optional embodiments, the vector encoding module 503 performs intramodal fusion and intermodal fusion on the plurality of first word vectors e times to obtain the n encoded feature vectors. used to do Here, the intra-modal fusion refers to semantic fusion between the first word vectors in the same modal, and the inter-modal fusion is the semantic fusion between the first word vectors of different modals. , where e is a positive integer.

Alternatively, the multimodal fusion encoder comprises e encoding modules connected in series, e being a positive integer,
The vector encoding module 503 is used to perform intra-modal fusion and inter-modal fusion on the first word vector e times by e encoding modules connected in series to obtain an encoded feature vector. . Here, the intra-modal fusion refers to semantic fusion between the first word vectors in the same modal, and the inter-modal fusion is the semantic fusion between the first word vectors of different modals. point to
In some alternative embodiments, each encoding module includes n intra-modal fusion layers and n inter-modal fusion layers, each corresponding one-to-one to the n modals;
The vector encoding module 503 inputs the first word vector to each of the n intra-modal fusion layers in the first encoding module, and the n intra-modal fusion layers respectively generate the same modal intra-modal for the first word vector. to obtain n first hidden layer vectors, one said first hidden layer vector corresponds to one modal, that is, one-to-one correspondence to n modals obtaining n first hidden layer vectors that
Input the n first hidden layer vectors into each intermodal fusion layer in the first encoding module, and perform semantic fusion between different modals for the n first hidden layer vectors by each intermodal fusion layer. to obtain n first intermediate vectors, one intermediate vector corresponding to one modal, that is, n first intermediate vectors corresponding one-to-one to n modals to get the vector, and
The n first intermediate vectors are input to the j-th encoding module to perform the j-th encoding process until the last encoding module outputs n encoded feature vectors. Continuing, one encoding feature vector corresponds to one modal, that is, the last one encoding module creates n encoding feature vectors corresponding one-to-one to n modals. and j is a positive integer greater than 1 and less than or equal to e.

In some alternative embodiments, each encoding module further includes n first vector transformation layers, and one vector transformation layer corresponds to one modal, i.e., 1 in n modals. n first vector transformation layers corresponding one-to-one,
The vector encoding module 503 further inputs the n first intermediate vectors to the n first vector transformation layers corresponding to the respective modals to which they belong, and nonlinearly transforms them into n first intermediate vectors after the nonlinear transformation. Used to get intermediate vectors.

In some alternative embodiments, the hierarchical structure in each encoding module of the e serially connected encoding modules is the same.

In some alternative embodiments, different intra-modal fusion layers are assigned different or the same self-attention functions, and different inter-modal fusion layers are assigned different or the same feature fusion functions.

In some optional embodiments, the vector decoding module 504 performs feature extraction on a first target phrase to obtain a second word vector, wherein the first target phrase is is a translated word; performing feature extraction by combining the second word vector with the encoded feature vector to obtain a decoded feature vector; determining a probability distribution corresponding to the decoded feature vector; and determining a second target phrase after the first target phrase based on the probability distribution.

Optionally, the decoder comprises d decoding modules connected in series, d being a positive integer,
Vector decoding module 504 is to obtain a first target phrase through a second word vector layer, where the first target phrase is a translated phrase in the target statement; performing feature extraction on the phrase to obtain a second word vector;
performing feature extraction by combining the second word vector with the encoded feature vector by d decoding modules connected in series to obtain a decoded feature vector; inputting the decoded feature vector to a classifier; calculating a probability distribution corresponding to the decoded feature vector by and determining a second target phrase after the first target phrase based on the probability distribution.

In some alternative embodiments, each decoding module of the d serially connected decoding modules both includes a first self-attention layer and a second self-attention layer;
The vector decoding module 504 inputs the second word vector to the first self-attention layer in the first decoding module, performs feature extraction on the second word vector by the first self-attention layer, and extracts the second hidden layer vector and
The second hidden layer vector and the encoded feature vector are input to the second self-attention layer in the first decoding module, and the second self-attention layer combines the second hidden layer vector and the encoded feature vector to perform feature extraction. to obtain a second intermediate vector;
inputting the second intermediate vector to the k-th decoding module to perform the k-th decoding process, and continuing until the last one decoding module outputs a decoded feature vector, where k is greater than 1 and is a positive integer less than or equal to d.

In some optional embodiments, each decoding module further includes a second vector transformation layer;
The vector decoding module 504 is further used to input the second intermediate vector to the second vector transformation layer for non-linear transformation to obtain a second intermediate vector after non-linear transformation.

As described above, the translation device based on multimodal machine learning provided by the present embodiment performs semantic associations with n modal source languages through the multimodal graph representation layer, constructs a semantic association diagram, In the semantic relationship diagram, a first connecting edge is used to connect semantic nodes of the same modal, and a second connecting edge is used to connect semantic nodes of different modals, and the semantic relation diagram is used to connect multiple modal source languages. Express semantic associations well. Then a multimodal fusion encoder performs sufficient semantic fusion on the feature vectors in the semantic association diagram to obtain a coded feature vector after coding, and a more accurate target after decoding the coded feature vector. Get statement. The goal statement more closely approximates the content, emotion and linguistic environment, etc. that the multimodal source language collectively represents.

Referring to FIG. 12, it shows a structural schematic diagram of a server provided by one embodiment of the present application. The server is used to implement the steps of the multimodal machine learning based translation method provided in the above embodiment. in particular,
The server 600 includes a CPU (Central Processing Unit) 601, a system memory 604 including a RAM (Random Access Memory) 602 and a ROM (Read-Only Memory) 603, and a system memory 604 and a system bus 605 coupling the central processing unit 601 . The server 600 includes a basic I/O (Input/Output) system 606, an operating system 613, application programs 614, and other program modules 615 that support information transmission between devices in a computer. and a mass storage device 607 used to store the .

The basic input/output system 606 includes a display 608 used to display information and an input device 609 such as a mouse, keyboard, etc. used to input information by a user. Here the display 608 and input devices 609 are both coupled to the central processing unit 601 by an input/output controller 610 coupled to the system bus 605 . The basic input/output system 606 may further include an input/output controller 610, which is used to receive and process input from a number of other devices such as keyboards, mice or electronic styluses. Similarly, input/output controller 610 may also provide output to a display screen, printer, or other type of output device.

The mass storage device 607 is coupled to central processing unit 601 by a mass storage controller (not shown) coupled to system bus 605 . The mass storage device 607 and its associated computer-readable media provide nonvolatile storage for server 600 . In other words, the mass storage device 607 may include a computer readable medium (not shown) such as a hard disk or CD-ROM (Compact Disc Read-Only Memory) driver or the like.

Without loss of generality, the computer-readable media may include computer storage media and communication media. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented by any method or technology used for storage of information such as computer readable instructions, data structures, program modules or other data. include. Computer storage media include RAM, ROM, EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), Flash Memory. ) or other solid state memory technology, CD-ROM, DVD (Digital Versatile Disc) or other optical storage, tape cassette, magnetic tape, magnetic disk storage or other magnetic storage device. Of course, as will be appreciated by those skilled in the art, the computer storage medium is not limited to the few types described above. The system memory 604 and mass storage device 607 may be collectively referred to as memory.

According to various embodiments of the present application, the server 600 may also be operatively coupled to remote computers in a network, eg, via a network such as the Internet. That is, server 600 may be coupled to network 612 by network interface unit 611 coupled to system bus 605 described above, or may be connected to other types of networks or remote computer systems (not shown) using network interface unit 611 . ).

In an exemplary embodiment, further comprising a computer readable storage medium, e.g., a memory 602 containing instructions, the instructions being executed by the processor 601 of the server 600 to perform the multimodal machine learning based translation method. can be completed. Alternatively, the computer readable storage medium may be a non-transitory storage medium, such as ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk and optical data storage. It may be a device or the like.

In an exemplary embodiment, a computer program product is further provided, which includes a computer program, the computer program may be executed by a processor of an electronic device, thereby implementing the above multimodal machine learning based translation method. do.

As can be understood by those skilled in the art, all or part of the steps for implementing the above embodiments may be completed by hardware, or may be completed by a program issuing instructions to relevant hardware, The above program may be stored in a kind of computer-readable storage medium, and the storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, or the like.

The above descriptions are merely alternative embodiments of the present application and are not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall all fall within the protection scope of the present application.

11 intramodal fusion layer 12 intermodal fusion layer 13 first vector transformation layer 21 first self-attention layer 22 second self-attention layer 23 second vector transformation layer 31 translation target image 32 translation target statement 100 multimodal machine translation model 101 multi modal graph representation layer 102 first word vector layer 103 multimodal fusion encoder 104 decoder 105 second word vector layer 106 classifier 220 terminal 240 server 502 feature extraction module 503 vector encoding module 504 vector decoding module 600 server 601 central processing unit 601 processor 602 memory 604 system memory 605 system bus 606 output system 607 mass storage device 608 display 609 input device 610 input output controller 611 network interface unit 612 network 613 operating system 614 application program 615 program module 1031 encoding module 1042 decoding module

Claims (15)

A multimodal machine learning based translation method performed by a computer device, the method comprising:
A step of obtaining a semantic relationship diagram based on n source statements belonging to different modals, wherein the semantic relationship diagram is used to combine n different modal semantic nodes with the same modal semantic nodes. and a second connecting edge used to connect semantic nodes of different modals, wherein the semantic node indicates one semantic unit of the source statement in one kind of modal. wherein n is a positive integer greater than 1;
extracting a plurality of first word vectors from the semantic association diagram;
encoding the plurality of first word vectors to obtain n encoded feature vectors;
decoding n encoded feature vectors to obtain a post-translation target statement. The step of obtaining a semantic association diagram based on n source statements belonging to different modals comprises:
obtaining n sets of semantic nodes, one set of semantic nodes corresponding to one modal source statement;
adding the first connecting edge between any two of the semantic nodes of the same modal and adding the second connecting edge between any two of the semantic nodes of different modals to form the semantic relationship diagram and obtaining. the n modal source statements include a textual first source statement and a non-textual second source statement; the n sets of semantic nodes include a first semantic node and a second semantic node;
The step of obtaining n-tuples of semantic nodes includes:
obtaining the first semantic node, wherein the first semantic node is obtained by a multimodal graph representation layer processing the first source statement;
obtaining candidate semantic nodes, wherein the candidate semantic nodes are obtained by a multimodal graph representation layer processing the second source statement;
obtaining a first probability distribution of the candidate semantic nodes, the first probability distribution being computed by the multimodal graph representation layer according to semantic associations between the first semantic nodes and the candidate semantic nodes; a step obtained by
determining the second semantic node from the candidate semantic nodes, the second semantic node determined by the multimodal graph representation layer based on the first probability distribution; 3. The method of claim 2. the step of adding the first connecting edge between any two of the semantic nodes of the same modal;
adding an i-th kind of first connecting edge between any two semantic nodes in the same modal in the i-th set of semantic nodes, wherein the i-th kind of first connecting edge is the i-th 3. The method of claim 2, comprising steps corresponding to modal, i being a positive integer less than or equal to n. The step of encoding the plurality of first word vectors to obtain n encoded feature vectors comprises:
performing intra-modal fusion and inter-modal fusion on the plurality of first word vectors e times to obtain n encoded feature vectors, wherein the intra-modal fusion is the first modal fusion in the same modal; refers to performing semantic fusion between one word vectors, refers to performing semantic fusion between said first word vectors of modals different from said inter-modal fusion, wherein e is a positive integer. , the method according to any one of claims 1 to 4. A multimodal fusion encoder includes e encoding modules connected in series,
each of the encoding modules includes n intra-modal fusion layers and n inter-modal fusion layers corresponding to the n modals one-to-one;
performing e intra-modal and inter-modal fusions on the plurality of first word vectors to obtain n encoded feature vectors;
inputting the plurality of first word vectors into n intra-modal fusion layers in the first encoding module, respectively; to obtain n first hidden layer vectors, one said first hidden layer vector corresponding to one modal;
Input n first hidden layer vectors to each intermodal fusion layer in the first encoding module, and perform different intermodal fusion layers for the n first hidden layer vectors by each intermodal fusion layer. performing semantic fusion to obtain n first intermediate vectors, one said first intermediate vector corresponding to one modal;
inputting the n first intermediate vectors to the jth encoding module to perform the jth encoding process until the last one encoding module outputs n encoded feature vectors; and wherein one encoding feature vector corresponds to one modal, and j is a positive integer greater than 1 and less than or equal to e. each of the encoding modules further includes n first vector transformation layers, one of the first vector transformation layers corresponding to one modal;
The method further comprises:
inputting n first intermediate vectors to n first vector transformation layers corresponding to respective modals to which they belong, and performing nonlinear transformation to obtain n first intermediate vectors after nonlinear transformation; 7. The method of claim 6. 7. The method of claim 6, wherein the hierarchical structure in each of said serially connected e encoding modules is the same. 7. The method of claim 6, wherein different intra-modal fusion layers are set with different or the same self-attention functions, and different inter-modal fusion layers are set with different or the same feature fusion functions. The step of decoding the n encoded feature vectors to obtain a translated target statement comprises:
performing feature extraction on a first target phrase to obtain a second word vector, wherein the first target phrase is a translated phrase in the target statement;
performing feature extraction by combining the second word vector with the encoded feature vector to obtain a decoded feature vector;
determining a probability distribution corresponding to said decoding feature vector, and determining a second target phrase after said first target phrase based on said probability distribution. The method described in . The decoder includes d serially connected decoding modules, where d is a positive integer, and each decoding module of the d serially connected decoding modules is both a first self-attention layer and including a second self-attention layer,
The step of performing feature extraction by combining the second word vector with the encoded feature vector to obtain a decoded feature vector,
Inputting the second word vector into a first self-attention layer in a first decoding module, performing feature extraction on the second word vector by the first self-attention layer to obtain a second hidden layer vector. a step;
inputting the second hidden layer vector and the encoded feature vector to a second self-attention layer in the first decoding module, and converting the second hidden layer vector and the encoded feature vector by the second self-attention layer; performing combined feature extraction to obtain a second intermediate vector;
a step of inputting the second intermediate vector to the k-th decoding module to perform the k-th decoding process, and continuing until the last one decoding module outputs the decoded feature vector, where k is greater than 1; is a positive integer greater than d and less than or equal to d. each decoding module further includes a second vector transformation layer;
The method further comprises:
12. The method of claim 11, comprising inputting the second intermediate vector to the second vector transformation layer to perform nonlinear transformation to obtain a second intermediate vector after nonlinear transformation. A translation device based on multimodal machine learning, said device comprising:
A semantic association module for building a semantic association diagram based on n source statements belonging to different modals, wherein the semantic association diagram includes n different modal semantic nodes and the same modal semantic nodes. and a second connecting edge used to connect semantic nodes of different modals, wherein the semantic node is one semantic unit of the source statement in one kind of modal a semantic association module, wherein n is a positive integer greater than 1;
a feature extraction module used to extract a plurality of first word vectors from the semantic association diagram;
a vector encoding module used to encode the plurality of first word vectors to obtain n encoded feature vectors;
a vector decoding module used to decode the encoded feature vector to obtain a post-translation target statement. A computer device, said computer device comprising:
memory;
a processor coupled to the memory;
Computer equipment, wherein the processor is configured to implement a multimodal machine learning based translation method according to any one of claims 1 to 12 by loading and executing executable instructions. A computer readable storage medium, wherein at least one segment of a program is stored on said computer readable storage medium, said at least one segment of program being loaded and executed by a processor. A computer-readable storage medium that implements the multimodal machine learning-based translation method described in .

Images