CN111310484B - Automatic training method and platform of machine translation model, electronic device and storage medium - Google Patents
Automatic training method and platform of machine translation model, electronic device and storage medium Download PDFInfo
- Publication number
- CN111310484B CN111310484B CN202010170720.8A CN202010170720A CN111310484B CN 111310484 B CN111310484 B CN 111310484B CN 202010170720 A CN202010170720 A CN 202010170720A CN 111310484 B CN111310484 B CN 111310484B
- Authority
- CN
- China
- Prior art keywords
- link
- file
- translation
- training
- script
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Medical Informatics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Artificial Intelligence (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
The embodiment of the disclosure discloses an automatic training method, a platform, electronic equipment and a storage medium of a machine translation model, wherein the method comprises the following steps: visually displaying at least one translation link; acquiring a training instruction which is input by a user and aims at a target model to be trained; determining at least one link to be translated and configuration information according to the training instruction, acquiring a script file of the at least one link to be translated, and associating the acquired script file after parameter configuration according to the configuration information; and running the associated script file to train the target model. The automatic machine translation training method provided by the embodiment of the disclosure can realize visual control and convenient control of a machine translation process.
Description
Technical Field
The embodiment of the disclosure relates to the technical field of natural language processing, in particular to an automatic training method, a platform, electronic equipment and a storage medium for a machine translation model.
Background
The training process of the machine translation model at least needs to perform data cleaning (such as dirty character removal, punctuation standardization, dirty data removal and the like), data preprocessing (such as Chinese word segmentation, Japanese word segmentation and the like), model training, automatic evaluation (effect) of a machine, manual evaluation, model online and the like on sample data.
Generally, each translation link needs to be manually set, a high professional level is required for model training personnel, in addition, each link generally runs independently in the training process, and the model training personnel are required to perform too many operations, so that smooth one-key operation cannot be realized.
Disclosure of Invention
In view of this, the embodiments of the present disclosure provide an automatic training method, a platform, an electronic device, and a storage medium for a machine translation model, so as to implement visual control and convenient control of a machine translation process.
Additional features and advantages of the disclosed embodiments will be set forth in the detailed description which follows, or in part will be obvious from the description, or may be learned by practice of the disclosed embodiments.
In a first aspect of the present disclosure, an embodiment of the present disclosure provides an automatic training method for a machine translation model, including:
visually displaying at least one translation link;
acquiring a training instruction which is input by a user and aims at a target model to be trained;
determining at least one link to be translated and configuration information according to the training instruction, acquiring a script file of the at least one link to be translated, and associating the acquired script file after parameter configuration according to the configuration information;
and running the associated script file to train the target model.
In a second aspect of the present disclosure, an embodiment of the present disclosure further provides an automatic training platform of a machine translation model, including:
the display module is used for visually displaying at least one translation link;
the training instruction acquisition module is used for acquiring a training instruction which is input by a user and aims at a target model to be trained;
a link determination and script association module, configured to determine at least one to-be-executed translation link and configuration information according to the training instruction, obtain a script file of the at least one to-be-executed translation link, perform parameter configuration on the obtained script file according to the configuration information, and associate the obtained script file with the script file;
and the target model training module is used for operating the associated script file so as to train the target model.
In a third aspect of the disclosure, an electronic device is provided. The electronic device includes: a processor; and a memory for storing executable instructions that, when executed by the processor, cause the electronic device to perform the method of the first aspect.
In a fourth aspect of the disclosure, a computer-readable storage medium is provided, on which a computer program is stored, which computer program, when being executed by a processor, carries out the method in the first aspect.
According to the embodiment of the invention, at least one translation link is visually displayed, a training instruction which is input by a user and aims at a target model to be trained is obtained, so that at least one translation link to be executed and configuration information are determined, a script file of the at least one translation link to be executed is obtained, the obtained script file is subjected to parameter configuration according to the configuration information and then is associated together to run, and the target model is trained, so that the visual control and the convenient control of the machine translation process can be realized.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings needed to be used in the description of the embodiments of the present disclosure will be briefly described below, and it is obvious that the drawings in the following description are only a part of the embodiments of the present disclosure, and for those skilled in the art, other drawings can be obtained according to the contents of the embodiments of the present disclosure and the drawings without creative efforts.
FIG. 1 is a schematic flow chart diagram illustrating a method for automatic training of a machine translation model according to an embodiment of the present disclosure;
FIG. 2 is a schematic diagram of an exemplary visualization display effect provided by an embodiment of the disclosure;
FIG. 3 is a flow chart of another method for automatic training of a machine translation model provided by an embodiment of the present disclosure;
FIG. 4 is a schematic structural diagram of an automatic training platform of a machine translation model provided by an embodiment of the present disclosure;
FIG. 5 illustrates a schematic diagram of an electronic device suitable for use in implementing embodiments of the present disclosure.
Detailed Description
In order to make the technical problems solved, technical solutions adopted and technical effects achieved by the embodiments of the present disclosure clearer, the technical solutions of the embodiments of the present disclosure will be described in further detail below with reference to the accompanying drawings, and it is obvious that the described embodiments are only some embodiments, but not all embodiments, of the embodiments of the present disclosure. All other embodiments, which can be obtained by a person skilled in the art without making creative efforts based on the embodiments of the present disclosure, belong to the protection scope of the embodiments of the present disclosure.
It should be noted that the terms "system" and "network" are often used interchangeably in the embodiments of the present disclosure. Reference to "and/or" in embodiments of the present disclosure is meant to include any and all combinations of one or more of the associated listed items. The terms "first", "second", and the like in the description and claims of the present disclosure and in the drawings are used for distinguishing between different objects and not for limiting a particular order.
It should also be noted that, in the embodiments of the present disclosure, each of the following embodiments may be executed alone, or may be executed in combination with each other, and the embodiments of the present disclosure are not limited specifically.
The names of messages or information exchanged between multiple platforms in the embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.
The technical solutions of the embodiments of the present disclosure are further described by the following detailed description in conjunction with the accompanying drawings.
Fig. 1 is a flowchart illustrating an automatic training method for a machine translation model according to an embodiment of the present disclosure, where the present embodiment is applicable to a case of automatically training a machine translation model, and the method may be executed by an automatic training platform of a machine translation model configured in an electronic device, as shown in fig. 1, the automatic training method for a machine translation model according to the present embodiment includes:
in step S110, at least one translation link is visually displayed. The training and training process of the machine translation model can comprise a parallel sentence pair extraction link, an alignment analysis link, a data preprocessing link, a model training link, a model verification link and the like. Of course, the method is not limited to the above steps, and the method can also be divided into a predefined data cleaning step, a data preprocessing step, a model training step, a model inference evaluation step, a model online step and the like. And performing visual display, namely displaying the name of each translation link and the sequence information among the translation links through a visual display interface.
Fig. 2 is a schematic diagram of an exemplary visual display effect provided by an embodiment of the present disclosure, and as shown in fig. 2, the model training process interface includes a plurality of training links, including a parallel sentence pair extraction link, an alignment analysis link, a data preprocessing link, a model training link, a model verification link, and a link with an arrow to illustrate a sequence of each translation link.
If the display space of the visually displayed interface allows, some more detailed information or more man-machine interaction information can be displayed.
For example, metadata of a predetermined translation link may be displayed, such as displaying a scale of a data file, displaying an evaluation result of a model file, and the like.
For another example, a mapping interface of the core associated file of each translation link may also be displayed, and the mapping interface may be used to receive a visualization operation instruction of a user from the mapping interface of the core associated file, and then open the core associated file corresponding to the visualization operation instruction. For example, if a visualization operation (for example, a click operation, a double click operation, or the like) of the user on the mapping interface of the core associated file is received, the corresponding core associated file is opened on the interface displayed in the visualization.
For another example, the resource mapping interface of the core associated folder of each translation link may also be shown, and may be configured to open the core associated folder corresponding to the visualization operation instruction after receiving the visualization operation instruction of the user, so as to present the files and/or the sub-folders inside the core associated folder, for example, the files and the sub-folders inside the folder may be presented in a style similar to a windows resource manager, and the organization structure and the hierarchical relationship of the sub-elements such as the files inside the folder, the sub-folders at each level, and the like may be clearly presented in a tree structure.
And if receiving the visualization operation (such as click operation, double-click operation and the like) of the user on the resource mapping interface of the core associated folder, opening the corresponding core associated folder on the visually displayed interface.
For another example, a configuration information interface of each translation link may be further displayed, and the configuration information interface may be configured to display the configuration information of each translation link on the visually displayed interface, and modify the corresponding configuration information according to a visualization operation when the visualization operation (for example, a click operation, a double click operation, and the like) of the user on the configuration information interface is received.
If the user modifies the configuration information of a certain translation link through the visually displayed interface, when parameter configuration is carried out according to the configuration information, parameter configuration is carried out according to the modified configuration information.
For example, a competitive assessment BLEU (Bilingual Evaluation understatus), a popular machine translation Evaluation index, for analyzing the co-occurrence degree of n-tuples in the candidate translation and the reference translation, may also be displayed, and GSB visualization and case analysis may be manually evaluated. Common translation links can be combined through the workflow, and the operation times of a user are reduced.
In step S120, a training instruction for a target model to be trained, which is input by a user, is acquired. The training instructions include, but are not limited to, one-touch training instructions, instructions that instruct one or more translation stages to operate.
In step S130, at least one to-be-executed translation link and configuration information are determined according to the training instruction, a script file of the at least one to-be-executed translation link is obtained, and the obtained script files are associated together after parameter configuration is performed according to the configuration information. For example, a predetermined script may be executed to perform the following operations: taking out script files of each link to be translated according to a preset naming rule and a preset path rule; acquiring association rules corresponding to the training process of the target model, configuring parameters of the taken script files according to the configuration information, and associating the configured script files together according to the association rules.
In step S140, the associated script file is executed to train the target model. According to the technical scheme, the association processing of the script file and the data file can be performed according to the common functions (such as any link or organic combination of several links) in the training process, the association processing is displayed on a visual interface for a user to select the file, and the operation can be started by one key. The user can configure the parameters used in each link, for example, in the data preprocessing link, chinese participles are required when the source file is chinese, japanese participles are required when the source file is japanese, and the like. For the combination of several translation link tasks, a multi-task combined template can be preset, and the combination can be freely combined by a user.
HDFS is a master-slave architecture, a HDFS cluster is composed of a name node, which is the master server that manages file namespaces and regulates client access to files, and of course some data nodes, usually a node-machine, that manage storage for the corresponding nodes. HDFS opens file namespaces to the outside and allows user data to be stored in file form. The internal mechanism of HDFS is to split a file into one or more blocks, which are stored in a set of data nodes. The namenodes are used to manipulate file or directory operations of the file namespace, such as open, close, rename, and the like. It also determines the mapping of the block to the data node. The data nodes are responsible for read and write requests from file system clients. The data node also performs block creation, deletion, and block copy instructions from the name node.
The GPU cluster has strong large-scale data-level parallel computing capacity, and in addition, the GPU is a computing device facing throughput rate, so that the GPU cluster has stronger throughput rate than the traditional cluster. And by the amplification of the nodes and the update and upgrade of hardware, the GPU cluster also has the characteristic of expandability. In addition, the GPU cluster has a higher cost performance ratio and performance power consumption ratio.
In this embodiment, the associated script file may be run based on a distributed file system (e.g., HADOOP) and a GPU clustering technique during training.
Further, in the process of running the associated script file, naming the generated intermediate file and result file according to the preset naming rule, and storing according to the preset path rule.
The related files required by each translation link include, but are not limited to, a script file, an intermediate file, a result file, and the like, which are specifically shown in fig. 2, but are not limited to the files shown in fig. 2.
The files referred to above may be stored in various ways, which is not limited in this embodiment. If the amount of data involved is large, the files can be stored based on a distributed file system, for example, the files can be stored based on a Hadoop distributed file system HDFS and a distributed file system Ceph.
Ceph is a unified, distributed file system designed for excellent performance, reliability, and scalability. The file related to the embodiment is stored by adopting Ceph, so that the PB capacity can be easily expanded, the high performance of various workloads is supported, and the high reliability of file access can be maintained.
In the embodiment, at least one translation link is visually displayed, a training instruction aiming at a target model to be trained, which is input by a user, is obtained, so that at least one translation link to be executed and configuration information are determined, a script file of the at least one translation link to be executed is obtained, the obtained script file is associated with the configuration information to run after parameter configuration is carried out according to the configuration information, and the target model is trained, so that visual control and convenient control of a machine translation process can be realized.
Fig. 3 is a flow chart illustrating another automatic training method for a machine translation model according to an embodiment of the present disclosure, where the present embodiment is based on the foregoing embodiment and performs improved optimization. As shown in fig. 3, the automatic training method of a machine translation model according to this embodiment includes:
in step S310, at least one translation link is visually displayed. For example, the name of each translation link and the sequence information between each translation link can be displayed through a visual display interface.
In step S320, a training instruction for the target model to be trained, which is input by the user, is obtained, where the training instruction includes, but is not limited to, a one-key training instruction, an instruction indicating operation of one or more translation links, and the like.
In step S330, the script file of each link to be translated is extracted according to the predetermined naming rule and the predetermined path rule. The preset naming rule and the preset path rule can adopt a default form and can be self-configured by a user through a visual interface. The files are extracted and stored according to the preset naming rule and the preset path rule, and a user can be liberated from a plurality of data files, model files, data cleaning, data processing, model training and decoding scripts.
In step S340, association rules corresponding to the training process of the target model are obtained, and after the parameter configuration is performed on the extracted script files according to the configuration information, the configured script files are associated together according to the association rules.
In step S350, the associated script file is executed to train the target model. During training, the associated script file can be operated based on a distributed file system (HADOOP) and a GPU cluster technology.
In step S360, in the process of running the associated script file, naming the generated intermediate file and result file according to the predetermined naming rule, and storing the intermediate file and result file according to the predetermined path rule.
On the basis of the previous embodiment, the embodiment further discloses a method for associating links to be translated and configuration information according to the training instruction, wherein script files of the links to be translated are taken out according to a preset naming rule and a preset path rule, and the taken script files are associated together after parameters are configured according to an association rule corresponding to a training process of a target model, so that a user can shield fussy file management, and the efficiency of model training can be further improved.
As an implementation of the methods shown in the above diagrams, the present application provides an embodiment of an automatic training platform of a machine translation model, and fig. 4 illustrates a schematic structural diagram of the automatic training platform of the machine translation model provided in this embodiment, where the embodiment of the platform corresponds to the embodiment of the methods shown in fig. 1 and fig. 3, and the platform may be specifically applied to various electronic devices. As shown in fig. 4, the automatic training platform of the machine translation model according to this embodiment includes
The presentation module 410 is configured to visually present at least one translation link.
The training instruction obtaining module 420 is configured to obtain a training instruction input by a user and directed to a target model to be trained.
The link determination and script association module 430 is configured to determine at least one to-be-executed translation link and configuration information according to the training instruction, obtain a script file of the at least one to-be-executed translation link, and associate the obtained script file together after performing parameter configuration according to the configuration information.
The target model training module 440 is configured to run the associated script file to train the target model.
Further, the link determination and script association module 430 is configured to execute the predetermined script to perform the following operations:
taking out script files of each link to be translated according to a preset naming rule and a preset path rule;
acquiring association rules corresponding to the training process of the target model, configuring parameters of the taken script files according to the configuration information, and associating the configured script files together according to the association rules.
In an embodiment, the target model training module 440 is further configured to name the generated intermediate file and result file according to the predetermined naming rule and store the intermediate file and result file according to the predetermined path rule during the process of running the associated script file.
Further, script files, intermediate files and result files of each translation link are stored based on a distributed file system.
In an embodiment, the presentation module 410 is further configured to present names of the translation links and sequence information between the translation links.
In one embodiment, the display module 410 further includes a mapping interface of the core association file and/or a resource mapping interface of the core association folder;
the mapping interface of the core associated file is used for opening the corresponding core associated file after receiving the visual operation of the user;
and the resource mapping interface of the core associated folder is used for opening the corresponding core associated folder after receiving the visual operation of the user.
In one embodiment, the presentation module 410 further comprises an interface for configuration information;
the configuration information interface is configured to, after receiving a visualization operation by a user:
modifying corresponding configuration information according to the visualization operation;
the link determining and script associating module is used for performing parameter configuration on the acquired script file according to the configuration information and comprises the following steps: and performing parameter configuration on the acquired script file according to the modified configuration information.
In an embodiment, the presentation module 410 is configured to further present metadata of the predetermined translation link, where the metadata includes a scale of the data file or an evaluation result of the model file.
In an embodiment, the target model training module 440 is further configured to run the associated script file based on a distributed file system and a GPU clustering technique to train the target model.
In one embodiment, the translation links include at least one of: the method comprises a parallel sentence pair extraction link, an alignment analysis link, a data preprocessing link, a model training link and a model verification link.
In one embodiment, the translation links include at least one of: the method comprises a predefined data cleaning link, a data preprocessing link, a model training link, a model inference evaluation link and a model online link.
The automatic training platform of the machine translation model provided by the embodiment of the invention can execute the automatic training method of the machine translation model provided by the embodiment of the method disclosed by the invention, and has corresponding functional modules and beneficial effects of the execution method.
Referring now to FIG. 5, a block diagram of an electronic device 500 suitable for use in implementing embodiments of the present disclosure is shown. The terminal device in the embodiments of the present disclosure may include, but is not limited to, a mobile terminal such as a mobile phone, a notebook computer, a digital broadcast receiver, a PDA (personal digital assistant), a PAD (tablet computer), a PMP (portable multimedia player), a vehicle terminal (e.g., a car navigation terminal), and the like, and a stationary terminal such as a digital TV, a desktop computer, and the like. The electronic device shown in fig. 5 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present disclosure.
As shown in fig. 5, electronic device 500 may include a processing means (e.g., central processing unit, graphics processor, etc.) 501 that may perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)502 or a program loaded from a storage means 508 into a Random Access Memory (RAM) 503. In the RAM 503, various programs and data necessary for the operation of the electronic apparatus 500 are also stored. The processing device 501, the ROM 502, and the RAM 503 are connected to each other through a bus 504. An input/output (I/O) interface 505 is also connected to bus 504.
Generally, the following devices may be connected to the I/O interface 505: input devices 506 including, for example, a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 507 including, for example, Liquid Crystal Displays (LCDs), speakers, vibrators, and the like; storage devices 508 including, for example, magnetic tape, hard disk, etc.; and a communication device 509. The communication means 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. While fig. 5 illustrates an electronic device 500 having various means, it is to be understood that not all illustrated means are required to be implemented or provided. More or fewer devices may alternatively be implemented or provided.
In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated by the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network via the communication means 509, or installed from the storage means 508, or installed from the ROM 502. The computer program performs the above-described functions defined in the methods of the embodiments of the present disclosure when executed by the processing device 501.
It should be noted that the computer readable medium described above in the embodiments of the present disclosure may be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the disclosed embodiments, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In contrast, in the disclosed embodiments, a computer readable signal medium may comprise a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, optical cables, RF (radio frequency), etc., or any suitable combination of the foregoing.
The computer readable medium may be embodied in the electronic device; or may exist separately without being assembled into the electronic device.
The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: visually displaying at least one translation link; acquiring a training instruction which is input by a user and aims at a target model to be trained; determining at least one link to be translated and configuration information according to the training instruction, acquiring a script file of the at least one link to be translated, and associating the acquired script file after parameter configuration according to the configuration information; and running the associated script file to train the target model.
Computer program code for carrying out operations for embodiments of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units described in the embodiments of the present disclosure may be implemented by software or hardware. Where the name of a unit does not in some cases constitute a limitation of the unit itself, for example, the first retrieving unit may also be described as a "unit for retrieving at least two internet protocol addresses".
According to one or more embodiments of the present disclosure, in the automatic training method of a machine translation model, obtaining a script file of the at least one link to be translated, and associating the obtained script file with the obtained script file after performing parameter configuration according to the configuration information includes executing a predetermined script to perform the following operations: taking out script files of each link to be translated according to a preset naming rule and a preset path rule; acquiring association rules corresponding to the training process of the target model, configuring parameters of the taken script files according to the configuration information, and associating the configured script files together according to the association rules.
According to one or more embodiments of the present disclosure, the method for automatically training a machine translation model further includes naming the generated intermediate file and result file according to the predetermined naming rule and storing the intermediate file and result file according to the predetermined path rule during the process of running the associated script file.
According to one or more embodiments of the disclosure, in the automatic training method of the machine translation model, the script file, the intermediate file and the result file of each translation link are stored based on a distributed file system.
According to one or more embodiments of the present disclosure, in the automatic training method of a machine translation model, the visually displaying at least one translation link includes: and displaying the name of each translation link and the sequence information among the translation links.
According to one or more embodiments of the present disclosure, in the automatic training method of a machine translation model, the visually displaying at least one translation link further includes: displaying a mapping interface of a core associated file and/or a resource mapping interface of a core associated folder of each translation link; the method further comprises the following steps: after receiving a visual operation instruction of a user from a mapping interface of the core associated file, opening the core associated file corresponding to the visual operation instruction; and after receiving a visualization operation instruction of a user from a resource mapping interface of the core associated folder, opening the core associated folder corresponding to the visualization operation instruction to present files and/or sub-folders in the core associated folder.
According to one or more embodiments of the present disclosure, in the automatic training method of a machine translation model, the visually displaying at least one translation link further includes: displaying a configuration information interface of each translation link; the method further comprises the following steps: receiving the visual operation of a user on the configuration information interface, and modifying corresponding configuration information according to the visual operation; the parameter configuration of the acquired script file according to the configuration information comprises the following steps: and performing parameter configuration on the acquired script file according to the modified configuration information.
According to one or more embodiments of the present disclosure, in the automatic training method of a machine translation model, the visually displaying at least one translation link further includes: and displaying metadata of the preset translation link, wherein the metadata comprises the scale of the data file or the evaluation result of the model file.
According to one or more embodiments of the present disclosure, in the method for automatically training a machine translation model, the running an associated script file to train the target model includes: and running the associated script file based on a distributed file system and a GPU cluster technology to train the target model.
According to one or more embodiments of the present disclosure, in the automatic training method of the machine translation model, each translation link includes at least one of: the method comprises a parallel sentence pair extraction link, an alignment analysis link, a data preprocessing link, a model training link and a model verification link.
According to one or more embodiments of the present disclosure, in the automatic training method of the machine translation model, each translation link includes at least one of: the method comprises a predefined data cleaning link, a data preprocessing link, a model training link, a model inference evaluation link and a model online link.
According to one or more embodiments of the disclosure, in the automatic training platform of the machine translation model, the link determining and script associating module is used for executing a predetermined script to perform the following operations: taking out script files of each link to be translated according to a preset naming rule and a preset path rule; acquiring association rules corresponding to the training process of the target model, configuring parameters of the taken script files according to the configuration information, and associating the configured script files together according to the association rules.
According to one or more embodiments of the present disclosure, in the automatic training platform of the machine translation model, the target model training module is further configured to: and in the process of running the associated script file, naming the generated intermediate file and result file according to the preset naming rule and storing the intermediate file and the result file according to the preset path rule.
According to one or more embodiments of the disclosure, in the automatic training platform of the machine translation model, script files, intermediate files and result files of each translation link are stored based on a distributed file system.
According to one or more embodiments of the present disclosure, in the automatic training platform of the machine translation model, the presentation module is further configured to: and displaying the name of each translation link and the sequence information among the translation links.
According to one or more embodiments of the present disclosure, in the automatic training platform of the machine translation model, the presentation module further includes: a mapping interface of the core associated file and/or a resource mapping interface of the core associated folder; the mapping interface of the core associated file is used for opening the core associated file corresponding to the visual operation instruction after receiving the visual operation instruction of the user from the mapping interface of the core associated file; and the resource mapping interface of the core associated folder is used for opening the core associated folder corresponding to the visualization operation instruction after receiving the visualization operation instruction of the user from the resource mapping interface of the core associated folder so as to present the files and/or sub-folders in the core associated folder.
According to one or more embodiments of the present disclosure, in the automatic training platform of the machine translation model, the presentation module further includes a configuration information interface; the configuration information interface is configured to, after receiving a visualization operation by a user: modifying corresponding configuration information according to the visualization operation; the link determining and script associating module is used for performing parameter configuration on the acquired script file according to the configuration information and comprises the following steps: and performing parameter configuration on the acquired script file according to the modified configuration information.
According to one or more embodiments of the present disclosure, in the automatic training platform of the machine translation model, the display module is further configured to display metadata of a predetermined translation link, where the metadata includes a scale of a data file or an evaluation result of a model file.
According to one or more embodiments of the present disclosure, in the automatic training platform of the machine translation model, the target model training module is configured to: and running the associated script file based on a distributed file system and a GPU cluster technology to train the target model.
According to one or more embodiments of the present disclosure, in the automatic training platform of the machine translation model, each translation link includes at least one of the following: the method comprises a parallel sentence pair extraction link, an alignment analysis link, a data preprocessing link, a model training link and a model verification link.
According to one or more embodiments of the present disclosure, in the automatic training platform of the machine translation model, each translation link includes at least one of the following: the method comprises a predefined data cleaning link, a data preprocessing link, a model training link, a model inference evaluation link and a model online link.
The foregoing description is only a preferred embodiment of the disclosed embodiments and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the disclosure in the embodiments of the present disclosure is not limited to the particular combination of the above-described features, but also encompasses other embodiments in which any combination of the above-described features or their equivalents is possible without departing from the scope of the present disclosure. For example, the above features and (but not limited to) the features with similar functions disclosed in the embodiments of the present disclosure are mutually replaced to form the technical solution.
Claims (13)
1. A method for automatically training a machine translation model is characterized by comprising the following steps of;
visually displaying at least one translation link;
acquiring a training instruction which is input by a user and aims at a target model to be trained;
determining at least one link to be translated and configuration information according to the training instruction, acquiring a script file of the at least one link to be translated, and associating the acquired script file after parameter configuration according to the configuration information;
running the associated script file to train the target model;
the obtaining the script file of the at least one link to be translated and associating the obtained script file after parameter configuration according to the configuration information includes:
executing a predetermined script to perform the following operations:
taking out script files of each link to be translated according to a preset naming rule and a preset path rule;
acquiring association rules corresponding to the training process of the target model, configuring parameters of the taken script files according to the configuration information, and associating the configured script files together according to the association rules.
2. The automated training method of claim 1, wherein the method further comprises: and in the process of running the associated script file, naming the generated intermediate file and result file according to the preset naming rule and storing the intermediate file and the result file according to the preset path rule.
3. The automated training method of claim 2, wherein the script file, the intermediate file, and the result file for each translation link are stored based on a distributed file system.
4. The automated training method of claim 1, wherein visually presenting at least one translation stage comprises:
and displaying the name of each translation link and the sequence information among the translation links.
5. The automated training method of claim 4, wherein visually presenting at least one translation stage further comprises:
displaying a mapping interface of a core associated file and/or a resource mapping interface of a core associated folder of each translation link;
the method further comprises the following steps:
after receiving a visual operation instruction of a user from a mapping interface of the core associated file, opening the core associated file corresponding to the visual operation instruction;
and after receiving a visualization operation instruction of a user from a resource mapping interface of the core associated folder, opening the core associated folder corresponding to the visualization operation instruction to present files and/or sub-folders in the core associated folder.
6. The automated training method of claim 4, wherein visually presenting at least one translation stage further comprises:
displaying a configuration information interface of each translation link;
the method further comprises the following steps:
receiving the visual operation of a user on the configuration information interface, and modifying corresponding configuration information according to the visual operation;
the parameter configuration of the acquired script file according to the configuration information comprises the following steps:
and performing parameter configuration on the acquired script file according to the modified configuration information.
7. The automated training method of claim 4, wherein visually presenting at least one translation stage further comprises:
and displaying metadata of the preset translation link, wherein the metadata comprises the scale of the data file or the evaluation result of the model file.
8. The automated training method of claim 1, wherein the running the associated script file to train the target model comprises:
and running the associated script file based on a distributed file system and a GPU cluster technology to train the target model.
9. An automated training method according to any one of claims 1-8, wherein each translation stage comprises at least one of:
the method comprises a parallel sentence pair extraction link, an alignment analysis link, a data preprocessing link, a model training link and a model verification link.
10. An automated training method according to any one of claims 1-8, wherein each translation stage comprises at least one of:
the method comprises a predefined data cleaning link, a data preprocessing link, a model training link, a model inference evaluation link and a model online link.
11. An automated training platform for machine translation models, comprising:
the display module is used for visually displaying at least one translation link;
the training instruction acquisition module is used for acquiring a training instruction which is input by a user and aims at a target model to be trained;
a link determination and script association module, configured to determine at least one to-be-executed translation link and configuration information according to the training instruction, obtain a script file of the at least one to-be-executed translation link, perform parameter configuration on the obtained script file according to the configuration information, and associate the obtained script file with the script file; the method is specifically used for executing a predetermined script to execute the following operations:
taking out script files of each link to be translated according to a preset naming rule and a preset path rule;
acquiring association rules corresponding to the training process of the target model, configuring parameters of the taken script files according to the configuration information, and associating the configured script files together according to the association rules;
and the target model training module is used for operating the associated script file so as to train the target model.
12. An electronic device, comprising:
a processor; and
a memory to store executable instructions that, when executed by the one or more processors, cause the electronic device to perform the method of any of claims 1-8.
13. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010170720.8A CN111310484B (en) | 2020-03-12 | 2020-03-12 | Automatic training method and platform of machine translation model, electronic device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010170720.8A CN111310484B (en) | 2020-03-12 | 2020-03-12 | Automatic training method and platform of machine translation model, electronic device and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111310484A CN111310484A (en) | 2020-06-19 |
| CN111310484B true CN111310484B (en) | 2021-07-06 |
Family
ID=71162260
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010170720.8A Active CN111310484B (en) | 2020-03-12 | 2020-03-12 | Automatic training method and platform of machine translation model, electronic device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111310484B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112965803A (en) * | 2021-03-22 | 2021-06-15 | 共达地创新技术(深圳)有限公司 | AI model generation method and electronic equipment |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106126507A (en) * | 2016-06-22 | 2016-11-16 | 哈尔滨工业大学深圳研究生院 | A kind of based on character-coded degree of depth nerve interpretation method and system |
| CN110286885A (en) * | 2019-06-27 | 2019-09-27 | 江苏满运软件科技有限公司 | Automate initial development method, system, computer equipment and storage medium |
| CN110795077A (en) * | 2019-09-26 | 2020-02-14 | 北京你财富计算机科技有限公司 | Software development method and device based on artificial intelligence and electronic equipment |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7065744B2 (en) * | 2002-01-14 | 2006-06-20 | International Business Machines Corporation | System and method for converting management models to specific console interfaces |
| US7191404B2 (en) * | 2002-01-14 | 2007-03-13 | International Business Machines Corporation | System and method for mapping management objects to console neutral user interface |
-
2020
- 2020-03-12 CN CN202010170720.8A patent/CN111310484B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106126507A (en) * | 2016-06-22 | 2016-11-16 | 哈尔滨工业大学深圳研究生院 | A kind of based on character-coded degree of depth nerve interpretation method and system |
| CN110286885A (en) * | 2019-06-27 | 2019-09-27 | 江苏满运软件科技有限公司 | Automate initial development method, system, computer equipment and storage medium |
| CN110795077A (en) * | 2019-09-26 | 2020-02-14 | 北京你财富计算机科技有限公司 | Software development method and device based on artificial intelligence and electronic equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111310484A (en) | 2020-06-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11169789B2 (en) | Rich text box for live applications in a cloud collaboration platform | |
| Begoli et al. | Design principles for effective knowledge discovery from big data | |
| US11829853B2 (en) | Systems and methods for tracking and representing data science model runs | |
| US10108601B2 (en) | Method and system for presenting personalized content | |
| CN102520841B (en) | Collection user interface | |
| US11086891B2 (en) | Systems and methods for tracking and representing data science data runs | |
| US9015118B2 (en) | Determining and presenting provenance and lineage for content in a content management system | |
| US9324373B2 (en) | Determining updates for a video tutorial | |
| CN110275962B (en) | Method and apparatus for outputting information | |
| US20180260442A1 (en) | Self-tutoring graph of event data | |
| WO2024036662A1 (en) | Parallel graph rule mining method and apparatus based on data sampling | |
| CN111435367A (en) | Knowledge graph construction method, system, equipment and storage medium | |
| US9280401B2 (en) | Type agnostic data engine | |
| CN111310484B (en) | Automatic training method and platform of machine translation model, electronic device and storage medium | |
| CN112582073B (en) | Medical information acquisition method, device, electronic equipment and medium | |
| US20230161945A1 (en) | Automatic two-way generation and synchronization of notebook and pipeline | |
| Madaminov et al. | Firebase Database Usage and Application Technology in Modern Mobile Applications | |
| CN113806556A (en) | Method, device, equipment and medium for constructing knowledge graph based on power grid data | |
| CN113760317A (en) | Page display method, device, equipment and storage medium | |
| Milovanovic et al. | Python Data Visualization Cookbook | |
| CN111522548B (en) | Project function expansion method, apparatus, electronic device and computer readable medium | |
| CN115422131B (en) | Business audit knowledge base retrieval method, device, equipment and computer readable medium | |
| US20230081509A1 (en) | Cognitive programming paradigm assistant | |
| CN116149523A (en) | Automatic bidirectional generation and synchronization of notebooks and pipelines | |
| CN113672134A (en) | Media information editing method and device, computer readable medium and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |