EP3926512A1 - Verfahren und vorrichtung zur modellverbesserung basierend auf einem im voraus trainierten semantischen modell - Google Patents

Verfahren und vorrichtung zur modellverbesserung basierend auf einem im voraus trainierten semantischen modell Download PDF

Info

Publication number
EP3926512A1
EP3926512A1 EP21161686.7A EP21161686A EP3926512A1 EP 3926512 A1 EP3926512 A1 EP 3926512A1 EP 21161686 A EP21161686 A EP 21161686A EP 3926512 A1 EP3926512 A1 EP 3926512A1
Authority
EP
European Patent Office
Prior art keywords
model
semantic
improved
initial
improved model
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
EP21161686.7A
Other languages
English (en)
French (fr)
Inventor
Xuyi Chen
Shiwei HUANG
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Baidu Netcom Science and Technology Co Ltd
Original Assignee
Beijing Baidu Netcom Science and Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Baidu Netcom Science and Technology Co Ltd filed Critical Beijing Baidu Netcom Science and Technology Co Ltd
Publication of EP3926512A1 publication Critical patent/EP3926512A1/de
Withdrawn legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/30—Semantic analysis
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/31—Indexing; Data structures therefor; Storage structures
    • G06F16/316—Indexing structures
    • G06F16/325—Hash tables
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/36—Creation of semantic tools, e.g. ontology or thesauri
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N5/00—Computing arrangements using knowledge-based models
    • G06N5/02—Knowledge representation; Symbolic representation

Definitions

  • Embodiments of the present disclosure relate to the field of computer technology, in particular, to the technical fields of natural language processing and deep learning, and more in particular, to a method and apparatus for improving a model based on a pre-trained semantic model.
  • pre-trained semantic models is a development trend in the field of natural language processing.
  • current pre-trained semantic models are generally overly larger in parameter scale and complex in calculation, which makes them difficult to be deployed in a production environment.
  • a model distillation technique, a quantitative clipping technique and the like are generally used to compress the models to increase the processing speed of the models.
  • the compression ratio and processing speed of the compressed semantic models obtained based on the model distillation technique or the quantitative clipping technique need to be improved.
  • the present disclosure provides a method, apparatus, device and storage medium for improving a model based on a pre-trained semantic model.
  • the present disclosure provides a method for improving a model based on a pre-trained semantic model, and the method includes: based on the pre-trained semantic model, obtaining an initial improved model, where semantic result information of an input vector is determined in the initial improved model based on a hash search method; and based on a model distillation method, training the initial improved model to obtain an improved model.
  • the present disclosure provides a method for improving a model based on a pre-trained semantic model
  • the apparatus includes: an improvement unit configured to, based on the pre-trained semantic model, obtain an initial improved model, where semantic result information of an input vector is determined in the initial improved model based on a hash search method; and a training unit configured to, based on a model distillation method, train the initial improved model to obtain an improved model.
  • the present disclosure provides an electronic device, the device includes: at least one processor; and a memory communicating with the at least one processor, where the memory stores instructions executable by the at least one processor, and the instructions, when executed by the at least one processor, cause the at least one processor to execute the method according to any implementation of the first aspect.
  • the present disclosure provides a non-transitory computer readable storage medium storing computer instructions, where the computer instructions cause a computer to execute the method according to any implementation of the first aspect.
  • a computer program when executed by a computer, causes the computer to perform the method according to any implementation of the first aspect.
  • the semantic result information of the input vector is obtained by performing the hash search method on the input vector, and the complex iterative calculation process of a original semantic model is replaced, and the improved model with few model parameters and high compression ratio is obtained, and the processing speed of the improved model is improved.
  • the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105.
  • the network 104 serves as a medium for providing a communication link between the terminal devices 101, 102, 103 and the server 105.
  • the network 104 may include various types of connections, such as wired or wireless communication links, or optical fiber cables.
  • the terminal devices 101, 102 and 103 may be hardware or software supporting network connections for data interaction and data processing.
  • the terminal devices 101, 102 and 103 may be various electronic devices supporting functions such as information exchange, network connections and information processing, including but not limited to a smart phone, a tablet computer, an electronic book reader, a laptop portable computer, a desktop computer and the like.
  • the terminal devices 101, 102 and 103 are software, the terminal devices 101, 102 and 103 may be installed in the electronic devices, and may be implemented as multiple software pieces or software modules (such as for providing distributed services), or as a single software piece or software module. It is not specifically limited herein.
  • the server 105 may be a server providing various services, such as a background processing server improving and training the model based on a pre-trained semantic model sent by the terminal devices 101, 102, 103.
  • the background processing server may obtain an initial improved model based on the pre-trained semantic model and train the initial improved model to obtain an improved model.
  • the background processing server may feedback the improved model to the terminal devices for use by the terminal devices.
  • the server 105 may be a cloud server.
  • the server may be hardware or software.
  • the server may be implemented as a distributed server cluster composed of multiple servers, or as a single server.
  • the server may be implemented as multiple software pieces or software modules (such as for providing distributed services), or as a single software piece or software module. It is not specifically limited herein.
  • the method for improving the model based on the pre-trained semantic model provided by the embodiments of the present disclosure may be executed by the server or the terminal devices or the server and the terminal devices in cooperation with each other.
  • parts (such as units, sub-units, modules and sub-modules) included in the information processing apparatus may be all arranged in the terminal devices, or may be arranged in the server and the terminal devices respectively.
  • the number of the terminal devices, the network and the server in Fig. 1 is merely illustrative. Any number of terminal devices, networks and servers may be provided according to actual requirements.
  • the system architecture may only include the electronic device adapted to execute the method for improving the model based on the pre-trained semantic model (such as the server or the terminal devices).
  • a flow 200 of an embodiment of the method for improving the model based on the pre-trained semantic model is shown, and the flow 200 includes the following steps 201 to 202.
  • Step 201 includes based on the pre-trained semantic model, obtaining an initial improved model.
  • the execution body of the method for improving the model based on the pre-trained semantic model may: obtain the initial improved model based on the pre-trained semantic model, where semantic result information of an input vector is determined in the initial improved model based on a hash search method; and train the initial improved model based on a model distillation method to obtain an improved model.
  • the pre-trained semantic model is a semantic recognition model obtained by training based on large training data, and is configured to represent a corresponding relationship between a word or a sentence represented by an input vector and a semantic recognition result.
  • a complex large-scale model is often trained in order to obtain good performance.
  • the large-scale model obtained by training determines the semantic recognition result of the input vector through a complex iterative calculation process based on the determined and numerous model parameters.
  • factors such as real-time performance and calculation amount, do not need to be considered.
  • a small and refined model is easy to deploy, and many factors, such as the size, calculation complexity and speed of the model need to be considered.
  • the executive body may improve the pre-trained semantic model. For example, the execution body may obtain the semantic result information of the input vector by performing the hash search on the input vector and replace the original complex iterative calculation process of a semantic model with the hash search process. Specifically, the execution body may deploy a hash storage module storing large amount of semantic information. The hash storage module may be deployed together with the improved model, or may be deployed separately, which is not limited herein.
  • the input vector is used as an independent variable, and a corresponding function value (a hash address) is calculated through a certain function relationship (a hash function), and the function value is used as an address of a data element, and the data element is stored in a storage unit of the corresponding address.
  • the input vector is used as an independent variable, and a corresponding function value (a hash address) is calculated through a certain function relationship (a hash function), and semantic information stored in the hash address is used as the semantic result information of the input vector.
  • the execution body obtains a transformation vector of the input vector based on a fully connected layer; then, according to the hash search method, a target position corresponding to the transformation vector is determined in the hash storage module storing the semantic information, and the semantic information stored in the target position is used as the semantic result information of the input vector.
  • the vector transformation process and the hash search process of the fully connected layer are used to replace the original complex iterative calculation process of the semantic model, so that the model can be trained with few model parameters and high compression ratio; and compared with the complex iterative calculation process, the vector transformation process and the hash search process of the fully connected layer improve the processing speed of the input vector.
  • the initial improved model obtained based on the pre-trained semantic model is an untrained model, and the semantic result information obtained in the above operation process may be erroneous semantic result information, and the initial improved model needs to be trained through subsequent training steps.
  • Step 202 includes, based on a model distillation method, training the initial improved model to obtain an improved model.
  • the execution body may train the initial improved model obtained in step 201 based on the model distillation method to obtain the trained improved model.
  • the output result of a preset teacher network model replace true training data to a certain extent to guide the learning of the initial improved model used as a student network model, thereby encouraging the student network model to approximate the input and output vectors of the teacher model.
  • the execution body uses the pre-trained semantic model as the teacher network model to train the initial improved model used as the student network model to obtain the improved model.
  • the semantic model used as the teacher network model is a network model that is pre-trained and has good performance.
  • the training of the initial improved model used as the student network model includes two objectives. One is the original objective function, which is the cross entropy between the output result of the initial improved model and the true value of the training data tag, and the other is an improved objective function, which is the cross entropy between the output result of the initial improved model and the output result of the semantic model.
  • the improved objective function has a higher entropy, which can provide more information than the original objective function, and therefore, by using the improved objective function, less data and a greater learning rate are used during the training of the initial improved model.
  • the execution body may use the weight average of the original objective function and the improved objective function as the objective function of the student network model, and the weight of the improved objective function may be larger.
  • the training of the student network model is supervised at a neuron level by using the teacher network model with good performance, and the usage rate of model parameters and the training speed of the improved model are improved.
  • the pre-trained semantic model is the pre-trained semantic model obtained based on the model distillation method.
  • a semantic model is a semantic model obtained by model transfer based on a corresponding teacher network model.
  • the pre-trained semantic model has been compressed based on the model distillation method, and on this basis, the semantic model is further improved, thereby further improving the refinement degree of the improved model.
  • Fig. 3 is a schematic diagram of an application scenario of the method for improving the model based on the pre-trained semantic model according to the present disclosure.
  • the operation process of the semantic model 302 includes an input operation, an iterative calculation operation and an output operation.
  • the server 301 obtains an initial improved model 303 based on the pre-trained semantic model, where the operation process of the initial improved model 303 includes an input operation, a search operation and an output operation, that is, in the initial improved model 303, the semantic result information of the input vector is determined based on the hash search method, and the initial improved model 303 is then trained based on the model distillation method to obtain an improved model 304.
  • the semantic result information of the input vector is obtained by performing the hash search method on the input vector, and the original complex iterative calculation process of the semantic model is replaced, and the improved model with few model parameters and high compression ratio is obtained, and the processing speed of the improved model is improved.
  • a flow 400 of another embodiment of the method for improving the model based on the pre-trained semantic model according to the present disclosure is shown, and the flow 400 includes the following steps 401 to 403.
  • Step 401 includes, based on the pre-trained semantic model, obtaining an initial improved model.
  • step 401 is substantially identical to step 201 in the corresponding embodiment of Fig. 2 , and details are not described herein.
  • Step 402 includes, based on a model distillation method, training the initial improved model to obtain an improved model.
  • step 402 is substantially identical to step 202 in the corresponding embodiment of Fig. 2 , and details are not described herein.
  • Step 403 includes inputting a to-be-recognized vector into the improved model to obtain semantic result information of the to-be-recognized vector.
  • the execution body of the method for improving the model based on the pre-trained semantic model may input the to-be-recognized vector into the trained improved model to obtain the semantic result information of the to-be-recognized vector.
  • the to-be-recognized vector is used to represent a word, phrase or sentence corresponding to the to-be-recognized vector.
  • the word, phrase or sentence in this embodiment may be any word, phrase or sentence.
  • the execution body may process an input sentence to obtain the to-be-recognized vector of words or phrases in the sentence.
  • the execution body For the to-be-recognized vector, the execution body first obtains a transformation vector of the input vector based on a fully connected layer in the improved model; then, according to the hash search method, a target position corresponding to the transformation vector is determined in a hash storage module storing semantic information, and the semantic information stored in the target position is used as the semantic result information of the to-be-recognized vector.
  • the improved model is a trained model, and accurate semantic result information of the to-be-recognized vector can be obtained based on the improved model.
  • the flow 400 of the method for improving the model based on the pre-trained semantic model in this embodiment inputs the to-be-recognized vector into the improved model to obtain the semantic result information of the to-be-recognized vector.
  • the solution described in this embodiment can obtain accurate semantic result information according to the to-be-recognized vector, thereby enriching the obtaining ways of semantic information of the to-be-recognized vector.
  • the present disclosure provides an embodiment of an apparatus for improving the model based on the pre-trained semantic model, which corresponds to the method embodiment shown in Fig. 2 .
  • the apparatus embodiment may include the same or corresponding features as the method embodiment shown in Fig. 2 , and produce the same or corresponding effects as the method embodiment shown in Fig. 2 .
  • the apparatus may be specifically applicable to various electronic devices.
  • the apparatus for improving the model based on the pre-trained semantic model of this embodiment includes: an improvement unit 501 configured to, based on the pre-trained semantic model, obtain an initial improved model, where semantic result information of an input vector is determined in the initial improved model based on a hash search method; and a training unit 502 configured to, based on a model distillation method, train the initial improved model to obtain an improved model.
  • the determining semantic result information of an input vector in the initial improved model based on a hash search method includes: based on a fully connected layer, obtaining a transformation vector of the input vector in the initial improved model; and determining a target position corresponding to the transformation vector in a hash storage module storing semantic information according to the hash search method, and using semantic information stored in the target position as the semantic result information of the input vector.
  • the pre-trained semantic model is the pre-trained semantic model obtained based on the model distillation method.
  • the training unit 502 is further configured to use the semantic model as a teacher network model to train the initial improved model used as a student network model to obtain the improved model.
  • the apparatus further includes an operation unit (not shown) configured to input a to-be-recognized vector into the improved model to obtain semantic result information of the to-be-recognized vector.
  • an operation unit (not shown) configured to input a to-be-recognized vector into the improved model to obtain semantic result information of the to-be-recognized vector.
  • the improvement unit based on the pre-trained semantic model, obtains the initial improved model, where the semantic result information of the input vector is determined in the initial improved model based on the hash search method, and the training unit, based on the model distillation method, trains the initial improved model to obtain the improved model, such that the semantic result information of the input vector is obtained by performing the hash search method on the input vector, and the original complex iterative calculation process of the semantic model is replaced, and the improved model with few model parameters and high compression ratio is obtained, and the processing speed of the improved model is improved.
  • the present disclosure further provides an electronic device and a readable storage medium.
  • Fig. 6 is a block diagram of an electronic device adapted to execute the method for improving the model based on the pre-trained semantic model according to an embodiment of the present disclosure.
  • the electronic device is intended to represent various forms of digital computers, such as laptops, desktops, worktables, personal digital assistants, servers, blade servers, mainframe computers and other suitable computers.
  • the electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices.
  • the parts, their connections and relationships, and their functions shown herein are examples only, and are not intended to limit the implementations of the present disclosure as described and/or claimed herein.
  • the electronic device includes one or more processors 601, a memory 602, and interfaces for connecting components, including a high-speed interface and a low-speed interface.
  • the components are interconnected by using different buses and may be mounted on a common motherboard or otherwise as required.
  • the processor may process instructions executed within the electronic device, including instructions stored in memory or on memory to display graphical information of the GUI on an external input or output device (such as a display device coupled to an interface).
  • multiple processors and/or multiple buses and multiple memories may be used with multiple memories, if required.
  • multiple electronic devices may be connected (for example, used as a server array, a set of blade servers or a multiprocessor system), and each electronic device provides some of the necessary operations.
  • An example of a processor 601 is shown in Fig. 6 .
  • the memory 602 is a non-transitory computer readable storage medium according to the present disclosure.
  • the memory stores instructions executable by at least one processor to cause the at least one processor to execute the method for improving the model based on the pre-trained semantic model according to the present disclosure.
  • the non-transitory computer readable storage medium of the present disclosure stores computer instructions for causing a computer to execute the method for improving the model based on the pre-trained semantic model according to the present disclosure.
  • the memory 602 may be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as the program instructions or modules corresponding to the method for improving the model based on the pre-trained semantic model in the embodiment of the present disclosure (such as the improvement unit 501 and the training unit 502 shown in Fig. 5 ).
  • the processor 601 runs the non-transitory software programs, instructions and modules stored in the memory 602 to execute various functional applications and data processing of the server, thereby implementing the method for improving the model based on the pre-trained semantic model in the method embodiment.
  • the memory 602 may include a storage program area and a storage data area, where the storage program area may store an operating system and an application program required by at least one function; and the storage data area may store data created by the electronic device when executing the method for improving the model based on the pre-trained semantic model.
  • the memory 602 may include a high-speed random access memory, and may further include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory or other non-transitory solid state storage devices.
  • the memory 602 may alternatively include a memory disposed remotely relative to the processor 601, which may be connected through a network to the electronic device adapted to execute the method for improving the model based on the pre-trained semantic model. Examples of such networks include, but are not limited to, the Internet, enterprise intranets, local area networks, mobile communication networks and combinations thereof.
  • the electronic device adapted to execute the method for improving the model based on the pre-trained semantic model may further include an input device 603 and an output device 604.
  • the processor 601, the memory 602, the input device 603 and the output device 604 may be interconnected through a bus or other means, and an example of a connection through a bus is shown in Fig. 6 .
  • the input device 603 may receive input number or character information, and generate key signal input related to user settings and functional control of the electronic device adapted to execute the method for improving the model based on the pre-trained semantic model, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, a pointer bar, one or more mouse buttons, a trackball or a joystick.
  • the output device 604 may include a display device, an auxiliary lighting device (such as an LED) and a tactile feedback device (such as a vibration motor).
  • the display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display and a plasma display. In some embodiments, the display device may be a touch screen.
  • the various embodiments of the systems and technologies described herein may be implemented in digital electronic circuit systems, integrated circuit systems, ASICs (application specific integrated circuits), computer hardware, firmware, software and/or combinations thereof.
  • the various embodiments may include: being implemented in one or more computer programs, where the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, and the programmable processor may be a dedicated or general-purpose programmable processor, which may receive data and instructions from a memory system, at least one input device and at least one output device, and send the data and instructions to the memory system, the at least one input device and the at least one output device.
  • machine readable medium and “computer readable medium” refer to any computer program product, device and/or apparatus (such as magnetic disk, optical disk, memory and programmable logic device (PLD)) for providing machine instructions and/or data to a programmable processor, including a machine readable medium that receives machine instructions as machine readable signals.
  • machine readable signal refers to any signal used to provide machine instructions and/or data to a programmable processor.
  • the systems and technologies described herein may be implemented on a computer having: a display device (such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (such as a mouse or a trackball) through which the user may provide input to the computer.
  • a display device such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor
  • a keyboard and a pointing device such as a mouse or a trackball
  • Other types of devices may also be used to provide interaction with the user.
  • the feedback provided to the user may be any form of sensory feedback (such as visual feedback, auditory feedback or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input or tactile input.
  • the systems and technologies described herein may be implemented in: a computing system including a background component (such as a data server), or a computing system including a middleware component (such as an application server), or a computing system including a front-end component (such as a user computer having a graphical user interface or a web browser through which the user may interact with the implementation of the systems and technologies described herein), or a computing system including any combination of such background component, middleware component or front-end component.
  • the components of the system may be interconnected by any form or medium of digital data communication (such as a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN) and the Internet.
  • LAN local area network
  • WAN wide area network
  • the Internet the global information network
  • the computer system may include a client and a server.
  • the client and the server are generally remote from each other and interact generally through a communication network.
  • the relationship between the client and the server is generated by running the computer programs having a client-server relationship with each other on the corresponding computer.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Artificial Intelligence (AREA)
  • Data Mining & Analysis (AREA)
  • Computational Linguistics (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Mathematical Physics (AREA)
  • Computing Systems (AREA)
  • Biophysics (AREA)
  • Molecular Biology (AREA)
  • Biomedical Technology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Databases & Information Systems (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Medical Informatics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Machine Translation (AREA)
EP21161686.7A 2020-06-17 2021-03-10 Verfahren und vorrichtung zur modellverbesserung basierend auf einem im voraus trainierten semantischen modell Withdrawn EP3926512A1 (de)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010555885.7A CN111709252B (zh) 2020-06-17 2020-06-17 基于预训练的语义模型的模型改进方法及装置

Publications (1)

Publication Number Publication Date
EP3926512A1 true EP3926512A1 (de) 2021-12-22

Family

ID=72540956

Family Applications (1)

Application Number Title Priority Date Filing Date
EP21161686.7A Withdrawn EP3926512A1 (de) 2020-06-17 2021-03-10 Verfahren und vorrichtung zur modellverbesserung basierend auf einem im voraus trainierten semantischen modell

Country Status (5)

Country Link
US (1) US11775766B2 (de)
EP (1) EP3926512A1 (de)
JP (1) JP7269972B2 (de)
KR (1) KR20210036875A (de)
CN (1) CN111709252B (de)

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111709252B (zh) * 2020-06-17 2023-03-28 北京百度网讯科技有限公司 基于预训练的语义模型的模型改进方法及装置
CN112527127B (zh) * 2020-12-23 2022-01-28 北京百度网讯科技有限公司 输入法长句预测模型的训练方法、装置、电子设备及介质
US11823490B2 (en) * 2021-06-08 2023-11-21 Adobe, Inc. Non-linear latent to latent model for multi-attribute face editing
CN113408265B (zh) * 2021-06-22 2023-01-17 平安科技（深圳）有限公司 基于人机交互的语义解析方法、装置、设备及存储介质
CN115168537B (zh) * 2022-06-30 2023-06-27 北京百度网讯科技有限公司 语义检索模型的训练方法、装置、电子设备及存储介质

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2018126213A1 (en) * 2016-12-30 2018-07-05 Google Llc Multi-task learning using knowledge distillation
US20200034703A1 (en) * 2018-07-27 2020-01-30 International Business Machines Corporation Training of student neural network with teacher neural networks

Family Cites Families (24)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8429174B2 (en) * 2003-01-25 2013-04-23 Purdue Research Foundation Methods, systems, and data structures for performing searches on three dimensional objects
US7594258B2 (en) * 2005-06-27 2009-09-22 Yahoo! Inc. Access control systems and methods using visibility tokens with automatic propagation
JP2008084068A (ja) 2006-09-28 2008-04-10 Toshiba Corp 現場作業実施状況管理装置及び管理方法
US8442823B2 (en) * 2010-10-19 2013-05-14 Motorola Solutions, Inc. Methods for creating and searching a database of speakers
JP2013206187A (ja) 2012-03-28 2013-10-07 Fujitsu Ltd 情報変換装置、情報検索装置、情報変換方法、情報検索方法、情報変換プログラム、情報検索プログラム
WO2016179419A1 (en) * 2015-05-05 2016-11-10 Kyndi, Inc. Quanton representation for emulating quantum-like computation on classical processors
US10540611B2 (en) * 2015-05-05 2020-01-21 Retailmenot, Inc. Scalable complex event processing with probabilistic machine learning models to predict subsequent geolocations
JP6924571B2 (ja) * 2016-11-11 2021-08-25 ヤフー株式会社 情報処理装置、情報処理方法、および情報処理プログラム
US10109275B2 (en) * 2016-12-19 2018-10-23 Asapp, Inc. Word hash language model
CN107092661A (zh) * 2017-03-28 2017-08-25 桂林明辉信息科技有限公司 一种基于深度卷积神经网络的图像检索方法
US10565562B2 (en) * 2017-07-25 2020-02-18 Microsoft Technology Licensing, Llc Hashing query and job posting features for improved machine learning model performance
CN107730503B (zh) * 2017-09-12 2020-05-26 北京航空航天大学 三维特征嵌入的图像对象部件级语义分割方法与装置
US10491697B2 (en) * 2018-02-15 2019-11-26 Cognant Llc System and method for bot detection
US11138520B2 (en) * 2018-06-28 2021-10-05 International Business Machines Corporation Ranking and updating machine learning models based on data inputs at edge nodes
CN109933682B (zh) * 2019-01-11 2022-01-04 上海交通大学 一种基于语义与内容信息结合的图像哈希检索方法及系统
CN109977250B (zh) * 2019-03-20 2023-03-28 重庆大学 融合语义信息和多级相似性的深度哈希图像检索方法
CN110059740A (zh) * 2019-04-12 2019-07-26 杭州电子科技大学 一种针对嵌入式移动端的深度学习语义分割模型压缩方法
CN110134761A (zh) * 2019-04-16 2019-08-16 深圳壹账通智能科技有限公司 判决文书信息检索方法、装置、计算机设备和存储介质
CN118349673A (zh) * 2019-09-12 2024-07-16 华为技术有限公司 文本处理模型的训练方法、文本处理方法及装置
CN110795939A (zh) * 2019-10-15 2020-02-14 腾讯科技（深圳）有限公司 文本处理方法、装置
CN111104482A (zh) 2019-12-18 2020-05-05 北京百度网讯科技有限公司 数据处理方法和装置
CN111709252B (zh) * 2020-06-17 2023-03-28 北京百度网讯科技有限公司 基于预训练的语义模型的模型改进方法及装置
TWI746038B (zh) * 2020-07-02 2021-11-11 阿證科技股份有限公司 類神經網路人工智慧決策核心系統
US11699044B1 (en) * 2022-10-31 2023-07-11 Todd Allen Apparatus and methods for generating and transmitting simulated communication

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2018126213A1 (en) * 2016-12-30 2018-07-05 Google Llc Multi-task learning using knowledge distillation
US20200034703A1 (en) * 2018-07-27 2020-01-30 International Business Machines Corporation Training of student neural network with teacher neural networks

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
GOU JIANPING ET AL: "Knowledge Distillation: A Survey", 9 June 2020 (2020-06-09), pages 1 - 30, XP055830086, Retrieved from the Internet <URL:https://arxiv.org/pdf/2006.05525v1.pdf> [retrieved on 20210804], DOI: 10.1007/s11263-021-01453-z *
XIPENG QIU ET AL: "Pre-trained Models for Natural Language Processing: A Survey", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, 18 March 2020 (2020-03-18), XP081652050 *

Also Published As

Publication number Publication date
US20210397794A1 (en) 2021-12-23
US11775766B2 (en) 2023-10-03
JP7269972B2 (ja) 2023-05-09
CN111709252A (zh) 2020-09-25
KR20210036875A (ko) 2021-04-05
JP2021197156A (ja) 2021-12-27
CN111709252B (zh) 2023-03-28

Similar Documents

Publication Publication Date Title
US20210390428A1 (en) Method, apparatus, device and storage medium for training model
US11775766B2 (en) Method and apparatus for improving model based on pre-trained semantic model
US11403468B2 (en) Method and apparatus for generating vector representation of text, and related computer device
US11727216B2 (en) Method, apparatus, device, and storage medium for linking entity
EP3916614A1 (de) Verfahren und vorrichtung zum trainieren eines sprachmodells, elektronische vorrichtung, lesbares speichermedium und computerprogrammprodukt
EP3828719A2 (de) Verfahren und vorrichtung zur erzeugung von modellen zur darstellung von heterogenen graphknoten, elektronische vorrichtung, speichermedium und computerprogrammprodukt
CN111104514B (zh) 文档标签模型的训练方法及装置
CN111414482B (zh) 一种事件论元抽取方法、装置以及电子设备
US20210374343A1 (en) Method and apparatus for obtaining word vectors based on language model, device and storage medium
EP3852000A1 (de) Verfahren und vorrichtung zur verarbeitung der semantischen beschreibung einer texteinheit, vorrichtung und speichermedium
US20210200813A1 (en) Human-machine interaction method, electronic device, and storage medium
JP2022018095A (ja) マルチモーダル事前訓練モデル取得方法、装置、電子デバイス及び記憶媒体
US20210383233A1 (en) Method, electronic device, and storage medium for distilling model
KR20210157342A (ko) 언어 모델의 훈련 방법, 장치, 전자 기기 및 판독 가능 기록 매체
US11216615B2 (en) Method, device and storage medium for predicting punctuation in text
US11735168B2 (en) Method and apparatus for recognizing voice
US11200382B2 (en) Prosodic pause prediction method, prosodic pause prediction device and electronic device
EP3799036A1 (de) Sprachsteuerungsverfahren, sprachsteuerungsvorrichtung, elektronische vorrichtung und lesbares speichermedium
CN112329429B (zh) 文本相似度学习方法、装置、设备以及存储介质
US11900918B2 (en) Method for training a linguistic model and electronic device
EP3958183A1 (de) Deep learning modell anpassungsverfahren und geräte und elektronische geräte
CN115688796B (zh) 用于自然语言处理领域中预训练模型的训练方法及其装置

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20210310

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

B565 Issuance of search results under rule 164(2) epc

Effective date: 20210813

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20220825

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION IS DEEMED TO BE WITHDRAWN

18D Application deemed to be withdrawn

Effective date: 20230105