GB2625476A - Path dropout for natural language processing - Google Patents

Path dropout for natural language processing Download PDF

Info

Publication number
GB2625476A
GB2625476A GB2404323.4A GB202404323A GB2625476A GB 2625476 A GB2625476 A GB 2625476A GB 202404323 A GB202404323 A GB 202404323A GB 2625476 A GB2625476 A GB 2625476A
Authority
GB
United Kingdom
Prior art keywords
learning model
machine
connection
dropout
attention
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
GB2404323.4A
Other languages
English (en)
Other versions
GB202404323D0 (en
Inventor
Tien Vu Thanh
Quang Pham Tuyen
Edward Johnson Mark
Long Duong Thanh
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Oracle International Corp
Original Assignee
Oracle International Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Oracle International Corp filed Critical Oracle International Corp
Publication of GB202404323D0 publication Critical patent/GB202404323D0/en
Publication of GB2625476A publication Critical patent/GB2625476A/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/06—Creation of reference templates; Training of speech recognition systems, e.g. adaptation to the characteristics of the speaker's voice
    • G10L15/063—Training
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/082—Learning methods modifying the architecture, e.g. adding, deleting or silencing nodes or connections
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/20—Natural language analysis
    • G06F40/279—Recognition of textual entities
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/30—Semantic analysis
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/044—Recurrent networks, e.g. Hopfield networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/0985—Hyperparameter optimisation; Meta-learning; Learning-to-learn
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N7/00—Computing arrangements based on specific mathematical models
    • G06N7/01—Probabilistic graphical models, e.g. probabilistic networks
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/08—Speech classification or search
    • G10L15/18—Speech classification or search using natural language modelling
    • G10L15/1815—Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/22—Procedures used during a speech recognition process, e.g. man-machine dialogue

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Artificial Intelligence (AREA)
  • General Physics & Mathematics (AREA)
  • Computational Linguistics (AREA)
  • Health & Medical Sciences (AREA)
  • General Engineering & Computer Science (AREA)
  • General Health & Medical Sciences (AREA)
  • Software Systems (AREA)
  • Evolutionary Computation (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Data Mining & Analysis (AREA)
  • Biomedical Technology (AREA)
  • Molecular Biology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biophysics (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Acoustics & Sound (AREA)
  • Human Computer Interaction (AREA)
  • Multimedia (AREA)
  • Probability & Statistics with Applications (AREA)
  • Algebra (AREA)
  • Computational Mathematics (AREA)
  • Mathematical Analysis (AREA)
  • Mathematical Optimization (AREA)
  • Pure & Applied Mathematics (AREA)
  • Machine Translation (AREA)
GB2404323.4A 2021-11-17 2022-11-16 Path dropout for natural language processing Pending GB2625476A (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US202163280580P 2021-11-17 2021-11-17
PCT/US2022/050076 WO2023091468A1 (en) 2021-11-17 2022-11-16 Path dropout for natural language processing

Publications (2)

Publication Number Publication Date
GB202404323D0 GB202404323D0 (en) 2024-05-08
GB2625476A true GB2625476A (en) 2024-06-19

Family

ID=86323954

Family Applications (1)

Application Number Title Priority Date Filing Date
GB2404323.4A Pending GB2625476A (en) 2021-11-17 2022-11-16 Path dropout for natural language processing

Country Status (6)

Country Link
US (1) US12412563B2 (https=)
JP (1) JP2024543062A (https=)
KR (1) KR20240111760A (https=)
CN (1) CN118235143A (https=)
GB (1) GB2625476A (https=)
WO (1) WO2023091468A1 (https=)

Families Citing this family (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US12572852B2 (en) 2021-12-23 2026-03-10 Oracle International Corporation Lexical dropout for natural language processing
US12566921B2 (en) * 2021-12-23 2026-03-03 Oracle International Corporation Gazetteer integration for neural named entity recognition
US12493918B2 (en) * 2022-01-21 2025-12-09 Walmart Apollo, Llc Systems and methods for dispute resolution
US12554934B2 (en) * 2022-10-13 2026-02-17 Oracle International Corporation Multi-task model with context masking
US12530545B2 (en) 2022-10-13 2026-01-20 Oracle International Corporation Data augmentation and batch balancing for training multi-lingual model
US12554518B2 (en) * 2023-08-04 2026-02-17 Dell Products L.P. Smart input devices with user behavior predictions
CN118968975A (zh) * 2024-07-18 2024-11-15 安徽海轩教育科技有限公司 人工智能助理系统

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110349676A (zh) * 2019-06-14 2019-10-18 华南师范大学 时序生理数据分类方法、装置、存储介质和处理器
CN113139585A (zh) * 2021-03-30 2021-07-20 太原科技大学 一种基于统一多尺度密集连接网络的红外与可见光图像融合方法

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11138392B2 (en) * 2018-07-26 2021-10-05 Google Llc Machine translation using neural network models
US11922290B2 (en) * 2021-05-24 2024-03-05 Visa International Service Association System, method, and computer program product for analyzing multivariate time series using a convolutional Fourier network
US12417372B2 (en) * 2021-06-15 2025-09-16 Riiid Inc. System for predicting user drop-out rate and tracking user knowledge based on artificial intelligence learning and method therefor
US20220405575A1 (en) * 2021-06-18 2022-12-22 Riiid Inc. System for predicting user drop-out rate based on artificial intelligence learning and method therefor

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110349676A (zh) * 2019-06-14 2019-10-18 华南师范大学 时序生理数据分类方法、装置、存储介质和处理器
CN113139585A (zh) * 2021-03-30 2021-07-20 太原科技大学 一种基于统一多尺度密集连接网络的红外与可见光图像融合方法

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
arsson Gustav, Maire Michael, Shakhnarovich Gregory, "Fractalnet: Ultra-deep neural networks without residuals", 5th International Conference on Learning Representations (ICLR) 2017, doi:10.48550/arXiv.1605.07648, (20170401), pages 1 - 11, 5th International Conference on Learning Representations (IC *
Wangchunshu Zhou; Tao Ge; Ke Xu; Furu Wei; Ming Zhou, "Scheduled DropHead: A Regularization Method for Transformer Models", arXiv.org, Cornell University Library, 201 Olin Library Cornell University Ithaca, NY 14853, 201 Olin Library Cornell University Ithaca, NY 14853 , (20200428), XP081654072 [Y] *
Zhen Wu; Lijun Wu; Qi Meng; Yingce Xia; Shufang Xie; Tao Qin; Xinyu Dai; Tie-Yan Liu, "UniDrop: A Simple yet Effective Technique to Improve Transformer without Extra Cost", arXiv.org, Cornell University Library, 201 Olin Library Cornell University Ithaca, NY 14853, 201 Olin Library Cornell Universit *

Also Published As

Publication number Publication date
US12412563B2 (en) 2025-09-09
GB202404323D0 (en) 2024-05-08
CN118235143A (zh) 2024-06-21
KR20240111760A (ko) 2024-07-17
US20230154455A1 (en) 2023-05-18
JP2024543062A (ja) 2024-11-19
WO2023091468A1 (en) 2023-05-25

Similar Documents

Publication Publication Date Title
GB2625476A (en) Path dropout for natural language processing
AU2020250312B2 (en) Batch normalization layers
KR102656006B1 (ko) 심층 다항식 네트워크를 사용하는 암호화된 데이터의 분산 및 협업 분석
US20230360640A1 (en) Keyword-based dialogue summarizer
JP2024543062A5 (https=)
US11070673B1 (en) Call monitoring and feedback reporting using machine learning
US11282503B2 (en) Voice conversion training method and server and computer readable storage medium
US11647116B2 (en) Automated agent behavior recommendations for call quality improvement
CN107452385A (zh) 一种基于语音的数据评价方法及装置
GB2622755A (en) Evaluating output sequences using an auto-regressive language model neural network
CN111326168B (zh) 语音分离方法、装置、电子设备和存储介质
CN108986798B (zh) 语音数据的处理方法、装置及设备
CN107240395A (zh) 一种声学模型训练方法和装置、计算机设备、存储介质
WO2014035738A1 (en) Computer-implemented deep tensor neural network
US20180005626A1 (en) Obfuscating training data
CN111460303B (zh) 数据处理方法、装置、电子设备及计算机可读存储介质
Dominic et al. Onboarding bot for newcomers to software engineering
CN113889149A (zh) 语音情感识别方法及装置
US20250117579A1 (en) Generated automated agent instructions from user training materials
US12217762B1 (en) System and method for detecting synthetic speech based on prosody analysis
CN112863518A (zh) 一种语音数据主题识别的方法及装置
KR102731396B1 (ko) 언어 이해 알고리즘 학습 시스템 및 방법
KR20230149172A (ko) 문서 기계 독해 시스템 및 방법
CN113257237A (zh) 语音交互的意图识别方法、装置、电子设备及存储介质
CN115731920B (zh) 语音识别处理方法、装置及设备