WO2020257812A3 - Modeling dependencies with global self-attention neural networks - Google Patents

Modeling dependencies with global self-attention neural networks Download PDF

Info

Publication number
WO2020257812A3
WO2020257812A3 PCT/US2020/050995 US2020050995W WO2020257812A3 WO 2020257812 A3 WO2020257812 A3 WO 2020257812A3 US 2020050995 W US2020050995 W US 2020050995W WO 2020257812 A3 WO2020257812 A3 WO 2020257812A3
Authority
WO
WIPO (PCT)
Prior art keywords
attention
context
content
attention layer
positions
Prior art date
Application number
PCT/US2020/050995
Other languages
French (fr)
Other versions
WO2020257812A2 (en
Inventor
Zhuoran SHEN
Irwan BELLO
Xuhui JIA
Ching-Hui Chen
Raviteja Vemulapalli
Original Assignee
Google Llc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Google Llc filed Critical Google Llc
Priority to EP20781680.2A priority Critical patent/EP4154185A2/en
Priority to CN202080102596.XA priority patent/CN115885289A/en
Priority to PCT/US2020/050995 priority patent/WO2020257812A2/en
Priority to US18/044,842 priority patent/US20230359865A1/en
Publication of WO2020257812A2 publication Critical patent/WO2020257812A2/en
Publication of WO2020257812A3 publication Critical patent/WO2020257812A3/en

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2413—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on distances to training or reference patterns
    • G06F18/24133—Distances to prototypes
    • G06F18/24137—Distances to cluster centroïds
    • G06F18/2414—Smoothing the distance, e.g. radial basis function networks [RBFN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G06V10/443—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components by matching or filtering
    • G06V10/449—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters
    • G06V10/451—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters with interaction between the filter responses, e.g. cortical complex cells
    • G06V10/454—Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/044—Recurrent networks, e.g. Hopfield networks

Abstract

The present disclosure provides systems, methods, and computer program products for modeling dependencies throughout a network using a global-self attention model with a content attention layer and a positional attention layer that operate in parallel. The model receives input data comprising content values and context positions. The content attention layer generates one or more output features for each context position based on a global attention operation applied to the content values independent of the context positions. The positional attention layer generates an attention map for each of the context positions based on one or more content values of the respective context position and associated neighboring positions. Output is determined based on the output features generated by the content attention layer and the attention map generated for each context position by the positional attention layer. The model improves efficiency and can be used throughout a deep network.
PCT/US2020/050995 2020-09-16 2020-09-16 Modeling dependencies with global self-attention neural networks WO2020257812A2 (en)

Priority Applications (4)

Application Number Priority Date Filing Date Title
EP20781680.2A EP4154185A2 (en) 2020-09-16 2020-09-16 Modeling dependencies with global self-attention neural networks
CN202080102596.XA CN115885289A (en) 2020-09-16 2020-09-16 Modeling dependency with global self-attention neural networks
PCT/US2020/050995 WO2020257812A2 (en) 2020-09-16 2020-09-16 Modeling dependencies with global self-attention neural networks
US18/044,842 US20230359865A1 (en) 2020-09-16 2020-09-16 Modeling Dependencies with Global Self-Attention Neural Networks

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/US2020/050995 WO2020257812A2 (en) 2020-09-16 2020-09-16 Modeling dependencies with global self-attention neural networks

Publications (2)

Publication Number Publication Date
WO2020257812A2 WO2020257812A2 (en) 2020-12-24
WO2020257812A3 true WO2020257812A3 (en) 2021-07-29

Family

ID=72670816

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2020/050995 WO2020257812A2 (en) 2020-09-16 2020-09-16 Modeling dependencies with global self-attention neural networks

Country Status (4)

Country Link
US (1) US20230359865A1 (en)
EP (1) EP4154185A2 (en)
CN (1) CN115885289A (en)
WO (1) WO2020257812A2 (en)

Families Citing this family (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112883149B (en) * 2021-01-20 2024-03-26 华为技术有限公司 Natural language processing method and device
CN112802038B (en) * 2021-01-26 2022-05-24 桂林电子科技大学 Panorama segmentation method based on multi-scale edge attention
CN112802039B (en) * 2021-01-26 2022-03-01 桂林电子科技大学 Panorama segmentation method based on global edge attention
CN112949415B (en) * 2021-02-04 2023-03-24 北京百度网讯科技有限公司 Image processing method, apparatus, device and medium
CN113065550B (en) * 2021-03-12 2022-11-11 国网河北省电力有限公司 Text recognition method based on self-attention mechanism
CN113239981B (en) * 2021-04-23 2022-04-12 中国科学院大学 Image classification method of local feature coupling global representation
CN113159056B (en) * 2021-05-21 2023-11-21 中国科学院深圳先进技术研究院 Image segmentation method, device, equipment and storage medium
WO2023091925A1 (en) * 2021-11-16 2023-05-25 Qualcomm Incorporated Panoptic segmentation with panoptic, instance, and semantic relations
CN115035512B (en) * 2022-05-24 2023-04-18 合肥工业大学 Crop nutrition state diagnosis method and system based on multi-mode deep learning
CN116051810B (en) * 2023-03-30 2023-06-13 武汉纺织大学 Intelligent clothing positioning method based on deep learning
CN116644788B (en) * 2023-07-27 2023-10-03 山东交通学院 Local refinement and global reinforcement network for vehicle re-identification
CN116757369B (en) * 2023-08-22 2023-11-24 国网山东省电力公司营销服务中心(计量中心) Attention mechanism-based carbon emission analysis method and system

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111369543A (en) * 2020-03-07 2020-07-03 北京工业大学 Rapid pollen particle detection algorithm based on dual self-attention module

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111369543A (en) * 2020-03-07 2020-07-03 北京工业大学 Rapid pollen particle detection algorithm based on dual self-attention module

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
CAVERLEE JAMES ET AL: "Time Interval Aware Self-Attention for Sequential Recommendation", PROCEEDINGS OF THE 13TH INTERNATIONAL CONFERENCE ON WEB SEARCH AND DATA MINING, 3 February 2020 (2020-02-03), New York, NY, USA, pages 322 - 330, XP055811142, ISBN: 978-1-4503-6822-3, Retrieved from the Internet <URL:https://dl.acm.org/doi/pdf/10.1145/3336191.3371786> [retrieved on 20210604], DOI: 10.1145/3336191.3371786 *
HUIYU WANG ET AL: "Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, 6 August 2020 (2020-08-06), XP081735345 *
MOU LEI ET AL: "CS-Net: Channel and Spatial Attention Network for Curvilinear Structure Segmentation", 10 October 2019, ADVANCES IN INTELLIGENT DATA ANALYSIS XIX; [LECTURE NOTES IN COMPUTER SCIENCE; LECT.NOTES COMPUTER], SPRINGER INTERNATIONAL PUBLISHING, CHAM, PAGE(S) 721 - 730, ISBN: 978-3-030-71592-2, ISSN: 0302-9743, XP047522582 *

Also Published As

Publication number Publication date
EP4154185A2 (en) 2023-03-29
CN115885289A (en) 2023-03-31
US20230359865A1 (en) 2023-11-09
WO2020257812A2 (en) 2020-12-24

Similar Documents

Publication Publication Date Title
WO2020257812A3 (en) Modeling dependencies with global self-attention neural networks
WO2008142502A3 (en) Method for coordinating a plurality of sensors
RAMLI et al. Modeling undesirable factors in efficiency measurement using data envelopment analysis: A review
CN103942108A (en) Resource parameter optimization method under Hadoop homogenous cluster
CN104503847A (en) Data center energy saving method and device
Meddah et al. Parallel distributed patterns mining using hadoop mapreduce framework
Horváth Intelligent Content Driving of Engineering Model System in Modeling Platform.
Burmistrov et al. Project management life cycle models to improve management in high-rise construction
CN103092960A (en) Method for building software product feature tree model based on demand cluster
Kostromin et al. Service-oriented tools for automating digital twin development
WO2022117127A3 (en) Engineering forklift multi-objective performance optimization method based on deep surrogate model
Quan-Yin et al. A novel efficient adaptive sliding window model for week-ahead price forecasting
Horváth In the main stream of emerging engineering
Wang et al. Parallel Monte Carlo method with MapReduce for option pricing
Ren et al. Multibranch multilevel federated learning for a better feature extraction and a plug-and-play dynamic-adjusting double flow personalization approach
JP2010134926A (en) Multi-modal green computing fusion system, computer program, method, and developing method of system (multi-modal green computing fusion using problem analytics)
Schelén et al. A roadmap for big-data research and education
Hartenstein The paramountcy of reconfigurable computing
Fujisawa et al. Optimization in the real world
Fu et al. Coordinated development of energy-saving and emission-reduction evolution systems in the Yangtze River delta
Adaramola et al. OPEN ACCESS EDITED BY
Chashchina et al. COMPARISON OF SIMULATION PROGRAMS
Fulei et al. An Optimal Strategy for Global Carbon Reduction Using Evolutionary Game Theory: A Study Based on the Paris Agreement
Lam et al. Optimal schedules for multimodal transit services an activity-based approach
Shao et al. Logistics information management based on Internet of Things and data mining

Legal Events

Date Code Title Description
ENP Entry into the national phase

Ref document number: 2020781680

Country of ref document: EP

Effective date: 20221222

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20781680

Country of ref document: EP

Kind code of ref document: A2

NENP Non-entry into the national phase

Ref country code: DE