KR20250029051A - 특수 전문가 혼합 머신 러닝 모델의 계층을 분배하기 위한 시스템 및 방법 - Google Patents

특수 전문가 혼합 머신 러닝 모델의 계층을 분배하기 위한 시스템 및 방법 Download PDF

Info

Publication number
KR20250029051A
KR20250029051A KR1020247042066A KR20247042066A KR20250029051A KR 20250029051 A KR20250029051 A KR 20250029051A KR 1020247042066 A KR1020247042066 A KR 1020247042066A KR 20247042066 A KR20247042066 A KR 20247042066A KR 20250029051 A KR20250029051 A KR 20250029051A
Authority
KR
South Korea
Prior art keywords
accelerators
layers
sparse
experts
machine learning
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
KR1020247042066A
Other languages
English (en)
Korean (ko)
Inventor
데방쿠마르 라메쉬바이 파텔
웨이 주오
유안 유
Original Assignee
마이크로소프트 테크놀로지 라이센싱, 엘엘씨
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 마이크로소프트 테크놀로지 라이센싱, 엘엘씨 filed Critical 마이크로소프트 테크놀로지 라이센싱, 엘엘씨
Publication of KR20250029051A publication Critical patent/KR20250029051A/ko
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
    • G06N3/063—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using electronic means
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/042—Knowledge-based neural networks; Logical representations of neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/082—Learning methods modifying the architecture, e.g. adding, deleting or silencing nodes or connections
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T1/00—General purpose image data processing
    • G06T1/60—Memory management
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Software Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • Biophysics (AREA)
  • Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Mathematical Physics (AREA)
  • Computing Systems (AREA)
  • Computational Linguistics (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Neurology (AREA)
  • Medical Informatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Evolutionary Biology (AREA)
  • Complex Calculations (AREA)
KR1020247042066A 2022-06-24 2023-05-16 특수 전문가 혼합 머신 러닝 모델의 계층을 분배하기 위한 시스템 및 방법 Pending KR20250029051A (ko)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US17/848,679 US12579470B2 (en) 2022-06-24 2022-06-24 Systems and methods for distributing layers of special mixture-of-experts machine learning models
US17/848,679 2022-06-24
PCT/US2023/022447 WO2023249754A1 (en) 2022-06-24 2023-05-16 Systems and methods for distributing layers of special mixture-of-experts machine learning models

Publications (1)

Publication Number Publication Date
KR20250029051A true KR20250029051A (ko) 2025-03-04

Family

ID=87036771

Family Applications (1)

Application Number Title Priority Date Filing Date
KR1020247042066A Pending KR20250029051A (ko) 2022-06-24 2023-05-16 특수 전문가 혼합 머신 러닝 모델의 계층을 분배하기 위한 시스템 및 방법

Country Status (6)

Country Link
US (1) US12579470B2 (https=)
EP (1) EP4544451A1 (https=)
JP (1) JP2025522299A (https=)
KR (1) KR20250029051A (https=)
CN (1) CN119452368A (https=)
WO (1) WO2023249754A1 (https=)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117972293B (zh) * 2024-03-28 2024-06-07 北京思凌科半导体技术有限公司 基于混合专家模型的计算方法、装置、设备及存储介质

Family Cites Families (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2018085643A1 (en) * 2016-11-04 2018-05-11 Google Llc Mixture of experts neural networks
US10509846B2 (en) * 2017-12-13 2019-12-17 Intel Corporation Accelerator for processing data
US11893502B2 (en) 2017-12-20 2024-02-06 Advanced Micro Devices, Inc. Dynamic hardware selection for experts in mixture-of-experts model
EP3619654B1 (en) * 2018-07-23 2024-09-04 Google LLC Continuous parametrizations of neural network layer weights
US11003823B2 (en) 2018-08-09 2021-05-11 Palo Alto Research Center Incorporated Re-design of analog circuits
US20200117978A1 (en) 2018-10-12 2020-04-16 Alibaba Group Holding Limited Systems and methods for efficiently mapping neural networks to programmable logic devices
US20220230051A1 (en) 2018-11-18 2022-07-21 Innatera Nanosystems B.V. Spiking Neural Network
US12124941B2 (en) 2020-03-27 2024-10-22 Intel Corporation Methods and apparatus for dynamic batching of data for neural network workloads
US11586894B2 (en) 2020-05-04 2023-02-21 SiMa Technologies, Inc. Ordering computations of a machine learning network in a machine learning accelerator for efficient memory usage
US12530571B2 (en) 2020-07-08 2026-01-20 Nvidia Corporation Image generation using one or more neural networks
US20220036186A1 (en) * 2020-07-30 2022-02-03 Waymo Llc Accelerated deep reinforcement learning of agent control policies
US20220059200A1 (en) 2020-08-21 2022-02-24 Washington University Deep-learning systems and methods for medical report generation and anomaly detection
US12518135B2 (en) * 2021-02-05 2026-01-06 Google Llc Sparse and differentiable mixture of experts neural networks
US20230281510A1 (en) * 2022-03-07 2023-09-07 Qualcomm Incorporated Machine learning model architecture combining mixture of experts and model ensembling

Also Published As

Publication number Publication date
CN119452368A (zh) 2025-02-14
US12579470B2 (en) 2026-03-17
EP4544451A1 (en) 2025-04-30
US20230419166A1 (en) 2023-12-28
WO2023249754A1 (en) 2023-12-28
JP2025522299A (ja) 2025-07-15

Similar Documents

Publication Publication Date Title
Cordova et al. DBSCAN on resilient distributed datasets
CN110262901A (zh) 一种数据处理方法及数据处理系统
CN112269887A (zh) 一种基于图数据库的分布式系统
Cui et al. Distributed training of deep neural network acoustic models for automatic speech recognition: a comparison of current training strategies
US20240273346A1 (en) Self-balancing mixture of experts
Xia et al. Redundancy-free high-performance dynamic GNN training with hierarchical pipeline parallelism
Liu et al. Meta-mapreduce for scalable data mining
CN119046015B (zh) 神经网络模型训练处理的电子设备、方法和介质
WO2025081828A1 (zh) 训练模型分配方法、装置、计算机设备和存储介质
CN118446265A (zh) 神经网络加速器设计方法及装置
KR20250029051A (ko) 특수 전문가 혼합 머신 러닝 모델의 계층을 분배하기 위한 시스템 및 방법
Xia et al. Redundancy-free and load-balanced tgnn training with hierarchical pipeline parallelism
Fadnavis et al. Big data processing using Hadoop
CN117371517A (zh) 深度学习模型的元学习方法及深度学习模型的元学习系统
Chunduri et al. Haloop approach for concept generation in formal concept analysis
Zhan et al. Field programmable gate array‐based all‐layer accelerator with quantization neural networks for sustainable cyber‐physical systems
Perwej et al. An extensive investigate the mapreduce technology
Yang et al. EDGES: An efficient distributed graph embedding system on GPU clusters
CN117114055A (zh) 面向工业应用场景的fpga二值神经网络加速方法
CN117036898A (zh) 一种基于图神经网络和fpga的图数据处理方法
Liu et al. Parallelization Techniques for Large Language Models: A Review from Training to Inference
Bu et al. Research on parallel data processing of data mining platform in the background of cloud computing
Xu et al. EdgeMesh: A hybrid distributed training mechanism for heterogeneous edge devices
Fu et al. Spark and dask performance analysis based on ARL image library
Zhang et al. A review of FPGA-based graph neural network accelerator architectures

Legal Events

Date Code Title Description
PA0105 International application

St.27 status event code: A-0-1-A10-A15-nap-PA0105

PG1501 Laying open of application

St.27 status event code: A-1-1-Q10-Q12-nap-PG1501

P11 Amendment of application requested

Free format text: ST27 STATUS EVENT CODE: A-2-2-P10-P11-NAP-X000 (AS PROVIDED BY THE NATIONAL OFFICE)

P11-X000 Amendment of application requested

St.27 status event code: A-2-2-P10-P11-nap-X000

P13 Application amended

Free format text: ST27 STATUS EVENT CODE: A-2-2-P10-P13-NAP-X000 (AS PROVIDED BY THE NATIONAL OFFICE)

P13-X000 Application amended

St.27 status event code: A-2-2-P10-P13-nap-X000