CN117980916A - 用于媒体的变换译码的基于变换器的架构 - Google Patents

用于媒体的变换译码的基于变换器的架构 Download PDF

Info

Publication number
CN117980916A
CN117980916A CN202280063767.1A CN202280063767A CN117980916A CN 117980916 A CN117980916 A CN 117980916A CN 202280063767 A CN202280063767 A CN 202280063767A CN 117980916 A CN117980916 A CN 117980916A
Authority
CN
China
Prior art keywords
transformer
decoder
patches
patch
layer
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202280063767.1A
Other languages
English (en)
Chinese (zh)
Inventor
朱银浩
杨阳
T·S·科恩
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Qualcomm Inc
Original Assignee
Qualcomm Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Qualcomm Inc filed Critical Qualcomm Inc
Publication of CN117980916A publication Critical patent/CN117980916A/zh
Pending legal-status Critical Current

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/59—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/60—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G06N3/0455—Auto-encoder networks; Encoder-decoder networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/0895—Weakly supervised learning, e.g. semi-supervised or self-supervised learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/096—Transfer learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/0985—Hyperparameter optimisation; Meta-learning; Learning-to-learn
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T9/00—Image coding
    • G06T9/002—Image coding using neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T9/00—Image coding
    • G06T9/007—Transform coding, e.g. discrete cosine transform
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N1/00—Scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission; Details thereof
    • H04N1/387—Composing, repositioning or otherwise geometrically modifying originals
    • H04N1/3876—Recombination of partial images to recreate the original image
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
    • H04N19/146—Data rate or code amount at the encoder output
    • H04N19/147—Data rate or code amount at the encoder output according to rate distortion criteria
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/189—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the adaptation method, adaptation tool or adaptation type used for the adaptive coding
    • H04N19/19—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the adaptation method, adaptation tool or adaptation type used for the adaptive coding using optimisation based on Lagrange multipliers
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/44—Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
    • H04N19/51—Motion estimation or motion compensation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
    • H04N19/51—Motion estimation or motion compensation
    • H04N19/537—Motion estimation other than block-based
    • H04N19/543—Motion estimation other than block-based using regions
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/90—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using coding techniques not provided for in groups H04N19/10-H04N19/85, e.g. fractals
    • H04N19/91—Entropy coding, e.g. variable length coding [VLC] or arithmetic coding
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00—Details of colour television systems
    • H04N9/79—Processing of colour television signals in connection with recording
    • H04N9/87—Regeneration of colour television signals
    • H04N9/877—Regeneration of colour television signals by assembling picture element blocks in an intermediate memory
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N2013/0074—Stereoscopic image analysis
    • H04N2013/0088—Synthesising a monoscopic image signal from stereoscopic images, e.g. synthesising a panoramic or high resolution monoscopic image

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • General Physics & Mathematics (AREA)
  • Signal Processing (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Biophysics (AREA)
  • Mathematical Physics (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Software Systems (AREA)
  • Biomedical Technology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Discrete Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)
  • Compression Of Band Width Or Redundancy In Fax (AREA)
CN202280063767.1A 2021-09-27 2022-09-15 用于媒体的变换译码的基于变换器的架构 Pending CN117980916A (zh)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US17/486,732 US12120348B2 (en) 2021-09-27 2021-09-27 Transformer-based architecture for transform coding of media
US17/486,732 2021-09-27
PCT/US2022/076496 WO2023049655A1 (en) 2021-09-27 2022-09-15 Transformer-based architecture for transform coding of media

Publications (1)

Publication Number Publication Date
CN117980916A true CN117980916A (zh) 2024-05-03

Family

ID=83598683

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202280063767.1A Pending CN117980916A (zh) 2021-09-27 2022-09-15 用于媒体的变换译码的基于变换器的架构

Country Status (7)

Country Link
US (1) US12120348B2 (enExample)
EP (1) EP4409467A1 (enExample)
JP (1) JP2024536035A (enExample)
KR (1) KR20240090145A (enExample)
CN (1) CN117980916A (enExample)
TW (1) TW202318878A (enExample)
WO (1) WO2023049655A1 (enExample)

Families Citing this family (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20240065281A (ko) * 2021-10-05 2024-05-14 구글 엘엘씨 벡터-양자화된 이미지 모델링
EP4388451A1 (en) * 2021-10-19 2024-06-26 Huawei Technologies Co., Ltd. Attention-based method for deep point cloud compression
WO2023081095A1 (en) * 2021-11-05 2023-05-11 Subtle Medical, Inc. Systems and methods for multi-contrast multi-scale vision transformers
US12399033B2 (en) * 2022-03-01 2025-08-26 Here Global B.V. Method and apparatus for generating maps from aligned geospatial observations
US12573191B2 (en) * 2022-05-09 2026-03-10 Samsung Electronics Co., Ltd. Image processing apparatus and operating method thereof
CN119487857A (zh) * 2022-06-06 2025-02-18 谷歌有限责任公司 用于视频压缩Transformer的方法和系统
US12206874B1 (en) * 2022-06-27 2025-01-21 Amazon Technologies, Inc. Spatially lapped encoding
EP4552318A1 (en) * 2022-07-05 2025-05-14 Telefonaktiebolaget LM Ericsson (publ) Filtering for video encoding and decoding
US12444173B2 (en) * 2022-07-13 2025-10-14 Sony Group Corporation Image component generation based on application of iterative learning on autoencoder model and transformer model
US20240169715A1 (en) * 2022-11-23 2024-05-23 Google Llc Training large-scale vision transformer neural networks with variable patch sizes
US20240378947A1 (en) * 2023-05-10 2024-11-14 Sierra Artificial Neural Networks Systems and methods for slot machine game development utilizing artificial intelligence game math design systems
CN122395386A (zh) * 2023-06-01 2026-07-14 杭州海康威视数字技术股份有限公司 一种解码、编码方法、装置及其设备
CN116595421B (zh) * 2023-06-10 2024-04-09 北京航空航天大学 基于时频谱图和变换器算法的飞行器电信号预测方法
CN117061760B (zh) * 2023-07-27 2026-07-17 北京邮电大学 一种基于注意力机制的视频压缩方法和系统
CN118317114B (zh) * 2024-06-11 2024-08-16 中国科学技术大学 一种基于局部特征和非局部特征的图像压缩方法
CN119442310B (zh) * 2024-10-23 2025-12-12 四川大学 一种基于关键特征提取的工业软件资源共享方法
CN119228860B (zh) * 2024-11-28 2025-06-13 江西省肿瘤医院(江西省第二人民医院、江西省癌症中心) 一种基于深度学习的图像非刚体配准方法及系统

Also Published As

Publication number Publication date
KR20240090145A (ko) 2024-06-21
TW202318878A (zh) 2023-05-01
WO2023049655A1 (en) 2023-03-30
EP4409467A1 (en) 2024-08-07
US12120348B2 (en) 2024-10-15
US20230100413A1 (en) 2023-03-30
JP2024536035A (ja) 2024-10-04

Similar Documents

Publication Publication Date Title
US12120348B2 (en) Transformer-based architecture for transform coding of media
US11405626B2 (en) Video compression using recurrent-based machine learning systems
CN116114247B (zh) 用于处理视频数据的装置和方法
US12231666B2 (en) Front-end architecture for neural network based video coding
US12003734B2 (en) Machine learning based flow determination for video coding
TWI883294B (zh) 用於基於神經網路的視訊譯碼的前端架構
KR102831175B1 (ko) 머신 러닝 향상을 갖는 비디오 코딩을 위한 비트-레이트 추정
US11399198B1 (en) Learned B-frame compression
US20240013441A1 (en) Video coding using camera motion compensation and object motion compensation
US12177473B2 (en) Video coding using optical flow and residual predictors
WO2024015665A1 (en) Bit-rate estimation for video coding with machine learning enhancement
US20240214578A1 (en) Regularizing neural networks with data quantization using exponential family priors
EP4298795A1 (en) Machine learning based flow determination for video coding
CN116965032A (zh) 用于视频译码的基于机器学习的流确定

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination