JP2024536035A - メディアの変換コーディングのためのトランスフォーマベースのアーキテクチャ - Google Patents

メディアの変換コーディングのためのトランスフォーマベースのアーキテクチャ Download PDF

Info

Publication number
JP2024536035A
JP2024536035A JP2024517458A JP2024517458A JP2024536035A JP 2024536035 A JP2024536035 A JP 2024536035A JP 2024517458 A JP2024517458 A JP 2024517458A JP 2024517458 A JP2024517458 A JP 2024517458A JP 2024536035 A JP2024536035 A JP 2024536035A
Authority
JP
Japan
Prior art keywords
transformer
patches
decoder
patch
encoder
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
JP2024517458A
Other languages
English (en)
Japanese (ja)
Other versions
JP2024536035A5 (enExample
Inventor
ジュ、インハオ
ヤン、ヤン
コーエン、タコ・セバスティアーン
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Qualcomm Inc
Original Assignee
Qualcomm Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Qualcomm Inc filed Critical Qualcomm Inc
Publication of JP2024536035A publication Critical patent/JP2024536035A/ja
Publication of JP2024536035A5 publication Critical patent/JP2024536035A5/ja
Pending legal-status Critical Current

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/60—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/59—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G06N3/0455—Auto-encoder networks; Encoder-decoder networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/0895—Weakly supervised learning, e.g. semi-supervised or self-supervised learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/096—Transfer learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/0985—Hyperparameter optimisation; Meta-learning; Learning-to-learn
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T9/00—Image coding
    • G06T9/002—Image coding using neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T9/00—Image coding
    • G06T9/007—Transform coding, e.g. discrete cosine transform
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N1/00—Scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission; Details thereof
    • H04N1/387—Composing, repositioning or otherwise geometrically modifying originals
    • H04N1/3876—Recombination of partial images to recreate the original image
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
    • H04N19/146—Data rate or code amount at the encoder output
    • H04N19/147—Data rate or code amount at the encoder output according to rate distortion criteria
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/189—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the adaptation method, adaptation tool or adaptation type used for the adaptive coding
    • H04N19/19—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the adaptation method, adaptation tool or adaptation type used for the adaptive coding using optimisation based on Lagrange multipliers
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/44—Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
    • H04N19/51—Motion estimation or motion compensation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
    • H04N19/51—Motion estimation or motion compensation
    • H04N19/537—Motion estimation other than block-based
    • H04N19/543—Motion estimation other than block-based using regions
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/90—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using coding techniques not provided for in groups H04N19/10-H04N19/85, e.g. fractals
    • H04N19/91—Entropy coding, e.g. variable length coding [VLC] or arithmetic coding
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00—Details of colour television systems
    • H04N9/79—Processing of colour television signals in connection with recording
    • H04N9/87—Regeneration of colour television signals
    • H04N9/877—Regeneration of colour television signals by assembling picture element blocks in an intermediate memory
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N2013/0074—Stereoscopic image analysis
    • H04N2013/0088—Synthesising a monoscopic image signal from stereoscopic images, e.g. synthesising a panoramic or high resolution monoscopic image

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • General Physics & Mathematics (AREA)
  • Signal Processing (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Biophysics (AREA)
  • Mathematical Physics (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Software Systems (AREA)
  • Biomedical Technology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Discrete Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)
  • Compression Of Band Width Or Redundancy In Fax (AREA)
JP2024517458A 2021-09-27 2022-09-15 メディアの変換コーディングのためのトランスフォーマベースのアーキテクチャ Pending JP2024536035A (ja)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US17/486,732 US12120348B2 (en) 2021-09-27 2021-09-27 Transformer-based architecture for transform coding of media
US17/486,732 2021-09-27
PCT/US2022/076496 WO2023049655A1 (en) 2021-09-27 2022-09-15 Transformer-based architecture for transform coding of media

Publications (2)

Publication Number Publication Date
JP2024536035A true JP2024536035A (ja) 2024-10-04
JP2024536035A5 JP2024536035A5 (enExample) 2025-08-26

Family

ID=83598683

Family Applications (1)

Application Number Title Priority Date Filing Date
JP2024517458A Pending JP2024536035A (ja) 2021-09-27 2022-09-15 メディアの変換コーディングのためのトランスフォーマベースのアーキテクチャ

Country Status (7)

Country Link
US (1) US12120348B2 (enExample)
EP (1) EP4409467A1 (enExample)
JP (1) JP2024536035A (enExample)
KR (1) KR20240090145A (enExample)
CN (1) CN117980916A (enExample)
TW (1) TW202318878A (enExample)
WO (1) WO2023049655A1 (enExample)

Families Citing this family (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20240065281A (ko) * 2021-10-05 2024-05-14 구글 엘엘씨 벡터-양자화된 이미지 모델링
EP4388451A1 (en) * 2021-10-19 2024-06-26 Huawei Technologies Co., Ltd. Attention-based method for deep point cloud compression
WO2023081095A1 (en) * 2021-11-05 2023-05-11 Subtle Medical, Inc. Systems and methods for multi-contrast multi-scale vision transformers
US12399033B2 (en) * 2022-03-01 2025-08-26 Here Global B.V. Method and apparatus for generating maps from aligned geospatial observations
US12573191B2 (en) * 2022-05-09 2026-03-10 Samsung Electronics Co., Ltd. Image processing apparatus and operating method thereof
CN119487857A (zh) * 2022-06-06 2025-02-18 谷歌有限责任公司 用于视频压缩Transformer的方法和系统
US12206874B1 (en) * 2022-06-27 2025-01-21 Amazon Technologies, Inc. Spatially lapped encoding
EP4552318A1 (en) * 2022-07-05 2025-05-14 Telefonaktiebolaget LM Ericsson (publ) Filtering for video encoding and decoding
US12444173B2 (en) * 2022-07-13 2025-10-14 Sony Group Corporation Image component generation based on application of iterative learning on autoencoder model and transformer model
US20240169715A1 (en) * 2022-11-23 2024-05-23 Google Llc Training large-scale vision transformer neural networks with variable patch sizes
US20240378947A1 (en) * 2023-05-10 2024-11-14 Sierra Artificial Neural Networks Systems and methods for slot machine game development utilizing artificial intelligence game math design systems
CN122395386A (zh) * 2023-06-01 2026-07-14 杭州海康威视数字技术股份有限公司 一种解码、编码方法、装置及其设备
CN116595421B (zh) * 2023-06-10 2024-04-09 北京航空航天大学 基于时频谱图和变换器算法的飞行器电信号预测方法
CN117061760B (zh) * 2023-07-27 2026-07-17 北京邮电大学 一种基于注意力机制的视频压缩方法和系统
CN118317114B (zh) * 2024-06-11 2024-08-16 中国科学技术大学 一种基于局部特征和非局部特征的图像压缩方法
CN119442310B (zh) * 2024-10-23 2025-12-12 四川大学 一种基于关键特征提取的工业软件资源共享方法
CN119228860B (zh) * 2024-11-28 2025-06-13 江西省肿瘤医院(江西省第二人民医院、江西省癌症中心) 一种基于深度学习的图像非刚体配准方法及系统

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
DAVID MINNEN, ET.AL.: "Joint Autoregressive and Hierarchical Priors for Learned Image Compression", ARXIV, vol. arXiv:1809.02736v1, JPN6026020622, 18 September 2018 (2018-09-18), pages 1 - 4, ISSN: 0005869454 *
ZE LIU, ET.AL.: "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows", ARXIV, vol. arXiv:2103.14030v2, JPN6026020623, 17 August 2021 (2021-08-17), pages 1 - 9, ISSN: 0005869455 *

Also Published As

Publication number Publication date
KR20240090145A (ko) 2024-06-21
TW202318878A (zh) 2023-05-01
WO2023049655A1 (en) 2023-03-30
EP4409467A1 (en) 2024-08-07
CN117980916A (zh) 2024-05-03
US12120348B2 (en) 2024-10-15
US20230100413A1 (en) 2023-03-30

Similar Documents

Publication Publication Date Title
US12120348B2 (en) Transformer-based architecture for transform coding of media
US11405626B2 (en) Video compression using recurrent-based machine learning systems
KR102578967B1 (ko) 단대단 신경망 기반 비디오 코딩
US12231666B2 (en) Front-end architecture for neural network based video coding
US20220272355A1 (en) Machine learning based flow determination for video coding
JP7478911B2 (ja) 並列エントロピーコーディングのためのビットストリームインデックスの圧縮
JP7815247B2 (ja) ニューラルネットワークベースのビデオコーディングのためのフロントエンドアーキテクチャ
US11399198B1 (en) Learned B-frame compression
KR102831175B1 (ko) 머신 러닝 향상을 갖는 비디오 코딩을 위한 비트-레이트 추정
EP4229868A1 (en) Joint termination of bidirectional data blocks for parallel coding
US12177473B2 (en) Video coding using optical flow and residual predictors
EP4555726A1 (en) Video coding using camera motion compensation and object motion compensation
WO2024015665A1 (en) Bit-rate estimation for video coding with machine learning enhancement
US20240214578A1 (en) Regularizing neural networks with data quantization using exponential family priors
JP7840973B2 (ja) ビデオコーディングのための機械学習ベースのフロー決定
CN116965032A (zh) 用于视频译码的基于机器学习的流确定

Legal Events

Date Code Title Description
A521 Request for written amendment filed

Free format text: JAPANESE INTERMEDIATE CODE: A523

Effective date: 20250815

A621 Written request for application examination

Free format text: JAPANESE INTERMEDIATE CODE: A621

Effective date: 20250815

A977 Report on retrieval

Free format text: JAPANESE INTERMEDIATE CODE: A971007

Effective date: 20260410

A131 Notification of reasons for refusal

Free format text: JAPANESE INTERMEDIATE CODE: A131

Effective date: 20260526