KR20240090145A - 미디어의 변환 코딩을 위한 트랜스포머 기반 아키텍처 - Google Patents

미디어의 변환 코딩을 위한 트랜스포머 기반 아키텍처 Download PDF

Info

Publication number
KR20240090145A
KR20240090145A KR1020247009071A KR20247009071A KR20240090145A KR 20240090145 A KR20240090145 A KR 20240090145A KR 1020247009071 A KR1020247009071 A KR 1020247009071A KR 20247009071 A KR20247009071 A KR 20247009071A KR 20240090145 A KR20240090145 A KR 20240090145A
Authority
KR
South Korea
Prior art keywords
transformer
decoder
patches
patch
encoder
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
KR1020247009071A
Other languages
English (en)
Korean (ko)
Inventor
인하오 주
양 양
타코 세바스티안 코헌
Original Assignee
퀄컴 인코포레이티드
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 퀄컴 인코포레이티드 filed Critical 퀄컴 인코포레이티드
Publication of KR20240090145A publication Critical patent/KR20240090145A/ko
Pending legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/59—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/60—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using transform coding
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G06N3/0455—Auto-encoder networks; Encoder-decoder networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/0895—Weakly supervised learning, e.g. semi-supervised or self-supervised learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/096—Transfer learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/0985—Hyperparameter optimisation; Meta-learning; Learning-to-learn
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T9/00—Image coding
    • G06T9/002—Image coding using neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T9/00—Image coding
    • G06T9/007—Transform coding, e.g. discrete cosine transform
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N1/00—Scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission; Details thereof
    • H04N1/387—Composing, repositioning or otherwise geometrically modifying originals
    • H04N1/3876—Recombination of partial images to recreate the original image
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
    • H04N19/146—Data rate or code amount at the encoder output
    • H04N19/147—Data rate or code amount at the encoder output according to rate distortion criteria
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/189—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the adaptation method, adaptation tool or adaptation type used for the adaptive coding
    • H04N19/19—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the adaptation method, adaptation tool or adaptation type used for the adaptive coding using optimisation based on Lagrange multipliers
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/44—Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
    • H04N19/51—Motion estimation or motion compensation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
    • H04N19/51—Motion estimation or motion compensation
    • H04N19/537—Motion estimation other than block-based
    • H04N19/543—Motion estimation other than block-based using regions
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/90—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using coding techniques not provided for in groups H04N19/10-H04N19/85, e.g. fractals
    • H04N19/91—Entropy coding, e.g. variable length coding [VLC] or arithmetic coding
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N9/00—Details of colour television systems
    • H04N9/79—Processing of colour television signals in connection with recording
    • H04N9/87—Regeneration of colour television signals
    • H04N9/877—Regeneration of colour television signals by assembling picture element blocks in an intermediate memory
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N2013/0074—Stereoscopic image analysis
    • H04N2013/0088—Synthesising a monoscopic image signal from stereoscopic images, e.g. synthesising a panoramic or high resolution monoscopic image

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • General Physics & Mathematics (AREA)
  • Signal Processing (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Biophysics (AREA)
  • Mathematical Physics (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Software Systems (AREA)
  • Biomedical Technology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Discrete Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)
  • Compression Of Band Width Or Redundancy In Fax (AREA)
KR1020247009071A 2021-09-27 2022-09-15 미디어의 변환 코딩을 위한 트랜스포머 기반 아키텍처 Pending KR20240090145A (ko)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US17/486,732 US12120348B2 (en) 2021-09-27 2021-09-27 Transformer-based architecture for transform coding of media
US17/486,732 2021-09-27
PCT/US2022/076496 WO2023049655A1 (en) 2021-09-27 2022-09-15 Transformer-based architecture for transform coding of media

Publications (1)

Publication Number Publication Date
KR20240090145A true KR20240090145A (ko) 2024-06-21

Family

ID=83598683

Family Applications (1)

Application Number Title Priority Date Filing Date
KR1020247009071A Pending KR20240090145A (ko) 2021-09-27 2022-09-15 미디어의 변환 코딩을 위한 트랜스포머 기반 아키텍처

Country Status (7)

Country Link
US (1) US12120348B2 (enExample)
EP (1) EP4409467A1 (enExample)
JP (1) JP2024536035A (enExample)
KR (1) KR20240090145A (enExample)
CN (1) CN117980916A (enExample)
TW (1) TW202318878A (enExample)
WO (1) WO2023049655A1 (enExample)

Families Citing this family (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20240065281A (ko) * 2021-10-05 2024-05-14 구글 엘엘씨 벡터-양자화된 이미지 모델링
EP4388451A1 (en) * 2021-10-19 2024-06-26 Huawei Technologies Co., Ltd. Attention-based method for deep point cloud compression
WO2023081095A1 (en) * 2021-11-05 2023-05-11 Subtle Medical, Inc. Systems and methods for multi-contrast multi-scale vision transformers
US12399033B2 (en) * 2022-03-01 2025-08-26 Here Global B.V. Method and apparatus for generating maps from aligned geospatial observations
US12573191B2 (en) * 2022-05-09 2026-03-10 Samsung Electronics Co., Ltd. Image processing apparatus and operating method thereof
CN119487857A (zh) * 2022-06-06 2025-02-18 谷歌有限责任公司 用于视频压缩Transformer的方法和系统
US12206874B1 (en) * 2022-06-27 2025-01-21 Amazon Technologies, Inc. Spatially lapped encoding
EP4552318A1 (en) * 2022-07-05 2025-05-14 Telefonaktiebolaget LM Ericsson (publ) Filtering for video encoding and decoding
US12444173B2 (en) * 2022-07-13 2025-10-14 Sony Group Corporation Image component generation based on application of iterative learning on autoencoder model and transformer model
US20240169715A1 (en) * 2022-11-23 2024-05-23 Google Llc Training large-scale vision transformer neural networks with variable patch sizes
US20240378947A1 (en) * 2023-05-10 2024-11-14 Sierra Artificial Neural Networks Systems and methods for slot machine game development utilizing artificial intelligence game math design systems
CN122395386A (zh) * 2023-06-01 2026-07-14 杭州海康威视数字技术股份有限公司 一种解码、编码方法、装置及其设备
CN116595421B (zh) * 2023-06-10 2024-04-09 北京航空航天大学 基于时频谱图和变换器算法的飞行器电信号预测方法
CN117061760B (zh) * 2023-07-27 2026-07-17 北京邮电大学 一种基于注意力机制的视频压缩方法和系统
CN118317114B (zh) * 2024-06-11 2024-08-16 中国科学技术大学 一种基于局部特征和非局部特征的图像压缩方法
CN119442310B (zh) * 2024-10-23 2025-12-12 四川大学 一种基于关键特征提取的工业软件资源共享方法
CN119228860B (zh) * 2024-11-28 2025-06-13 江西省肿瘤医院(江西省第二人民医院、江西省癌症中心) 一种基于深度学习的图像非刚体配准方法及系统

Also Published As

Publication number Publication date
TW202318878A (zh) 2023-05-01
WO2023049655A1 (en) 2023-03-30
EP4409467A1 (en) 2024-08-07
CN117980916A (zh) 2024-05-03
US12120348B2 (en) 2024-10-15
US20230100413A1 (en) 2023-03-30
JP2024536035A (ja) 2024-10-04

Similar Documents

Publication Publication Date Title
US12120348B2 (en) Transformer-based architecture for transform coding of media
US11405626B2 (en) Video compression using recurrent-based machine learning systems
KR102578967B1 (ko) 단대단 신경망 기반 비디오 코딩
US12003734B2 (en) Machine learning based flow determination for video coding
US12231666B2 (en) Front-end architecture for neural network based video coding
KR20230074137A (ko) 머신 러닝 시스템들을 이용한 인스턴스 적응적 이미지 및 비디오 압축
US20220295095A1 (en) Learned b-frame coding using p-frame coding system
US11399198B1 (en) Learned B-frame compression
US12394100B2 (en) Video coding using camera motion compensation and object motion compensation
KR102831175B1 (ko) 머신 러닝 향상을 갖는 비디오 코딩을 위한 비트-레이트 추정
US12177473B2 (en) Video coding using optical flow and residual predictors
KR20230117346A (ko) 뉴럴 네트워크 기반 비디오 코딩을 위한 프론트-엔드 아키텍처
WO2024015665A1 (en) Bit-rate estimation for video coding with machine learning enhancement
US20240214578A1 (en) Regularizing neural networks with data quantization using exponential family priors
EP4298795A1 (en) Machine learning based flow determination for video coding
TW202616720A (zh) 用於在碼流中指示立方體工具啟用的方法和裝置

Legal Events

Date Code Title Description
PA0105 International application

St.27 status event code: A-0-1-A10-A15-nap-PA0105

PG1501 Laying open of application

St.27 status event code: A-1-1-Q10-Q12-nap-PG1501

D18 Deferred examination requested

Free format text: ST27 STATUS EVENT CODE: A-1-2-D10-D18-EXM-X000 (AS PROVIDED BY THE NATIONAL OFFICE)

D18-X000 Deferred examination requested

St.27 status event code: A-1-2-D10-D18-exm-X000