SG11202003269SA - Optimizing audio delivery for virtual reality applications - Google Patents

Optimizing audio delivery for virtual reality applications

Info

Publication number
SG11202003269SA
SG11202003269SA SG11202003269SA SG11202003269SA SG11202003269SA SG 11202003269S A SG11202003269S A SG 11202003269SA SG 11202003269S A SG11202003269S A SG 11202003269SA SG 11202003269S A SG11202003269S A SG 11202003269SA SG 11202003269S A SG11202003269S A SG 11202003269SA
Authority
SG
Singapore
Prior art keywords
virtual reality
reality applications
audio delivery
optimizing audio
optimizing
Prior art date
Application number
SG11202003269SA
Inventor
Adrian Murtaza
Harald Fuchs
Bernd Czelhan
Jan Plogsties
Matteo Agnelli
Ingo Hofmann
Original Assignee
Fraunhofer Ges Forschung
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Fraunhofer Ges Forschung filed Critical Fraunhofer Ges Forschung
Publication of SG11202003269SA publication Critical patent/SG11202003269SA/en

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/16—Sound input; Sound output
    • G06F3/165—Management of the audio stream, e.g. setting of volume, audio stream path
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/81—Monomedia components thereof
    • H04N21/8106—Monomedia components thereof involving special audio data, e.g. different tracks for different languages
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/0093—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/0179—Display position adjusting means not related to the information to be displayed
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/012—Head tracking input arrangements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/16—Sound input; Sound output
    • G06F3/167—Audio in a user interface, e.g. using voice commands for navigating, audio feedback
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/60—Network streaming of media packets
    • H04L65/70—Media network packetisation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/293—Generating mixed stereoscopic images; Generating mixed monoscopic and stereoscopic images, e.g. a stereoscopic image overlay window on a monoscopic image background
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30—Image reproducers
    • H04N13/361—Reproducing mixed stereoscopic images; Reproducing mixed monoscopic and stereoscopic images, e.g. a stereoscopic image overlay window on a monoscopic image background
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/21—Server components or server architectures
    • H04N21/218—Source of audio or video content, e.g. local disk arrays
    • H04N21/21805—Source of audio or video content, e.g. local disk arrays enabling multiple viewpoints, e.g. using a plurality of cameras
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
    • H04N21/23439—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements for generating different versions
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/439—Processing of audio elementary streams
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/439—Processing of audio elementary streams
    • H04N21/4394—Processing of audio elementary streams involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client 
    • H04N21/65—Transmission of management data between client and server
    • H04N21/658—Transmission by the client directed to the server
    • H04N21/6587—Control parameters, e.g. trick play commands, viewpoint selection
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/81—Monomedia components thereof
    • H04N21/816—Monomedia components thereof involving special video data, e.g 3D video
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
    • H04R5/00—Stereophonic arrangements
    • H04R5/033—Headphones for stereophonic communication
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/0179—Display position adjusting means not related to the information to be displayed
    • G02B2027/0187—Display position adjusting means not related to the information to be displayed slaved to motion of at least a part of the body of the user, e.g. head, eye

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Optics & Photonics (AREA)
  • Databases & Information Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Health & Medical Sciences (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Acoustics & Sound (AREA)
  • Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
  • Stereophonic System (AREA)
  • Position Input By Displaying (AREA)
  • User Interface Of Digital Computer (AREA)
SG11202003269SA 2017-10-12 2018-10-11 Optimizing audio delivery for virtual reality applications SG11202003269SA (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
EP17196259 2017-10-12
PCT/EP2018/077770 WO2019072984A1 (en) 2017-10-12 2018-10-11 Optimizing audio delivery for virtual reality applications

Publications (1)

Publication Number Publication Date
SG11202003269SA true SG11202003269SA (en) 2020-05-28

Family

ID=60191107

Family Applications (2)

Application Number Title Priority Date Filing Date
SG10202106080XA SG10202106080XA (en) 2017-10-12 2018-10-11 Optimizing audio delivery for virtual reality applications
SG11202003269SA SG11202003269SA (en) 2017-10-12 2018-10-11 Optimizing audio delivery for virtual reality applications

Family Applications Before (1)

Application Number Title Priority Date Filing Date
SG10202106080XA SG10202106080XA (en) 2017-10-12 2018-10-11 Optimizing audio delivery for virtual reality applications

Country Status (16)

Country Link
US (2) US11354084B2 (en)
EP (2) EP3695613B1 (en)
JP (2) JP7295851B2 (en)
KR (2) KR20230130729A (en)
CN (5) CN116193213A (en)
AR (6) AR113357A1 (en)
AU (2) AU2018348762B2 (en)
BR (1) BR112020008073A2 (en)
CA (6) CA3078858A1 (en)
ES (1) ES2970490T3 (en)
MX (6) MX2020003450A (en)
RU (2) RU2750505C1 (en)
SG (2) SG10202106080XA (en)
TW (1) TWI713911B (en)
WO (1) WO2019072984A1 (en)
ZA (3) ZA202002064B (en)

Families Citing this family (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10242486B2 (en) * 2017-04-17 2019-03-26 Intel Corporation Augmented reality and virtual reality feedback enhancement system, apparatus and method
EP3756355A1 (en) * 2018-02-22 2020-12-30 Dolby International AB Method and apparatus for processing of auxiliary media streams embedded in a mpeg-h 3d audio stream
CN109151565B (en) * 2018-09-04 2019-12-20 北京达佳互联信息技术有限公司 Method and device for playing voice, electronic equipment and storage medium
JP2020137044A (en) * 2019-02-25 2020-08-31 ソニーセミコンダクタソリューションズ株式会社 Audio signal processing device
US11211073B2 (en) * 2019-04-22 2021-12-28 Sony Corporation Display control of different verbatim text of vocal deliverance of performer-of-interest in a live event
US11429340B2 (en) * 2019-07-03 2022-08-30 Qualcomm Incorporated Audio capture and rendering for extended reality experiences
US11432097B2 (en) 2019-07-03 2022-08-30 Qualcomm Incorporated User interface for controlling audio rendering for extended reality experiences
US20210006976A1 (en) * 2019-07-03 2021-01-07 Qualcomm Incorporated Privacy restrictions for audio rendering
CN111246225B (en) * 2019-12-25 2022-02-08 北京达佳互联信息技术有限公司 Information interaction method and device, electronic equipment and computer readable storage medium
JP7371595B2 (en) * 2020-09-15 2023-10-31 横河電機株式会社 Apparatus, system, method and program
GB2599359A (en) * 2020-09-23 2022-04-06 Nokia Technologies Oy Spatial audio rendering
US12010496B2 (en) * 2020-09-25 2024-06-11 Apple Inc. Method and system for performing audio ducking for headsets
US11914157B2 (en) 2021-03-29 2024-02-27 International Business Machines Corporation Adjustable air columns for head mounted displays
US20220391167A1 (en) * 2021-06-02 2022-12-08 Tencent America LLC Adaptive audio delivery and rendering
CN113660347B (en) * 2021-08-31 2024-05-07 Oppo广东移动通信有限公司 Data processing method, device, electronic equipment and readable storage medium
US20230086248A1 (en) * 2021-09-21 2023-03-23 Meta Platforms Technologies, Llc Visual navigation elements for artificial reality environments
US20230260537A1 (en) * 2022-02-16 2023-08-17 Google Llc Single Vector Digital Voice Accelerometer
CN116709162B (en) * 2023-08-09 2023-11-21 腾讯科技（深圳）有限公司 Audio processing method and related equipment

Family Cites Families (26)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20020103554A1 (en) * 2001-01-29 2002-08-01 Hewlett-Packard Company Interactive audio system
AUPR989802A0 (en) 2002-01-09 2002-01-31 Lake Technology Limited Interactive spatialized audiovisual system
JP2004072694A (en) 2002-08-09 2004-03-04 Sony Corp Information providing system and method, information providing apparatus and method, recording medium, and program
DE102005008366A1 (en) * 2005-02-23 2006-08-24 Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. Device for driving wave-field synthesis rendering device with audio objects, has unit for supplying scene description defining time sequence of audio objects
JP2007029506A (en) * 2005-07-28 2007-02-08 Konami Digital Entertainment:Kk Game device, sound data creating method, and program
US20080022348A1 (en) 2006-07-03 2008-01-24 Samoa Opulence Investment Inc. Interactive video display system and a method thereof
KR101576294B1 (en) * 2008-08-14 2015-12-11 삼성전자주식회사 Apparatus and method to perform processing a sound in a virtual reality system
US10326978B2 (en) * 2010-06-30 2019-06-18 Warner Bros. Entertainment Inc. Method and apparatus for generating virtual or augmented reality presentations with 3D audio positioning
EP3258671B1 (en) * 2011-10-28 2018-09-05 Magic Leap, Inc. System and method for augmented and virtual reality
US20150296247A1 (en) 2012-02-29 2015-10-15 ExXothermic, Inc. Interaction of user devices and video devices
US10375439B2 (en) * 2014-05-30 2019-08-06 Sony Corporation Information processing apparatus and information processing method
US9787846B2 (en) * 2015-01-21 2017-10-10 Microsoft Technology Licensing, Llc Spatial audio signal processing for objects with associated audio content
WO2016138502A1 (en) * 2015-02-27 2016-09-01 Arris Enterprises, Inc. Adaptive joint bitrate allocation
GB2536025B (en) * 2015-03-05 2021-03-03 Nokia Technologies Oy Video streaming method
EP4254145A3 (en) 2015-09-16 2023-11-01 Magic Leap, Inc. Head pose mixing of audio files
US20170109131A1 (en) * 2015-10-20 2017-04-20 Bragi GmbH Earpiece 3D Sound Localization Using Mixed Sensor Array for Virtual Reality System and Method
US10229540B2 (en) 2015-12-22 2019-03-12 Google Llc Adjusting video rendering rate of virtual reality content and processing of a stereoscopic image
WO2017120681A1 (en) * 2016-01-15 2017-07-20 Michael Godfrey Method and system for automatically determining a positional three dimensional output of audio information based on a user's orientation within an artificial immersive environment
US10229541B2 (en) 2016-01-28 2019-03-12 Sony Interactive Entertainment America Llc Methods and systems for navigation within virtual reality space using head mounted display
US10291910B2 (en) * 2016-02-12 2019-05-14 Gopro, Inc. Systems and methods for spatially adaptive video encoding
US11017712B2 (en) * 2016-08-12 2021-05-25 Intel Corporation Optimized display image rendering
CN106774891A (en) * 2016-12-15 2017-05-31 北京小鸟看看科技有限公司 The audio effect generating method of virtual reality scenario, equipment and virtual reality device
US10659906B2 (en) * 2017-01-13 2020-05-19 Qualcomm Incorporated Audio parallax for virtual reality, augmented reality, and mixed reality
GB2560923A (en) * 2017-03-28 2018-10-03 Nokia Technologies Oy Video streaming
KR102517906B1 (en) * 2017-06-15 2023-04-05 돌비 인터네셔널 에이비 Method, apparatus and system for optimizing communication between transmitter and receiver in computer-assisted reality applications
US11164606B2 (en) * 2017-06-30 2021-11-02 Qualcomm Incorporated Audio-driven viewport selection

Also Published As

Publication number Publication date
MX2023012965A (en) 2023-11-15
EP3695613A1 (en) 2020-08-19
AU2023263436A1 (en) 2023-11-23
ES2970490T3 (en) 2024-05-29
US20200278828A1 (en) 2020-09-03
CA3230231A1 (en) 2019-04-18
CA3230221A1 (en) 2019-04-18
MX2023012967A (en) 2023-11-15
ZA202208388B (en) 2023-03-29
CA3078858A1 (en) 2019-04-18
BR112020008073A2 (en) 2020-11-03
MX2023012963A (en) 2023-11-15
SG10202106080XA (en) 2021-07-29
AR125884A2 (en) 2023-08-23
EP4329319A3 (en) 2024-04-24
EP3695613C0 (en) 2024-01-03
ZA202208364B (en) 2023-03-29
EP3695613B1 (en) 2024-01-03
CA3230304A1 (en) 2019-04-18
MX2023012964A (en) 2023-11-15
AR125882A2 (en) 2023-08-23
TWI713911B (en) 2020-12-21
AR125880A2 (en) 2023-08-23
MX2020003450A (en) 2020-10-01
CN116193215A (en) 2023-05-30
CN116193212A (en) 2023-05-30
TW201924362A (en) 2019-06-16
CN116193213A (en) 2023-05-30
CN111466122A (en) 2020-07-28
KR102568373B1 (en) 2023-08-18
CN116193214A (en) 2023-05-30
EP4329319A2 (en) 2024-02-28
JP2020537418A (en) 2020-12-17
MX2023012966A (en) 2023-11-15
CA3230205A1 (en) 2019-04-18
WO2019072984A1 (en) 2019-04-18
KR20200078537A (en) 2020-07-01
AU2018348762B2 (en) 2023-08-10
RU2765569C1 (en) 2022-02-01
CA3230310A1 (en) 2019-04-18
RU2750505C1 (en) 2021-06-29
AR113357A1 (en) 2020-04-22
ZA202002064B (en) 2023-02-22
AR125881A2 (en) 2023-08-23
KR20230130729A (en) 2023-09-12
AU2018348762A1 (en) 2020-05-07
US20220261215A1 (en) 2022-08-18
JP7295851B2 (en) 2023-06-21
US11354084B2 (en) 2022-06-07
CN111466122B (en) 2023-02-28
AR125883A2 (en) 2023-08-23
JP2023116635A (en) 2023-08-22

Similar Documents

Publication Publication Date Title
ZA202208364B (en) Optimizing audio delivery for virtual reality applications
GB201702700D0 (en) Virtual reality
AU201813289S (en) Virtual reality headset
AU201616338S (en) Virtual reality headset
EP3318958A4 (en) Virtual reality experience device
GB2567530B (en) Virtual reality parallax correction
GB2563276B (en) Virtual reality systems
GB2560004B (en) Virtual reality
GB201712569D0 (en) Virtual reality video processing
GB201900434D0 (en) Virtual reality headset
GB201620985D0 (en) Virtual reality
EP3510474A4 (en) Virtual reality input
GB2558284B (en) Virtual reality
GB201802888D0 (en) Virtual reality
GB2562808B (en) Reality viewer
EP3556094A4 (en) Virtual reality buffering
GB201622170D0 (en) Virtual reality
GB2560003B (en) Virtual reality
EP3673331A4 (en) Virtual reality system
GB2556114B (en) Virtual reality
AU2017902751A0 (en) Virtual reality apparatus
GB201513050D0 (en) Virtual reality headset user input system