JP7477600B2 - マルチストリーム中心窩ディスプレイトランスポート - Google Patents
マルチストリーム中心窩ディスプレイトランスポート Download PDFInfo
- Publication number
- JP7477600B2 JP7477600B2 JP2022515065A JP2022515065A JP7477600B2 JP 7477600 B2 JP7477600 B2 JP 7477600B2 JP 2022515065 A JP2022515065 A JP 2022515065A JP 2022515065 A JP2022515065 A JP 2022515065A JP 7477600 B2 JP7477600 B2 JP 7477600B2
- Authority
- JP
- Japan
- Prior art keywords
- stream
- image
- regions
- receiver
- frame
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000004434 saccadic eye movement Effects 0.000 claims description 45
- 238000009877 rendering Methods 0.000 claims description 39
- 238000000034 method Methods 0.000 claims description 27
- 238000002156 mixing Methods 0.000 claims description 13
- 230000004044 response Effects 0.000 claims description 10
- 239000002131 composite material Substances 0.000 claims description 8
- 238000010586 diagram Methods 0.000 description 19
- 239000000872 buffer Substances 0.000 description 11
- 239000000203 mixture Substances 0.000 description 9
- 230000003139 buffering effect Effects 0.000 description 6
- 238000012545 processing Methods 0.000 description 6
- 230000000007 visual effect Effects 0.000 description 5
- 230000005540 biological transmission Effects 0.000 description 4
- 230000007246 mechanism Effects 0.000 description 4
- 230000010363 phase shift Effects 0.000 description 4
- 238000004891 communication Methods 0.000 description 3
- 238000001514 detection method Methods 0.000 description 3
- 239000000284 extract Substances 0.000 description 3
- 230000003190 augmentative effect Effects 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 230000004424 eye movement Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000001711 saccadic effect Effects 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 210000004556 brain Anatomy 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 230000005764 inhibitory process Effects 0.000 description 1
- 230000015654 memory Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000004461 rapid eye movement Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/013—Eye tracking input arrangements
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- H04N13/361—Reproducing mixed stereoscopic images; Reproducing mixed monoscopic and stereoscopic images, e.g. a stereoscopic image overlay window on a monoscopic image background
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/04—Context-preserving transformations, e.g. by using an importance map
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4092—Image resolution transcoding, e.g. by using client-server architectures
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/60—Network streaming of media packets
- H04L65/61—Network streaming of media packets for supporting one-way streaming services, e.g. Internet radio
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/60—Network streaming of media packets
- H04L65/70—Media network packetisation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/60—Network streaming of media packets
- H04L65/75—Media network packet handling
- H04L65/764—Media network packet handling at the destination
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/80—Responding to QoS
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/131—Protocols for games, networked simulations or virtual reality
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
- H04N13/156—Mixing image signals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
- H04N13/161—Encoding, multiplexing or demultiplexing different image signal components
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
- H04N13/167—Synchronising or controlling image signals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/194—Transmission of image signals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- H04N13/366—Image reproducers using viewer tracking
- H04N13/383—Image reproducers using viewer tracking for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- H04N13/398—Synchronisation thereof; Control thereof
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
- H04N19/132—Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
- H04N19/162—User input
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
- H04N19/164—Feedback from the receiver or from the transmission channel
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
- H04N19/167—Position within a video image, e.g. region of interest [ROI]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
- H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/46—Embedding additional information in the video signal during the compression process
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
- H04N19/59—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
- H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
- H04N21/234345—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements the reformatting operation being performed only on part of the stream, e.g. a region of the image or a time segment
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
- H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
- H04N21/234363—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements by altering the spatial resolution, e.g. for clients with a lower screen resolution
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
- H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
- H04N21/234381—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements by altering the temporal resolution, e.g. decreasing the frame rate by frame skipping
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/236—Assembling of a multiplex stream, e.g. transport stream, by combining a video stream with other content or additional data, e.g. inserting a URL [Uniform Resource Locator] into a video stream, multiplexing software data into a video stream; Remultiplexing of multiplex streams; Insertion of stuffing bits into the multiplex stream, e.g. to obtain a constant bit-rate; Assembling of a packetised elementary stream
- H04N21/2365—Multiplexing of several video streams
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/4302—Content synchronisation processes, e.g. decoder synchronisation
- H04N21/4307—Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen
- H04N21/43072—Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen of multiple content streams on the same device
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/434—Disassembling of a multiplex stream, e.g. demultiplexing audio and video streams, extraction of additional data from a video stream; Remultiplexing of multiplex streams; Extraction or processing of SI; Disassembling of packetised elementary stream
- H04N21/4347—Demultiplexing of several video streams
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/442—Monitoring of processes or resources, e.g. detecting the failure of a recording device, monitoring the downstream bandwidth, the number of times a movie has been viewed, the storage space available from the internal hard disk
- H04N21/44213—Monitoring of end-user related data
- H04N21/44218—Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/81—Monomedia components thereof
- H04N21/816—Monomedia components thereof involving special video data, e.g 3D video
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Computer Networks & Wireless Communication (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Social Psychology (AREA)
- Databases & Information Systems (AREA)
- Controls And Circuits For Display Device (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
- Image Processing (AREA)
Description
ディスプレイの解像度とリフレッシュレートが上がるにつれて、表示される画像に必要な伝送帯域幅が主要な制限要因になってくる。仮想現実(VR)及び拡張現実(AR)ヘッドセット又はヘッドマウントディスプレイ(HMD)の場合、これは、ケーブルの物理的な太さがユーザの移動を妨げるためにさらに大きな問題になり、より高解像度の画像を伝送するためにより多くのワイヤを追加するのが受け入れられない解決策である。ユーザに対する没入型環境を作り出すために、VR及びARビデオストリーミングアプリケーションは、典型的には、高解像度、及び、高データレートと同等である高フレームレートを有する。VR及びARディスプレイの場合、今日一般的に行われているように、完全に均一な解像度の画像を送信するのは無駄である。
Claims (20)
- 回路を含み、フレームの複数の領域をレンダリングするように構成されたレンダリングユニットであって、前記複数の領域の各領域は、他の領域とは異なる解像度でレンダリングされる、レンダリングユニットと、
回路を含む送信機と、を備え、
前記送信機は、
前記複数の領域の各領域を個別のストリーム内の個別の画像として受信機に送信することと、
第1のストリームが送信される時間を変更することで前記第1のストリームの位相を第2のストリームに対しシフトすることによって、前記第1のストリームを送信することであって、前記位相は、前記フレーム内の中心窩領域の位置に対応する量だけシフトされ、前記第1のストリームは前記中心窩領域を含み、前記第2のストリームは背景画像を含む、ことと、
を行うように構成されている、
システム。 - 前記中心窩領域は、第1のリフレッシュレートで更新され、
前記背景画像は、第2のリフレッシュレートで更新され、
前記第2のリフレッシュレートは、前記第1のリフレッシュレートとは異なる、
請求項1のシステム。 - 前記レンダリングユニットは、眼球サッケードが検出されたという指標を受信したことに応じて、異なる解像度でレンダリングされる領域の数を増やすように構成されている、
請求項1のシステム。 - 前記レンダリングユニットは、
前記指標を受信する前に、第1のフレームに対して2つの個別の領域をレンダリングし、
前記指標を受信した後に、第2のフレームに対して3つ以上の個別の領域をレンダリングするように構成されている、
請求項3のシステム。 - 前記レンダリングユニットは、前記指標を受信したことに応じて、前記中心窩領域のリフレッシュレートを調整するように構成されている、
請求項3のシステム。 - 前記送信機は、前記複数の領域を組み合わせて合成画像を生成する方法を指定するために、ブレンドマスクを前記受信機に伝達するように構成されている、
請求項1のシステム。 - 前記送信機は、前記背景画像をスケーリングする方法を指定するために、背景画像スケーリング属性を前記受信機に伝達するように構成されている、
請求項1のシステム。 - 回路を含むレンダリングユニットが、フレームの複数の領域をレンダリングすることであって、前記複数の領域の各領域は、他の領域とは異なる解像度でレンダリングされる、ことと、
回路を含む送信機が、各領域を個別のストリーム内の個別の画像として受信機に送信することと、
第1のストリームが送信される時間を変更することで前記第1のストリームの位相を第2のストリームに対してシフトすることによって、前記第1のストリームを送信することであって、前記位相は、前記フレーム内の中心窩領域の位置に対応する量だけシフトされ、前期第1のストリームは前記中心窩領域を含み、前記第2のストリームは背景画像を含む、ことと、を含む、
方法。 - 前記中心窩領域は、第1のリフレッシュレートで更新され、
前記背景画像は、第2のリフレッシュレートで更新され、
前記第2のリフレッシュレートは、前記第1のリフレッシュレートとは異なる、
請求項8の方法。 - 眼球サッケードが検出されたという指標を受信したことに応じて、異なる解像度でレンダリングされる領域の数を増やすことをさらに含む、
請求項8の方法。 - 前記指標を受信する前に、第1のフレームに対して2つの個別の領域をレンダリングすることと、
前記指標を受信した後に、第2のフレームに対して3つ以上の個別の領域をレンダリングすることと、をさらに含む、
請求項10の方法。 - 前記指標を受信したことに応じて、前記中心窩領域のリフレッシュレートを調整することをさらに含む、
請求項10の方法。 - 前記複数の領域を組み合わせて合成画像を生成する方法を指定するために、ブレンドマスクを前記受信機に伝達することをさらに含む、
請求項8の方法。 - 前記背景画像をスケーリングする方法を指定するために、背景画像スケーリング属性を前記受信機に伝達することをさらに含む、
請求項8の方法。 - 回路を含む送信機と、
受信機と、を備え、
前記送信機は、
フレームの複数の領域をレンダリングすることであって、前記複数の領域の各領域は、他の領域とは異なる解像度でレンダリングされる、ことと、
各領域を個別のストリーム内の個別の画像として前記受信機に送信することと、
第1のストリームが送信される時間を変更することで前記第1のストリームの位相を第2のストリームに対してシフトすることによって、前記第1のストリームを送信することであって、前記位相は、前記フレーム内の中心窩領域の位置に対応する量だけシフトされ、前期第1のストリームは前記中心窩領域を含み、前記第2のストリームは背景画像を含む、ことと、
を行うように構成されており、
前記受信機は、
複数のストリームからの前記複数の領域を組み合わせて、合成画像を生成することと、
前記合成画像をディスプレイに送ることと、
を行うように構成されている、
システム。 - 前記中心窩領域は、第1のリフレッシュレートで更新され、
前記背景画像は、第2のリフレッシュレートで更新され、
前記第2のリフレッシュレートは、前記第1のリフレッシュレートとは異なる、
請求項15のシステム。 - 前記送信機は、眼球サッケードが検出されたという指標を受信したことに応じて、異なる解像度でレンダリングされる領域の数を増やすように構成されている、
請求項15のシステム。 - 前記送信機は、
前記指標を受信する前に、第1のフレームに対して2つの個別の領域をレンダリングし、
前記指標を受信した後に、第2のフレームに対して3つ以上の個別の領域をレンダリングするように構成されている、
請求項17のシステム。 - 前記送信機は、前記指標を受信したことに応じて、前記中心窩領域のリフレッシュレートを調整するように構成されている、
請求項17のシステム。 - 前記送信機は、前記複数の領域を組み合わせて前記合成画像を生成する方法を指定するために、ブレンドマスクを前記受信機に伝達するように構成されている、
請求項15のシステム。
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US16/575,804 US11307655B2 (en) | 2019-09-19 | 2019-09-19 | Multi-stream foveal display transport |
US16/575,804 | 2019-09-19 | ||
PCT/IB2020/058783 WO2021053644A1 (en) | 2019-09-19 | 2020-09-21 | Multi-stream foveal display transport |
Publications (3)
Publication Number | Publication Date |
---|---|
JP2022548534A JP2022548534A (ja) | 2022-11-21 |
JPWO2021053644A5 JPWO2021053644A5 (ja) | 2023-07-20 |
JP7477600B2 true JP7477600B2 (ja) | 2024-05-01 |
Family
ID=74880843
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
JP2022515065A Active JP7477600B2 (ja) | 2019-09-19 | 2020-09-21 | マルチストリーム中心窩ディスプレイトランスポート |
Country Status (6)
Country | Link |
---|---|
US (1) | US11307655B2 (ja) |
EP (1) | EP4032283A4 (ja) |
JP (1) | JP7477600B2 (ja) |
KR (1) | KR20220065794A (ja) |
CN (1) | CN114402613A (ja) |
WO (1) | WO2021053644A1 (ja) |
Families Citing this family (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11353723B2 (en) * | 2019-10-31 | 2022-06-07 | Tectus Corporation | Saccade detection and endpoint prediction for electronic contact lenses |
FR3116351B1 (fr) * | 2020-11-18 | 2023-06-16 | Thales Sa | Dispositif d'affichage tête haute porté et procédé d'affichage associé |
US20220201271A1 (en) * | 2020-12-23 | 2022-06-23 | Facebook Technologies, Llc | Temporal foveated rendering |
CN113362450B (zh) * | 2021-06-02 | 2023-01-03 | 聚好看科技股份有限公司 | 一种三维重建方法、装置及系统 |
TWI783588B (zh) * | 2021-07-23 | 2022-11-11 | 瑞昱半導體股份有限公司 | 可應用於在顯示裝置中進行多畫面處理之縮放器積體電路 |
US11874958B2 (en) * | 2021-09-24 | 2024-01-16 | Apple Inc. | Eye detection methods and devices |
CN118215954A (zh) * | 2021-11-10 | 2024-06-18 | 索尼半导体解决方案公司 | 显示装置、显示系统及显示驱动方法 |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2018227100A1 (en) | 2017-06-09 | 2018-12-13 | Sony Interactive Entertainment Inc. | Optimized shadows and adaptive mesh skinning in a foveated rendering system |
US20190043167A1 (en) | 2017-12-29 | 2019-02-07 | Intel Corporation | Foveated image rendering for head-mounted display devices |
Family Cites Families (29)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6141062A (en) * | 1998-06-01 | 2000-10-31 | Ati Technologies, Inc. | Method and apparatus for combining video streams |
US20050047504A1 (en) | 2003-09-03 | 2005-03-03 | Sung Chih-Ta Star | Data stream encoding method and apparatus for digital video compression |
US8704834B2 (en) * | 2007-12-03 | 2014-04-22 | Nvidia Corporation | Synchronization of video input data streams and video output data streams |
US9024964B2 (en) | 2008-06-06 | 2015-05-05 | Omnivision Technologies, Inc. | System and method for dithering video data |
US20110317770A1 (en) * | 2010-06-24 | 2011-12-29 | Worldplay (Barbados) Inc. | Decoder for multiple independent video stream decoding |
US8615140B2 (en) | 2011-11-18 | 2013-12-24 | Canon Kabushiki Kaisha | Compression of image data in accordance with depth information of pixels |
US9485515B2 (en) | 2013-08-23 | 2016-11-01 | Google Inc. | Video coding using reference motion vectors |
US9998750B2 (en) | 2013-03-15 | 2018-06-12 | Cisco Technology, Inc. | Systems and methods for guided conversion of video from a first to a second compression format |
US9426475B2 (en) | 2013-06-25 | 2016-08-23 | VIXS Sytems Inc. | Scene change detection using sum of variance and estimated picture encoding cost |
WO2015138008A1 (en) | 2014-03-10 | 2015-09-17 | Euclid Discoveries, Llc | Continuous block tracking for temporal prediction in video encoding |
JP2016225849A (ja) * | 2015-05-29 | 2016-12-28 | 株式会社ジャパンディスプレイ | 表示システム及び表示方法 |
US9857871B2 (en) * | 2015-09-04 | 2018-01-02 | Sony Interactive Entertainment Inc. | Apparatus and method for dynamic graphics rendering based on saccade detection |
US10460704B2 (en) | 2016-04-01 | 2019-10-29 | Movidius Limited | Systems and methods for head-mounted display adapted to human visual mechanism |
EP3440495A1 (en) | 2016-04-08 | 2019-02-13 | Google LLC | Encoding image data at a head mounted display device based on pose information |
US10341650B2 (en) | 2016-04-15 | 2019-07-02 | Ati Technologies Ulc | Efficient streaming of virtual reality content |
US10805592B2 (en) | 2016-06-30 | 2020-10-13 | Sony Interactive Entertainment Inc. | Apparatus and method for gaze tracking |
JP2019525248A (ja) * | 2016-08-01 | 2019-09-05 | イマジン・コーポレイション | 再構成可能なディスプレイおよび再構成可能なディスプレイのための方法 |
CN109792562B (zh) * | 2016-09-26 | 2021-04-13 | 杜比实验室特许公司 | 视频数据的基于内容的流分割 |
US10291931B2 (en) | 2016-10-13 | 2019-05-14 | Ati Technologies Ulc | Determining variance of a block of an image based on a motion vector for the block |
US11290699B2 (en) * | 2016-12-19 | 2022-03-29 | Dolby Laboratories Licensing Corporation | View direction based multilevel low bandwidth techniques to support individual user experiences of omnidirectional video |
US10490157B2 (en) | 2017-01-03 | 2019-11-26 | Screenovate Technologies Ltd. | Compression of distorted images for head-mounted display |
EP3370419B1 (en) | 2017-03-02 | 2019-02-13 | Axis AB | A video encoder and a method in a video encoder |
US20180262758A1 (en) | 2017-03-08 | 2018-09-13 | Ostendo Technologies, Inc. | Compression Methods and Systems for Near-Eye Displays |
US10665209B2 (en) * | 2017-05-18 | 2020-05-26 | Synaptics Incorporated | Display interface with foveal compression |
WO2019018365A1 (en) | 2017-07-17 | 2019-01-24 | Tobii Ab | CLOTHING DEVICES FOR EYE FOLLOW-UP |
US10553016B2 (en) * | 2017-11-15 | 2020-02-04 | Google Llc | Phase aligned foveated rendering |
US10798455B2 (en) * | 2017-12-22 | 2020-10-06 | Comcast Cable Communications, Llc | Video delivery |
US10904552B2 (en) | 2018-02-15 | 2021-01-26 | Intel Corporation | Partitioning and coding mode selection for video encoding |
US20200195944A1 (en) | 2018-12-14 | 2020-06-18 | Advanced Micro Devices, Inc. | Slice size map control of foveated coding |
-
2019
- 2019-09-19 US US16/575,804 patent/US11307655B2/en active Active
-
2020
- 2020-09-21 CN CN202080064752.8A patent/CN114402613A/zh active Pending
- 2020-09-21 EP EP20866101.7A patent/EP4032283A4/en active Pending
- 2020-09-21 JP JP2022515065A patent/JP7477600B2/ja active Active
- 2020-09-21 WO PCT/IB2020/058783 patent/WO2021053644A1/en unknown
- 2020-09-21 KR KR1020227011803A patent/KR20220065794A/ko active IP Right Grant
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2018227100A1 (en) | 2017-06-09 | 2018-12-13 | Sony Interactive Entertainment Inc. | Optimized shadows and adaptive mesh skinning in a foveated rendering system |
US20190043167A1 (en) | 2017-12-29 | 2019-02-07 | Intel Corporation | Foveated image rendering for head-mounted display devices |
Also Published As
Publication number | Publication date |
---|---|
CN114402613A (zh) | 2022-04-26 |
EP4032283A1 (en) | 2022-07-27 |
US11307655B2 (en) | 2022-04-19 |
EP4032283A4 (en) | 2023-09-27 |
JP2022548534A (ja) | 2022-11-21 |
KR20220065794A (ko) | 2022-05-20 |
US20210089119A1 (en) | 2021-03-25 |
WO2021053644A1 (en) | 2021-03-25 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
JP7477600B2 (ja) | マルチストリーム中心窩ディスプレイトランスポート | |
US11290699B2 (en) | View direction based multilevel low bandwidth techniques to support individual user experiences of omnidirectional video | |
US11330262B2 (en) | Local image enhancing method and apparatus | |
CN109891906A (zh) | 视图感知360度视频流化 | |
US11711530B2 (en) | Tile shuffling for 360 degree video decoding | |
JP2023512937A (ja) | 無線仮想拡張現実システムでのレイテンシの隠蔽 | |
KR20220054604A (ko) | 단일-스트림 포빌 디스플레이 전송 | |
WO2021199205A1 (ja) | 画像データ転送装置、画像表示システム、および画像データ転送方法 | |
US20230011586A1 (en) | Electronic device, server and methods for viewport prediction based on head and eye gaze | |
KR102228025B1 (ko) | 고화질의 vr 컨텐츠의 스트리밍 제공 방법, 단말 및 서버 | |
US11134236B2 (en) | Image processing device and system | |
WO2022028684A1 (en) | Improved split rendering for extended reality (xr) applications | |
US20240107110A1 (en) | Changing video tracks in immersive videos | |
WO2021199128A1 (ja) | 画像データ転送装置、画像生成方法およびコンピュータプログラム | |
WO2021193362A1 (ja) | 画像処理装置および画像処理方法 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
A521 | Request for written amendment filed |
Free format text: JAPANESE INTERMEDIATE CODE: A523 Effective date: 20230711 |
|
A621 | Written request for application examination |
Free format text: JAPANESE INTERMEDIATE CODE: A621 Effective date: 20230711 |
|
A871 | Explanation of circumstances concerning accelerated examination |
Free format text: JAPANESE INTERMEDIATE CODE: A871 Effective date: 20230711 |
|
A131 | Notification of reasons for refusal |
Free format text: JAPANESE INTERMEDIATE CODE: A131 Effective date: 20230822 |
|
A601 | Written request for extension of time |
Free format text: JAPANESE INTERMEDIATE CODE: A601 Effective date: 20231122 |
|
A521 | Request for written amendment filed |
Free format text: JAPANESE INTERMEDIATE CODE: A523 Effective date: 20240117 |
|
TRDD | Decision of grant or rejection written | ||
A01 | Written decision to grant a patent or to grant a registration (utility model) |
Free format text: JAPANESE INTERMEDIATE CODE: A01 Effective date: 20240326 |
|
A61 | First payment of annual fees (during grant procedure) |
Free format text: JAPANESE INTERMEDIATE CODE: A61 Effective date: 20240418 |
|
R150 | Certificate of patent or registration of utility model |
Ref document number: 7477600 Country of ref document: JP Free format text: JAPANESE INTERMEDIATE CODE: R150 |