WO2018185534A1 - Conversion de 2d en 3d à faible latence pour un jeu de réalité virtuelle - Google Patents

Conversion de 2d en 3d à faible latence pour un jeu de réalité virtuelle Download PDF

Info

Publication number
WO2018185534A1
WO2018185534A1 PCT/IB2017/052055 IB2017052055W WO2018185534A1 WO 2018185534 A1 WO2018185534 A1 WO 2018185534A1 IB 2017052055 W IB2017052055 W IB 2017052055W WO 2018185534 A1 WO2018185534 A1 WO 2018185534A1
Authority
WO
WIPO (PCT)
Prior art keywords
pixels
image
source
images
color
Prior art date
Application number
PCT/IB2017/052055
Other languages
English (en)
Inventor
Shubham MISHRA
Vrushali PRASADE
Harikrishna VALIYATH
Original Assignee
Absentia Virtual Reality Pvt. Ltd.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Absentia Virtual Reality Pvt. Ltd. filed Critical Absentia Virtual Reality Pvt. Ltd.
Publication of WO2018185534A1 publication Critical patent/WO2018185534A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—3D [Three Dimensional] image rendering
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/12—Edge-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/194—Segmentation; Edge detection involving foreground-background segmentation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/261—Image signal generators with monoscopic-to-stereoscopic image conversion

Definitions

  • Virtual-reality, three-dimensional (3D), games can provide an immersive and exciting gaming experience.
  • 3D titles include: Kitchen for the Project Morpheus platform, Minecraft Hololens for the Microsoft Hololens platform, Time Machine for the Oculus Rift platform, and Keep Talking and Nobody Explodes for the Gear VR platform.
  • 3D versions of some conventional two-dimensional (2D) games are being released, there is a vast trove of conventional 2D games that are not scheduled for conversion to 3D. It would be desirable to provide for an immersive virtual-reality experience for such games, and to allow owners of games that are to be rereleased to benefit from immersive virtual-reality without requiring them to buy the new versions of already-owned games.
  • FIGURE 1 is a schematic diagram of a gaming system and associated process for converting a 2D game to 3D.
  • FIGURE 2 is a grey-scale rendition of a screen shot day-map source image to be converted to a stereo pair according to the process of FIG. 1.
  • FIGURE 3 is a grey-scale rendition of a screen shot of aside-by-side stereo pair of images resulting from a 2D-to-3D conversion of the image of FIG. 2.
  • a nose has been added to minimize disorientation of a player using the gaming system of FIG. 1.
  • FIGURE 4 is a grey-scale rendition of a screen shot of a night-map source image that may be converted to a stereo pair according to the process of FIG. 1.
  • FIGURE 5 is a schematic diagram of a source image showing selected pixels of a source image according to the process of FIG. 1.
  • FIGURE 6 is a schematic diagram of the source image of FIG. 5for explaining edge detection and object resizing as implemented in the process of FIG. 1.
  • a subset of source- image pixels is selected to reduce the number of pixels to be handled while providing effective immersion.
  • Edge detection is used to identify image objects. Identified objects are resized in at least one image of a stereo pair derived from the source image. Presentation of the stereo pair to respective left and right eyes provides a 3D effect. The simple operations called for by the process along with the data reduction due to image sampling allows a low-latency conversion suitable for gaming.
  • a virtual-reality gaming system 100 includes a game controller 102, a game computer 104, and a virtual-reality (VR) headset 106, i.e., a VR head-mounted display (HMD).
  • Game controller 102 may include such features as multiple buttons, directional pads, joysticks, and motion detection, for example.
  • Computer 104 is a programmable hardware system for running a 2D computer game 108 and 2D-to-3D conversion software 110.
  • computer 104 can include an input buffer 112, left and right (L&R) back buffers 114, and L&R front buffers 116.
  • Input buffer 112 is used to store a 2D source image 118 generated by computer game 108.
  • Front buffers 116 are used to store for display stereo image pairs resulting from the 2D- to-3D conversion. Each stereo pair includes a left-eye 120 image and a right-eye image 122.
  • Back buffers 114 are used to pipeline converted images to front buffers 116 for sequential display.
  • VR headset 106 includes a display system 124, an inter-pupil distance dial 126, a head-tracking system 128, and a haptic feedback system 130.
  • Display system 124 includes: a left (L-) display 132 for display a left-eye image 120; and a right (R-) display 134 for display a right-eye image 122.
  • Inter-pupil distance dial 126 can be used to adjust the separation of displays 132 and 134 to match an inter-pupillary distance of a user.
  • the default distance is set at 55millimeters (mm).
  • the dial is capable of adjusting the distance in the range 50 to 60 mm. Also vertical adjustment is provided to adjust the distance between screen and the eyes for myopic vision correction.
  • 2D-to-3D converter 114 is a hardware device, e.g., media encoded with code that, when executed by hardware, implements a 2D-to-3D conversion process 150 for converting a 2D source image, e.g., image 200, FIG. 2, to a stereo image pair, e.g., image pair 300, FIG. 3.
  • Each source image includes a two-dimensional array of pixels, each of which is characterized by a respective position (x, y) vector and a respective color (R.G.B) vector.
  • the effective distance between the left-eye and right-eye images is adjusted using dial 126.
  • the adjustment allows matching of a distance between displays 132 and 134 with the inter-pupillary distance of the player.
  • a color map for the image is characterized.
  • a day map such as the one characterizing source image 200 (FIG. 2) may be distinguished from a night map, such as the one characterizing a source image 400 of FIG. 4.
  • This characterizing can involve identifying the minimum and the maximum value of 8-bit-per-color-channel pixel data in each of the R, G and B color dimensions. If the difference between the minimum and the maximum in each scalar is less than, for example, 100, the map is identified to be low color range, which is characteristic of a night map, otherwise, the image is considered high color range, which is characteristic of a day map.
  • a background color range is set as a function of the
  • a range of 50 per color channel can be set for a day map while a narrower range of 25 pre color channel can be used for a night map.
  • a background portion of the source image is identified using the selected background range. For example, if a day map is determined at 153, then for each RGB color dimension, a range of 50, topping out at the maximum for that color value in the source image, is used. In one scenario, the color ranges could be R: 210-260, G: 190-240, B: 130-180. Pixels falling within this range are considered part of the background. If a night map is determined, the range is 25 values to the maximum for each color dimension. In FIG. 5, the unshaded portion 502 of image 500 is considered background, in contrast to a red image object 504 and a blue image object 506. Note that FIG. 5 is highly schematized for expository purposes.
  • a subset of the pixels of the 2D source image is selected. That is, a subset of the pixels of the source image is selected for further processing. This reduces the amount of data to be processed, and, concomitantly, provides a lower latency. For example, as shown in the example of FIG. 5, an image can be divided into 3x3 arrays of pixels, with the center pixel of each array being selected and the remaining eight pixels in each array being unselected.
  • the sampling is performed before action 152, to reduce the amount of processing required for identifying background images.
  • the background of the source image is determined using all pixels in the image for greater accuracy.
  • edges are detected. Edge detection can be performed by scanning columns of selected pixels for breakpoints. In the illustrated embodiment, one in 3 pixels in every vertical strip is selected and the pixel data retrieved. If the difference between vertically adjacent pixels in an individual color-dimension scalar is more than 50 and the sum across color dimensions of the differences is more than 150, then the vertically adjacent pixels represent an edge.Each edge is extrapolated so that it extends three pixels horizontally.
  • FIG. 6 which presents an alternative representation of source image 500
  • the selected pixels are arranged in rows R1-R8 and columns C1-C8. Scanning is column-wise from top to bottom, as indicated by arrow 602. Scanning from pixel R8C5 to R7C5 does not detect an edge as the pixels involved have similar colors. However, pixels R7C5 and R6C5 have sharply differing colors, so an edge is detectedtherebetween. No edge is detected between pixels R6C5 and R5C5, or between pixels R5C5 and R4C4. An edge is detected between pixels R4C5 and R3C5.
  • image objects are identified. Once a next edge (e.g., between R4C5 and R3C5) is determined, the color vectors associated with the pixel (e.g., R6C5) below the previous edge (between R7C5 and R6C5) and the pixel (R4C5) above the present edge (e.g., between R4C5 and R3C5) are compared. If the color difference in individual scalars is less than 50, then this block is identified as a single object that may be considered for resizing. However, if the vertical extent of this block is less than 80 pixels or greater than 600 pixels, it is not considered for resizing and/or other modification. Anarrow strip of pixels is not be consequential even after adding depth, while a strip taller than 600 is treated as a background object (sky/ground or similar). Note that the identified image objects are 3-pixel-wide vertical strips such as image object 604 in FIG. 6.
  • the identified image objects are differentially modified, i.e., an image object is modified one way for the left image and another way for the right image.
  • “Differentially modified” encompasses modifying an image object for one of the left and right images, but not for the other.
  • image objects are resized for the left image while the right image remains unchanged.
  • identified image objects are scaled up 1096 vertically. More specifically, the top extends 596 higher and the bottom extends 596 lower, as shown
  • an object that extends 100 pixels from 250 to 350 vertically is resized to extend from 245 to 355.
  • an object that extends 500 pixels is resized so that is extends 550 pixels.
  • a nose 302, or rather a left-half nose 304 and a right-half nose 306 are added to the left and right images308 and 310 of a stereo image pair 300, as shown in FIG. 3. This helps prevent disorientation of the player that might occur as the player's head turns to change the view in the VR field.
  • the left and right images on route to displays 120 and 124 are time-staggered, e.g., 4-5 ms, so that they are slightly out-of- phase to enhance the sensation of depth.
  • the left and right images are presented respectively to the left and right eyes. Actions 155-162 can be repeated for each image in a video sequence.
  • VR headset 198 includes two or more (as opposed to only one) inertial measurement units (IMUs), the outputs of which can be time-multiplexed to improve time resolution.
  • IMUs can be 9-axis IMUs, with 3-axis accelerometers, 3- axis gyroscopes, and 3-axis magnetometers.
  • the IMU outputs are time- staggered with a small phase shift to allow measurement of intermediate values.
  • the signals can then be converted to human-interface-device (HID) signals to emulate mouse, keyboard, or joysticks with complete
  • the user can map keys of a keyboard in a game with any particular action captured by the gyro or the
  • VR headset 106 also employs haptics involving feature abstraction and haptic feedback.
  • Audiofeature abstraction can capture features like gun shots, bomb blasts, steps (e.g., walking) and recoil are to be gauged, e.g., by building a directory for games.accessing the sounds, and
  • the audio features can be game specific; the user selects the game to be played from a list.
  • pixel mapping can be done to extract features like: the health bar falls if the player gets shot by a gun, so once the pixels in that region change, it can depict a gunshot. Recoil can be checked by the mouse click as it relates to gunshot.
  • other game- specific features can be determined.
  • Haptic feedback can take the form of vibrations and push-pull effects .
  • An array of vibrationsensors is triggered by a microcontroller which receives feedback from the feature extraction software.
  • the vibrations vary in terms of intensity, patterns and time. These sensors are placed in the cushion of the VR headsetlOS.
  • a solenoid based pushback is produced.
  • the solenoid is placed on the top of the head-mounted display (HMD) and the tip is connected to a strap which goes over the head.
  • HMD head-mounted display
  • VR headset 106 Whenever the solenoid is triggered, VR headset 106 is slightly pulled giving a push-back sensation to the user.
  • Signals from game computer 104 can be taken from the video (e.g., HDMI or VGA) output, and these video signals can be converted to a Mobile Industry Processor Interface (MIPI) specification or a Low-voltage
  • MIPI Mobile Industry Processor Interface
  • LVDS differential signaling
  • the system is USB powered.
  • the ARM Cortex is used to interact with the two IMU's and emulate an HID.
  • spot jogger band developed which has an
  • accelerometer chip This band can be attached to the foot and track the motion of walking and running and sending this to gaming computer 104 in real time to interact with games to move around in real time.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Computer Graphics (AREA)
  • Processing Or Creating Images (AREA)

Abstract

L'invention concerne un casque de réalité virtuelle (VR) qui fournit une meilleure immersion dans un jeu bidimensionnel (2D) à l'aide d'une conversion de 2D en 3D à faible latence. Des colonnes de pixels sont balayées en vue d'identifier des bords d'objet d'image dans les formes de pixels verticalement adjacents dont la couleur est différente de plus d'un seuil prédéterminé. Afin de réduire le temps de traitement, seule chaque troisième colonne est considérée et seulement chaque troisième pixel dans chaque colonne considérée est considéré. Des bandes d'objets sont identifiées par corrélation de paires verticales de bords. Des bandes d'objet identifiées sont agrandies verticalement pour fournir une image d'une paire d'images stéréo. Des bandes d'objet ne sont pas redimensionnées pour l'autre image de la paire stéréo. Les images de la paire stéréo sont respectivement présentées aux yeux gauche et droit d'un joueur afin de fournir une expérience plus immersive.
PCT/IB2017/052055 2017-04-05 2017-04-10 Conversion de 2d en 3d à faible latence pour un jeu de réalité virtuelle WO2018185534A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
IN201741012256 2017-04-05
IN201741012256 2017-04-05

Publications (1)

Publication Number Publication Date
WO2018185534A1 true WO2018185534A1 (fr) 2018-10-11

Family

ID=63712055

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/IB2017/052055 WO2018185534A1 (fr) 2017-04-05 2017-04-10 Conversion de 2d en 3d à faible latence pour un jeu de réalité virtuelle

Country Status (1)

Country Link
WO (1) WO2018185534A1 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190058857A1 (en) * 2017-08-15 2019-02-21 International Business Machines Corporation Generating three-dimensional imagery

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP2656315B1 (fr) * 2010-12-22 2016-10-05 Legend3D, Inc. Système et procédé pour flux de travail à itération minimale pour améliorer la profondeur de séquence d'images

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP2656315B1 (fr) * 2010-12-22 2016-10-05 Legend3D, Inc. Système et procédé pour flux de travail à itération minimale pour améliorer la profondeur de séquence d'images

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190058857A1 (en) * 2017-08-15 2019-02-21 International Business Machines Corporation Generating three-dimensional imagery
US10735707B2 (en) * 2017-08-15 2020-08-04 International Business Machines Corporation Generating three-dimensional imagery
US10785464B2 (en) 2017-08-15 2020-09-22 International Business Machines Corporation Generating three-dimensional imagery

Similar Documents

Publication Publication Date Title
US9910509B2 (en) Method to control perspective for a camera-controlled computer
KR100812905B1 (ko) 입체 화상 처리 방법, 장치 및 컴퓨터 판독가능 기록 매체
US9495800B2 (en) Storage medium having stored thereon image processing program, image processing apparatus, image processing system, and image processing method
AU2005242447B2 (en) Image frame processing method and device for displaying moving images to a variety of displays
US9064335B2 (en) System, method, device and computer-readable medium recording information processing program for superimposing information
US11231587B2 (en) Information processing apparatus and image display method
US20170150108A1 (en) Autostereoscopic Virtual Reality Platform
EP2278818A2 (fr) Système et procédé d'affichage pour optimiser l'affichage de vues suivant un suivi actif
US20120056992A1 (en) Image generation system, image generation method, and information storage medium
CN103329165B (zh) 放缩三维场景中的用户控制的虚拟对象的像素深度值
JP2014238731A (ja) 画像処理装置、画像処理システム、および画像処理方法
US10431006B2 (en) Multisensory augmented reality
EP2394710A2 (fr) Système de génération d'images, procédé de génération d'images et support de stockage d'informations
GB2481366A (en) 3D interactive display and pointer control
CN105611267B (zh) 现实世界和虚拟世界图像基于深度和色度信息的合并
JP7353782B2 (ja) 情報処理装置、情報処理方法、及びプログラム
CN102385438A (zh) 信息处理设备,信息处理方法和程序
US10792567B2 (en) Terminal device, system, program, and method
CN107077199B (zh) 用于在三维显示器上呈现虚拟对象的装置及用于控制装置的方法
JP5876983B2 (ja) 表示制御プログラム、表示制御装置、表示制御方法および表示制御システム
US10674140B2 (en) Method, processing device, and computer system for video preview
WO2018185534A1 (fr) Conversion de 2d en 3d à faible latence pour un jeu de réalité virtuelle
EP3346375A1 (fr) Programme, support d'enregistrement, dispositif de fourniture de contenu, et procédé de commande
JP5950701B2 (ja) 画像表示システム、パズルゲームシステム、画像表示方法、パズルゲーム方法、画像表示装置、パズルゲーム装置、画像表示プログラム、および、パズルゲームプログラム
WO2020194973A1 (fr) Système de distribution de contenu, procédé de distribution de contenu, et programme de distribution de contenu

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17904758

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 17904758

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 17904758

Country of ref document: EP

Kind code of ref document: A1