US20220116577A1 - Volumetric video-based augmentation with user-generated content - Google Patents
Volumetric video-based augmentation with user-generated content Download PDFInfo
- Publication number
- US20220116577A1 US20220116577A1 US17/556,873 US202117556873A US2022116577A1 US 20220116577 A1 US20220116577 A1 US 20220116577A1 US 202117556873 A US202117556873 A US 202117556873A US 2022116577 A1 US2022116577 A1 US 2022116577A1
- Authority
- US
- United States
- Prior art keywords
- video
- volumetric
- dimensional
- source video
- volumetric video
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
- 230000003416 augmentation Effects 0.000 title description 2
- 238000012545 processing Methods 0.000 claims abstract description 66
- 238000000034 method Methods 0.000 claims description 50
- 238000001514 detection method Methods 0.000 claims description 16
- 238000013507 mapping Methods 0.000 claims description 16
- 238000010801 machine learning Methods 0.000 claims description 10
- 230000000007 visual effect Effects 0.000 description 25
- 230000006870 function Effects 0.000 description 18
- 238000004891 communication Methods 0.000 description 8
- 230000001413 cellular effect Effects 0.000 description 5
- 230000008859 change Effects 0.000 description 5
- 241000282326 Felis catus Species 0.000 description 4
- 238000012986 modification Methods 0.000 description 4
- 230000004048 modification Effects 0.000 description 4
- 230000007613 environmental effect Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 230000009471 action Effects 0.000 description 2
- 239000003086 colorant Substances 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 239000011159 matrix material Substances 0.000 description 2
- 238000009877 rendering Methods 0.000 description 2
- 238000012706 support-vector machine Methods 0.000 description 2
- 238000012549 training Methods 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- 230000008901 benefit Effects 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 238000003066 decision tree Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 210000003128 head Anatomy 0.000 description 1
- 230000000977 initiatory effect Effects 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- QCAWEPFNJXQPAN-UHFFFAOYSA-N methoxyfenozide Chemical compound COC1=CC=CC(C(=O)NN(C(=O)C=2C=C(C)C=C(C)C=2)C(C)(C)C)=C1C QCAWEPFNJXQPAN-UHFFFAOYSA-N 0.000 description 1
- 201000001997 microphthalmia with limb anomalies Diseases 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000012552 review Methods 0.000 description 1
- 239000004984 smart glass Substances 0.000 description 1
- 230000002123 temporal effect Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
- 238000012800 visualization Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/006—Mixed reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/04—Texture mapping
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/20—Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
- G06T7/55—Depth or shape recovery from multiple images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
- G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
- G06V10/751—Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/20—Scenes; Scene-specific elements in augmented reality scenes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G06V20/46—Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
-
- G—PHYSICS
- G11—INFORMATION STORAGE
- G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
- G11B27/00—Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
- G11B27/02—Editing, e.g. varying the order of information signals recorded on, or reproduced from, record carriers
- G11B27/031—Electronic editing of digitised analogue information signals, e.g. audio or video signals
- G11B27/036—Insert-editing
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
- H04N13/111—Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
- H04N13/117—Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
- H04N13/156—Mixing image signals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/81—Monomedia components thereof
- H04N21/8146—Monomedia components thereof involving graphical data, e.g. 3D object, 2D graphics
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/81—Monomedia components thereof
- H04N21/816—Monomedia components thereof involving special video data, e.g 3D video
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
- G06T2207/10021—Stereoscopic video; Stereoscopic image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20072—Graph-based image processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30196—Human being; Person
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30232—Surveillance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2219/00—Indexing scheme for manipulating 3D models or images for computer graphics
- G06T2219/20—Indexing scheme for editing of 3D models
- G06T2219/2004—Aligning objects, relative positioning of parts
Abstract
A processing system having at least one processor may obtain a two-dimensional source video, select a volumetric video associated with at least one feature of the source video from a library of volumetric videos, identify a first object in the source video, and determine a location of the first object within a space of the volumetric video. The processing system may further obtain a three-dimensional object model of the first object, texture map the first object to the three-dimensional object model of the first object to generate an enhanced three-dimensional object model of the first object, and modify the volumetric video to include the enhanced three-dimensional object model of the first object in the location of the first object within the space of the volumetric video.
Description
- This application is a continuation of U.S. patent application Ser. No. 16/201,525, filed on Nov. 27, 2018, now U.S. Pat. No. 10,771,763, which is herein incorporated by reference in its entirety.
- The present disclosure relates generally to visual communication sessions, and more particularly to methods, computer-readable media, and devices for modifying a volumetric video to include an enhanced three-dimensional object model of an object from a two-dimensional source video.
- The teachings of the present disclosure can be readily understood by considering the following detailed description in conjunction with the accompanying drawings, in which:
-
FIG. 1 illustrates an example network related to the present disclosure; -
FIG. 2 illustrates a flowchart of an example method for modifying a volumetric video to include an enhanced three-dimensional object model of an object from a two-dimensional source video; and -
FIG. 3 illustrates a high level block diagram of a computing device specifically programmed to perform the steps, functions, blocks and/or operations described herein. - To facilitate understanding, identical reference numerals have been used, where possible, to designate identical elements that are common to the figures.
- In one example, the present disclosure describes a method, computer-readable medium, and device for modifying a volumetric video to include an enhanced three-dimensional object model of an object from a two-dimensional source video. For instance, in one example, a method may include a processing system having at least one processor obtaining a two-dimensional source video, selecting a volumetric video associated with at least one feature of the source video from a library of volumetric videos, identifying a first object in the source video, and determining a location of the first object within a space of the volumetric video. The processing system may further obtain a three-dimensional object model of the first object, texture map the first object to the three-dimensional object model of the first object to generate an enhanced three-dimensional object model of the first object, and modify the volumetric video to include the enhanced three-dimensional object model of the first object in the location of the first object within the space of the volumetric video.
- Examples of the present disclosure match two-dimensional (2D) objects from user-generated video (also referred to herein as user-generated content (UGC) to three-dimensional (3D) counterparts for volumetric video (VV). For instance, a 2D object may be detected in a source video, a counterpart 3D object model of a same object type may be obtained and enhanced via texture mapping with the visual information of the 2D object, and the enhanced 3D object model may be placed into the volumetric video. The volumetric video may then be presented and/or experienced by a user including the enhanced 3D object model associated with the 2D object. In addition, in one example, a traversal of the volumetric video including the enhanced 3D object model may be captured and exported to a new 2D video. In one example, user preferences and/or user interactions with an automated processing system may guide the matches between objects in the 2D user-generated video and 3D object models. In accordance with the present disclosure, 2D user-generated video may comprise analog video, e.g., 35 mm video reels, digitized 35mm, digital videos, e.g., H.264 (Advanced Video Coding (AVC)), H.265 (High Efficiency Video Coding (HEVC)), Moving Picture Expert Group (MPEG) .mpeg files, .mov files, .mp4 files, .3gp files, .f4f files, .m3u8 files, and so forth. Examples of the present disclosure provide the bridging of these “historical” assets into new volumetric video experiences as well as the opportunity to regenerate two-dimensional videos with new volumetric video details (e.g., environments, historical events, etc.).
- An example of the operations of the present disclosure may proceed as follows. A processing system may first obtain a 2D source video (e.g., a user-generated video). The source video may be historic (e.g., recorded and stored in one or more machine-readable formats) or may be a video stream from a video capture device, such as any user endpoint device including but not limited to a digital camera, a smartphone, a web-camera, a home-base or business-based security camera, and so forth. The processing system may then search for and rank one or more volumetric videos overlapping with the content of the source video. The content matching may be based upon one or more user-provided criteria (or “factors”), or may be based upon one or more default criteria, such as the two-dimensional source video and the volumetric video being of/from a same location, being recorded at a same time, including a same object (e.g., a landmark, a moveable object, etc.), and so forth. For instance, a user can specify certain criteria as providing a greater weight for matching, e.g., a same location may be required, while the user may be flexible as to the time of recording. In another example, a user may express a preference for a volumetric video of a same location as the 2D source video, but specifically for a different time period. For example, the user may wish to associate one or more objects from the 2D source video with content from the volumetric video relating to a historic event at the same location (e.g., a concert, a speech, a party, a sporting event, etc.).
- In one example, matching between the 2D source video and a stored volumetric video may be made directly via the respective metadata (e.g., time tags, location tags, object tags, event tags, keywords, etc.). Alternatively, or in addition, a matching may be made by performing object detection/recognition processes (e.g., via machine learning-based object detection/recognition models) with respect to either or both of the 2D source video or the volumetric video, and then confirming the presence of the same factors in both the 2D source video and the volumetric video. In one example, a user associated with the 2D source video may be prompted to annotate one or more objects and/or the 2D source video with keywords, tags, or the like.
- In one example, a user may be presented with a ranked list of volumetric videos comprising potential matches to the 2D source video from which the user may select one for further processing. For instance, results may be ranked by time (e.g., date and time, by the time of year/season, etc.), location, and/or environmental similarity (e.g., rainy, snowy, cloudy, windy, sunny, bright, dark, clear, etc.) to the source video. When the user does not have a device capable of experiencing volumetric video, a 2D snapshot or video clip representation of the volumetric video may be provided for the user to preview in connection with the selection. In another example, the processing system may select a top matching volumetric video for further processing.
- In one example, the processing system may determine a position of at least one object of the 2D source video within a space of the volumetric video. The determining of the positon may account for a location of a device recording the video, a bearing to an object, an estimated range to the object, a translation of a position of the device in a physical space to a position in a space of the volumetric video, and so forth. In one example, the processing system may calculate one or more adjustment factors to position the visual information of the 2D source video correctly within the space of the volumetric video. For instance, the physical space may include reference points having known corresponding locations/positions in the space of the volumetric video from which the processing system may compute a transformation matrix to translate coordinates of objects in the source video into corresponding coordinates in the space of the volumetric video. In one example, 2D/planar objects from the source video may be matched against a precomputed library of 2D objects to identify an object type. In one example, the matching may be a visual matching, e.g., using machine learning over low level and/or high level visual features of the object from the 2D source video. In an example where the object is tagged with a label, the matching may comprise performing a lookup of an object matching the label. In addition, in one example, the processing system may also store or have access to 3D object models which are associated with the 2D objects. For instance, a database may store 2D objects and corresponding “mapped” 3D object models.
- In one example, the processing system may retrieve a corresponding 3D object model, and may texture map the visual content of the 2D object from the source video to the 3D object model to create an enhanced 3D object model. The texture mapping may include diffuse mapping, two-part mapping/intermediate mapping to an intermediate object, bump mapping, etc. For instance, visual information of the 2D object or “texture” (e.g., textels) may replace visual information in certain locations of the surface of the 3D object model. In one example, the visual information of the 2D object may be integrated with the visual information of the 3D object model at textel/texture boundaries derived from the visual information of the 2D object model. As such, the enhanced 3D object model may be considered as an estimate of what the 2D object would look like if full 3D information were available from the source video or elsewhere. The enhanced 3D object model may then be placed into the volumetric video at the location in the space of the volumetric video that is determined. In one example, an audio portion of the volumetric video may also be modified to account for the presence of the object (e.g., to include audio from the 2D source video).
- In one example, a user may review a temporary volumetric video (e.g., modified as described above with the enhanced 3D object model) and then choose to manipulate or explore output view positions. The user may also modify object(s) or scene(s), followed by further automatic optimization. For instance, a user interface may enable a user to interact with and change the enhanced 3D object model, such as to change colors, shading, texture, size, position, orientation, etc. The user may also select a different 3D object model. For instance, the processing system may identify the 2D object as a “cat,” but the user may manually specify that the object is a “dog.” As such, the processing system may perform another texture mapping to a 3D object model for “dog” to make this correction. In such an example, the volumetric video may then be re-adjusted to account for such changes.
- In one example, the modified volumetric video may be stored, e.g., as a new volumetric video or new version of the volumetric video, by overwriting a previous version of the volumetric video, etc. Alternatively, or in addition, a 2D traversal, or “exploration” of the volumetric video may be stored and/or exported as a new output 2D video. In one example, the processing system may update one or more 2D object detection/recognition models in accordance with user feedback or based upon user actions for improved indexing. For example, if a user has manually corrected a 3D object model to “dog” instead of “cat,” this fact may be used as a negative example/feedback to retrain the 2D detection/recognition model for “cat.”
- Examples of the present disclosure provide a potential market for owners of content libraries by offering augmentation and upgrade capabilities. In addition, examples of the present disclosure enable users to regenerate video content from different angles that may capture more interesting details or other background visual information obtained from volumetric videos. Furthermore, an object from a 2D source video may be represented in a 3D format in a volumetric video without having to recapture from multiple angles, without having to provide precise lighting and other condition information, and so forth. In particular, 3D representations of 2D objects may be estimated from texture mapping to 3D object models of the same object types. Thus, efficient access to 3D rendering is provided to users without each user having to individually utilize a full volumetric video capture system. These and other aspects of the present disclosure are described in greater detail below in connection with the examples of
FIGS. 1-3 . - To further aid in understanding the present disclosure,
FIG. 1 illustrates anexample system 100 in which examples of the present disclosure for modifying a volumetric video to include an enhanced three-dimensional object model of an object from a two-dimensional source video may operate. Thesystem 100 may include any one or more types of communication networks, such as a traditional circuit switched network (e.g., a public switched telephone network (PSTN)) or a packet network such as an internet Protocol (IP) network (e.g., an IP Multimedia Subsystem (IMS) network), an asynchronous transfer mode (ATM) network, a wireless network, a cellular network (e.g., 2G, 3G, and the like), a long term evolution (LTE) network, 5G and the like, related to the current disclosure. It should be noted that an IP network is broadly defined as a network that uses internet Protocol to exchange data packets. Additional example IP networks include Voice over IP (VoIP) networks, Service over IP (SoIP) networks, and the like. - In one example, the
system 100 may comprise anetwork 102, e.g., a telecommunication service provider network, a core network, an enterprise network comprising infrastructure for computing and providing communications services of a business, an educational institution, a governmental service, or other enterprises. Thenetwork 102 may be in communication with one ormore access networks network 102 may combine core network components of a cellular network with components of a triple play service network; where triple-play services include telephone services, internet or data services and television services to subscribers. For example,network 102 may functionally comprise a fixed mobile convergence (FMC) network, e.g., an IP Multimedia Subsystem (IMS) network. In addition,network 102 may functionally comprise a telephony network, e.g., an internet Protocol/Multi-Protocol Label Switching (IP/MPLS) backbone network utilizing Session Initiation Protocol (SIP) for circuit-switched and Voice over internet Protocol (VoIP) telephony services.Network 102 may further comprise a broadcast television network, e.g., a traditional cable provider network or an internet Protocol Television (IPTV) network, as well as an internet Service Provider (ISP) network. In one example,network 102 may include a plurality of television (TV) servers (e.g., a broadcast server, a cable head-end), a plurality of content servers, an advertising server (AS), an interactive TV/video on demand (VoD) server, and so forth. - In accordance with the present disclosure, application server (AS) 104 may comprise a computing system or server, such as
computing system 300 depicted inFIG. 3 , and may be configured to provide one or more operations or functions for modifying a volumetric video to include an enhanced three-dimensional object model of an object from a two-dimensional source video, as described herein. It should be noted that as used herein, the terms “configure,” and “reconfigure” may refer to programming or loading a processing system with computer-readable/computer-executable instructions, code, and/or programs, e.g., in a distributed or non-distributed memory, which when executed by a processor, or processors, of the processing system within a same device or within distributed devices, may cause the processing system to perform various functions. Such terms may also encompass providing variables, data values, tables, objects, or other data structures or the like which may cause a processing system executing computer-readable instructions, code, and/or programs to function differently depending upon the values of the variables or other data structures that are provided. As referred to herein a “processing system” may comprise a computing device including one or more processors, or cores (e.g., as illustrated inFIG. 3 and discussed below) or multiple computing devices collectively configured to perform various steps, functions, and/or operations in accordance with the present disclosure. - Thus, although only a single application server (AS) 104 is illustrated, it should be noted that any number of servers may be deployed, and which may operate in a distributed and/or coordinated manner as a processing system to perform operations for modifying a volumetric video to include an enhanced three-dimensional object model of an object from a two-dimensional source video, in accordance with the present disclosure. In one example, AS 104 may comprise a physical storage device (e.g., a database server), to store various types of information in support of systems for modifying a volumetric video to include an enhanced three-dimensional object model of an object from a two-dimensional source video, in accordance with the present disclosure. For example, AS 104 may store a library of volumetric videos, a plurality of 2D object detection/recognition models (e.g., machine learning-based image detection models), a catalog matching 2D objects to 3D object models, user preferences for volumetric video matching, and so forth that may be processed by AS 104 in connection with modifying a volumetric video to include an enhanced three-dimensional object model of an object from a two-dimensional source video. AS 104 may further store additional information such as a lexicon of topic models, e.g., machine learning-based models to identify topics and/or themes in 2D and/or volumetric video, and so forth. For ease of illustration, various additional elements of
network 102 are omitted fromFIG. 1 . - As referred to herein, a machine learning model (MLM) (or machine learning-based model) may comprise a machine learning algorithm (MLA) that has been “trained” or configured in accordance with input data (e.g., training data) to perform a particular service, e.g., to detect a type of 2D object in image and/or video content. Examples of the present disclosure are not limited to any particular type of MLA/model, but are broadly applicable to various types of MLAs/models that utilize training data, such as support vector machines (SVMs), e.g., linear or non-linear binary classifiers, multi-class classifiers, deep learning algorithms/models, decision tree algorithms/models, k-nearest neighbor (KNN) clustering algorithms/models, and so forth.
- The types of features from which 2D object detection/recognition models may be derived may include visual features from 2D video. For instance, the visual features may include low-level invariant image data, such as colors (e.g., RGB (red-green-blue) or CYM (cyan-yellow-magenta) raw data (luminance values) from a CCD/photo-sensor array), shapes, color moments, color histograms, edge distribution histograms, etc. Visual features may also relate to movement in a video and may include changes within images and between images in a sequence (e.g., video frames or a sequence of still image shots), such as color histogram differences or a change in color distribution, edge change ratios, standard deviation of pixel intensities, contrast, average brightness, and the like.
- In one example, the
access networks network 102 may provide a cable television service, an IPTV service, or any other types of telecommunication service to subscribers viaaccess networks access networks network 102 may be operated by a telecommunication network service provider. Thenetwork 102 and theaccess networks - In one example, the
access network 120 may be in communication with adevice 131. Similarly,access network 122 may be in communication with one or more devices, e.g.,devices Access networks devices network 102, devices reachable via the Internet in general, and so forth. In one example, devices 131-133 may each comprise a mobile device, a cellular smart phone, a wearable computing device (e.g., smart glasses) a laptop, a tablet computer, a desktop computer, an application server, a bank or cluster of such devices, and the like. In one example, devices 131-133 may each comprise programs, logic or instructions for performing functions in connection with examples of the present disclosure for modifying a volumetric video to include an enhanced three-dimensional object model of an object from a two-dimensional source video. For example, devices 131-133 may each comprise a computing system or device, such ascomputing system 300 depicted inFIG. 3 , and may be configured to provide one or more operations or functions in connection with examples of the present disclosure for providing a communication session including video content of a physical environment that is modified in accordance with a visualization action for a physical object, as described herein. - In one example, the
device 131 is associated with afirst user 191 at a firstphysical environment 140. As illustrated inFIG. 1 , thedevice 131 may comprise a smartphone with a camera to capture video (e.g., 2D video) from thephysical environment 140. For illustrative purposes, thephysical environment 140 may include areference point 160 which may be associated with thebuilding 170. For instance, thebuilding 170 may comprise a known landmark, andreference point 160 may denote a corner of thebuilding 170. Alternatively, or in addition, wireless beacons (not shown) inphysical environment 140 having known locations may be used as reference points, such as Institute of Electrical and Electronics Engineers (IEEE) 802.15 based-beacons, IEEE 802.11 wireless routers and/or access points (AP), cellular base stations (or more particularly, the base station antennas), etc. For instance,device 131 may determine its position via observed time difference of arrival (OTDA), barycentric triangulation, or a similar technique with reference to one or more wireless beacons. In one example,device 131 may communicate viaaccess network 120 to provide a video of the firstphysical environment 140 toAS 104. In the present example, thedevice 131 may capture 2D video that includes images of abuilding 170 and a bird 180 (and which may also include sounds). Theperspective 137 of thedevice 131 is also illustrated inFIG. 1 . In one example, thedevice 131 may provide information from which theperspective 137 may be quantified, such as: position information (e.g., GPS coordinates, coordinates and/or a position vector in relation to reference point 160), orientation information (e.g., azimuth and/or elevation information derived from a gyroscope and compass), and so forth. - In one example, the 2D video may be processed by AS 104 as a “source video” as described herein. For instance AS 104 may select a volumetric video associated with at least one feature of the 2D source video from a library of volumetric videos stored at
AS 104 and/or accessible toAS 104. AS 104 may then identify a first object in the source video, such as thebird 180, determine a location of thebird 180 within a space of the volumetric video, obtain a 3D object model of the bird, texture map thebird 180 to the 3D object model to produce a modified/enhanced 3D object model, modify the volumetric video to include the enhanced 3D object model, and so forth. - In one example, the match between the 2D source video and the volumetric video may be in accordance with one or more features that are shared by both the 2D source video and the volumetric video, such as time tags, location tags, object tags, event tags, keywords, etc. Alternatively, or in addition, a matching may be made by performing object detection/recognition processes with respect to either or both of the 2D source video or the volumetric video, and then confirming the presence of the same factors in both the 2D source video and the volumetric video. In one example,
user 191 may be prompted byAS 104 and/ordevice 131 to annotate one or more objects and/or the 2D source video with keywords, tags, or the like. In one example, AS 104 may present to theuser 191, viadevice 131, a ranked list of volumetric videos comprising potential matches to the 2D source video from which theuser 191 may select one for further processing. For instance, results may be ranked by time (e.g., date and time, by the time of year/season, etc.), location, and/or environmental similarity (wet, snowy, clear, etc.) to the source video. - In one example, the position of the
bird 180 within the space of the volumetric video may be determined based upon a position and orientation of thedevice 131. In one example, the range (distance) fromdevice 131 to thebird 180 may be determined from a focus of thedevice 131. For instance, zoom and focus settings may indicate the approximate distance at which the image of a field of view of a camera ofdevice 131 is placed in focus. In one example, the zoom and focus settings may be manually set by theuser 191. However, thedevice 131 may alternatively or additionally calculate the distance based upon the settings of the zoom and/or focus controls. In one example, the position of thebird 180 may be determined by using features of building 170 as reference points/key points. For instance, building 170 may be a landmark for which full 3D information has been captured and stored in the volumetric video. As such, the position of thebird 180 may be determined by estimating distances to the one or more key points of thebuilding 170 in accordance with the visual information of the 2D source video. - As stated above, the volumetric video may be modified to include the enhanced 3D object mode. In addition, AS 104 may also store the modified volumetric video, e.g., as a new volumetric video, as a replacement to the volumetric video, as a new version of the volumetric video, etc. The
user 191, e.g., viadevice 131, may then obtain a presentation of the modified volumetric video fromAS 104. For instance, AS 104 may create a 2D output video, e.g., a 2D projection/traversal of the 3D space of the volumetric video, and stream or otherwise transmit the output video todevice 131. Alternatively, or in addition, the modified volumetric video may be provided todevice 131, anddevice 131 may render the volumetric video in accordance with the 2D display capabilities of thedevice 131. For example, theuser 191 may select one or more perspectives from within the space ofvolumetric video 150 from which to render the 2D output video. For instance, a user interface ofdevice 131 may enable theuser 191 to move a viewpoint left/right, up/down, forward/backward, etc., to select coordinates or define a position with respect to a reference point (e.g., reference point 160), and so forth. It should be noted that the perspective(s) represented in the 2D output video may be entirely different from the perspective(s) from which theuser 191 captured the source video via device 131 (and may be entirely different from the current location of theuser 191 and device 131). In addition, in a server-based example, AS 104 may also store the 2D output video that is created. - In one example, AS 104 may make the modified volumetric video and/or 2D output video available to other users (e.g.,
users 192 and 193). For example, as illustrated inFIG. 1 , thedevice 132 may comprise a wearable computing device (e.g., a VR headset) and may present aspace 150 of a volumetric video foruser 192. In one example,device 132 may communicate viaaccess network 122 to request the modified volumetric video, to receive the volumetric video and/or to receive at least a portion of the volumetric video to be used in rendering a portion of thespace 150 that is currently within the perspective/view of theuser 192, to provide position and orientation information of a view of the user to AS 104 to determine which portion of the volumetric video to provide, to provide commands to start, stop, pause, resume, etc., (e.g., in an example where the volumetric video is streamed from AS 104), and so forth. - To illustrate,
user 192 may set a perspective from which to experience the visual contents of the volumetric video. For instance, a user interface ofdevice 131 may provide for theuser 191 to move a viewpoint left/right, up/down, forward/backward, etc., to select coordinates or define a position with respect to a reference point (e.g.,reference point 160 in space 150), and so forth. In one example, theuser 192 may change perspectives by changing a location, e.g., by walking, changing a focus, e.g., by moving the user's head and/or eyes, changing a zoom level via a specific command and/or via eye adjustment, and so on. As illustrated inFIG. 1 ,user 192 may be viewing thebird 180 and building 170 from an entirely different perspective from which theuser 191 recorded the 2D source video in thephysical environment 140. Notably, while the original volumetric video may not have included thebird 180, the modified volumetric video now includes thebird 180. In addition, thebird 180 is represented as a full 3D object that has been created from a 3D object model texture mapped with the 2D visual information of thebird 180 obtained from the 2D source video fromdevice 131. - Similarly, a
third user 193 may obtain and experience the volumetric video, or the 2D output video, via thedevice 133. For instance,device 133 may comprise a mobile computing device, such as a smartphone, a tablet, or the like with a 2D display screen. For instance, AS 104 may provide the 2D output video todevice 133. Alternatively, or in addition, AS 104 may provide the modified volumetric video todevice 133, anddevice 133 may render the volumetric video in accordance with the 2D display capabilities of thedevice 133. For example, theuser 193 may select one or more perspectives from within the space of (modified)volumetric video 150 from which to render the 2D output video. - It should also be noted that the
system 100 has been simplified. Thus, it should be noted that thesystem 100 may be implemented in a different form than that which is illustrated inFIG. 1 , or may be expanded by including additional endpoint devices, access networks, network elements, application servers, etc. without altering the scope of the present disclosure. In addition,system 100 may be altered to omit various elements, substitute elements for devices that perform the same or similar functions, combine elements that are illustrated as separate devices, and/or implement network elements as functions that are spread across several devices that operate collectively as the respective network elements. For example, thesystem 100 may include other network elements (not shown) such as border elements, routers, switches, policy servers, security devices, gateways, a content distribution network (CDN) and the like. For example, portions ofnetwork 102,access networks access networks 120 and/or 122 may each comprise a plurality of different access networks that may interface withnetwork 102 independently or in a chained manner. In one example, thesystem 100 may further include wireless or wired connections to sensors, radio frequency identification (RFID) tags, or the like from which devices may determine locations/positions, ranges/distances, bearings, and so forth withinphysical environment 140. - In one example,
device 131 may capture additional objects in the 2D source video and may present to theuser 191 options for removing or excluding such objects from the volumetric video to be modified. For instance, theuser 191 may select thebird 180 to be texture mapped to a corresponding 3D object model and inserted into the volumetric video. However, imagery of one or more individuals may be captured in the 2D source video who did not consent to have their images included in a volumetric video. As such, theuser 191 will skip the images of such individuals. It should also be noted that in one example, the original volumetric video may include detailed 3D imagery ofbuilding 170. As such,user 191 may create a volumetric video based upon the 2D source video captured viaendpoint device 131 without having to utilize multiple cameras, without having to perform multiple takes from different perspectives, and so forth. In addition, other users who may generate their own 2D source videos at or near the samephysical environment 140 may similarly benefit from the stored 3D content of the volumetric video including thebuilding 170. In one example, new/modified volumetric videos may be stored in an efficient form by storing thebuilding 170 as an object that may be re-used across different volumetric videos. Furthermore, other objects atphysical environment 140 may be present in a volumetric video but may be selectively included or excluded from user edits/modifications to the volumetric video as described herein. Thus, these and other modifications are all contemplated within the scope of the present disclosure. -
FIG. 2 illustrates a flowchart of anexample method 200 for modifying a volumetric video to include an enhanced three-dimensional object model of an object from a two-dimensional source video, in accordance with the present disclosure. In one example, themethod 200 is performed by a component of thesystem 100 ofFIG. 1 , such as byapplication server 104,device 131, and/or any one or more components thereof (e.g., a processor, or processors, performing operations stored in and loaded from a memory), or byapplication server 104, in conjunction with one or more other devices, such as devices 131-133, and so forth. In one example, the steps, functions, or operations ofmethod 200 may be performed by a computing device orsystem 300, and/orprocessor 302 as described in connection withFIG. 3 below. For instance, the computing device orsystem 300 may represent any one or more components ofapplication server 104,device 131, etc. inFIG. 1 that is/are configured to perform the steps, functions and/or operations of themethod 200. Similarly, in one example, the steps, functions, or operations ofmethod 200 may be performed by a processing system comprising one or more computing devices collectively configured to perform various steps, functions, and/or operations of themethod 200. For instance, multiple instances of the computing device orprocessing system 300 may collectively function as a processing system. For illustrative purposes, themethod 200 is described in greater detail below in connection with an example performed by a processing system. Themethod 200 begins instep 205 and proceeds to step 210. - At
step 210, the processing system may obtain a source video comprising a two-dimensional (2D) video. The source video may be a 2D video in a digital format, or may comprise a non-digital 2D video that may be converted to a digital format atstep 210. In one example, step 210 may further include obtaining at least one feature of the source video, such as: location information, time information, an event tag, or a keyword. For instance, the location information may provide GPS coordinates, a landmark identifying the location, a name of a place, or the like. An event tag may distinguish between different events at a same place but at different times, e.g., a concert at a stadium versus a sporting event on a different day. A keyword can include information regarding the presence of an object (e.g., a name of a known object, such as Statue of Liberty, Empire State Building, the aircraft carrier Intrepid, etc.). In one example, features may also be specified by a user to be used in a later step for matching volumetric videos and the source video obtained by the processing system atstep 210. For instance, the user may specify an identity of a producer of volumetric videos for which the user has a preference. - At
optional step 215, the processing system detects an object in the source video (e.g., at least a “second” object). For instance, the second object could be a famous landmark, a street corner, a statute, a monument, or the like, where the second object is not specified as a feature in connection withstep 210. In one example, the second object may be detected in accordance with one or more 2D object detection/recognition models (e.g., MLM/machine learning-based image detection models). It should also be noted that although the terms, “first,” “second,” “third,” etc., are used herein, the use of these terms are intended as labels only. Thus, the use of a term such as “third” in one example does not necessarily imply that the example must in every case include a “first” and/or a “second” of a similar item. In other words, the use of the terms “first,” “second,” “third,” and “fourth,” do not imply a particular number of those items corresponding to those numerical values. In addition, the use of the term “third” for example, does not imply a specific sequence or temporal relationship with respect to a “first” and/or a “second” of a particular type of item, unless otherwise indicated. - At
step 220, the processing system selects a volumetric video associated with at least one feature of the source video from a library of volumetric videos. In one example, step 220 may include determining that the volumetric video matches the at least one feature of the source video. For example, the at least one feature of the source video may be received atstep 210 as described above. Alternatively, or in addition, the at least one feature of the source video may comprise the at least the second object that may be detected atoptional step 215. In such an example, step 220 may include: detecting the at least the second object in the volumetric video, and selecting the volumetric video when the presence of the second object is so detected. For instance, as described above in connection withoptional step 215, the second object could be a famous landmark, a street corner, a statute, a monument, or the like. Similarly, a second object could be a large portable exhibit, such that a volumetric video generated at one or more installation locations may still be paired with a 2D source video of the same exhibit from an entirely different physical location. In such case, the 2D source video and the volumetric video may be matched atstep 220 based upon the same visual features. For example, the processing system may identify the second object because it is tagged in both the 2D source video and the volumetric video (e.g., keyword matching). In another example, the processing system may perform a matching of visual features, e.g., not necessarily detecting the second object per se, but detecting sufficient matching between low level and/or high level visual features of the respective 2D source video and the volumetric video. - In one example, the selecting may be in accordance with a ranked list of volumetric videos from the library of volumetric videos based upon a level of matching to the at least one feature of the source video. For instance, the volumetric video may be automatically selected, or the processing system may provide a ranked list from which the volumetric video is selected by a user. In one example, the ranking can be based upon how well the at least one feature matches, the number of features for which the match is confirmed, etc. For instance, ranking may be ordered by location, time, or environmental similarity, or a combination thereof. Similarly, in one example a user may specify that the volumetric video should match both the location and the time, or can specify that it just matches the location. A user may also specify a particular time that is different from the time of the source video. For instance, a user may want to place visual information recorded in the source video at a landmark/location during a historic event.
- At
step 225, the processing system identifies a “first” object in the source video. In one example, the first object may be detected in the source video in accordance with one or more 2D object detection/recognition models. The first object may be identified as a specific instance of an object type (e.g., a famous building, a specific person for which a 2D object detection/recognition model is available, etc.), or may be identified as an instance of an object type (e.g., “person,” “building,” “dog,” “cat,” etc., without identifying a specific person, building, dog, cat, etc.). - At
optional step 230, the processing system may perform an alignment of the source video to the volumetric video. For example, the alignment may include a spatial alignment where the processing system may detect key points (e.g., of at least one object, such as the second object) in both the source video and the volumetric video, and calculate one or more adjustment factors to position the visual information of the source video correctly within the space of the volumetric video. For instance, the physical space may include reference points having known corresponding locations/positions in the space of the volumetric video from which the processing system may compute a transformation matrix to translate coordinates of objects in the source video into corresponding coordinates in the space of the volumetric video. The alignment may further include a time alignment of the source video and the volumetric video. For instance, the time of a frame of the source video may be determined in accordance with one or both of a start time and an end time, and the frame rate of the source video. In another example, each frame may be tagged with timing information. In addition, the volumetric video may similarly have a start time, end time and/or duration, frame rate, and similar information stored as metadata along with the visual information of the volumetric video. As such, frames of the source video may be paired with frames of the volumetric video having corresponding times. Alternatively, or in addition, alignment may be achieved with reference to one or more objects in motion. For instance, a ball may be bouncing and then come to rest in both the source video and the volumetric video. Thus, the processing system may determine that a frame in the source video and a frame in the volumetric video where the ball comes to rest are at a same time. However, it should again be noted that in some cases the intention of a user may be that the source video and the volumetric video be from different recording times. Thus, in one example, the user may be provided with a user interface to manually select a time alignment of the source video to the volumetric video. - At
step 235, the processing system determines a location of the first object within a space of the volumetric video. In one example, the determining of the location of the first object within the space of the volumetric video is in accordance with the alignment performed atoptional step 230. - At
step 240, the processing system obtains a three-dimensional object model of the first object. In one example, the obtaining of the three-dimensional object model of the first object is in accordance with a catalog of two-dimensional objects and three-dimensional object models that are matched to the two-dimensional objects. In one example, the obtaining of the three-dimensional object model comprises matching the first object to one of the two-dimensional objects in the catalog, and obtaining, from the catalog, the three-dimensional object model that is matched to the two-dimensional object. In one example, the matching may be based upon keyword matching. For instance, a user may tag objects (e.g., including the first object and possibly other objects) in the 2D source video. These keywords, or tags, may be stored as metadata along with the 2D source video, and may be referenced by the processing system atstep 240. Alternatively, or in addition, in one example, the matching may be in accordance with a machine learning-based image detection model. For instance, the matching may be based upon low level or high-level visual features. Matching may also be enhanced or confirmed with audio features, movement features, etc. - At
step 245, the processing system texture maps the first object to the three-dimensional object model of the first object to generate an enhanced three-dimensional object model of the first object. The texture mapping may include diffuse mapping, two-part mapping/intermediate mapping to an intermediate object, bump mapping, etc. - At
step 250, the processing system modifies the volumetric video to include the enhanced three-dimensional object model of the first object in the location of the first object within the space of the volumetric video. For instance, voxel data associated with the location of the first object within the space of the volumetric video may be changed to correspond to the three-dimensional object model of the first object. In one example, the voxel data may be changed across a plurality of frames in the volumetric video corresponding to times in which the three-dimensional object model of the first object are to be present in the volumetric video, e.g., in accordance with a time-alignment ofoptional step 230. Thus, frames of the source video may be matched to frames of the volumetric video. In another example, the time at which the source video was recorded may be different from when the volumetric video was recorded, or the volumetric video may be composited from video content taken at a plurality of different times. Accordingly, in one example, a user may be prompted to time align the source video to the volumetric video. Then the modification of the voxel data of the volumetric video may be in accordance with the temporally appropriate data of the enhanced three-dimensional object model. - At
optional step 255, the processing system may generate an output video comprising a two dimensional traversal of the volumetric video that is modified. For instance, one or more perspective(s) in the output video may be different from the view(s)/perspective(s) of the source video and may comprise any number of changes in views/perspective, zooms, focuses, etc. In one example, the output video is generated in accordance with a selection by a user of at least one viewing perspective within the space of the volumetric video. - At
optional step 260, the processing system may present, via an endpoint device, one or both of: the output video, or the volumetric video that is modified. The endpoint device may comprise a same endpoint device from which the source video may be obtained atstep 210, or a different endpoint device. - Following
step 250, or either of theoptional steps method 200 proceeds to step 295 where the method ends. - It should be noted that the
method 200 may be expanded to include additional steps, or may be modified to replace steps with different steps, to combine steps, to omit steps, to perform steps in a different order, and so forth. For instance, in one example the processor may repeat one or more steps of themethod 200, such as steps 222-250, steps 225-260, etc. to continue to detect additional objects in the source video, to include in the volumetric video 3D object models enhanced in accordance with the respective additional objects, etc. In another example, a feature of the source video used for matching the volumetric video atstep 220 may include a topic of the source video. For instance, in such an example, themethod 200 may be expanded to include topic (e.g., theme and/or concept) detection in the source video and then selecting the volumetric video (or a plurality of matching volumetric videos) in accordance with the topic. For instance, the volumetric videos may be tagged with topics contained therein (manually or via application of topic detection algorithms). The processing system may then apply topic models (e.g., classifiers) for a number of topics to the source video to identify a topic, and match the source video to one or more volumetric videos tagged with the same topic. Thus, these and other modifications are all contemplated within the scope of the present disclosure. - In addition, although not expressly specified above, one or more steps of the
method 200 may include a storing, displaying and/or outputting step as required for a particular application. In other words, any data, records, fields, and/or intermediate results discussed in the method can be stored, displayed and/or outputted to another device as required for a particular application. Furthermore, operations, steps, or blocks inFIG. 3 that recite a determining operation or involve a decision do not necessarily require that both branches of the determining operation be practiced. In other words, one of the branches of the determining operation can be deemed as an optional step. Thus, the use of the term “optional step” is intended to only reflect different variations of a particular illustrative embodiment and is not intended to indicate that steps not labelled as optional steps to be deemed to be essential steps. Furthermore, operations, steps or blocks of the above described method(s) can be combined, separated, and/or performed in a different order from that described above, without departing from the example embodiments of the present disclosure. -
FIG. 3 depicts a high-level block diagram of a computing device or processing system specifically programmed to perform the functions described herein. For example, any one or more components or devices illustrated inFIG. 1 or described in connection with themethod 200 may be implemented as theprocessing system 300. As depicted inFIG. 3 , theprocessing system 300 comprises one or more hardware processor elements 302 (e.g., a microprocessor, a central processing unit (CPU) and the like), amemory 304, (e.g., random access memory (RAM), read only memory (ROM), a disk drive, an optical drive, a magnetic drive, and/or a Universal Serial Bus (USB) drive), amodule 305 for modifying a volumetric video to include an enhanced three-dimensional object model of an object from a two-dimensional source video, and various input/output devices 306, e.g., a camera, a video camera, storage devices, including but not limited to, a tape drive, a floppy drive, a hard disk drive or a compact disk drive, a receiver, a transmitter, a speaker, a display, a speech synthesizer, an output port, and a user input device (such as a keyboard, a keypad, a mouse, and the like). - Although only one processor element is shown, it should be noted that the computing device may employ a plurality of processor elements. Furthermore, although only one computing device is shown in the Figure, if the method(s) as discussed above is implemented in a distributed or parallel manner for a particular illustrative example, i.e., the steps of the above method(s) or the entire method(s) are implemented across multiple or parallel computing devices, e.g., a processing system, then the computing device of this Figure is intended to represent each of those multiple general-purpose computers. Furthermore, one or more hardware processors can be utilized in supporting a virtualized or shared computing environment. The virtualized computing environment may support one or more virtual machines representing computers, servers, or other computing devices. In such virtualized virtual machines, hardware components such as hardware processors and computer-readable storage devices may be virtualized or logically represented. The
hardware processor 302 can also be configured or programmed to cause other devices to perform one or more operations as discussed above. In other words, thehardware processor 302 may serve the function of a central controller directing other devices to perform the one or more operations as discussed above. - It should be noted that the present disclosure can be implemented in software and/or in a combination of software and hardware, e.g., using application specific integrated circuits (ASIC), a programmable logic array (PLA), including a field-programmable gate array (FPGA), or a state machine deployed on a hardware device, a computing device, or any other hardware equivalents, e.g., computer readable instructions pertaining to the method(s) discussed above can be used to configure a hardware processor to perform the steps, functions and/or operations of the above disclosed method(s). In one example, instructions and data for the present module or
process 305 for modifying a volumetric video to include an enhanced three-dimensional object model of an object from a two-dimensional source video (e.g., a software program comprising computer-executable instructions) can be loaded intomemory 304 and executed byhardware processor element 302 to implement the steps, functions or operations as discussed above in connection with theexample method 200. Furthermore, when a hardware processor executes instructions to perform “operations,” this could include the hardware processor performing the operations directly and/or facilitating, directing, or cooperating with another hardware device or component (e.g., a co-processor and the like) to perform the operations. - The processor executing the computer readable or software instructions relating to the above described method(s) can be perceived as a programmed processor or a specialized processor. As such, the
present module 305 for modifying a volumetric video to include an enhanced three-dimensional object model of an object from a two-dimensional source video (including associated data structures) of the present disclosure can be stored on a tangible or physical (broadly non-transitory) computer-readable storage device or medium, e.g., volatile memory, non-volatile memory, ROM memory, RAM memory, magnetic or optical drive, device or diskette and the like. Furthermore, a “tangible” computer-readable storage device or medium comprises a physical device, a hardware device, or a device that is discernible by the touch. More specifically, the computer-readable storage device may comprise any physical devices that provide the ability to store information such as data and/or instructions to be accessed by a processor or a computing device such as a computer or an application server. - While various embodiments have been described above, it should be understood that they have been presented by way of example only, and not limitation. Thus, the breadth and scope of a preferred embodiment should not be limited by any of the above-described example embodiments, but should be defined only in accordance with the following claims and their equivalents.
Claims (21)
1.-20. (canceled)
21. A method comprising:
obtaining, by a processing system including at least one processor, a source video, wherein the source video is a two-dimensional video;
selecting, by the processing system, a volumetric video from a library of volumetric videos;
identifying, by the processing system, a first object in the source video and an audio portion in the source video associated with the first object;
determining, by the processing system, a position of the first object within a space of the volumetric video;
obtaining, by the processing system, a three-dimensional object model of the first object;
texture mapping, by the processing system, the first object to the three-dimensional object model of the first object to generate an enhanced three-dimensional object model of the first object; and
modifying, by the processing system, the volumetric video to include the enhanced three-dimensional object model of the first object in the position of the first object within the space of the volumetric video, wherein the modifying further includes modifying the volumetric video to include the audio portion associated with the first object.
22. The method of claim 21 , further comprising:
performing an alignment of the source video to the volumetric video, wherein the determining the position of the first object within the space of the volumetric video is in accordance with the alignment.
23. The method of claim 22 , wherein the alignment further comprises a time alignment of the source video and the volumetric video.
24. The method of claim 21 , wherein the obtaining the source video further comprises:
obtaining at least one feature of the source video, wherein the at least one feature comprises at least one of:
location information;
time information;
an event tag; or
a keyword.
25. The method of claim 24 , wherein the selecting comprises:
determining that the volumetric video shares the at least one feature of the source video.
26. The method of claim 24 , wherein the selecting is in accordance with a ranked list of volumetric videos from the library of volumetric videos based upon a level of matching to the at least one feature of the source video.
27. The method of claim 21 , wherein the obtaining the source video further comprises:
detecting a second object that appears in the source video, wherein at least one feature of the source video comprises the second object.
28. The method of claim 27 , wherein the selecting comprises:
detecting the second object appearing in the volumetric video.
29. The method of claim 21 , further comprising:
presenting, via an endpoint device, the volumetric video that is modified.
30. The method of claim 21 , further comprising:
generating an output video comprising a two dimensional traversal of the volumetric video that is modified.
31. The method of claim 30 , wherein the output video is generated in accordance with a selection by a user of at least one viewing perspective within the space of the volumetric video.
32. The method of claim 31 , wherein the at least one viewing perspective is one of a plurality of viewing perspectives within the space of the volumetric video for which a two dimensional traversal of the volumetric video is available.
33. The method of claim 30 , further comprising:
presenting, via an endpoint device, the output video.
34. The method of claim 33 , wherein the source video is obtained from the endpoint device.
35. The method of claim 21 , wherein the obtaining the three-dimensional object model of the first object is in accordance with a catalog of two-dimensional objects and three-dimensional object models that are matched to the two-dimensional objects.
36. The method of claim 35 , wherein the obtaining the three-dimensional object model comprises:
matching the first object to one of the two-dimensional objects in the catalog; and
obtaining, from the catalog, the three-dimensional object model that is matched to the one two-dimensional object.
37. The method of claim 36 , wherein the matching is in accordance with a machine learning-based image detection model.
38. The method of claim 21 , wherein the modifying comprises changing voxel data of the volumetric video that corresponds to the first object so that the voxel data corresponds to the enhanced three-dimensional object model.
39. A non-transitory computer-readable medium storing instructions which, when executed by a processing system including at least one processor, cause the processing system to perform operations, the operations comprising:
obtaining a source video, wherein the source video is a two-dimensional video;
selecting a volumetric video from a library of volumetric videos;
identifying a first object in the source video and an audio portion in the source video associated with the first object;
determining a position of the first object within a space of the volumetric video;
obtaining a three-dimensional object model of the first object;
texture mapping the first object to the three-dimensional object model of the first object to generate an enhanced three-dimensional object model of the first object; and
modifying the volumetric video to include the enhanced three-dimensional object model of the first object in the position of the first object within the space of the volumetric video, wherein the modifying further includes modifying the volumetric video to include the audio portion associated with the first object.
40. A device comprising:
a processing system including at least one processor; and
a computer-readable medium storing instructions which, when executed by the processing system, cause the processing system to perform operations, the operations comprising:
obtaining a source video, wherein the source video is a two-dimensional video;
selecting a volumetric video from a library of volumetric videos;
identifying a first object in the source video and an audio portion in the source video associated with the first object;
determining a position of the first object within a space of the volumetric video;
obtaining a three-dimensional object model of the first object;
texture mapping the first object to the three-dimensional object model of the first object to generate an enhanced three-dimensional object model of the first object; and
modifying the volumetric video to include the enhanced three-dimensional object model of the first object in the position of the first object within the space of the volumetric video, wherein the modifying further includes modifying the volumetric video to include the audio portion associated with the first object.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US17/556,873 US20220116577A1 (en) | 2018-11-27 | 2021-12-20 | Volumetric video-based augmentation with user-generated content |
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US16/201,525 US10771763B2 (en) | 2018-11-27 | 2018-11-27 | Volumetric video-based augmentation with user-generated content |
US17/013,808 US11206385B2 (en) | 2018-11-27 | 2020-09-07 | Volumetric video-based augmentation with user-generated content |
US17/556,873 US20220116577A1 (en) | 2018-11-27 | 2021-12-20 | Volumetric video-based augmentation with user-generated content |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/013,808 Continuation US11206385B2 (en) | 2018-11-27 | 2020-09-07 | Volumetric video-based augmentation with user-generated content |
Publications (1)
Publication Number | Publication Date |
---|---|
US20220116577A1 true US20220116577A1 (en) | 2022-04-14 |
Family
ID=70771031
Family Applications (3)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/201,525 Active US10771763B2 (en) | 2018-11-27 | 2018-11-27 | Volumetric video-based augmentation with user-generated content |
US17/013,808 Active US11206385B2 (en) | 2018-11-27 | 2020-09-07 | Volumetric video-based augmentation with user-generated content |
US17/556,873 Abandoned US20220116577A1 (en) | 2018-11-27 | 2021-12-20 | Volumetric video-based augmentation with user-generated content |
Family Applications Before (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/201,525 Active US10771763B2 (en) | 2018-11-27 | 2018-11-27 | Volumetric video-based augmentation with user-generated content |
US17/013,808 Active US11206385B2 (en) | 2018-11-27 | 2020-09-07 | Volumetric video-based augmentation with user-generated content |
Country Status (1)
Country | Link |
---|---|
US (3) | US10771763B2 (en) |
Families Citing this family (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
KR102595787B1 (en) * | 2018-02-27 | 2023-11-24 | 삼성전자주식회사 | Electronic device and control method thereof |
KR102526700B1 (en) * | 2018-12-12 | 2023-04-28 | 삼성전자주식회사 | Electronic device and method for displaying three dimensions image |
CN113243112A (en) * | 2018-12-21 | 2021-08-10 | 皇家Kpn公司 | Streaming volumetric and non-volumetric video |
CN111869201B (en) * | 2019-01-08 | 2023-01-31 | 三星电子株式会社 | Method for processing and transmitting three-dimensional content |
US11189100B2 (en) * | 2019-11-25 | 2021-11-30 | Verizon Patent and Acessing Inc. | Systems and methods for optimizing extended reality experiences |
US11683453B2 (en) * | 2020-08-12 | 2023-06-20 | Nvidia Corporation | Overlaying metadata on video streams on demand for intelligent video analysis |
US20220165024A1 (en) * | 2020-11-24 | 2022-05-26 | At&T Intellectual Property I, L.P. | Transforming static two-dimensional images into immersive computer-generated content |
US11941080B2 (en) | 2021-05-20 | 2024-03-26 | Retrocausal, Inc. | System and method for learning human activities from video demonstrations using video augmentation |
CN115812173A (en) * | 2021-05-20 | 2023-03-17 | 京东方科技集团股份有限公司 | Method for dynamically displaying three-dimensional image objects in a stereoscopic display device, dynamic stereoscopic display device and computer program product |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20160189334A1 (en) * | 2014-12-29 | 2016-06-30 | Nbcuniversal Media, Llc | Apparatus and method for generating virtual reality content |
US20190243530A1 (en) * | 2018-02-06 | 2019-08-08 | Adobe Inc. | Immersive Media Content Navigation and Editing Techniques |
Family Cites Families (24)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7797740B2 (en) | 2006-01-06 | 2010-09-14 | Nokia Corporation | System and method for managing captured content |
US20090021513A1 (en) * | 2007-07-18 | 2009-01-22 | Pixblitz Studios Inc. | Method of Customizing 3D Computer-Generated Scenes |
US8538678B2 (en) | 2009-09-30 | 2013-09-17 | Nokia Corporation | Method and apparatus for thematically modifying location and related information |
US20110123117A1 (en) * | 2009-11-23 | 2011-05-26 | Johnson Brian D | Searching and Extracting Digital Images From Digital Video Files |
US8447136B2 (en) * | 2010-01-12 | 2013-05-21 | Microsoft Corporation | Viewing media in the context of street-level images |
US9160938B2 (en) * | 2010-04-12 | 2015-10-13 | Wsi Corporation | System and method for generating three dimensional presentations |
US9277164B2 (en) | 2010-07-06 | 2016-03-01 | Mark Lane | Apparatus, system, and method for tagging objects in a video stream |
US20120050491A1 (en) | 2010-08-27 | 2012-03-01 | Nambi Seshadri | Method and system for adjusting audio based on captured depth information |
US20120142415A1 (en) * | 2010-12-03 | 2012-06-07 | Lindsay L Jon | Video Show Combining Real Reality and Virtual Reality |
SG192884A1 (en) | 2011-02-21 | 2013-09-30 | Univ Singapore | Apparatus, system, and method for annotation of media files with sensor data |
US9424255B2 (en) | 2011-11-04 | 2016-08-23 | Microsoft Technology Licensing, Llc | Server-assisted object recognition and tracking for mobile devices |
US10127722B2 (en) | 2015-06-30 | 2018-11-13 | Matterport, Inc. | Mobile capture visualization incorporating three-dimensional and two-dimensional imagery |
US20140286624A1 (en) | 2013-03-25 | 2014-09-25 | Nokia Corporation | Method and apparatus for personalized media editing |
US10521671B2 (en) * | 2014-02-28 | 2019-12-31 | Second Spectrum, Inc. | Methods and systems of spatiotemporal pattern recognition for video content development |
US10852838B2 (en) * | 2014-06-14 | 2020-12-01 | Magic Leap, Inc. | Methods and systems for creating virtual and augmented reality |
US20160041998A1 (en) * | 2014-08-05 | 2016-02-11 | NFL Enterprises LLC | Apparatus and Methods for Personalized Video Delivery |
CN104754359B (en) | 2015-01-26 | 2017-07-21 | 清华大学深圳研究生院 | A kind of depth map encoding distortion prediction method of Two Dimensional Free viewpoint video |
WO2016130935A1 (en) * | 2015-02-13 | 2016-08-18 | Harvey Gregory Lawrence | System and method to integrate content in real time into a dynamic 3-dimensional scene |
US10708569B2 (en) | 2016-09-29 | 2020-07-07 | Eric Wilson | Turbine-Powered Pool Scrubber |
US20180160194A1 (en) * | 2016-12-01 | 2018-06-07 | Google Inc. | Methods, systems, and media for enhancing two-dimensional video content items with spherical video content |
US10489975B2 (en) | 2017-01-04 | 2019-11-26 | Daqri, Llc | Environmental mapping system |
US10796723B2 (en) * | 2017-05-26 | 2020-10-06 | Immersive Licensing, Inc. | Spatialized rendering of real-time video data to 3D space |
US10769411B2 (en) * | 2017-11-15 | 2020-09-08 | Qualcomm Technologies, Inc. | Pose estimation and model retrieval for objects in images |
US20190230317A1 (en) * | 2018-01-24 | 2019-07-25 | Blueprint Reality Inc. | Immersive mixed reality snapshot and video clip |
-
2018
- 2018-11-27 US US16/201,525 patent/US10771763B2/en active Active
-
2020
- 2020-09-07 US US17/013,808 patent/US11206385B2/en active Active
-
2021
- 2021-12-20 US US17/556,873 patent/US20220116577A1/en not_active Abandoned
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20160189334A1 (en) * | 2014-12-29 | 2016-06-30 | Nbcuniversal Media, Llc | Apparatus and method for generating virtual reality content |
US20190243530A1 (en) * | 2018-02-06 | 2019-08-08 | Adobe Inc. | Immersive Media Content Navigation and Editing Techniques |
Also Published As
Publication number | Publication date |
---|---|
US20200169717A1 (en) | 2020-05-28 |
US20200404237A1 (en) | 2020-12-24 |
US10771763B2 (en) | 2020-09-08 |
US11206385B2 (en) | 2021-12-21 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11206385B2 (en) | Volumetric video-based augmentation with user-generated content | |
US20210368155A1 (en) | Volumetric video creation from user-generated content | |
US10810430B2 (en) | Augmented reality with markerless, context-aware object tracking | |
US11470297B2 (en) | Automatic selection of viewpoint characteristics and trajectories in volumetric video presentations | |
US11956546B2 (en) | Selecting spectator viewpoints in volumetric video presentations of live events | |
US9058757B2 (en) | Systems and methods for image or video personalization with selectable effects | |
US11431953B2 (en) | Opportunistic volumetric video editing | |
WO2017196670A1 (en) | Bit depth remapping based on viewing parameters | |
US11670099B2 (en) | Validating objects in volumetric video presentations | |
US11663725B2 (en) | Selecting viewpoints for rendering in volumetric video presentations | |
US11145065B2 (en) | Selection of video frames using a machine learning predictor | |
US11941816B2 (en) | Automated cropping of images using a machine learning predictor | |
US11080549B1 (en) | Automated cropping of images using a machine learning predictor | |
US20220086470A1 (en) | Transcoding ultra-high-definition panoramic videos | |
US20230209003A1 (en) | Virtual production sets for video content creation | |
Kim et al. | Heterogeneous Terminals-Aware Viewer Centric Specific Scenes Extracting Scheme for Personal Broadcasting IP-TV Service | |
CN113327308A (en) | Method and device for generating expression package picture |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
AS | Assignment |
Owner name: AT&T INTELLECTUAL PROPERTY I, L.P., GEORGIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:ZAVESKY, ERIC;LIU, ZHU;GIBBON, DAVID CRAWFORD;AND OTHERS;SIGNING DATES FROM 20181113 TO 20181121;REEL/FRAME:058707/0105 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |