CN106416239B - Method and apparatus for delivering content and/or playing back content - Google Patents
Method and apparatus for delivering content and/or playing back content Download PDFInfo
- Publication number
- CN106416239B CN106416239B CN201580028645.9A CN201580028645A CN106416239B CN 106416239 B CN106416239 B CN 106416239B CN 201580028645 A CN201580028645 A CN 201580028645A CN 106416239 B CN106416239 B CN 106416239B
- Authority
- CN
- China
- Prior art keywords
- content
- stream
- image
- environment
- content stream
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 93
- 238000003860 storage Methods 0.000 claims description 68
- 239000000463 material Substances 0.000 claims description 61
- 230000000007 visual effect Effects 0.000 claims description 57
- 238000009877 rendering Methods 0.000 claims description 48
- 230000005540 biological transmission Effects 0.000 claims description 41
- 238000012913 prioritisation Methods 0.000 claims description 17
- 239000000872 buffer Substances 0.000 claims description 12
- 238000009826 distribution Methods 0.000 claims description 9
- 238000013459 approach Methods 0.000 claims description 7
- 230000001052 transient effect Effects 0.000 claims description 3
- 238000012360 testing method Methods 0.000 claims description 2
- 238000005243 fluidization Methods 0.000 abstract description 32
- 210000003128 head Anatomy 0.000 description 127
- 230000008859 change Effects 0.000 description 50
- 238000013507 mapping Methods 0.000 description 36
- 230000006870 function Effects 0.000 description 21
- 238000009434 installation Methods 0.000 description 18
- 238000004891 communication Methods 0.000 description 17
- 238000004088 simulation Methods 0.000 description 14
- 238000010276 construction Methods 0.000 description 13
- 230000015572 biosynthetic process Effects 0.000 description 11
- 238000012545 processing Methods 0.000 description 11
- 238000003786 synthesis reaction Methods 0.000 description 11
- 230000036961 partial effect Effects 0.000 description 9
- NJPPVKZQTLUDBO-UHFFFAOYSA-N novaluron Chemical compound C1=C(Cl)C(OC(F)(F)C(OC(F)(F)F)F)=CC=C1NC(=O)NC(=O)C1=C(F)C=CC=C1F NJPPVKZQTLUDBO-UHFFFAOYSA-N 0.000 description 8
- 230000008569 process Effects 0.000 description 8
- 230000011218 segmentation Effects 0.000 description 8
- 230000011664 signaling Effects 0.000 description 8
- 238000007906 compression Methods 0.000 description 7
- 230000006835 compression Effects 0.000 description 7
- 230000002829 reductive effect Effects 0.000 description 7
- 230000000694 effects Effects 0.000 description 6
- 230000033001 locomotion Effects 0.000 description 6
- 239000002131 composite material Substances 0.000 description 5
- 238000001914 filtration Methods 0.000 description 5
- 238000002156 mixing Methods 0.000 description 5
- 239000007787 solid Substances 0.000 description 5
- 230000003068 static effect Effects 0.000 description 5
- 238000001514 detection method Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 238000007654 immersion Methods 0.000 description 4
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 239000004033 plastic Substances 0.000 description 3
- 229920003023 plastic Polymers 0.000 description 3
- 230000004044 response Effects 0.000 description 3
- 238000006467 substitution reaction Methods 0.000 description 3
- 241000251468 Actinopterygii Species 0.000 description 2
- OKTJSMMVPCPJKN-UHFFFAOYSA-N Carbon Chemical compound [C] OKTJSMMVPCPJKN-UHFFFAOYSA-N 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 230000002860 competitive effect Effects 0.000 description 2
- 230000003111 delayed effect Effects 0.000 description 2
- 230000001419 dependent effect Effects 0.000 description 2
- 238000000151 deposition Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 239000003365 glass fiber Substances 0.000 description 2
- 229910002804 graphite Inorganic materials 0.000 description 2
- 239000010439 graphite Substances 0.000 description 2
- 238000005259 measurement Methods 0.000 description 2
- 210000001747 pupil Anatomy 0.000 description 2
- 230000009467 reduction Effects 0.000 description 2
- 238000013468 resource allocation Methods 0.000 description 2
- 230000002194 synthesizing effect Effects 0.000 description 2
- 230000002123 temporal effect Effects 0.000 description 2
- 241001269238 Data Species 0.000 description 1
- 235000010627 Phaseolus vulgaris Nutrition 0.000 description 1
- 244000046052 Phaseolus vulgaris Species 0.000 description 1
- XUIMIQQOPSSXEZ-UHFFFAOYSA-N Silicon Chemical compound [Si] XUIMIQQOPSSXEZ-UHFFFAOYSA-N 0.000 description 1
- 229910000831 Steel Inorganic materials 0.000 description 1
- 238000013475 authorization Methods 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 150000001875 compounds Chemical class 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 238000013144 data compression Methods 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 210000000887 face Anatomy 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 239000012530 fluid Substances 0.000 description 1
- 210000001061 forehead Anatomy 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- PCHJSUWPFVWCPO-UHFFFAOYSA-N gold Chemical compound [Au] PCHJSUWPFVWCPO-UHFFFAOYSA-N 0.000 description 1
- 239000010931 gold Substances 0.000 description 1
- 229910052737 gold Inorganic materials 0.000 description 1
- 230000012447 hatching Effects 0.000 description 1
- 235000019692 hotdogs Nutrition 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- JEIPFZHSYJVQDO-UHFFFAOYSA-N iron(III) oxide Inorganic materials O=[Fe]O[Fe]=O JEIPFZHSYJVQDO-UHFFFAOYSA-N 0.000 description 1
- 238000004020 luminiscence type Methods 0.000 description 1
- 230000014759 maintenance of location Effects 0.000 description 1
- 239000002184 metal Substances 0.000 description 1
- 229910052751 metal Inorganic materials 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000000465 moulding Methods 0.000 description 1
- 230000000149 penetrating effect Effects 0.000 description 1
- 238000005375 photometry Methods 0.000 description 1
- 238000011112 process operation Methods 0.000 description 1
- 230000000717 retained effect Effects 0.000 description 1
- 230000033764 rhythmic process Effects 0.000 description 1
- 230000000630 rising effect Effects 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 229910052710 silicon Inorganic materials 0.000 description 1
- 239000010703 silicon Substances 0.000 description 1
- 239000010959 steel Substances 0.000 description 1
- 235000019640 taste Nutrition 0.000 description 1
- 210000003454 tympanic membrane Anatomy 0.000 description 1
- 238000005303 weighing Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/194—Transmission of image signals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
- H04N13/161—Encoding, multiplexing or demultiplexing different image signal components
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/204—Image signal generators using stereoscopic image cameras
- H04N13/243—Image signal generators using stereoscopic image cameras using three or more 2D image sensors
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- H04N13/332—Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
- H04N13/344—Displays for viewing with the aid of special glasses or head-mounted displays [HMD] with head-mounted left-right displays
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- H04N13/366—Image reproducers using viewer tracking
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/21—Server components or server architectures
- H04N21/218—Source of audio or video content, e.g. local disk arrays
- H04N21/21805—Source of audio or video content, e.g. local disk arrays enabling multiple viewpoints, e.g. using a plurality of cameras
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/21—Server components or server architectures
- H04N21/218—Source of audio or video content, e.g. local disk arrays
- H04N21/2187—Live feed
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
- H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
- H04N21/23439—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements for generating different versions
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/24—Monitoring of processes or resources, e.g. monitoring of server load, available bandwidth, upstream requests
- H04N21/2402—Monitoring of the downstream path of the transmission network, e.g. bandwidth available
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/25—Management operations performed by the server for facilitating the content distribution or administrating data related to end-users or client devices, e.g. end-user or client device authentication, learning user preferences for recommending movies
- H04N21/266—Channel or content management, e.g. generation and management of keys and entitlement messages in a conditional access system, merging a VOD unicast channel into a multicast channel
- H04N21/2662—Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/442—Monitoring of processes or resources, e.g. detecting the failure of a recording device, monitoring the downstream bandwidth, the number of times a movie has been viewed, the storage space available from the internal hard disk
- H04N21/44213—Monitoring of end-user related data
- H04N21/44218—Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client
- H04N21/63—Control signaling related to video distribution between client, server and network components; Network processes for video distribution between server and clients or between remote clients, e.g. transmitting basic layer and enhancement layers over different transmission paths, setting up a peer-to-peer communication via Internet between remote STB's; Communication protocols; Addressing
- H04N21/64—Addressing
- H04N21/6405—Multicasting
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client
- H04N21/65—Transmission of management data between client and server
- H04N21/658—Transmission by the client directed to the server
- H04N21/6587—Control parameters, e.g. trick play commands, viewpoint selection
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/81—Monomedia components thereof
- H04N21/816—Monomedia components thereof involving special video data, e.g 3D video
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/83—Generation or processing of protective or descriptive data associated with content; Content structuring
- H04N21/845—Structuring of content, e.g. decomposing content into time segments
- H04N21/8456—Structuring of content, e.g. decomposing content into time segments by decomposing the content in the time domain, e.g. in time segments
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Databases & Information Systems (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Social Psychology (AREA)
- Computer Networks & Wireless Communication (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
- Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
- Information Transfer Between Computers (AREA)
Abstract
Describe content delivery and back method and device.The method and device are very suitable for the delivery and playback of the content corresponding to 360 degree of environment, and can be used to support correspond to such as sports tournament etc event content (such as, 3D content) fluidisation and/or in real time deliver, for example, when event is carrying out or after event terminates.The part of environment is captured by the camera for being located at different location.The content captured from different location is encoded and it is made to can be used for delivering.Playback apparatus selects to want received content based on the head position of user.Stream can present viewing field based on user and/or end rotation direction be prioritized and be selected for delivering.Still image or composograph can be used and be combined with the content from one or more streams (such as background, sky and/or above ground portion).
Description
Technical field
The present invention relates to content delivery and/or playback, for example, the playback of stereoscopic image content.
Background technique
The display equipment for being intended to provide immersion experience allows generally for user to rotate his head and experiences shown field
Corresponding variation in scape.Head-mounted display supports 360 degree of viewings sometimes, because user can be when wearing head-mounted display
Rotation, as the head position of user changes, the scene change of display.
Using such equipment, when eyes front, the scene captured before camera position should be presented on to user, when with
When family is turned round completely, the scene that camera position captures below should be presented on.Although user can go to his head below,
It is that at any given time, since people perceives the essence of the ability of Limited eyeshot at any given time, the visual field of user is logical
Often it is limited to 120 degree or smaller.
In order to support 360 degree of viewing angle, 360 degree of scene can use multiple cameras to capture, and image is combined
To generate the 360 degree of scenes that will can be used for watching.
It should be appreciated that 360 degree of views include than being usually common TV and the capture of many other Video Applications, coding
The much more image data of simple frontal view, in normal tv and many other Video Applications, user has no chance to change
It will be in the viewing angle for the image that particular point in time is shown for determination.
It is given to be constrained with the associated transmission of content that is being fluidized, for example, network data constraint, it may not be possible to seeking to receive
Content and all consumers interacted fluidize complete 360 degree of views with overall height clarity video.Being when content includes being intended to
When stereo content corresponding to the picture material of left eye and right-eye view to allow 3D viewing effect, especially such case.
In view of above discussion, it should be understood that for being changed with allowing individual consumer for example, by rotating his or her head
Become his viewing location and sees that the mode of the expectations section of environment supports the fluidisation of content and/or the method and apparatus of playback to deposit
It is needing.If can be while abiding by due to bandwidth or other data streaming constraints for delivering related constraint and possible application
The option for changing his/her head position and thus changing view direction is provided a user, this will be expected to.Although no
It is all required to all embodiments, but it is expected that at least some embodiments allow the receiving stream simultaneously of multiple users in different location
And any different piece in their desired environment is watched, but regardless of other users in viewing which or which part.
Summary of the invention
Describe the delivery (for example, fluidisation) for supporting the video for corresponding to 360 degree of viewing areas or other contents
Method and apparatus.Fluidisation of the methods and apparatus of the present invention particularly suitable for picture materials three-dimensional and/or other in following situation,
The transmission constraint of middle data may 360 degree of content to deliver to be difficult to (such as utilize best matter with the quality level of maximum support
The frame rate that amount coding and highest are supported) it is delivered.But the method is not limited to stereo content.
In various embodiments, corresponding to the 3D model and/or 3D dimensional information of the environment of video content will be obtained from it
It is generated and/or accesses.Camera position in environment is recorded.May exist multiple and different camera positions in the environment.Example
Such as, different end objectives camera positions and one or more midfields camera position can be supported and be used to capture real time camera
Feeding.
3D model and/or other 3D information be stored in be used to video streaming to the server of one or more users or
In image capture device.
3D model is provided to user's playback apparatus with image rendering and synthesis capability, for example, consumer guard station is set
It is standby.The 3D of consumer's premises equipment build environment indicates that 3D expression is for example shown to consumer via head-mounted display and stays
The user of ground equipment.
In various embodiments, it is streamed to individual consumer at any given time less than complete 360 degree of environment
Premises equipment.Consumer's premises equipment is based on user's input and indicates which camera feed is to be streamed.User can be via conduct
A part of consumer's premises equipment is attached to the input equipment of consumer's premises equipment to select place and/or phase seat in the plane
It sets.
In some embodiments, 180 degree video flowing is from the server and/or video camera of responsible streamable content with for example real
Condition, real-time or near real-time stream are sent to client's playback apparatus.Playback apparatus monitors the head position of user and therefore plays back
Equipment knows the viewing areas watched in 3D environment that the user of playback apparatus is being generated by playback apparatus.When the one of 3D environment
When part can be used for being watched, video is presented in consumer's premises equipment, and wherein video content is replaced or is shown as not have
The substitution for the simulation 3D environment being presented in the case where video content.When the user of playback apparatus rotates his or her head, it is in
Now the part to the environment of user can come from the video content for being supplied to (such as being streamed to) playback apparatus, wherein other parts
What is from 3D model and/or previously provided synthetically generates with video content different time captured image content.
Therefore, playback apparatus can show the video for example provided via fluidisation, while compete, concert or other events
Still carrying out, 180 degree camera view before corresponding to for example, and the rear and/or lateral parts of 3D environment or complete synthesis or
Person generates from the picture material of side or Background Region in different time environment.
Although user can by provide flow content server signaling position variation come between camera position into
Row selection, but the server for being to provide flow content may provide for generating synthetic environment for the part for the 3D environment not being fluidized
Useful information.
For example, in some embodiments, multiple rear portions and lateral plan are captured in different time, for example, in fluidisation one
Before partial content or since time point earlier.Image is buffered in playback apparatus.The server for providing content can
With (and in some embodiments really) to which of one group of non real-time scene of playback apparatus signaling or image will by with
In the synthesis for the environment division not provided in video streaming.For example, image that concert participant is seated and concert participant
Standing can be provided in the subsequent another image of camera position and be stored in playback apparatus.Server can be with signaling when specific
Between point should use which group storage image data.Therefore, when crowd stands, server can be stood with signaling corresponding to crowd
Image background 180 degree view should be used for during image synthesizes, and when crowd is seated, server can be stayed to consumer
Ground equipment indicates that it should be when synthesizing the side or rear portion of 3D camera environment using the image or figure for corresponding to the crowd being seated
As composite signal.
In at least some embodiments, camera direction of each of one or more positions position in 3D environment exists
It is tracked during image capture.Label and/or identification point in environment can be used to promote institute's captured image, such as activity
Image, alignment and/or other mappings with the previous 3D environment for modeling and/or mapping to be simulated by consumer's premises equipment.
The mixing of the environment division of synthesis and true (video of fluidisation) provides immersion video tastes.Environment can with and
Sometimes measure or model using 3d photometric measurement, be used to creation when video is unavailable, for example, before the environment not by
When modeling, the 3D information of simulated environment.
Reference mark is in the 3D model that the use of commitment positions facilitates video and is previously generated in real world space
Calibration and alignment.
The realization when video is captured of the position tracking of each camera.Relative to the camera location information in place, for example, with
Degree is that unit maps X, Y, Z and yaw (yaw) (therefore, it is understood that position that each camera is directed toward).This allows easily to examine
What part that institute's captured image corresponds to environment is surveyed, and, it is set when being sent to playback together with the video captured
When standby, allow to play back automatically to capture our video and the synthetic environment by playback apparatus generation during image is presented
Overlapping, for example, being played back to user.The content of fluidisation can be limited to less than 360 degree views, for example, captured in phase seat in the plane
Set the 180 degree view in the region of front.When viewer looks around, when back track, viewer will be seen that the background of simulation (no
It is black blank), and when turning to front, viewer will be seen that video.
Synthetic environment can be (and being in some embodiments) interactive mode.In some embodiments, multiple reality
Viewer (such as user of different consumer's premises equipment) be included in simulated environment, allow user in virtual 3D
Environment neutralizes his/her friend and comes to matches together, and it is practical in stadium to appear to user.
The image of user can (and in some embodiments) by include or be attached to the phase of consumer's premises equipment
Machine capture is supplied to server and is supplied to other users, such as the member of group, uses for generating simulated environment.User's figure
As needing not be realtime graphic, but it can be realtime graphic.
The method can be used in real time or near real-time encode and provide content, but be not limited to this real-time application.
Given that real-time and near real-time is supported to encode and be streamed to the ability of multiple users, method described herein and device are very suitable for flowing
Change the scene of competitive sports, concert and/or other places, wherein individual like viewing event and not only observe stage or
Place, and the view of environment can be rotated and appreciate, such as stadium or crowd.By supporting 360 degree of viewings and 3D, this hair
Bright method and apparatus are very suitable for being used together with the head-mounted display for being intended to provide a user the experience of 3D immersion, have
Rotate and from different viewing angles scene freedom degree, just as user be in environment and the head port of user,
Right or rear.
Describe the method and apparatus for transmitting picture material (for example, the content for corresponding to 360 degree of visuals field).Various
In embodiment, the visual field corresponds to the different piece of environment, such as front, at least one rear portion, top and bottom.In some implementations
In example, the left back and right back (for example, rear portion) of environment is separately generated and/or is transmitted.Playback apparatus monitors user's head
Position and generate image, such as the stereo-picture of environment division seen corresponding to user in given time then will figure
As being shown to user.In the case where solid playback, the left-eye image and eye image of separation are generated.Image generated can be with
(and in some embodiments really) correspond to one or more scenes, such as environment division.
When playing back beginning, the head level position of user's eyes front is arranged to be default to correspond to preceding to scene portion
Point.When user rotates his/her head and raises or reduce his or her head, the other parts of environment, which can enter, to be used
The visual field at family.
Bandwidth and image decoding ability on many playback apparatus is by the processing capacity of equipment and/or for receiving in image
The limitation of the bandwidth of appearance.In some embodiments, playback apparatus determine environment which partially correspond to user main view it is wild.So
Afterwards, equipment will be with the received part of high-speed (such as specified stream using full resolution) as main from preferential angle Selection
Stream.It can also be received from the content for corresponding to one or more of the other stream of the content of other parts of environment is provided, but
Usually with lower data rate.Content delivery for specific stream can be started by playback apparatus, such as be used to by sending
Trigger the signal of content delivery.The signal can be used to that multicast group is added, to provide the content for corresponding to a part of environment
Or the delivery of starting switched digital data interconnect.In the other signals that broadcasted content does not need request or signal is added in such as multicast group
In the case where, equipment can start to receive by being transferred to the available channel of content.
It is assumed that user is usually mainly interested in the front view part of environment, because this is the ground that main actions are normally carried out
Side, especially when content corresponds to competitive sports, Springsteen, fashion show or multiple and different events, in some embodiments
In, the frontal view part of environment is given data transport priority.In at least some embodiments, correspond to front and watch position
The image set is fluidized with the higher rate in one or more of the other part than 360 degree of environment.Other portions corresponding to environment
The image divided is sent using lower data rate or is sent as still image.For example, can send top (for example, sky) and
One or more still images of bottom (for example, ground).
In some embodiments, it is sent for one or more parts (such as rear view section or sky portion) of environment more
A static state captured image.In some embodiments, which of the still image of a part that instruction is used for environment is sent
It should be in the control information that given time is used.In the case where the still image of a part for environment is sent, it
Can be sent and be then stored in memory with decoded form in a coded form, for being combined with other picture materials.
In this way, decoding resource required during event can be reduced, because multiple streams are not needed with identical frame rate by simultaneously
Row decoding.Still image can be sent before the content of fluidisation main matter.As an alternative, it is contemplated that the head of user
Position is from the change of viewing location forward, for environment in the case that a few image can need them during playback
Different piece sends and stores.Static or image infrequently may be provided as being used for mainly, such as forward direction, view direction
A part of the content stream of content is encoded and sends, or can be used as individual content stream and sent.
Corresponding to rear portion still image can (and sometimes really) be captured image before the event, and it is right
Should in the forward portion of environment content can (and in many cases really) be included in event for example in real time into
The content for being captured and fluidizing when row.
Consider that such as two different rearview scenes are delivered and stored in the situation in playback apparatus.One scene can
To correspond to the crowd in standing place, another image can correspond to the crowd in seated position.Controlling information can
With (and in some embodiments really) if the visible position in rear portion that his/her head is turned to environment by instruction user
Words are taken one's seat or whether standing place crowd's image is used in given time.
Similarly, the multiple images of sky can be sent to playback apparatus and are stored in back with coding or decoded form
It puts in equipment.In some embodiments, which image of sky portion will be used in given time and be transmitted in the control information.
In other embodiments, it is based on corresponding to preceding one or more images to scene areas which scene of sky is to be used
Brightness automatically determine, wherein sky portion and selected forward direction environment scene part are consistent or close, for example, before bright
Scape scene areas can be detected and be used to control the selection to the bright sky image with a small amount of cloud.Similarly, some
It will lead to dark cloudy sky image to the detection before darkness to environmental area in embodiment to be used.
For in the not available situation of image of a part of environment, scene parts can be for example from from ring in the visual field
The information or content of the available other parts in border are come into.For example, coming from frontal scene if rear images part is unavailable
The left side in region and/or the content on right side can be replicated and be used to fill the rear portion of the missing of environment.Replicate it unless the context
Outside, it obscures and/or other image processing operations can also be used to fill the lack part of environment in some embodiments.As replacing
Dai Di, in some embodiments, pictorial information provides in the content stream and playback apparatus is that the part lost generates conjunction completely
At image.As video game content, this content substantially can be reality and may include from draw and/or
The various image effects and/or content that the other image creation rules being stored in playback apparatus generate.
According to some embodiments, the illustrative methods for operating playback system include the head position of determining viewer, described
Head position corresponds to present viewing field；It receives and the first content stream of the content for the first part for corresponding to environment is provided；Based on packet
Include the storage at least some of the first content stream received content and i) corresponding to the second part of the environment
Inside perhaps ii) the simulation environment second part composograph, generate the one or more outputs for corresponding to present viewing field
Image；And output or display the first output image, the first output image be one or more output images generated it
One.According to some embodiments, example content playback system includes viewer's head position determining module, is configured to determine that sight
The head position for the person of seeing, the head position correspond to present viewing field；Content stream receiving module is configured as receiving offer correspondence
In the first content stream of the content of the first part of environment；Based on the generation module of output picture material stream, it is configured as being based on
It is included in the received content of at least some of the first content stream and i) depositing corresponding to the second part of the environment
Perhaps ii in storing up) the simulation environment second part composograph, generate correspond to present viewing field one or more it is defeated
Image out；And it is following at least one: be configured as exporting the output module of the first output image or be configured
For the display module for showing the first output image.
Numerous variations and embodiment are possible and discuss in the following detailed description.
Detailed description of the invention
Fig. 1 shows the exemplary system realized according to some embodiments of the invention, which can be used to capture and flow
Change content, is then shown with one or more composite parts together with environment for one or more users.
Fig. 2A shows exemplary stereo scene, for example, not yet divided complete 360 degree of stereo scenes.
Fig. 2 B shows the exemplary stereo field for being divided into 3 exemplary scenes according to an exemplary embodiment
Scape.
Fig. 2 C shows the exemplary stereo scene for being divided into 4 scenes according to an exemplary embodiment.
Fig. 3 show according to an exemplary embodiment, the example process of 360 degree of stereo scenes of encoding examples.
Fig. 4, which is shown, indicates how input picture part is encoded using various encoders to generate identical input picture portion
The example for the different coding version divided.
Fig. 5 shows the coded portion stored for being divided into the input stereo scene of 3 parts.
Fig. 6 is the illustrative methods for showing the streamable content for the exemplary embodiment realized according to the system using Fig. 1
The flow chart of step.
Fig. 7 shows feature according to the present invention, code capacity including that can be used to coding and streamable content example
Property content delivery system.
Fig. 8, which is shown, can be used to receive, decodes and show that the example content of the content by the system fluidisation of Fig. 7 plays back
System.
Fig. 9 shows the exemplary camera equipment for indicating to have 3 cameras pair for being mounted on 3 different installation sites
(rig) together with the figure for the calibration target that can be used for calibrating camera equipment.
Figure 10 shows the more concentration view for indicating to have the camera equipment for 3 cameras pair being mounted in camera equipment
Figure.
Figure 11 shows the detailed diagram for the exemplary camera equipment realized according to an exemplary embodiment.
Figure 12 shows exemplary 360 degree of scene environments, such as 360 degree of scene areas, can be divided into and capture this
The corresponding different viewing areas/parts in the different cameral position of the respective camera of the different piece of 360 degree of scenes.
Figure 13 includes three different figures for indicating the different piece of exemplary 360 degree of scene areas of Figure 12, these three portions
Dividing can be caught by the viewing areas/part different cameral for corresponding to and/or being positioned to exemplary 360 degree of scene areas
It obtains.
Figure 14 A is stream the step of showing the illustrative methods of operation playback apparatus according to an exemplary embodiment of the present invention
The first part of journey figure.
Figure 14 B is stream the step of showing the illustrative methods of operation playback apparatus according to an exemplary embodiment of the present invention
The second part of journey figure.
Figure 14 includes the combination of Figure 14 A and Figure 14 B.
Figure 15 is flow chart the step of showing stream selection subroutine accoding to exemplary embodiment.
Figure 16 is the flow chart for showing the step of stream accoding to exemplary embodiment is prioritized subroutine.
Figure 17 is flow chart the step of showing rendering subroutine accoding to exemplary embodiment.
Figure 18 shows the exemplary table including stream information corresponding with multiple content streams.
Figure 19 shows the exemplary playback system realized according to the present invention.
Figure 20 A is first of the flow chart of the illustrative methods of operation content playback system accoding to exemplary embodiment
Point.
Figure 20 B is second of the flow chart of the illustrative methods of operation content playback system accoding to exemplary embodiment
Point.
Figure 20 C is the third portion of the flow chart of the illustrative methods of operation content playback system accoding to exemplary embodiment
Point.
Figure 20 D is the 4th of the flow chart of the illustrative methods of operation content playback system accoding to exemplary embodiment
Point.
Figure 20 E is the 5th of the flow chart of the illustrative methods of operation content playback system accoding to exemplary embodiment
Point.
Figure 20 includes the combination of Figure 20 A, Figure 20 B, Figure 20 C, Figure 20 D and Figure 20 E.
Figure 21 is the figure of example content playback system accoding to exemplary embodiment, wherein example content playback system
Such as it is coupled to the content playback device or computer system of display.
Figure 22 is the figure that may include example modules component in the example content playback system of Figure 21.
Figure 23 be show it is in accordance with some embodiments can exemplary stream used in the playback system of Figure 19 select mould
The figure of block.
Figure 24 is a part for showing the stream selecting module that can be implemented as Figure 23 or the example for being embodied as separate modular
The figure of property stream prioritization module.
Specific embodiment
Fig. 1 shows the exemplary system 100 realized according to some embodiments of the invention.System 100 support to be located at disappear
One or more consumer devices (such as playback apparatus/content player) of the person of expense guard station carry out content delivery, such as are imaged
Content delivery.System 100 includes example images capture device 102, content delivery system 104, communication network 105 and multiple disappears
The person of expense guard station 106...110.The capture of the support stereo-picture of image capture device 102.Image capture device 102 is according to the present invention
Feature capture and handle image content.Communication network 105 can be such as Hybrid Fiber Coax (HFC) network, satellite network
The internet and/or.
Content delivery system 104 includes code device 112 and 114. code device 112 of stream content device/server can
With one or more encoders that (and certain in some embodiments) includes for coded image according to the present invention.Encoder
It can be used in parallel, to encode the different piece of scene and/or encode the given part of scene, to generate with different data speed
The version of code of rate.It the use of multiple encoders can be parallel particularly useful when to support in real time or near real-time fluidizes.
Stream content equipment 114 is configured as the content of fluidisation (such as transmission) coding, so as to for example through communication network 105
The picture material of coding is delivered to one or more consumer devices.Via network 105, content delivery system 104 can be sent
And/or be located at consumer guard station 106,110 devices exchange information, as in figure by pass through communication network 105 120 institute of link
As expression.
Although code device 112 and content delivery server 114 are illustrated as individual physical equipment in the example in fig 1,
But in some embodiments, they are implemented as the individual equipment of coding and streamable content.Cataloged procedure can be 3D (such as
It is three-dimensional) image encoding process, wherein being encoded corresponding to the left eye of scene parts and the information of right-eye view and being included in coding
Image data in, allow to that 3D rendering is supported to watch.Used specific coding method be not for the application it is crucial,
And extensive encoder is used as or for realizing code device 112.
Each consumer guard station 106,110 may include multiple equipment/player, such as decoding and playing back/show
By the playback system for the image content that stream content equipment 114 fluidizes.Consumer guard station 1 106 includes being coupled to display equipment
124 decoding apparatus/playback apparatus 122, and consumer guard station N 110 include be coupled to display equipment 128 decoding apparatus/return
Put equipment 126.In some embodiments, display equipment 124,128 is wear-type stereoscopic display device.In some embodiments,
Playback apparatus 122/126 and headset equipment 124/128 constitute playback system together.
In various embodiments, image content is presented in corresponding display equipment 124,128 in decoding apparatus 122,126.
Decoding apparatus/player 122,126, which can be, is able to carry out equipment below: decoding from content delivery system 104 it is received at
Image content is generated as content, using decoded content and renders image content, such as 3D rendering in display equipment 124,128
Content.Any one of decoding apparatus/playback apparatus 122,126 is used as decoding apparatus/playback apparatus shown in fig. 8
800.System/playback apparatus shown in such as Fig. 8 and 19 is used as any in decoding apparatus/playback apparatus 122,126
One.
Fig. 2A shows exemplary stereo scene 200, for example, not yet divided complete 360 degree of stereo scenes.Three-dimensional field
Scape can be and be usually combine from be usually mounted in single video capture platform or camera pedestal multiple cameras (such as
Video camera) captured image data result.
Fig. 2 B shows the segmentation version 2 50 of exemplary stereo scene 200, and wherein scene is according to an exemplary implementation
Example is divided into 90 degree of parts after 3 (N=3) a sample portions, such as front 180 degree part, left back 90 degree of parts and the right side.
Fig. 2 C shows another segmentation version 2 80 of exemplary stereo scene 200, according to an exemplary reality
It applies example and is divided into 4 a parts (N=4).
Although Fig. 2 B and 2C show two example divisions, but it is to be understood that other segmentations are possible.For example,
Scene 200 can be divided into a 30 degree of parts 12 (n=12).It in one suchembodiment, is not that individually coding is every
A part, but multiple portions are grouped together and are encoded as group.Partial different groups can be encoded and fluidize
To user, wherein each group of size is identical for total degree of scene, but correspond to may rely on user's
The different piece for the image that head position (such as viewing angle measured with the scale of 0 to 360 degree) is fluidized.
Fig. 3 show according to an exemplary embodiment, the example process of 360 degree of stereo scenes of encoding examples.
The input of method 300 shown in Fig. 3 includes being captured by the multiple cameras for being for example arranged to the 360 of capturing scenes degree of views
360 degree of stereoscopic image datas.Stereoscopic image data (such as three-dimensional video-frequency) can be in various known formats it is any also,
In most embodiments, left eye and right eye image data including being used to that 3D is allowed to experience.Although the method particularly suitable for
Three-dimensional video-frequency, but technique described herein and method also can be applied to the 2D image of such as 360 degree or small scene areas.
In step 304, contextual data 302 is divided into the data corresponding to different scenes region, such as corresponding to not
With N number of scene areas of view direction.For example, 360 degree of scene areas are divided in one embodiment shown in such as Fig. 2 B
It is cut into three subregions: corresponding to 90 degree of parts behind the left back portion of 90 degree of parts, preceding 180 degree part and the right side.Different parts can
To be captured by different camera, but this is not required, in fact, 360 degree scenes can by be divided into as Fig. 2 B with
The data building captured before N number of scene areas shown in 2C from multiple cameras.
Within step 306, the data corresponding to different scenes part are encoded according to the present invention.In some embodiments
In, each scene parts are independently encoded by multiple encoders, to support to be used for multiple possible bit rate flows of each part.
In step 308, the scene parts of coding are stored in the content delivery server 114 of such as content delivery system 104, with
For being streamed to consumer's playback apparatus.
Fig. 4 is to show how display input picture part (such as 180 degree front of scene) is compiled using various encoders
Code is to generate Figure 40 0 of the example of the different coding version of identical input picture part.
As shown in diagram 400, input scene part 402 (such as 180 degree front of scene) is supplied to multiple encoders
It is encoded.In this embodiment, there are K different encoders, these encoders are using different resolution and utilize different volumes
Code technology for encoding input data supports the different data rate stream of picture material to generate the data of coding.This K encoder
Including fine definition (HD) encoder 1 404, standard definition (SD) encoder 2 406, the frame rate SD encoder 3 reduced
408... and high compression reduce frame rate SD encoder K 410.
HD encoder 1 404 is configured as executing overall height clarity (HD) coding, to generate high bit rate HD coded image
412.SD encoder 2 406 is configured as executing low resolution standard definition coding, encodes version to generate the SD of input picture
This 2 414.Reduced frame rate SD encoder 3 408 is configured as executing reduced frame rate low resolution SD coding, to produce
The reduced rate SD version of code 3 416 of raw input picture.Reduced frame rate, which can be, for example to be used by SD encoder 2 406
In the half of the frame rate of coding.The frame rate SD encoder K 410 that high compression reduces, which is configured as executing, has high compression
Frame rate low resolution SD coding is reduced, reduces rate SD version of code K420 to generate the high compression of input picture.
It will thus be appreciated that the control of space and/or temporal resolution can be used to generate the data of different data rate
Stream, the control of other encoders setting of such as data compression level etc can also by individually or in addition to space and/or
It is used other than the control of temporal resolution, corresponds to the scene parts with one or more desired data rates to generate
Data flow.
Fig. 5 shows the coded portion 500 stored for being divided into the input stereo scene of 3 sample portions.
The coded portion stored can store in content delivery system 104, such as be used as data/information in memory.It is three-dimensional
The coded portion 500 of scene stored includes 3 different sets of coded portion, wherein each part corresponds to different fields
Scene area and each set include multiple and different version of code of corresponding scene parts.Each version of code is encoded video number
According to version and therefore indicate the multiple frames being encoded.It should be appreciated that each version of code 510,512,516 corresponds to
The video of multiple periods, and when fluidisation, the part (such as frame) corresponding to the period being just played will be used to transmit mesh
's.
As above for shown by Fig. 4 and discussing, each scene parts (such as forward and backward scene parts) can use more
A different encoder is to encoding, to generate K different editions of same scene part.Corresponding to given input scene
The output of each encoder is grouped together and is stored as set.The first set of coding scene parts 502 corresponds to
Preceding 180 degree scene parts, and the version of code 2 of the version of code 1 510 including preceding 180 degree scene, preceding 180 degree scene
512 ..., and preceding 180 degree scene version of code K 516.The second set for encoding scene parts 504 corresponds to scene portion
Divide 2, such as 90 degree of left back scene parts, and including 1 520,90 degree of left back scenes of the version of code of 90 degree of left back scene parts
Partial version of code 2 522 ..., and 90 degree of left back scene parts version of code K 526.Similarly, scene portion is encoded
506 third set is divided to correspond to scene parts after scene parts 3, such as 90 degree of right sides, and including scene parts behind 90 degree of right sides
1530,90 degree of version of code it is right after scene parts version of code 2 332 ..., and 90 degree right sides after scene parts coding
Version K 536.
A variety of different stored coded portions of 360 degree of scenes can be used to generate for being sent to consumer's playback
The various different bit rate flows of equipment.
Fig. 6 is the flow chart 600 the step of showing the illustrative methods of offer picture material accoding to exemplary embodiment.
In some embodiments, the method for flow chart 600 is realized using capture systems shown in Fig. 1.
This method starts in step 602, for example, delivery system is powered and initializes.This method is before starting step 602
Enter step 604.In step 604, content delivery system 104 (such as server 114 in system 104) is received to content
Request, such as to the request of the program of previous coding alternatively, in some cases, being encoded and being fluidized by real-time or near real-time
Live events, for example, when event still carry out when.
In response to the request, in step 606, server 114 determines the data rate that can be used for delivering.Data rate can
With according to include instruction support data rate request in information and/or according to such as instruction can be used for content delivery
It is determined to the other information of the network information of maximum bandwidth for requesting equipment etc.It should be appreciated that available data rate can
To change and can change during the period that content is fluidized dependent on network load.Change can be by user equipment report
It accuses or is detected from message or signal, wherein the message or signal designation grouping are dropped or are delayed over instruction network and be difficult to
Support that the expected time of data rate currently in use measures and currently available data rate is available original lower than being confirmed as
Data rate.
Operation proceeds to step 608 from step 606, there, the user being initialised since it to the request of content
The current head position (for example, current head position in request) of equipment will be 0 degree of position.In some embodiments, 0 degree
Or front view position can be signaled using playback apparatus by user and be reinitialized to reinitialize.With when
Between passage, variation of the head position and/or user's head position of user for example relative to original header position be reported to
Content delivery system 104, and the position as will be discussed below updated is used to make content delivery decision.
Operation proceeds to step 610 from step 608, wherein corresponding to the part quilt of 360 degree of scenes of requested content
It sends, to initialize playback apparatus.In at least some embodiments, initialization is related to sending complete 360 degree of collection of contextual data
It closes, such as when 360 degree of scenes are divided into N number of part is N number of part.
As being initialized in step 610 as a result, playback apparatus, which will have, corresponds to each of 360 degree of possible viewing areas
The contextual data of different piece.Thus, if the unexpected back track of the user of playback apparatus, even if not being and user exists
The part watched before rotation head is equally newest, and also at least some data will can be used for showing to user.
Operation proceeds to step 612 and step 622 from step 610.Step 622 corresponds to global scene more new route, it
It is used to assure that playback apparatus each global update cycle receives the more new version of entire 360 degree of scenes at least once.In step
It has been initialised in 610, global renewal process is delayed by the scheduled period in waiting step 622.Then in step 624
In, execute 360 degree of scene updates.Which scene dotted arrow 613 indicates during the auxiliary period for corresponding to step 622 about
Part is sent to the transmission of the information of playback apparatus.In step 624, entire 360 degree of scenes can be sent.But
In some embodiments, not all part is sent all in step 624.In some embodiments, during waiting period 622
The part for the scene being updated is omitted from the update executed in step 624, because they are in the head based on user
Refreshed during the normal fluid mapper process of at least some parts of portion position transmission scene.
Operation is advanced back waiting step 622 from step 624, there, executes waiting before next overall situation update.It should
Understand, by adjusting the waiting period used in step 622, can support different global refresh rates.In some implementations
In example, when content server selects waiting period and the thus reference of the selection overall situation based on the type for the scene content being provided
Section.It is the reason of wherein main actions are in face forward region and refresh first is that the possibility of outdoor lighting condition changed
In the case where sport event, waiting period can be relatively long, such as about one minute or a few minutes.Movement in wherein crowd
With activity may be with the Springsteen that different songs are frequently changed by performance in the case where, global refresh rate can be with, and
And be sometimes exactly to be higher than sport event, user may want to turn round and see crowd reaction and watch in addition to obtaining on foreground
Region is occurring also to want to obtain the feeling what has occurred in crowd except what feeling.
In some embodiments, global to change with reference to the period as the function of the part for the presentation being just fluidized.For example,
During the Competition Segment of sport event, global refresh rate can be relatively low, but wherein in event or via playback
During time-out or intermission that the people of equipment viewing event is more likely to rotate his or her head from main region forward or
Between later period at touchdown moment, global reference rate can (and in some embodiments) made in step 622 by reducing
Waiting (such as refresh period control) and increase.
Although describing global refresh process by reference to step 622 and 624, the normal confession of the part of scene is described
It gives.As it should be appreciated, scene or the normal refresh of scene parts are by the view in the case where data rate allows to be supported
At least one portion occurs for frequency frame rate.Accordingly, with respect at least one frame part, (such as his/her head is indicated as face
To part) will be provided with full video streaming frame rate, it is assumed that available data rate is enough.
In step 612, field to be offered is selected based on the indicated head position (for example, viewing angle) of user
Scape part.Selected part is for example periodically sent (such as fluidisation) to playback apparatus.In some embodiments, corresponding
The rate that data in these parts are fluidized depends on video frame rate.For example, at least one selected part will be to be propped up
The full motion held is fluidized.Although having selected at least one scene parts in step 612, multiple scenes are generally selected
Part, for example, user just towards scene parts and next nearest scene parts.If available data rate is enough
Support the communication of multiple frame parts, then additional scene parts can also be selected and be provided.
After selecting the scene parts to be fluidized in step 612, operation proceeds to step 614, wherein for example based on can
The viewing location of data rate and user select the version of code of selected stream part.For example, such as by the head of current report
The full rate high-resolution version for the scene parts that user indicated by portion part faces can with and will usually be fluidized.Working as
One or more scene parts on the fore head position left side and/or the right can be chosen so as to low resolution, speed of lower time
Rate is fluidized using another coding method of amount of bandwidth needed for reducing the scene areas for sending and not watched currently.Phase
The selection of the version of code of adjacent scene parts will be depended in the quality version for the scene parts currently watched by transmission
The amount of bandwidth left afterwards.Although the scene parts that do not watched currently can be used as low resolution version of code or as in frame
Between sent with the version of code of larger time gap, but if there is enough bandwidth are available, then can periodically or
Continually send full resolution quality version.
In step 616, the selected version of code of selected scene parts is sent to the playback apparatus of request content.
Therefore, in step 616, the encoded content corresponding to one or more parts, such as the stereopsis corresponding to multiple successive frames
Frequency content, is streamed to playback apparatus.
Operation proceeds to step 618 from step 616, wherein receiving the information of the current head position of instruction user.It is this
Information can be periodically and/or in response to the variation of detection head position and send from playback apparatus.In addition to head position
Except variation, what content the variation of available data rate, which also will affect, is fluidized.Operation proceeds to step 620 from step 618,
Wherein determine the current data rate that can be used for that content delivery is carried out to playback apparatus.Therefore, content delivery system can be examined
Survey the variation that can be used for supporting the amount of bandwidth of the fluidisation to request equipment.
Operation from step 620 proceeds to step 612, wherein fluidisation continues, until content is delivered completely, for example, program or
Event terminates, or until from the playback apparatus of request content receive instruction session signal to be terminated or fail from
Playback apparatus receives expected signal, such as detects the head position that instruction playback apparatus is no longer communicated with content server 114
Set update.
According to the contextual data delivered in the above described manner, playback apparatus will at least have can be used for user's quick rotation he
Or some data corresponding to each scene parts shown in the case where her head.It should be appreciated that user is seldom very
Head is fully turned in the short time, because this is uncomfortable variation of viewing location for many people.Thus, although
The height for the scene parts that complete 360 degree of scenes may not always be sent, but most possibly be watched at any given time
Quality version can be fluidized and make user available.
Content delivery system 104 can support a large amount of concurrent user because cataloged procedure allow scene N number of part with
Different modes are sent to different user and processed, without for each individually user individually encoded content.Therefore,
Although multiple parallel encoders can be used to support real-time coding, to allow the real-time or near real-time stream of sport or other events
Change, but the quantity of used encoder tends to the quantity for the playback apparatus being streamed to much smaller than content.
Although the part of content is described as the part corresponding to 360 degree of views, but it is to be understood that scene can (
In some embodiments really the flat version in the also space with vertical dimensions) is represented.Playback apparatus can utilize 3D environment
Model (such as space) comes mapping scenarios part, and adjusts vertical viewing location.Therefore, discussed herein 360 degree
Refer to just look like that user changes it while keeping his eye level to the left or to the right relative to horizontal head position
Viewing angle.
Fig. 7 shows feature according to the present invention, code capacity including that can be used to coding and streamable content example
Property content delivery system 700.
The system can be used to execute the coding of feature according to the present invention, storage, and transmission and/or content output.
In some embodiments, system 700 or in which element execute correspond to Fig. 6 shown in process operation.Content delivery system
700 are used as the system 104 of Fig. 1.Although system shown in fig. 7 is used for coding, processing and the fluidisation of content, answer
Work as understanding, system 700 can also include decoding and the energy that the image data for being handled and/or being encoded for example is shown to operator
Power.
System 700 includes display 702, input equipment 704, input/output (I/O) interface 706, processor 708, network
Interface 710 and memory 712.The various parts of system 700 are coupled via bus 709, this allows data in system 700
Component between transmit.
Memory 712 includes various modules, such as routine, and when it is executed by processor 708, control system 700 is realized
Segmentation, coding, storage and fluidisation/transmission and/or output operation according to the present invention.
Memory 712 includes that various modules, such as routine control computer system when it is executed by processor 708
700 realizations immersion stereo video acquisition according to the present invention, coding, storage, and transmission and/or output method.Memory
712 include control routine 714, segmentation module 716, encoder 718, fluidisation controller 720, received input picture 732 (such as
360 degree of three-dimensional video-frequencies of scene), coding scene parts 734 and timing information 736.In some embodiments, module quilt
It is embodied as software module.In other embodiments, module is implemented within hardware, such as is embodied as individual circuit, wherein often
A module is implemented as the circuit for the corresponding function of execution module.There are also in other embodiments, module is to utilize software
Combination with hardware is realized.
Control routine 714 includes equipment control routine and Communications routines, with the operation of control system 700.Divide module 716
It is configured as feature according to the present invention and 360 degree of versions of solid of the scene received is divided into N number of scene parts.
Encoder 718 can (and in some embodiments really) include being configured as feature coding according to the present invention
Multiple encoders of the picture material received, wherein the 360 of picture material such as scene and/or one or more scene parts
Spend version.In some embodiments, encoder includes multiple encoders, wherein each encoder is configured as encoded stereoscopic scene
And/or the scene parts of segmentation, to support given bit rate flow.Therefore, in some embodiments, each scene parts can be with
It is encoded using multiple encoders, to support to be used for multiple and different bit rate flows of each scene.The output of encoder 718 is
The scene parts 734 of coding, are stored in memory, for being streamed to consumer device, such as playback apparatus.Coding
Content one or more different equipment can be streamed to via network interface 710.
Fluidisation controller 720 is configured as the fluidisation of control encoded content, for will for example encode through communication network 105
Picture material consign to one or more consumer devices.In various embodiments, each step of flow chart 600 is by fluidizing
The element of controller 720 is realized.Fluidize controller 720 include request processing module 722, it is data rate determination module 724, current
Head position determining module 726, selecting module 728 and fluidisation control module 730.Request processing module 722 is configured as handling
From the received request to image content of consumer's playback apparatus.It in various embodiments, is via network to the request of content
Receiver in interface 710 is received.It in some embodiments, include the identity of instruction request playback apparatus to the request of content
Information.It in some embodiments, may include the data rate supported by consumer's playback apparatus, user to the request of content
Current head position, such as the position of head-mounted display.Request processing module 722 handles the request received and will retrieval
To information be supplied to fluidisation controller 720 other elements, to take further movement.Although can be with to the request of content
Including data-rate information and current head location information, but in various embodiments, the data supported by playback apparatus are fast
Rate can between system 700 and playback apparatus network test and the exchange of the other network informations determine.
Data rate determination module 724 be configured to determine that can be used to for image content being streamed to consumer device can
With data rate, for example, due to supporting multiple coding scene parts, content delivery system 700 can be supported with multiple numbers
According to rate to consumer device streamable content.Data rate determination module 724 is additionally configured to determine in by requesting from system 700
The data rate that the playback apparatus of appearance is supported.In some embodiments, data rate determination module 724 is configured as based on network
Measurement determines the available data rate delivered for picture material.
Current head position determination module 726 is configured as determining that user's is current according to from the received information of playback apparatus
Viewing angle and/or current head position, such as the position of head-mounted display.In some embodiments, the playback apparatus period
Property to system 700 send current head location information, wherein current head position determination module 726 receives and processes the letter
Breath, with the currently viewing angle of determination and/or current head position.
Selecting module 728 is configured as determining 360 degree of scenes based on currently viewing angle/head position information of user
Which to be partially streamed to playback apparatus.Selecting module 728 is additionally configured to select to determine based on available data rate
Scene parts version of code, to support the fluidisation of content.
Fluidisation control module 730 is configured as feature according to the present invention in the data rate control image of various supports
The fluidisation of appearance, wherein multiple portions of such as 360 degree stereo scenes of picture material.In some embodiments, control module is fluidized
730 be configured as control 360 degree stereo scenes N number of part to request content playback apparatus fluidisation, with initialize play back
Scene memory in equipment.In various embodiments, fluidisation control module 730 is configured as example with determining velocity periods
Property send determined by scene parts selected version of code.In some embodiments, fluidisation control module 730 is also configured
To send 360 degree of scene updates to playback apparatus according to time interval (such as once every minute).In some embodiments, it sends
360 degree of scene updates include the N number of scene parts or N-X scene parts for sending complete 360 degree of stereo scenes, and wherein N has been
The sum and X for the part that whole 360 degree of stereo scenes are divided into indicate the selected field for being sent to playback apparatus recently
Scape part.In some embodiments, fluidisation control module 730 waits the predetermined time after initially sending N number of scene parts, uses
It is initialized in front of 360 degree of scene updates of transmission.In some embodiments, the timing of the transmission of 360 degree of scene updates is controlled
Information is included in timing information 736.In some embodiments, fluidisation control module 730 is additionally configured to identify and refresh
Interim is not yet sent to the scene parts of playback apparatus；And playback is not sent to during being sent in refresh interval and is set
The more new version of the standby scene parts identified.
In various embodiments, fluidisation control module 730 is configured as periodically transmitting to playback apparatus at least enough
N number of part of quantity, to allow playback apparatus to refresh 360 degree of the scene completely at least once during each refresh cycle
Version.
Fig. 8 shows the playback system 800 realized according to the present invention, can be used to receive, decode, store and show from
The received image content of content delivery system shown in such as Fig. 1 and 7.System 800 can be implemented as including display 802
Single playback apparatus 800', or it is embodied as such as being coupled to the external display of computer system 800' (for example, wear-type is aobvious
Show device 805) element combination.
In at least some embodiments, playback system 800 includes 3D head-mounted display.Head-mounted display can use
It may include the OCULUS RIFT of head-mounted display 805TMVR (virtual reality) earphone is realized.Other head-mounted displays
It can be used as.In some embodiments, the wear-type helmet or other headset equipments, wherein one or more display screens by with
To show left eye from content to user and right eye.By the way that different images is shown to left eye and right eye on single screen, wherein
Headset equipment is configured as the different piece of single screen being exposed to different eyes, and individual monitor can be used to show
Show the left eye and eye image that will be perceived respectively by the left eye of viewer and right eye.In some embodiments, mobile phone screen quilt
Display as head-mounted display apparatus.In at least some such embodiments, mobile phone be inserted into headset equipment and
Mobile phone is used to display image.
3D figure of the coded image data and generation that there is playback system 800 decoding to receive for being shown to consumer
As the ability of content, wherein coded image data such as left eye and eye image and/or corresponding to environment or the different portions of scene
The monophonic (single image) divided, wherein display is for example to be rendered by the different left eyes and right-eye view that perceive user
And it is shown as 3D rendering.In some embodiments, playback system 800 is located at consumer guard station position, such as family or office, but
Image capture place can also be located at.System 800 can execute signal according to the present invention receive, decoding, display and/or other
Operation.
System 800 includes display 802, display device interfaces 803, input equipment 804, input/output (I/O) interface
806, processor 808, network interface 810 and memory 812.The various parts 800 of system are via permission data in system 800
It the bus 809 that is communicated between component and/or is coupled by other connections or by wireless interface.Although in some implementations
Display 802 is included as optional element in example, and as shown in using dotted line frame, but in some embodiments, outside is aobvious
Show that equipment 805 (such as wear-type stereoscopic display device) can be coupled to playback apparatus via display device interfaces 803.
For example, being used as processor 808 in cell phone processor and mobile phone generates in headset equipment and show image
In the case where, system may include processor 808, display 802 and memory 812, a part as headset equipment.Place
Reason device 808, display 802 and memory 812 can be a part of mobile phone.In the other embodiments of system 800, processing
Device 808 can be a part of the game system of such as XBOX or PS4, and wherein display 805 is mounted in headset equipment simultaneously
And it is coupled to game system.Processor 808 or memory 812 whether be located at wear be not in overhead equipment it is crucial and
And although as it would be appreciated, in some cases, co-located in headwear (headgear) processor can be it is convenient,
But from the perspective of power, heat and weight, at least some cases, it may be desirable to make processor 808 and memory
812 are coupled to the headwear including display.
Although various embodiments contemplate head-mounted display 805 or 802, this method and device can also with can prop up
The non-head-mounted display for holding 3D rendering is used together.Thus, although system 800 includes that wear-type is shown in many examples
Device, but it can also be realized using non-head-mounted display.
Memory 812 includes that various modules, such as routine control playback apparatus 800 when it is executed by processor 808
Execute decoding and output operation according to the present invention.Memory 812 includes control routine 814, the request generation module to content
816, head position and/or viewing angle determining module 818, decoder module 820, also referred to as 3D rendering generation module is vertical
Body image rendering module 822, and the coded image content 824 including receiving, decoding 826,360 degree of decoded fields of picture material
The data/information of scape buffer 828 and the stereo content 830 generated.
Control routine 814 includes equipment control routine and Communications routines, to control the operation of equipment 800.Request generates mould
Block 816 is configurable to generate the request to content, to be sent to content delivery system for providing content.In various embodiments
In, the request to content is sent via network interface 810.Head position and/or viewing angle determining module 818 are configured
Currently viewing angle and/or current head position for determining user, for example, the position of head-mounted display, and will be determining
Position and/or viewing angle information are reported to content delivery system 700.In some embodiments, playback apparatus 800 is periodically
Current head location information is sent to system 700.
Decoder module 820 is configured as decoding from the received coded image content 824 of content delivery system 700, to produce
Raw decoded image data 826.Decoded image data 826 may include decoded stereo scene and/or decoded scene portion
Point.
The feature according to the present invention of 3D rendering rendering module 822, such as 3D figure is generated using decoded picture material 826
Picture, such as the left eye and eye image that are shown in a manner of it will be perceived as 3D rendering, to be used in display 802 and/or show
Show and is shown in equipment 805 to user.Stereoscopic image content 830 generated is the output of 3D rendering generation module 822.Therefore,
3D rendering content 830 is rendered into display by rendering module 822.In some embodiments, display equipment 805 can be such as
A part of the 3D display equipment of Oculus Rift.The operator of playback apparatus 800 can control one via input equipment 804
A or multiple parameters and/or selection operation to be performed, such as selection display 3D scene.
Fig. 9 shows the figure for indicating exemplary camera component 900 (also sometimes referred to as camera equipment or camera array),
With being mounted on 3 cameras of 3 different installation sites to 902,904,906 and can be used for the school of calibration camera component 900
Quasi- target 915.According to some embodiments of the present invention, camera equipment 900 is used to capture picture material.In some embodiments
In, camera equipment 900 is used as the image capture apparatus 102 of Fig. 1.Camera equipment 900 includes that camera is maintained at indicating positions
Support construction (shown in Figure 11), 3 pair 902,904,906 stereoscopic cameras (901,903), (905,907), (909,911) it is total
Totally 6 cameras.Support construction is included herein the pedestal of also referred to as mounting plate (referring to element 1120 shown in Figure 11)
1120, the base supports camera and plate for being equipped with camera thereon can be fixed to the pedestal.Support construction can be by plastics, gold
Belong to or the composite material of such as graphite or glass fibre is made, and the line by constituting triangle indicates, is also used to show
Spacing and relationship between camera out.The central point of dotted line intersection indicates central node, some but be not necessarily all implementations
In example, camera can be rotated to 902,904,906 around the central node.In some embodiments, central node corresponds to example
Such as the steel pole or thread center's installation part of tripod mount, the camera supporting framework 912 indicated by triangle line can surround it
Rotation.Braced frame can be the plastic shell either tripod structure for being wherein equipped with camera.
In Fig. 9, each pair of camera 902,904,906 corresponds to different cameras to position.First camera corresponds to 902
0 degree of position forward.This position generally corresponds to interested home court scene area, for example, just carrying out sports tournament on it
Some the other region that may occur of place, stage or main actions.Second camera corresponds to 120 degree of phase seats in the plane to 904
It sets and is used to viewing areas behind the capture right side.Third camera corresponds to 240 degree of positions (relative to 0 degree of position) and left back to 906
Viewing areas.Note that three camera positions separate 120 degree.Each camera viewing location includes a camera in Fig. 9 embodiment
It is right, wherein each camera is to including left camera and right camera for capturing image.Left camera capture is sometimes referred to as left eye figure
The content of picture, right camera capture are sometimes referred to as the content of eye image.Image can be in the capture of one or more times
A part of view sequence or static image.In general, the preceding camera position at least corresponding to camera to 902 will be imaged with high quality
Machine filling.Other camera positions can use the single camera or high quality for being used to capture static or monochrome image (mono image)
Video camera, lower quality video camera are filled.In some embodiments, second and third camera embodiment holding be not filled simultaneously
And the support plate for being equipped with camera thereon is rotated, so that first camera be allowed to correspond to all three phase seats in the plane to 902 captures
It sets but in the image of different time.In some such embodiments, left and right rear images are captured and stored by relatively early, then
The video of forward direction camera position is captured during event.Captured image can still carried out by real time, such as in event
While, it is encoded and is streamed to one or more playback apparatus.
First camera shown in Fig. 9 is to including left camera 901 and right camera 903.Left camera 901, which has, is fixed to first
First lens subassembly 920 of camera, right camera 903 have the second lens subassembly 920' fixed to right camera 903.Lens subassembly
920,920' includes the lens for allowing to capture wide-angle view.In some embodiments, each lens subassembly 920,920' include fish
Glasses head.Therefore, each of camera 902,903 can capture the 180 degree visual field or about 180 degree.In some embodiments,
It is captured less than 180 degree, but in some embodiments, from adjacent camera to still having at least one in captured image
A little overlappings.In the embodiment in fig. 9, camera is in first (0 degree), second (120 degree) and third (240 degree) camera installation site
Each of place's positioning, wherein at least 120 degree or more of each pair of capturing ambient, but in many cases, each camera
180 degree to capturing ambient or about 180 degree.
2nd 904 and the 3rd 906 camera pair and first camera are same or similar to 902, but relative to preceding 0 degree of position position
In 120 and 240 degree of camera installation sites.Second camera includes left camera 905 and left lens assembly 922 and right camera to 904
907 and right camera lens assembly 922'.Third camera includes left camera 909 and left lens assembly 924 and right camera to 906
911 and right camera lens assembly 924'.
In Fig. 9, D indicate the first stereoscopic camera to 901,903 axle base.In Fig. 9, example D is 117mm, this
The distance between the pupil of the left eye of ordinary people and right eye is same or similar.Dotted line 950 in Fig. 9 is depicted from panorama array
Central point to right camera lens 920' entrance pupil distance (offset of aka node).In a reality for corresponding to Fig. 9 example
It applies in example, is 315mm by the distance that appended drawing reference 950 indicates, but other distances are possible.
In a particular embodiment, the area of coverage (footprint) of camera equipment 900 is relatively small, horizontal area
For 640mm2Or it is smaller.This small size allows camera to equip to be placed in spectators, for example, bean vermicelli or spectators usually can position
In or positioning seated position at.Therefore, in some embodiments, camera equipment is placed in audience area, to allow to see
The person of seeing has the feeling as a member in the spectators for it is expected this effect.In some embodiments, the area of coverage corresponds to one
It include that the support construction of centre post is installed to the size that itself or support tower navigate to its pedestal in a little embodiments.It should manage
Solution, in some embodiments, camera equipment can be around the central point rotations of the pedestal of the central point corresponded between 3 pairs of cameras
Turn.In other embodiments, camera is fixed and does not rotate around the center of camera array.
Camera equipment, which can capture, to be relatively close to and remote object.In a particular embodiment, camera array
Minimum image-forming range is 649mm, but other distances are also possible and this distance is not crucial.
From the center of photomoduel to first and the crosspoint 951 of the view of third camera part distance indicate can by with
In calibration by the first and second cameras to the example calibration distance of captured image.It should be pointed out that target 915 can be placed in away from
Be located at or just over maximum distortion region camera pair known distance at.Calibrating target includes known fixed calibration mould
Formula.Calibration target can be and be used to calibrate by the size of the camera captured image of camera pair.This calibration is possible
, because the camera of the image relative to capture calibration target 915, the size and position for calibrating target are known.
Figure 10 is more detailed Figure 100 0 of camera array shown in Fig. 9.Although camera equipment is shown to have 6 again
A camera, but in some embodiments, camera equipment are filled that there are two cameras, such as camera is to 902.As shown,
There are 120 degree of intervals between installation site for each camera.Consider for example, if the center between each camera pair corresponds to
The direction of camera installation site, then in this case, first camera installation site corresponds to 0 degree, second camera installation position
It sets corresponding to 120 degree and third camera installation site corresponds to 240 degree.Therefore, each camera installation site separates 120 degree.
If extended through, each camera is extended the center line at 902,904,906 center and the angle between line is measured,
Then it can be seen that this point.
In the example of Figure 10, camera can be equipped around camera (in some embodiments really) to 902,904,906
Central point rotation, thus allow different time capture different views, without change camera equipment pedestal position.That is,
Camera can rotate around the center support of equipment and be allowed to capture different scenes in different time, to allow only to fill out at it
Equipment shown in Figure 10 is utilized to carry out 360 degree of scene captures while camera there are two filling.In view of the cost of stereoscopic camera,
This configuration is especially desired to from the perspective of cost, and is very suitable for many applications, it may be desirable to be shown from identical
Viewpoint but the time that can occur from the preceding scene of the main actions during including sport event or other events it is different when
Between capture background.Consider that object can be placed in behind camera for example during event, then not showing during main event
Show that it will be preferred.In this case, rear image can (sometimes) be captured before main event and together with taking charge
The real-time captured image of part can be used together, to provide 360 degree of set of image data.
Figure 11 shows the detailed diagram for the exemplary camera equipment 1100 realized according to an exemplary embodiment.Such as from
Figure 11 is it should be understood that camera rack 1100 includes 3 pairs of cameras 1102,1104 and 1106, some but be not all of embodiment
In, they are stereoscopic cameras.In some embodiments, each camera is to including two cameras.Camera is to 1102,1104 and 1106
It is same or similar to 902,904,906 with the camera that is discussed above for Fig. 9-10.In some embodiments, camera to 1102,
1104 and 1106 are mounted in the support construction 1120 of camera equipment 1100.In some embodiments, three pairs of camera (six phases
Machine) 1102,1104 and 1106 mounting plate is mounted in support construction 1120 via corresponding camera.Support construction 1120 includes
For installing three installation sites of stereoscopic camera pair, wherein each installation site corresponds to 120 degree of different viewing areas.?
In the illustrated embodiment of Figure 11, first pair of stereoscopic camera 1102 is mounted on first in three installation sites, such as positive position
It sets, and corresponds to 120 degree of front viewing areas.Second pair of stereoscopic camera 1104 is mounted on second in three installation sites
It is a, such as 120 degree of background positions are rotated clockwise relative to front position, and correspond to 120 degree of different viewing areas.
Third is mounted on the third in three installation sites to stereoscopic camera 1106, such as rotates clockwise relative to front position
240 degree of background positions, and correspond to another 120 degree viewing areas.Although three cameras on camera equipment 1100
Installation site offsets relative to each other 120 degree, but in some embodiments, each camera installed on the camera frame has
The about visual field of 180 degree.In some embodiments, this widened visual field be by camera apparatus using fish eye lens come
It realizes.
Although as it can be seen that still showing for installing camera to 1102 in not every mounting plate all attached drawings shown in
Camera installing plate 1110.Mounting plate for camera has the slit for screw, narrow in support base 1120 to pass through
Slot simultaneously enters in the threaded hole in mounting plate from bottom.This allows then tighten screw from the close screw in bottom by unclamping
It adjusts the mounting plate for this pair, camera is fixed to support construction to mounting plate.Each camera position can also be adjusted
It is whole, then it is locked after the adjustment.In some embodiments, each camera can be adjusted/be fixed to mounting plate from top, and
And camera installing plate can be adjusted/fix from bottom.
In various embodiments, camera equipment 1100 includes pedestal 1122, and support construction 1120 is rotatably mounted to base
Seat 1122.Therefore, in various embodiments, the photomoduel in support construction 1120 can pass around the axis rotation of base central
Three-sixth turn.In some embodiments, pedestal 1122 can be a part of tripod or another installation equipment.Support construction can
It is made with the composite material by plastics, metal or such as graphite or glass fibre.In some embodiments, camera is to can enclose
Central point around sometimes referred to as central node rotates.
In addition to the aforementioned components, in some embodiments, camera equipment 1100 further include two simulation lugs 1130,
1132.These simulation lugs 1130,1132 imitate human ear and in some embodiments by with the silicon tree of human ear shape molding
Rouge is made.Simulating lug 1130,1132 includes microphone, and two of them lug is separated from each other equal or approximately equal to common
The distance being spaced between the human ear of people.The microphone being mounted in simulation lug 1130,1132 is mounted on Front camera pair
On 1102, but it can be used as and be alternatively mounted on support construction (such as platform 1120).With with human ear perpendicular on the number of people
Eyes front surface positioning similar fashion, simulation lug 1130,1132 perpendicular to camera to 1102 front surface position.
Simulation lug 1130,1132 sides hole serve as the audio gateway hole of simulation lug side, wherein simulation lug and hole
Audio, is directed toward the microphone being mounted in each simulation lug by combination operation, and audio sound is directed toward by extraordinary image human ear
Including the eardrum in human ear.Left and right simulates the microphone in lug 1130,1132 and provides stereo capture, is similar to place
If will be perceived via the left and right ear of people when the people that camera equips 1100 positions is located at the position of camera equipment vertical
Body sound.
Although Figure 11 shows tool there are three a kind of configuration of the exemplary camera of stereoscopic camera pair equipment 1100, answer
Work as understanding, other modifications are possible and in range.For example, in one implementation, camera equipment 1100 includes single phase
Machine pair, for example, can be around a pair of of stereoscopic camera of the central point rotation of camera equipment, to allow to capture in different time
The different piece of 360 degree of scenes.Therefore, single camera is to the central supported that may be mounted in support construction and surround equipment
It rotates and is allowed to capture different scenes in different time, to allow 360 degree of scene captures.
Figure 12 shows exemplary 360 degree of scene environments 1200, such as 360 degree of scene areas, can be divided into correspondence
In different viewing areas/parts of the different cameral position phase of the respective camera of the different piece of 360 degree of scenes of capture.Shown
Example in, 360 degree of scene areas 1200 be divided into correspond to by three different cameral/cameras to (for example, being such as mounted on
Camera equipment 1100 on and the positioning as shown in 9,10 and 11 camera) capture three 180 degree areas three parts.360
0 scale designation in degree scene 1200 is considered the center of scene.In not using fish-eye some embodiments, often
The visual field of a camera is about 120 degree, therefore the scene areas for allowing camera to capture about 120 degree.In such embodiments,
The boundary of 120 degree of different scene parts utilizes the solid black for 3 parts that 360 degree of scenes are divided into each 120 degree in figure
Line is shown.In camera equipped in fish-eye embodiment, the visual field of each camera expands to about 180 degree (± 5 degree), thus
Camera is allowed to capture the scene areas of about 180 degree (± 5 degree).
90 degree of left and right side of first area of 0 scale designation is covered (corresponding to the 180 degree front court scenic spot from 270 to 90
The area 1 in domain) it can be captured by first camera, first camera is for example oriented to capture preceding scene areas, is equipped with permission camera
Equipment has the fish-eye camera in the about 180 degree visual field to 1102.Second area (area 2) corresponds to 180 from 30 to 210
Right scene areas after degree, can be captured by second camera, and second camera is for example oriented capture right back scene areas, matches
Fish-eye camera is had to 1104, third area (area 3) corresponds to the left back scene areas of 180 degree from 150 to 330,
It can be captured by third camera, third camera is for example oriented to capture left back scene areas, equipped with fish-eye camera
To 1106.Legend 1250 includes the information that identification is used to refer to the not different line images on same district boundary, for example, label is in difference
The beginning and end for the scene areas being capped below area.It can be understood from the graph that in three differences covered by different cameral
There is substantive overlapping between scene areas below area.In the example shown in Figure 12, the overlapping between area 1 and area 2 is
60 degree, that is, the scene areas below 30 to 90, the overlapping between area 2 and area 3 are also 60 degree, that is, the scene below 150 to 210
Region, the overlapping between area 3 and area 1 are 60 degree, that is, the scene areas below 270 to 330.Although weighing in an example shown
Folded is 60 degree, but it is to be understood that different variation overlappings are possible.In some embodiments, two different scenes are covered
Overlapping between 30 degree to 60 degree between cover area.
In some embodiments, content provider includes corresponding to by not to consumer's playback apparatus publication (such as multicast)
With the content stream of the content of the different piece of 360 degree of scene areas of camera capture.In some embodiments, correspond to by
The content of multiple versions in the different scenes region differently encoded is supported and/or is preferred by content supplier's multicast
The playback apparatus of particular version can choose content stream appropriate to decode and play back.One side in accordance with some embodiments is returned
The current head position of the present viewing field of equipment tracking instruction user is put, and determines to select to include corresponding to 360 degree of scenes
Which of available content stream of the content of a part in region or multiple available content streams are received for playing back.Example
Such as, if user's head position instruction user is seeing/look straight ahead, the 180 degree of playback apparatus decoding 360 degree of scenes of transmission
The stream of front, but when the head position and viewing angle that detect user have changed, playback apparatus decodes 360 degree of scenes
Region corresponds to the stream for being suitble to the scene parts of currently viewing angle (for example, behind the right side, left back, rear) of user.In some realities
It applies in example, the stream of the content of first part's (for example, preceding 180 degree) including corresponding to 360 degree of scene areas includes by being used to catch
Obtain the scene areas of the left and right camera capture of the Front camera pair of the front of 360 degree of scenes.
Figure 13 shows the example for indicating the different piece of exemplary 360 degree of scene areas of Figure 12, can be by for example
It can be positioned so that the different phases in viewing areas/part for covering exemplary 360 degree of scene areas on exemplary camera equipment 1100
Machine capture.The example of Figure 13 includes the Figure 130 0,1320 and 1350 for showing the different piece of exemplary 360 degree of scene areas.So
And it should be understood that in some embodiments such as from figure, by each in the different piece of the scene of different cameral capture
It is least partially overlapped.Different hatching patterns is used in each in Figure 130 0,1320,1350, is corresponded to not with showing
With the part of the scene areas of camera position.In some embodiments, the different scenes shown in Figure 130 0,1320,1350
It is transmitted via the different content stream for providing the content for corresponding to different view directions part.Although the scene in the example of Figure 13
Part is illustrated as the viewing areas of covering about 180 degree, but scene parts can cover 120 degree and arrive in some embodiments
Between 180 degree.
Figure 130 0 shows the first exemplary scene part 1305 of 360 degree of scenes 1200.First exemplary scene part
1305 correspond to front view direction and cover the viewing areas of the 180 degree or about 180 degree in 360 degree of scene environments.The
One scene parts 1305 can be by for example capturing positioned at the first camera of 0 degree of camera position to 902 or 1102.First exemplary field
Region below scape part 1305 is shown in Figure 130 0 using pattern of diagonal lines.First exemplary scene part 1305 can be by
Transmission corresponds to the first-class transmission of the content frame of first (for example, front) view direction.
Figure 132 0 shows the second exemplary scene part 1307 of 360 degree of scenes 1200, and (it is included in Figure 132 0 and shows
Part 1307' and 1307 " out).Second exemplary scene part 1307 corresponds to right back view direction and covers 360 degree of fields
The 180 degree extended from 30 to 210 in scape environment or about 180 degree viewing areas.Second scene parts 1307 can be by being located at such as
Such as second camera of 120 degree of camera positions shown in Fig. 9-10 is captured to 904.Below the second exemplary scene part 1307
Region shown in Figure 130 0 using horizontal line pattern.Consider Figure 130 0 and 1320.Pay attention in the first and second scene portions
Divide the lap 1308 of the scene areas between 1305 and 1307.Lap 1308 shows to scene areas 1200
One and second the public institute's capturing scenes region of scene parts 1305,1307 a part.In some embodiments, first and
Overlapping between 30 degree to 60 degree between two scene parts 1305,1307 has the variation of plus-minus 2-3 degree.Shown in Figure 13
Example in, lap 1308 is 60 degree, for example, the region from 30 to 90.Therefore, in some embodiments, correspond to by
Different content stream is provided and/or is overlapped by least part of the scene areas of the different view directions of different cameral capture.?
In some other embodiments, it is not overlapped between the scene areas for corresponding to the different view directions captured by different cameral.
In some embodiments, the second exemplary scene part 1307 can be by transmitting the of the content frame for corresponding to the second view direction
Second-rate transmission.
Figure 135 0 shows the third exemplary scene part 1309 of 360 degree of scenes 1200 (including shown in Figure 135 0
Part 1309' and 1309 ").Third exemplary scene part 1309 corresponds to left back view direction and covers 360 degree of scene rings
The 180 degree extended from 150 to 330 in border or about 180 degree viewing areas.Third scene parts 1309 can be by being located at such as Fig. 9-
Such as third camera of 240 degree of camera positions shown in 10 is captured to 906.Region below third exemplary scene part 1309
It is shown in Figure 130 0 using vertical-line pattern.Consider Figure 132 0 and 1350.Pay attention to lap 1310,1310', they are combined
Constitute the overlapping region part between second and third scene parts 1307 and 1309.Below lap 1310,1310' altogether
Same region shows the scene area that is captured public to second and the third scene parts 1307,1309 of scene areas 1200
The part in domain.In some embodiments, between second and third scene parts 1307,1309 overlap 30 degree to 60 degree it
Between, there is the variation of plus-minus 2-3 degree.In the example shown in Figure 13, lap 1310,1310' include about 60 degree together
Covering, for example, the region from 150 to 210.Figure 130 0 and 1350 is further considered now.Pay attention to lap 1312, it refers to
Show the overlapping region part between first and third scene parts 1305 and 1309.In some embodiments, third exemplary field
Scape part 1309 can be transmitted by the third stream for transmitting the content frame for corresponding to third view direction.
It is of the invention some in order to understand although showing an example as shown in a part of Figure 13 example
Aspect, but it is to be understood that other modifications are possible and scope of the present disclosure interior.
Combined Figure 14 including Figure 14 A and 14B is to show operation playback system according to an exemplary embodiment of the present invention
Illustrative methods the step of flow chart 1400.The system can be playback system 800 shown in fig. 8 either the application
Shown in any other attached drawing playback system.
Illustrative methods start from step 1402, wherein playback apparatus (such as Figure 19 playback apparatus 1900 or it is any its
The playback apparatus of its attached drawing) it is turned on and initializes.For discussion purposes, consider that playback system includes being coupled to wear-type to show
Show that the computer system 1900' of equipment 1905, head-mounted display apparatus 1905 include the display that picture material is presented on it,
For example, different images is presented to the left eye of user and right eye in the case where stereo content.Although computer system 1900' quilt
It is shown as including the outside of the headset equipment of display, but computer system 1900' can be incorporated into head-mounted display
In, rather than outside it.
Operation proceeds to step 1404 from step 1402 is started.In step 1404, playback system 1900 is received about more
The information of a content stream and/or initialization data, such as a part as program guide.The received information of institute can be Figure 18
Shown in type and including indicating which content stream is or by available information, together with the letter that can be used to receive stream
Breath, such as multicast group identifier or can be used to request content or be tuned to content other identifiers.For example, being closed with content stream
The multicast address of connection may include that in the received information of institute or can be used to ask when content is provided via switched digital video
It asks in the program identifier of content.In the case of broadcast content, the received information of institute can (and sometimes really) include
Indicate playback apparatus in order to receive particular content stream and should be tuned to channel and/or frequency tuning information.
Received information may include the information for one or more programs in step 1404.For given program,
Such as sport event, concert etc., different stream can be used for providing at the environment corresponding to content and correspond to relative to camera
Position is in the content of different view directions.Camera position corresponds to the viewing location during playback.Therefore, it is used during playback
The viewing angle at family is related to the part of environment indicated in the content that can be received.The different piece of environment can be in difference
Stream in transmitted.For each part of environment, such as the part of 360 degree of environment, one corresponding to different data rate
Or multiple streams can be listed in provided information.The top and bottom of environment can also be provided.In some embodiments,
The content of each stream is stereo content, wherein different information provides for left eye and eye image, to allow to show to user
Different image, to provide desired 3D effect.In some embodiments, the top and bottom of spherical environment are as monochrome image
It is provided, wherein left eye and right-eye view are identical, therefore only need provide an image rather than two images.
The given finite bandwidth that can be used for flow content, the information of the data rate about program, the part of environment and stream
Can (and in some embodiments really) be played system to be used to be prioritized which stream to be received.It is received in given time
The priorization and selection of which stream can (and in some embodiments really) head position and/or user based on user work as
Preceding or past end rotation direction.
In the case where the fluidisation of stereo content, limited bandwidth and/or data constraint are given, to wanting received appropriate stream
Selection the satisfactory viewing experience with high quality is important.It is received in step 1404 can be with Figure 18 institute
The same or similar information of the information shown (such as stream information 1405) is stored in memory and be used for will be in specific time
One or more selections flowed that point is received, and be used to start the delivery of selected stream, such as be chosen by being added to correspond to
The multicast group of stream, be tuned to the channel of selected stream, and/or the phase by be provided to network equipment instruction to playback apparatus are provided
The delivery for flowing through the switched digital video channel that it is transmitted and carrying out requested stream hoped.
Operation proceeds to step 1406 from step 1404.In step 1406, during initialization, detection user works as front
Portion position.User knows that the head position detected during initial phase will be assumed to for eyes front position, therefore in step
His head is usually maintained into comfort level towards front position during rapid 1406.
Operation proceeds to step 1408 from step 1406.In step 1408, what is detected in step 1406 uses account
Portion position 1407 is considered as (0 degree) environment viewing location forward, and when his/her head of user is in this position Shi Suoxian
The view location shown will correspond to 0 degree of environment position, that is, be then encoded and by being used to capture including in the spy for corresponding to environment
Determine the forward location of the camera capture of the image in the content stream of part.In the case where sport event, this position usually will
Corresponding to the main actions region in environment, for example, being stage in the case where one or more stream corresponds to concert, flowing
It is the center in place in the case where corresponding to sport event.Therefore, in step 1408, the viewing location of user will be arranged to
It is interpreted zero degree viewing location, such as forward direction/forward portion of scene areas.It should be pointed out that the part of 360 degree of views is corresponding
In horizontal viewing location, if user rotates his/her head, different piece is visible.By moving up or down
User's head, user can see in combination sky portion and/or above ground portion individually or with one or more of the other part.
Due to assuming that home court scene area is divided into multiple portions along 360 degree rotation, therefore right in the case where horizontal head position
More bandwidth should be commonly accorded in the stream of these parts, and top/bottom scene parts can use still image or not frequently
The image of numerous variation is presented.
Operation proceeds to step 1410 from step 1408, and wherein environment depth map is received.Depth map defines content
The image of stream will be mapped to that the surface of 3D environment thereon.In the case where no reception depth map, sphere is environment
Default assumption shape, the image of environment will be mapped to table thereon during wherein the inner surface of sphere is rendering before display
Face.By providing and using depth map, more true experience is realized, because the image in content stream will be mapped to that more very
The shape for the environment to be modeled and the surface on surface are reproduced on the spot.Therefore, received depth map is corresponding in step 1410
In by user selection want received content also corresponding environment.Received environment mapping in step 1410, or do not connecing
The default map in the case where mapping is received, environment mapping 1411 is stored as, for then using when rendering image.
Operation proceeds to step 1412 from step 1410.In step 1412, reception will be used to reflect 2D picture material
The one or more UV mapping being mapped in at least part on the surface 3D.In one embodiment, for can be by different individual
The each part for the environment that image indicates receives at least one UV mapping.In some such embodiments, it is spread when by content
When the image sent is captured by different camera (such as left eye and right eye camera), different UV mappings can with and sometimes really
It is provided for different cameras.Therefore, correspond to environment such as although receiving in step 1414 in the example of Figure 14 A
First UV of the first part of forward portion maps, but such as left back portion for corresponding to environment is received in step 1416
2nd UV of second part maps, and the third for corresponding to the Part III of such as right-rearward portion of environment is received in step 1417
UV mapping.The UV mapping of top and bottom corresponding to environment receives in step 1418 and 1420 respectively.If these parts
It is of the same size, then identical UV mapping can be used for multiple portions.But in some embodiments, different UV reflects
It penetrates for by different cameral captured image.
Thus, in one suchembodiment, each part of the environment for providing stereoscopic image data for it, example
Such as left eye and eye image, individual UV mapping, therefore UV can be received and stored for each of left eye and eye image
Mapping is it is contemplated that be used to capture the specific feature of the photomoduel of specific left eye or eye image content.
Each UV mapping provides the surface for being used to for the two dimensional image of the corresponding content stream of the mapping being mapped to 3D environment
Corresponding part mapping.In this way, it can be used as 2D image by camera captured image to be sent, be then used as texture
It is mapped in a part on the surface or surface that are defined by 3D model.
It, can quilt in the case where other contents for scene parts as 3D model information and UV mapping have been received
Image as default value can be received and store.In step 1422, correspond to first, second, third and the 5th scene
One or more content (such as image) is received and stores in step 1422 in part.In some embodiments, corresponding
It is received and deposits in step 1422 in multiple alternate images of a part (for example, background parts or sky portion) of environment
Storage.It can receive control information, which indicates that given point in time is for for which stores multiple default during event
Which default image is a part of image will use.For example, in some embodiments, the background image and crowd station that crowd is seated
Vertical background area image is stored as two default images.Control information is used to refer to which background image is used for event
Given part.For example, during corresponding to the part of standing rhythm of drama or concert, if user turns to background side
To then crowd's standing image will be background image to be shown by denoting signaling.But in the event being usually seated as crowd
During major part, if user rotates his/her head towards background, controlling information will should be seated by signaling using crowd
Default image.Control information can separate signaling with content stream, or can be with the content stream one of a part for environment
Rising is included that this is partially different than the relevant part of one or more default images.For example, corresponding to the content stream of forward direction
Image corresponding to forward direction, such as left eye and eye image can be provided, and indicate which default image should be in thing
Each time during part for sky, ground, right background parts and left background parts control information.As an alternative, it returns
Putting equipment can determine to use based on the brightness in particular point in time and the similarity of the one or more features of foreground image
Which background or sky portion.For example, this can be detected and very cloudy sky image is examined automatically when foreground image is dark
It measures, and when foreground image is brighter, this can also be detected and from the available default day empty graph being received and stored
Partly more cloudy, brighter sky image is automatically selected as in.
In step 1422, the default image corresponding to varying environment part is usually received and stored in a coded form.
In step 1424, one or more is decoded in the received image of institute, and then decoded content is stored in step 1426
In one or more frame buffers.In this way, default image can be decoded and be stored with decoded form, therefore
It does not need to be decoded again when they need to render during playback.Since default image can be used multiple times, solution
Code and storage can reduce decoding request, otherwise may need in rendering or before rendering to decode image.It is assumed that processing money
Source may be insufficient, is being decoded before and then showing and then once decoding image is no longer required with wherein image by it
The embodiment deleted from such as memory is compared, the pre decoding of default image and with decoded form storage improve processor money
The use in source.
Although identical default decoding image can be used multiple times, such as sky etc., it can with its
It is processed before its received picture material combination, so that its closer other image for matching its environment in combination, with
Build environment is watched part.For example, in some embodiments, decoded default image is based on their images in combination
Part receives brightness adjustment, or in default image by least edge when combination with the image of the another part for corresponding to environment
Edge be blurred.Therefore, at least some embodiments, during use, image luminescence and/or color characteristics filtered or
Modification, so that they are more closely similar to the identical characteristic of their ground images in combination.
As initialization data and default image have been stored for using in the future, operation proceeds to step 1428, wherein
The set (such as set of the stream currently selected) of one or more streams to be received is initialised.Each stream can provide vertical
Body or monochromatic image data.Optionally, corresponding audio can also be received in selected stream, but more commonly be at one
Or it is received in multiple individual audio streams.Description will focus on the reception of video flowing, but it is to be understood that audio stream is generally also
It will be received the decode by playback apparatus, and audio may include stereo audio.
In illustrative steps 1428, the set of currently selected stream is set equal to transmission corresponding to before environment
To/forward portion content it is first-class.This is because on startup initial position be arranged to viewing location forward and because
This, as initialization as a result, user expectation will see frontal scene region when starting.
Operation proceeds to step 1429 from step 1428.In step 1429, resource allocation information is received.Resource allocation
Information can be the form of bandwidth and/or data rate allocation control information.In some embodiments, it is received in step 1429
Information include one or more that different piece corresponding to environment should be distributed to about how many bandwidth or data communication capacity
The information of a communication stream.Information can be indicated with regard to bandwidth or data rate, but it is to be understood that data rate is usually and bandwidth
Unanimously.For example, it is contemplated that being used to the type of the data encoding through bandwidth communication, the data volume that can be received can be according to amount of bandwidth
And change.
In step 1429 received information can indicate to distribute to the image corresponding to environment specific part is received can
With the relative maximum amount of message capacity.For example, it can indicate that at most 80% bandwidth or supported data rate should divide
Dispensing primary traffic, such as forward data flow, and last the 20% of bandwidth is assigned to one or more of the other stream.Resource is not to
Equidirectional distribution can according in the corresponding part of (and certain basis in some embodiments) environment picture material and/
Or the audience feedback that detects and change.For example, in some embodiments, the midfield occurred during the corresponding event of content is stopped
During breath, in step 1429 received information can indicate the resource of incrementss should distribute to corresponding to one of environment or
The reception image at two rear portions.This is because user more likely rotates their head and starts remote during intermission
From home court or stage look and may expect for rear portion to provide some video so that seem during intermission
There is movement to carry out in spectators.For example, the image bought hot dog in baseball game or change the people at seat can be (and some
In embodiment really it) is sent, so that seeming that background is movable during intermission, and is static at other times
's.Similarly, the image of billboard can change during intermission in background, for advertisement and/or amusement purpose.Cause
This, it may be desirable to trigger playback apparatus, with during intermission than distributing more resources during the other parts of event
To receive background parts.Received control information can be with during the major part of event in step 1429, (and sometimes
Really) different from during the intermission of event or other discrete portions.In at least some embodiments, in the main matter phase
Between, received control information makes more bandwidth and/or data rate compared with rear portion be assigned to ring in step 1429
Main (such as forward direction) region in border.But during intermission or other discrete portions, it can force and distribute to one
Or the data rate at two rear portions increases.
Distribute between the resource of assigned direction segmentation can based on presented in the part of environment content, spectators note
The measurement for power of anticipating and/or the part of ongoing event, such as the part after major part, intermission part, performance.?
In some embodiments, the specified one or more portions that distribute for corresponding to environment of the control information provided in step 1429
The maximum and/or minimum of the received bandwidth of image or data rate divided, such as when event is carrying out.In some implementations
In example, information instruction does not have bandwidth or data that should be allocated for receiving ground to sky image part during event,
And therefore these parts will be filled using still image when needed in this case.
Bandwidth/data rate allocation control information can change over time, and different information is received in different times.
Control information is sent into content stream and/or individually before being for example embedded into as individual control information set.
Operation proceeds to step 1430 from step 1429, wherein starting the content flowed in the selected adfluxion conjunction being not yet received
It delivers.This can be related to being added the corresponding multicast group of selected stream, message is sent to the network equipment of the delivery of the selected stream of request and/
Or be tuned to the broadcast channel that is sent on it of selected one or more streams.The case where first time passing through step 1430
Under, this corresponds to the delivery of the content stream of forward portion of environment by involving starting up, because this is arranged to be selected for
The initial viewing part delivered.But when the head position of user changes, for example, user rotates his/her to the left or to the right
Head, the set of the stream selected can with and will usually change.For example, if user turns left his head, so that preceding
The a part in square region and right back region enters the visual field, then the set of selected stream will be varied so that corresponding to front area
The content of domain and left back part is received.If left back portion is not received and is selected, left back portion is corresponded in stream
The delivery for the content divided will be activated in step 1430.If maximum data rate stream cannot be supported all in two directions,
It then can choose lower data rate forward-flow, and therefore the starting of the forward-flow of lower data rate and left back content stream is all
It will be activated.Stream except selected set is terminated before the stream in the set newly selected receives.The termination and starting of stream
It is performed in a smooth manner, so that time slot and/or significantling change for the content being received are minimized, wherein obscuring
And/or filtering is used to reduce the quality of image or the significant changes in source when stream switching occurs.For example, fuzzy can be across conduct
The part for the image that a part of rendering or display process is stitched together is implemented.
As current collection of the content delivery to selected stream starts, operation proceeds to step 1432 from step 1430.In step
Stream reception content in rapid 1432, from selected content stream set.This can (and in various embodiments really) be related to
The content for corresponding to highest priority stream is received, for example, providing the content for corresponding to most of visual field and corresponding to the one of environment
The stream of the content of a or a number of other parts (for example, the part for corresponding to the sub-fraction of picture material in the visual field is provided).It mentions
It can be described as minor flow for the stream of the sub-fraction of the content for present viewing field.Single stream is provided for regarding entirely wherein
In one embodiment of wild content, 20% or less available bandwidth/it can support that receiving data rate is retained and is used for
One or more secondary or lower priority streams are received, for example, these one or more secondary or lower priority streams are in user
Content except the visual field is provided in the case where the side except the visual field is rotated up.Corresponding to two different contents in the visual field
Between the region of stream in the case where generally uniform segmentation, each stream can be assigned approximately half of available data rate/reception number
According to rate because they contribute in a substantially even manner and user would be impossible to change position quickly to watch two streams be
The region of environment outside its region that image is provided.
As content reception has been activated, in step 1432, the reception content from the set of selected stream, such as scheme
Picture.In the case where main (for example, highest priority) stream, content usually will be stereo content, wherein left eye and eye image
Content is all received in stream.For low priority flows and/or it is assigned the stream of low data rate, monochrome image can be connect
It receives, wherein single image is received, for being shown to left eye and eye image.Forward direction scene parts are usually as in solid
Appearance is received, but one or more rear portions can be used as monochrome image stream and be provided.
In step 1432, the picture material of coding is usually received in stream.In step 1434, the content received
It is decoded, then in the step 1438 reached via connecting node A 1436, decoded content is stored in one or more
In frame buffer.In some embodiments, decoded picture buffer is maintained to each part of environment.Although the figure received
The only a part of picture is finally shown, but the frame completely received is usually decoded and buffered.Decoded buffers images can
With (and in some embodiments really) keep in memory, until it is used for the more recent figure of same scene part
As replacement.Therefore, at any given time, decoding image can be used for each part of environment, to be based on present viewing field root
According to needing to render final output image.Since the content of decoding image is maintained in memory, until it is by more recent
Image replacement is decoded, therefore the decoding for corresponding to the image of each part of 360 degree of environment does not need during each frame time
Occur.Therefore, it although the frame rate of 30 frame per second can be supported, does not need in each 150 frame of frame cycle decoder, for example,
There is a frame for each of top, bottom, preceding, left back, right-rearward portion, and is equal to or a little higher than frame rate to be supported
The frame of low amount can be decoded, wherein still image or early decoding of some parts of image from early decoding
The image section updated with rate more lower than the part for corresponding to main view open country.
Therefore, after the update in step 1438, current decoded frame can be used for the present viewing field based on user come wash with watercolours
Contaminate image.
Operation proceeds to step 1440 from step 1438.In step 1440, using rendering content of such as getting off for display:
From the available solution digital content of frame buffer, defining one or more image sections for example will be applied to table thereon as texture
The environment in face maps, and provides on how to which 2D decoding image to be applied to the UV of the information of the model on the surface 3D and is mapped.Ring
The 3D grid model in border can be the form of grid, and the image that wherein point in UV mapping corresponds to definition environment will be applied to it
Surface grid model vertex.
As a part of rendering, the part in the user visual field will be fully taken up when that cannot provide from the content individually flowed
When, the image section corresponding to the received content of never cocurrent flow will be combined, to generate the figure for the environment for corresponding to the user visual field
Picture.Filtering or it is fuzzy can (and in some embodiments really) by across image application, wherein image is connected, to be formed pair
It should be in the composograph in the user visual field.This, which tends to reduction seam, mostly obviously user.In addition, in some embodiments, providing
The brightness of the image section of a part of composograph is adjusted, and divides it with the image portion that reduction is combined to form composograph
Between luminance difference, wherein when to be combined to form composograph image section carry out brightness adjustment when, frontal view
Brightness value starts the priority for being given the brightness value higher than rear portion, top or side.
In step 1442, the synthesis of one or more image sections from the received image of different content streams can be
Image is rendered by storage, display or transmission.This indicates the current collection based on selected content stream to one or more content frames
Generation and display.
Over time, due to the variation of communication channel condition or network problem, maximum support data rate meeting
Change.This can be detected in step 1443 and be considered when selecting which content stream that should be received and processed.?
In step 1443, maximum support data rate and/or bandwidth are confirmed as being indicated by data 1444, to make in the next steps
With.In step 1445, the current head position of user is detected.This can be by using on the headwear including head-mounted display
Position sensor complete.
Operation proceeds to step 1446 from step 1445, wherein determining whether the head position of user has changed.If with
The head position at family does not change, then operation proceeds to step 1447, wherein checking that currently available maximum bandwidth or maximum can
Whether the data rate of support has changed, for example, since the last time carries out stream selection.If not detecting that maximum can prop up
Hold the change of data rate or head position, then previous stream selection keep effectively and not to selected content stream set into
Row changes.Therefore, playback system will continue to correspond to the content of the present viewing field of the user remained unchanged.In step 1447
In change is not detected in the case where, operation via connecting node B 1456 return to step 1429.
But if detecting change in step 1446 or 1447, operation proceeds to stream selection step 1448, relates to
And the calling of convection current selection subroutine.In this way, the variation of the head position and/or support data rate that detect can
To be considered, and the selection flowed in view of the support data rate that can be used to receive data (for example, picture material) and/or
User's head position is reconsidered.
It is selected once stream has passed through stream selection subroutine, operation just proceeds to step 1450, wherein selected by checking
Stream whether be different from the set of stream currently selected.If the set and the currently selected set of stream currently in use that newly select
It is identical, then it does not need to being just changed in received stream, and the adfluxion conjunction currently selected in step 1452 remains unchanged,
Operation proceeds to step 1429 via connecting node B 1456.But if the adfluxion that newly selects closes and is different from currently selecting
Adfluxion is closed, then the adfluxion conjunction currently selected in step 1454 is updated, to reflect the change of selected adfluxion conjunction, for example, working as
The set of preceding selection is set equal to be closed by the adfluxion of stream selection subroutine selection newly selected.
Operation from step 1454 proceeds to step 1455, wherein not currently selected adfluxion in the updated close in stream connect
Receipts are terminated.This may relate to playback system signaling, and it is no longer desire to become the one of the multicast group for corresponding to the stream being no longer received
Member, or another movement is taken to use resource for another object (for example, reception of the one or more streams newly selected)
(tuner for the stream being terminated is used to receive before such as).
Operation proceeds to step 1429 from step 1455 via connecting node B 1456.Then, any in step 1430
The reception of the stream newly selected will be activated, and the content received will be used to render one or more images.Therefore, with
The passage of time, it is selected when user changes his or her head position and/or the data rate that can be supported changes
Stream also can change.
Figure 15 is flow chart the step of showing stream selection subroutine accoding to exemplary embodiment, is used as scheming
The stream selection subroutine called in the step 1448 of 14 methods.
When subroutine is called by the step 1448 of such as Figure 14, stream selection subroutine 1500 starts in step 1502.Behaviour
Make to proceed to stream selection step 1504 from beginning step 1502.The input of stream selection step 1504 includes bandwidth and/or data speed
Rate constraint 1503.These may include about should be assigned to the available of one or more scene parts, stream and/or stream direction
The largest portion of bandwidth or total information for receiving data rate.Constraint can also include to scene parts, stream and/or view direction
Minimum bandwidth or data distribution.Therefore, constraint, which can limit, distributes to the resource for receiving the content for corresponding to specific direction
The least resource amount in direction is distributed in maximum and/or authorization (mandate), so that corresponding to the part in direction in event midfield
When movement occurs in a part (such as Background Region) of environment at least infrequently or in specific time during rest
Place is updated.
Maximum support data rate and/or maximum available bandwidth 1444 are another inputs for selecting step 1504.This
Information 1444 indicates to can be used for the maximum bandwidth of reception content and/or can be used to support to receive a content stream or content stream
The maximum support data rate of combination.The current head position 1407 of user and information about available stream 1405 are (for example, all
Stream guidance information as shown in Figure 18) it is also input to step 1504.
In stream selection step 1504, selection corresponds to program or event (for example, being ongoing in some cases
Real-time event) one or more content streams.Band is such as flowed in current head position of the selections based on user of one or more stream
Wide or stream data rate demand stream information and/or the information that bandwidth can be supported about maximum support data rate or maximum.
Can be about view direction or flow priority basis bandwidth and/or data rate constraints 1503 can (and sometimes really
It is considered and uses when the stream selection carried out in step 1504 in fact).
It includes step 1506 that exemplary stream, which selects step 1504, and wherein content stream is based on the current and/or past of user
Head position is prioritized.This can be related to the calling that subroutine is prioritized to stream shown in such as Figure 16.
In some embodiments, the stream corresponding to identical environment direction is assigned identical priority.Therefore, correspond to
Multiple streams of the corresponding content of the same section of environment and/or view direction can (and sometimes really) be assigned it is identical
Priority.Although stream corresponds to identical view direction, in some cases, stream has different data rates, wherein
Some streams provide high-resolution stereo content with high frame rate, and low resolution stream is provided in monochrome image in some cases
Appearance and/or low-resolution image and/or low frame (image) rate of support.Therefore, although specific direction, which may be considered that, has height
All streams for the content that priority and offer correspond to high priority direction are all identical, but in some embodiments to available
It is selected in the amount of bandwidth for receiving the content for corresponding to specific direction.
After the priorization of content stream, operation proceeds to step 1508 from step 1506.In step 1508, quilt is determined
Maximum bandwidth and/or data rate for the stream with highest priority.This determination can be based on bandwidth or other constraints
1503 make, and constraint can indicate that the maximum of highest priority stream distribution to be, minimum can use reception resource or part thereof.One
In a little embodiments, minimum bandwidth/data rate allocation for highest priority stream is 50% or more, but other distribution are also
It is possible.
In step 1510, maximum bandwidth and/or data rate that determination will be used to have each stream of lower priority.
In some embodiments, at least the 20% of data rate or bandwidth is used for secondary or lower priority stream.
With the data rate for having determined that the stream for different priorities in step 1508,1510, operation proceeds to step
Rapid 1512, wherein based on for highest priority stream maximum bandwidth and/or data rate check to determine whether can be with
Support highest priority stream.If any one stream for corresponding to highest priority can be supported, in step 1512
Determining will be "Yes" and operation proceeds to step 1514, will be chosen corresponding to the best quality stream of highest priority there
It selects.This is usually directed to the selection peak data rate stream from the set of stream for being assigned highest priority.Consider for example, if preceding
It is assigned highest priority to the stream in direction, then, the given data rate that can be used for highest priority content stream can be supported
Peak data rate forward direction content stream will be selected.
Operation proceeds to step 1516 from step 1514.In step 1516, it is determined whether can support that second is high preferential
Grade stream.In some embodiments, this is related to determining that how many bandwidth/data can be used to after highest priority stream is selected
And determine that how many can be used for the second highest priority stream in that bandwidth/data based on the received constraint of institute.If not right
Second high priority flows are set to constrain, then bandwidth/data reception capabilities of whole amount can be used for the second high priority flows.If
Determine that the second highest priority stream can be supported in step 1516, then operation proceeds to step 1518, wherein the second highest
Priority flow is for example to select from the set of one or more streams for being assigned the second highest priority.Step 1518 can be with
(and in some embodiments really) it is related to the peak data rate stream that can be supported that selection has the second high priority.
For example, step 1518 corresponds to the right side of environment by selection is related to if the second high priority corresponds to the right-rearward portion of environment
The peak data rate stream that can be supported of part afterwards.
Although in most cases highest priority and minor flow will be supported, two highest priorities are being selected
Possible remaining enough bandwidth after stream, to receive the one of the another part (for example, unviewed part) for corresponding to environment
A little contents.If the second highest priority stream cannot be supported, operation from step 1518 proceed to step 1520 or directly
Step 1520 is proceeded to from step 1516.
In step 1520, such as utilize available bandwidth/number after the first and/or second priority flow is selected
Check whether third highest priority stream can be supported according to resource is received.If determining that third is high preferential in step 1520
Grade stream can be supported, then operation proceeds to step 1522, wherein for example being selected using the remaining bandwidth/data rate that can be supported
Select third high priority flows.If given available bandwidth and/or received data and/or bandwidth allocation constraint, third highest
Priority flow cannot be supported, then operation proceeds to step 1524 from step 1522 or directly proceeds to step from step 1520
1524。
In step 1524, checked to determine whether residue can be used to receive in additional later in the selection of other streams
Any bandwidth (for example, the ability for receiving data) held.If remaining additional bandwidth, operation proceed to step 1526,
Middle one or more lower priority stream is selected, to use remaining available bandwidth/data rate.Operation is before step 1526
Enter step 1530.If available without additional bandwidth, operation proceeds to return step 1530 from step 1524.
Return step 1530 makes processing returns to the called point of stream selection subroutine 1500 is arrived, for example, the adfluxion newly selected
It closes and is determined via routine 1500.
Figure 16 is the flow chart for showing the step of stream accoding to exemplary embodiment is prioritized subroutine.Figure 160 0 is shown
The exemplary stream prioritization routine of (such as calling) can be used by one or more of the other routine described herein or subroutine.
For example, the step 1506 of Figure 15 can be realized by the calling to subroutine 1600.When routine is called, preferential beggar is flowed
Routine 1600 starts in step 1602.Operation proceeds to step 1604 from step 1602 is started.In step 1604, it is based on user
The head position detected determine the present viewing field of user.Assuming that the visual field of user is less than 180 degree, present viewing field can be corresponded to
In a part of the environment captured from single camera position (for example, camera position of eyes front).But when user moves his
When head, such as rotation and/or when seeing upward or downward to the left or to the right, the visual field of user can correspond to by being located at different phases
The part of the environment for the camera capture that seat in the plane is set.For example, by different cameral captured image being met or being overlapped when see to correspond to
When a part of the environment of point, the visual field of user can correspond to the content that will be transmitted in two not cocurrent flow.In some realities
It applies in example, the stream for providing the image of the largest portion corresponding to the visual field will usually be given highest priority.
As the visual field of user has been determined in step 1604, operation proceeds to step 1605, wherein identification transmission pair
Should be in the stream of the content of user's present viewing field, wherein content such as monochrome image or the stereo-picture including left eye and eye image
It is right.Then, operation proceeds to step 1606, wherein determining the field corresponding to user's present viewing field available from the stream identified
The size of one or more parts (for example, image section) of scene area.Which therefore, at the end of step 1606, flowed about
The information for providing the picture material and partial relative size that correspond to present viewing field, which can be used and can be used to convection current, is arranged
Name, such as be prioritized.
Operation proceeds to step 1608 from step 1606.In step 1608, priority is assigned to offer and corresponds to use
One or more streams of the picture material of the present viewing field at family.Being prioritized (such as ranking) is the user visual field provided based on stream
The size of one or more parts.For example, the stream for providing 80% picture material for corresponding to the user visual field will be than providing correspondence
High in the stream ranking of 15% picture material in the user visual field, the stream for providing remaining 5% (for example, top or bottom) will be assigned
Third priority, this is lower than the highest priority for distributing to the stream for the image for providing 80% part for corresponding to the visual field.
Step 1608 can (and in some embodiments really) include step 1610 and/or step 1612.In step
In 1610, highest priority, which is assigned to, provides the stream of the largest portion in the visual field.Step 1610 may include that will provide the visual field
The stream of largest portion is appointed as mainstream.Step 1612 includes that next highest priority is assigned to by highest priority stream tribute
The stream of visual field contribution part except the part offered.What these streams can contribute present viewing field in step 1612 based on them
Partial size is prioritized, wherein the stream of more fraction is contributed to be assigned lower priority.
Assuming that for example multiple streams with different data rate can contribute identical part to the visual field, although potential be located in
In different resolution ratio or frame rate, but multiple streams can be assigned identical priority, for example, corresponding to frontal view
Stream can be assigned identical priority, and the stream for providing left back view can be assigned identical priority, for example, with appointment
The stream given forward-flow different priority, and provide right back view can be provided identical priority, for example, with appointment
To it is preceding to or the different priority of left back view stream.
Therefore, in step 1608, stream contributive to the visual field will be ranked, that is, is prioritized.Priorization can lead to
It crosses and lists stream in ranked list to indicate, wherein mainstream is assigned highest priority, and other streams are assigned lower priority.
Not all stream can correspond to the visual field.For example, top or two scene parts can except the visual field, and
Therefore provide this view stream may there are no be prioritized in step 1608.Operation proceeds to step from step 1608
1614.In step 1614, it is determined whether there is any remaining stream to be prioritized.If there is no the residue to be prioritized
Stream, such as because they both correspond to the present viewing field of user, then operation proceeds to return step 1630.But if in step
Determine that one or more streams still will be prioritized in rapid 1614, then operation proceeds to step 1616.
In step 1616, priority is assigned to one or more additional streams of transmission content, such as in present viewing field
Except stream.In some embodiments, the priorization executed in the step 161 is based on the content provided by the stream being prioritized
With the degree of approach of content visible in present viewing field and/or the end rotation direction of the current or past based on user.For example,
If stream provides the picture material for corresponding to a part of the environment close to present viewing field, in some embodiments, it
Higher priority will be assigned than providing the stream corresponded to further from the content of the image section of user's present viewing field.It is similar
Ground, it is assumed that in the content on end rotation direction than in the opposite direction in the end rotation direction detected far from user
Content more likely rapidly enters the visual field of user, provide the picture material on user's head direction of rotation stream can than
Content on the end rotation direction detected far from user is given higher priority.
In at least one embodiment, step 1616 includes step 1618, wherein being checked to determine head position
Whether variation indicates end rotation, for example, tilting opposite leftward or rightward rotation upward or downward with head.If in step
End rotation is not detected in 1618, then operation proceeds to step 1620, wherein in some embodiments, stream is based on them
Present viewing field relative to user image data is provided which be partially prioritized.When top and bottom part and left or right
When except the visual field, the stream for providing top and/or bottom part can be default to than providing left or right rear portion rear portion
Stream be assigned lower priority.Operation proceeds to return step 1630 from step 1620.
User's head rotation is had been detected by if determined in step 1618, operation proceeds to step 1622.In step
In rapid 1622, the direction of end rotation is determined, for example, user's head is to the left or to turn right.This allows to consider head rotation
Turn direction because will enter the user visual field environment next part usually more likely on the direction of end rotation without
It is far from it.
Operation proceeds to step 1624 from step 1622, wherein the direction based on end rotation assigns a priority to one
Or multiple content streams, such as the stream except the visual field.In at least one embodiment, step 1624 includes step 1626, wherein
Next minimum unused priority, which is assigned to, provides the content of a part on end rotation direction for corresponding to environment
Stream.For example, if the forward portion for seeing environment user by his head rotated right, provide behind the right side except the visual field
The stream of square content will be assigned higher priority than the stream for providing the also left back content except the visual field.It operates from step 1624
Proceed to step 1628, there, there are no the residual stream being prioritized than be assigned the stream of priority be assigned it is lower
Priority, indicate lower importance.
Operation proceeds to return step 1630 from step 1628.When reaching return step 1630, content stream is by root
It is prioritized according to priority, such as ranking or sequence.
Figure 17 is the flow chart for showing the step 1700 of rendering subroutine accoding to exemplary embodiment.
When needing image rendering, rendering subroutine 1700 can be by one or more routine calls.In stereo content
In the case of, it is that every eye in the left eye and right eye of user renders individual image.In the case where monochromatic content, to user's
Every eye renders and uses single image in left eye and right eye.Rendering often refers to the picture material from one or more stream
Combination.Therefore, although some parts of environment can be used as monochromatic content and be provided, other parts be can be used as in solid
Appearance is provided, and in this case, different left eyes and eye image can be rendered, and some of contents are three-dimensional, and
Other contents are monochromatic, are left eye and eye image when at least part of environment is presented as stereo content still
Each of generate single image.
It renders routine 1700 and is starting step 1702 beginning, and proceed to rendering step 1706.Rendering step 1706 it is defeated
Enter including environment mapping 1411, corresponding to the decoding picture material 1703 of one or more views, and be used to one or
Multiple decoding images or image section are mapped to one or more UV mapping 1704 on the surface by 1411 definition of environment mapping.
As discussed above, in the case where not providing more complicated geometry, environment mapping 1411 can be default to be sphere,
Wherein image is mapped to the inner surface of sphere.
In some embodiments, rendering step 1706 includes step 1708, which is included in including the use of by decoding
Content and the environment mapping generated corresponding to the image in one or more content streams of user's present viewing field and at least one
A UV mapping is to generate at least one image corresponding to user's present viewing field.Three-dimensional (for example, 3D rendering content) the case where
Under, rendering will lead to the format generation left eye and eye image for being suitable for display.In some embodiments, rendered left eye and
It is 3D that the difference of eye image, which causes user to perceive image,.
Operation proceeds to step 1710 from step 1706, and step 1710 is return step, it makes rendered image quilt
Back to program or routine, in order to provide to display equipment, storage and/or output.
Every time when the frame for wanting the visual field or when more new version is to be displayed, rendering subroutine 1700 can be called.Therefore, wash with watercolours
Dye is usually to show that the consistent rate of rate occurs with image (for example, frame).
Figure 18 shows the exemplary table 1800 of the stream information including corresponding to multiple content streams.In some embodiments,
A part including the stream information in exemplary table 1800 as tutorial message (such as program guide) is received, to provide
It can choose the information of received content stream about playback system.Legend 1840 is used as in table 1800 including instruction
The information of the meaning of the various letters of the abbreviation of information.
Including content stream corresponding to access information can be used in the information in table 1800.As will in some embodiments
It discusses, for multiple available content streams, stream information includes that can be added into receive the multicast group of given corresponding content stream
Multicast address, can be used to request access to the information of the switched digital video channel for providing given content stream or can quilt
For control playback system tuner be tuned to the channel tuning information of broadcast channel broadcasted on it of given content stream work as
In at least one.
In table 1800, every row corresponds to the independent content stream of transmission content, wherein going corresponding content stream by column 1812
The identification of flow identifier shown in middle corresponding entry.Each entry identification in column 1804 is spread by the corresponding independent content of the row
The programme content sent.Such as from table 1800 it should be understood that first group of row 1820 corresponds to programme content " football ", this can be indicated
The program/event title as shown in the corresponding entry in column 1804.There may be correspond to a variety of different program/events
Multiple such groups.Each group includes content stream, and each content stream corresponds to view direction and supports given data rate, such as
As discussing.For simplicity, two groups are illustrated only in figure, wherein corresponding to second group 1822 of row only by part
Ground is shown, this is simply to illustrate that concept.Second group of row 1822 corresponds to programme content " HI ", such as by the correspondence item in column 1804
As mesh is indicated.
Each entry in column 1806 indicates a part by the streamed scene areas of corresponding content, such as 360 degree
The area 1200 of scene areas.Therefore, the first three rows in group 1820 (every a line corresponds to different content streams) transmission frontal scene
Partially (for example, being mapped to area 1 shown in Figure 12, the viewing areas of 270 ° to 90 ° of covering).Next three row in group 1820
(every a line corresponds to different content streams) transmission right back scene parts (for example, being mapped to area 2 shown in Figure 12, cover
30 ° to 210 ° of viewing areas).Left after last three row (every a line corresponds to different content streams) transmission in group 1820
Scape part (for example, being mapped to area 3 shown in Figure 12, the viewing areas of 150 ° to 330 ° of covering).
Each entry in column 1808 indicates the data rate that corresponding content stream is supported.Each entry in column 1810 refers to
Showing can be added into receive the multicast of the corresponding content stream of the identification of the flow identifier as shown in the corresponding entry in column 1812
The identifier/address of group.Each entry in column 1814 includes for the traffic identifier as shown in the corresponding entry in column 1812
Accord with the flow descriptors of the corresponding content stream of identification.Each entry in column 1816 includes that can be used to access or request corresponding content
The access information of stream, such as tuner parameters and/or other access parameters.
Such as from exemplary table 1800 it should be understood that in the example shown, existing and corresponding to given view direction
Multiple (such as three) of each content stream are available in different editions used in playback, and wherein each version of content stream is supported
Different data rates.Therefore, feature according to the present invention, playback system can factor based on one or more, for example, such as
Bandwidth, data rate, user's head position of support etc., come select will the one or more used in playback stream, such as about
As Figure 14-17 is discussed in detail.
It is used to select and/or access one in order to which how the information being more clearly understood that in table 1800 can be played system
Or multiple content streams, in the first row and column 1804,1806,1808,1810,1812,1814 and 1816 in consideration group 1820
First entry in each column.First entry of column 1804 indicates the flow identifier S by including in column 18121D1Identification
The streamed event/program of first content " football ".Corresponding entry indicates that first stream transmission corresponds to front in column 1806
The content of scene parts (for example, 270 ° to 90 ° viewing areas).This viewing areas information is played system and is used to identify transmission
One or more streams of content corresponding to user/spectators current head position, which corresponds to and works as forward sight
It is wild.Continue the example, data rate D is supported and/or needed to corresponding first entry instruction first content stream in column 18081.Column
Corresponding entry instruction first content stream can be accessed by the way that multicast group M1 is added in 1810, and wherein M1 indicates Multicast group address
And/or it is mapped to the identifier of address.Corresponding entry includes the flow descriptors corresponding to first content stream in column 1814
“V1C1D1F1", indicate first-class corresponding camera viewing angles (V1), correspond to first frame rate (F1), support number
According to rate (D1) and codec type (C1).The instruction of corresponding entry can be used to access or request the in last column 1816
The access tuner parameters of one content stream and/or other access parameters (being illustrated as A123).
As this type of information discussed above about available content stream can be used, such as playback system 1900 is returned
Place system can be with feature selecting according to the present invention and the one or more content streams of access to use in playback.In order to better
Understand, considers a simply example, wherein playback system determines that user's head position instruction user is seeing 360 degree of scenes
Front.In this case, in one embodiment, at least one content of playback system selection transmission frontal scene part
Stream.Dependent on the various other factors such as discussed about Figure 14-17, such as available bandwidth, supported data rate, stream bandwidth
And/or data rate constraints, playback system can be from three different available stream (S1D1、S1D2、S1D3) middle selection transmission front
One stream of scene parts.If constraint allows, playback system will be from the multiple content streams for corresponding to frontal scene part
Select best quality stream, such as stream S1D1.Appropriate stream of the information provided in table 1800 convenient for selection for playback, because can quilt
It is provided for carrying out at least some information of selection by stream information 1800.After stream selection, playback system can be reused
Stream information 1800 to correspond to the multicast group (for example, M1) of selected stream or by using access information acquisition content by being added
Stream starts content delivery (for example, content reception).
Figure 19 shows the playback system 1900 realized according to the present invention, can be used to receive, decodes, stores and show
From the received image content of content delivery system.The system 1900 that can be implemented is the single playback apparatus for including display 1902
1900', or be embodied as such as being coupled to the external display (for example, head-mounted display 1905) of computer system 1900'
Element combination.
In at least some embodiments, playback system 1900 includes 3D head-mounted display.Head-mounted display can benefit
With the OCULUSRIFT that may include head-mounted display 1905TMVR (virtual reality) earphone is realized.It can also use other
Head-mounted display.In some embodiments, wherein one or more display screens are used to show to the left eye and right eye of user
The wear-type helmet of content or other headset equipments are used as equipment 1905.By aobvious to left eye and right eye on single screen
Show different images, wherein headset equipment is configured as the different piece of single screen being exposed to different eyes, individually
Display can be used to show the left eye and eye image that will separately be perceived by the left eye and right eye of viewer.In some embodiments
In, mobile phone screen is used as the display of head-mounted display apparatus.In at least some such embodiments, mobile phone is inserted into head
It wears in formula equipment and mobile phone is used to display image.In some embodiments, display equipment 1905 can be such as Oculus
A part of the 3D display device of Rift.
Playback system 1900 has ability below: the received coded image data of decoding institute (for example, correspond to environment or
The left eye and eye image and/or monochrome (single image) of the different piece of scene) and for example by render and show by with
Family is perceived as the different left eyes and right-eye view of 3D rendering to generate 3D rendering content for showing to consumer.Some
In embodiment, playback system 1900 is located at consumer guard station position, such as family or office, but can also be located at image capture
Place.System 1900 can execute signal reception, decoding, display and/or other operations according to the present invention.
System 1900 connects including display 1902, display device interfaces 1903, input equipment 1904, input/output (I/O)
Mouth 1906, processor 1908, network interface 1910 and memory 1912.The various parts of system 1900 via allow data be
It the bus 1909 that communicates and/or is coupled by other connections or by wireless interface between the component of system 1900.Although
The optional element of the conduct of display 1902 in some embodiments is included, as shown in using dotted line frame, but in some implementations
In example, external display device 1905, such as wear-type stereoscopic display device, it can be coupled to back via display device interfaces 1903
Put equipment.
For example, being used as processor 1908 in cell phone processor and mobile phone generates in headset equipment and show image
In the case where, system may include processor 1908, display 1902 and memory 1912, and one as headset equipment
Point.Processor 1908, display 1902 and memory 1912 can be a part of mobile phone.In other implementations of system 1900
In example, processor 1908 can be a part of the game system of such as XBOX or PS4, and wherein display 1905, which is mounted on, wears
In formula equipment and it is coupled to game system.Whether processor 1908 or memory 1912, which are located at, is worn in overhead equipment simultaneously
Although be not it is crucial also, as it would be appreciated, co-located in headwear in some cases processor can be it is convenient,
But from the perspective of power, heat and weight, at least some cases, it may be desirable to make processor 1908 and memory
1912 are coupled to the headwear including display.
Although various embodiments contemplate head-mounted display 1905 or 1902, this method and device can also with can
The non-head-mounted display of 3D rendering is supported to be used together.Thus, although system 1900 includes wear-type in many examples
Display, but it can use non-head-mounted display also to realize.
The operator of playback system 1900 can control one or more parameters via input equipment 1904 and/or selection is wanted
The operation of execution, such as selection display 3D scene.Via I/O interface 1906, system 1900 may be coupled to external equipment and/or
With other devices exchange signals and/or information.In some embodiments, via I/O interface 1906, system 1900 can receive by
Various camera captured images, these cameras can be a part of the camera equipment of such as camera equipment 900.
Processor 1908 (such as CPU) is executed routine 1914 and is controlled playback system 1900 using various modules with basis
The present invention operates.Processor 1908 is responsible for the overall general operation of control playback system 1900.In various some embodiments
In, processor 1908 is configured as executing the function of being discussed as being executed by playback apparatus.
Via network interface 1610, system 1900 is set through communication network (for example, such as communication network 105) to various outsides
It is standby to transmit and/or receive signal and/or information (e.g., including image and/or video content).Network interface 1910 includes receiving
Device 1911 and transmitter 1913 execute via them and send and receive operation.In some embodiments, system is via network interface
1910 receive one or more selected content streams from content provider.In some embodiments, system 1900 is via interface
1910 receiver 1911 receives one or more selected content streams for playing back.The received content stream of institute can be used as volume
Code data (such as scene parts 1952 of coding) are received.Receiver 1911 be additionally configured to receive stream information 1946 and/or
Initialization data, such as a part as program guide.System 1900 also for example via 1911 reception bandwidth of receiver and/or
Data rate allocation controls information 1952, this include for different view directions bandwidth constraint, specified to be used to receive one
Or the individual bandwidth constraint of the maximum bandwidth of multiple content streams, in order to provide viewing side corresponding with individual bandwidth constraint is corresponded to
To content.In some embodiments, receiver 1911 is additionally configured to receive the mapping of at least one environment, such as limits justice 3D
The 3D depth map on surface, and be used to for picture material being mapped to the one or more in at least part on the surface 3D
UV mapping, for example, during initial phase or other time.In some embodiments, receiver 1911, which receives, corresponds to field
The first part of scape environment the first UV mapping, corresponding to scene environment second part the 2nd UV mapping, correspond to third
Partial 3rd UV mapping, corresponding to Part IV the 4th UV mapping and the Part V corresponding to scene environment the 5th
Mapping.In some embodiments, during initialization, system 1900, which is for example received via the receiver of interface 1910, corresponds to field
One or more content in first, second, third, fourth and fifth part of scape, such as image.
Memory 1912 includes various modules, for example, control playback system 1900 is held when it is executed by processor 1908
Row decoding according to the present invention and output operation.Memory 1912 include control routine 1914, head position determination module 1916, when
Preceding viewing location initialization module 1918, decoder module 1920, currently selected stream initialization module 1922, content delivery starting
Module 1924, frame buffer 1926, frame buffer update module 1928, the image rendering for being also referred to as image generation module
Module 1930, available bandwidth and/or the data rate determination module of support 1932, head position change determining module 1934, can
It is closed with bandwidth and/or the data rate of support change determining module 1936, stream selecting module 1938, selected adfluxion and is changed determining mould
Block 1940, selected adfluxion close update module 1942, stream terminates module 1944, and the stream information 1946 including receiving, reception
Bandwidth and/or data rate assignment information 1948, determining current maximum available bandwidth and/or the support data rate arrived
1950, UV mapping 1956, decoding figure that the coded image content 1952 received, the environment received map 1954, receive
As the data/information including content 1958 and the 3D content 1960 generated.
Control routine 1914 includes equipment control routine and Communications routines, with the operation of control system 1900.Head position
Determining module 1916 is configured to determine that the current head position of user, such as the position of head-mounted display.Head position is true
Cover half block 1916 can be integrated with position sensor and/or cooperates, and Position Sensor can include for example wearing
On the headwear of formula display.Current viewing position initialization module 1918 be configured as example passing through during initial phase by
The current head position of the user detected is initial by the current viewing position of user to (zero degree) viewing location before being set as
To (0 degree) environment viewing location before turning to.
Decoder module 1920 is configured as decoding from the received coded image content 1952 of content delivery system 700, with
Generate decoded image data 1958.Decoded image data 1958 may include decoded stereo scene and/or decoded field
Scape part.In some embodiments, decoded content is stored in one or more frame buffers 1926.At the beginning of currently selected stream
Beginningization module 1922 is configured as initializing the current collection of the one or more content streams to be received selected.Currently quilt
Choosing stream initialization module 1922 be configured as by the adfluxion currently selected close be set as transmission correspond to environment/scene forward direction/
The content of forward portion it is first-class.
Content delivery starting module 1924 is configured as starting the delivery of selected content stream.In some embodiments,
Content delivery starting module 1924 starts the delivery for the content stream being not yet received in selected set.In some embodiments, interior
Hold delivery starting module 1924 to be configured as sending request signal, the multicast group for corresponding to selected content stream is added, such as
The multicast group of corresponding content stream is closed with the adfluxion currently selected corresponding to transmission.In some other embodiments, content delivery
Starting module 1924 is configurable to generate and equipment into network sends request, request deliver selected content stream on it by
The exchange digital channel of transmission.
Frame buffer update module 1928 is configured as in the update for example by selected content stream set transmission
Appearance updates frame buffer 1926 using the content of update when being received and decoded.
The feature according to the present invention of image rendering module 1930, such as using decoded picture material 1958, generate 3D figure
Picture, such as the left eye and eye image that are shown in a manner of it will be perceived as 3D rendering, to be used in display 1902 and/or show
Show and is shown in equipment 1905 to user.In some embodiments, image rendering module 1930 is configured as using corresponding to user
The decoded picture material 1958 in currently viewing region, environment map 1954 and UV mapping and carry out rendering content for showing.Cause
This, in some embodiments, image rendering module 1930 is configured as executing the function of the step discussion shown in Figure 17.
Picture material 1960 generated is the output of 3D rendering generation module 1930.Therefore, rendering module 1930 is rendered to display
3D rendering content 1960.In some embodiments, image rendering module 1930 is configured as the image for generating one or more
It is output to and for example shows equipment or another equipment.Image generated can connect via network interface 1910 and/or display equipment
Mouth 1903 exports.
Available bandwidth and/or the data rate determination module of support 1932, which are configured to determine that, can be used (example in given time
Such as, be used for reception content stream) current maximum available bandwidth and/or current maximum support data rate.Due to available bandwidth
And/or supported data rate can due to communication channel condition change or network problem and change over time, one
In a little embodiments, determining module 1932 executes monitoring and/or is determined on the basis of continuing, to detect available bandwidth and/or branch
The change for the data rate held.Identified current maximum support data rate and/or bandwidth 1950 are determining modules 1932
Output, and can update when needed.
Head position changes determining module 1934 and is configured as example by checking and comparing head position determining module
The change of 1916 output determines whether user's head position has changed.Available bandwidth and/or the data rate of support change
Become current maximum available bandwidth and/or current maximum that determining module 1936 is configured as detection with is determined by determining module 1932
In comparison available bandwidth and/or the data rate of support whether there is any change to support data rate.
Which stream selecting module 1938 is configured as selecting in multiple content streams based on the current head position of user
It to be received for using in playback in given time.The change of current head position based on user and/or it is other because
Element, stream selecting module 1938 can select different stream in different time.It is stream selecting module that currently selected adfluxion, which closes 1961,
1938 output, and indicate that current selection is used for the set of received content stream.In some embodiments, selecting module is flowed
1938 include the multiple submodule for being configured as performing various functions as a part of stream selection operation.Figure 23 shows in more detail
The various modules for having gone out stream selecting module 1938 and being included therein, and be discussed later.
Selected adfluxion closes change determining module 1940 and is configured to determine that currently selected adfluxion closes whether 1961 have change,
For example, since selecting module has selected one or more additional content streams and/or one or more streams due to being received
It is terminated/stops.Selected adfluxion closes update module 1942 and is configured as when selected adfluxion closes and there is change that (such as content flows to
Selected adfluxion closes 1961 addition or termination) it updates currently selected adfluxion and closes 1961, any changed with reflecting to what selected adfluxion was closed
Become.Stream termination module 1944 is configured as termination/stopping reception and is previously received but no longer closes in 1961 in currently selected adfluxion
One or more content streams, such as closed and 1961 be updated due to flowing the change of selection due to currently selected adfluxion.
Stream information 1946 includes the information about the multiple content streams that can be used for receiving and using in playback.It is included in stream
It is shown in information and Figure 18 in information 1946 and previously discussed same or similar.The received bandwidth of institute and/or data rate
Distribution controls information 1948 and includes the bandwidth constraint for different view directions and/or indicate that about corresponding to be various different sights
See that direction provides the information of the constraint of the data rate of the content stream of content.Identified current maximum support data rate
And/or bandwidth 1950 indicates the maximum support data rate and/or bandwidth that are determined in given time by playback system 1900.
Received environment mapping 1954 include the 3D depth map for defining the environment on the surface 3D.In some embodiments,
Depth map as one or more corresponding to environment interested can be received by playback system 1900.The received UV of institute reflects
Penetrating 1956 includes the one or more UV mapping for corresponding to the part of interested environment/scene.Decoded data 1958 include
By the decoded data according to the present invention of decoder 1920.Decoded data 1958 include the environment comprising closing transmission by being chosen adfluxion
Scene or scene parts content.
In some embodiments, various modules discussed above are implemented as software module.In other embodiments, module
It realizes within hardware, such as is embodied as individual circuit, wherein each module is implemented as executing the corresponding function of the module
The circuit of energy.There are also in other embodiments, module is realized using the combination of software and hardware.
Although being illustrated as in memory 1912 to be included in the example of Figure 19, it is shown as including in playback apparatus
Module in 1900 can (and in some embodiments really) realized in the hardware in processor 1908 completely, such as make
For individual circuit.Module can (and in some embodiments really) realize within hardware completely, such as corresponding to
The independent circuit of disparate modules.In other embodiments, some modules are implemented as such as circuit in processor 1908, and its
Its module is implemented as example outside processor 1908 and is coupled to the circuit of processor 1908.As it should be appreciated, place
The level of the integrated horizontal and/or some modules portion outside the processor of managing device upper module can be one of design alternative.As replacing
In generation, is not implemented as circuit, but all or some modules can be realized in software and be stored in depositing for system 1900
In reservoir 1912, wherein the module control operation of system 1900 is real when module is executed by processor (for example, processor 1908)
Now correspond to the function of module.There are also in other embodiments, various modules are implemented as the combination of hardware and software, for example,
Another circuit in portion provides input to processor 1908 outside the processor, and then processor 1908 operates under software control, with
Execute a part of execution module function.
Figure 23 illustrate in greater detail used in playback system 1900 stream selecting module 1938 and including it is each
Kind module.Stream selecting module be configured as being selected according to the method for the invention being such as discussed in detail about Figure 14-16 one or
Multiple content streams.In some embodiments, stream selecting module be configured as head position based on user, stream information 1946 and/
Or which maximum support data rate selects to receive in multiple content streams.In some embodiments, selecting module is flowed
1938 include the stream prioritization module 2306 for being configured as the head position based on user and being prioritized content stream.Flow prioritization module
2306 output is, for example, the prioritized list of the content stream with the priority assigned.It is discussed in more detail below with reference to Figure 24
Flow prioritization module 2306.
Stream selecting module 1938 further includes being configured as example determining based on bandwidth and/or data rate constraints for having
The maximum bandwidth of the stream of highest priority and/or the highest priority stream maximum bandwidth and/or data rate of data rate determine
Module 2308, and it is configured to determine that the maximum bandwidth and/or data rate that be used to have each stream of lower priority
Lower priority stream maximum bandwidth and/or data rate determination module 2310.In some embodiments, determining module 2308,
2310 execute corresponding determination using the output of bandwidth control information 1948 and stream prioritization module 2306.Therefore, stream selection
Module 1938 may include one or more is configured as based on the band for being for example transmitted to playback system from the network equipment/server
The stream bandwidth determination module that width constrains to determine the bandwidth of at least one content stream.
Stream selecting module 1938 further include be configured as based on the identified maximum bandwidth for highest priority stream with/
Or data rate and determine what whether highest priority stream can be supported based on available bandwidth and/or support data rate
Module 2312, and it is configured as the module 2314 that selection has the peak data rate stream for the highest priority that can be supported.
In some embodiments, selecting module 2314 is configured as selecting in one from the multiple content streams for be assigned highest priority
Rong Liu, be assigned highest priority each content stream provide correspond to identical view direction content, as be configured as from
Multiple content streams with equal priority carry out a part of selection.In some embodiments, module 2314 is configured as base
It is selected from multiple content streams with equal priority (for example, highest priority) in identified amount of bandwidth available.
Therefore, in some embodiments, when multiple streams with equal priority are available, for example, some there is high data rate to need
It asks, and it is other with lower data rate demand, if available bandwidth and/or support data rate and bandwidth constraint allow this
The selection of sample, then selecting module 2314 selects best quality stream, such as high data rate stream.
Flowing selecting module 1938 further includes being configured as based on the identified maximum belt for being used for the second highest priority stream
Wide and/or data rate and based on available bandwidth (for example, total available or remaining can be used) and/or supported data rate come
Determine the module 2316 whether the second highest priority stream can be supported, being configured as selection has the second Gao You that can be supported
The module 2318 of the peak data rate stream of first grade is configured as based on the identified maximum for third highest priority stream
Bandwidth and/or data rate union are based on available bandwidth (for example, total is available or remaining available) and/or supported data speed
Rate determines the module 2320 whether third highest priority stream can be supported, and is configured as selection and has can be supported the
The module 2322 of the peak data rate stream of three highest priorities.Therefore, in some embodiments, stream selecting module 1938 is matched
It is set to the one or more content streams for for example selecting to be assigned highest priority by prioritization module 2306.
Flowing selecting module 1938 further includes additional capacity/bandwidth determining module 2324, is configured as example existing
Have been selected one or more higher priority flows for receive after determine whether there is any residue or additional available bandwidth
For receiving additional content stream.In some embodiments, stream selecting module 1938 further includes module 2326, is configured as base
In the identified maximum bandwidth for one or more lower priority streams and/or data rate and it is based on available bandwidth
And/or supported data rate selects the one or more lower priority streams that can be supported.
Figure 24 shows stream prioritization module 2306, can be implemented as example flowing a part (example of selecting module 1938
Such as, as its submodule) or as individual module.Stream prioritization module 2306 is configured as the head position based on user
To be prioritized content stream.Once content stream has been prioritized, stream selecting module 1938 can be held from the content stream of priorization
Row stream selection.In some embodiments, stream prioritization module 2306 include be configured as the current head position based on user come
The present viewing field identification module 2404 of user's present viewing field of a part for the scene areas that identification instruction user is watching, with
And be configured as identification transmission and the stream corresponding to the corresponding content in the part of scene areas of user's present viewing field works as forward sight
Open country stream identification module 2404.The output of present viewing field stream identification module 2404 can be stored in memory in some embodiments
The list of the stream identified in 1912, and the list can user head position and therefore the visual field change when quilt
It updates.Therefore, in various embodiments, in order to be prioritized various available content streams, the use for corresponding to head position is identified first
The present viewing field at family, and identify that transmission corresponds to the stream of the content in the visual field.
In some embodiments, stream prioritization module 2306 further include be configured to determine that it is available from the stream identified
The module 2406 of the size of the part of scene areas corresponding to user's present viewing field, and be configured as providing based on each stream
Part size to providing the one or more stream assigned priorities for the part for corresponding to the scene areas of user's present viewing field
Priority appointment/distribution module 2408.In some embodiments, priority appointment/distribution module 2408 includes module 2410,
It is configured as assigning highest priority to the stream for the largest portion for providing the visual field, such as will provide the field for corresponding to present viewing field
The stream of the largest portion of scape is appointed as mainstream.In some embodiments, priority appointment/distribution module 2408 further includes module
2412, the size of the view sections provided based on each remaining stream is configured as to assign next highest priority simultaneously
Remaining stream (for example, as secondary, third level, etc.) is specified, such as the stream ratio offer correspondence of the major part in the visual field is provided
It is given a higher priority and specifies in the stream of the smaller portions of the scene of present viewing field.
In some embodiments, stream prioritization module 2306 further includes module 2414, is configured to determine whether exist
Residual stream will be prioritized, for example, providing the stream of the content of the scene areas corresponded to except present viewing field.
In some embodiments, stream prioritization module 2306 further includes module 2416, is configured as image content-based
One of the content except user's present viewing field is provided with the degree of approach of present viewing field or the direction of end rotation to be prioritized
A or multiple additional streams, for example, to its assigned priority.In some embodiments, module 2416 is configured as being based on being transmitted
Picture material and the degree of approach of present viewing field be prioritized the content that transmission corresponds to part except the present viewing field
One or more additional streams, be transmitted in the content stream ratio of the picture material of present viewing field close proximity except present viewing field simultaneously
Further away from content stream be assigned higher priority.
In some embodiments, module 2416 includes end rotation determining module 2418, is configured to determine whether to examine
Measure the end rotation of user, such as a part as user's head position change.In some but not every embodiment
In, when user looks down towards sky or roof upwards or towards ground, although head position has change, this head fortune
Dynamic is not considered as end rotation.In some embodiments, module 2416 be configured as the end rotation direction based on user come
One or more additional content streams are prioritized, the picture material except present viewing field but on end rotation direction is provided
Another content stream quilt of the content stream than the picture material except present viewing field and on the direction far from end rotation direction
Assign higher priority.In some such embodiments, module 2416 further includes module 2420, is configured as to offer
The stream of content corresponding to the scene parts (for example, top or bottom of scene environment) except present viewing field assign it is next compared with
Low priority (for example, after higher priority is assigned to the stream for providing the content for corresponding to the visual field) and stream are specified, such as
The third level.In some embodiments, when determining there is no when end rotation, end rotation determining module provides control to module 2420
System input, so as to additional stream assigned priority.
In some embodiments, module 2416 further includes end rotation direction determining mould 2422, is configured to determine that
Direction of rotation of the user's head relative to previous head position, for example, to the left or to the right.In some embodiments, module 2416
Further include module 2424, is configured as considering end rotation direction and corresponding to the part except present viewing field to transmission
One or more additional streams assigned priorities of content.In some embodiments, module 2424 includes module 2426, is configured
For to the stream for the content that a part in cephalad direction corresponding to scene is provided assign next lower priority (for example, from
Next available priority of higher beginning) and it is specified, such as third level stream.It will thus be appreciated that if detecting that head is revolved
Turn, then the priority for being executed convection current based on the direction of end rotation in some embodiments is assigned.In some embodiments, mould
Block 2416 further includes the add-on module 2428 for being configured as assigning more low priority to any residual stream of consideration.
Although being illustrated as single processor (such as computer) in the embodiment of figure 19, but it is to be understood that processor
1908 may be implemented as one or more processors, such as computer.When implemented in software, module is included in by handling
Configuration processor 1908 corresponds to the code of the function of the module with realization when device 1908 executes.Shown in Figure 19,23 and 24
Various modules are stored in the embodiment in memory 1912, and memory 1912 is the computer journey for including computer-readable medium
Sequence product, computer-readable medium include code, such as the independent code of each module, for making at least one computer
(such as processor 1908) realizes the corresponding function of module.
Completely hardware-based or fully software-based module can be used.It is understood, however, that can be used software and
Hardware appoints any combination, for example, the module that circuit is realized can be used to realize function.As it should be appreciated, Figure 19,23 and 24
Shown in module control and/or configuration system 1900 or in which element (such as processor 1908) Lai Zhihang Figure 14 stream
The function for the correspondence step for showing and/or describing in the method for journey Figure 140 0 simultaneously executes and corresponds to step shown in Figure 15-17
Function.
Combined Figure 20 including Figure 20 A, Figure 20 B, Figure 20 C, Figure 20 D and Figure 20 E is according to various exemplary embodiments
Operation content playback system illustrative methods flow chart 2000.According to various embodiments, contents playback system is, for example,
It is coupled to the content playback device or computer system of display.
The operation of illustrative methods starts from step 2002, and wherein contents playback system is powered on and initialized.It operates from step
Rapid 2002 proceed to step 2004, and wherein contents playback system receives the first of the first rear view section for corresponding to the environment
Image.Operation from step 2004 proceeds to step 2006, and wherein contents playback system storage corresponds to described the of the environment
First image received of one rear view section.Operation proceeds to step 2008 from step 2006, wherein content playback
System receives the one or more additional images for corresponding to first rear view section of the environment, including corresponds to described
At least the second image of first rear view section of environment.Operation proceeds to step 2010 from step 2008, wherein content
Playback system storage corresponds to the one or more appended drawings received of first rear view section of the environment
Picture.Operation proceeds to step 2012 from step 2010.
In step 2012, contents playback system receives the first figure for corresponding to the second rear view section of the environment
Picture.Operation proceeds to step 2014 from step 2012, and wherein contents playback system storage corresponds to described the second of the environment
First image received of rear view section.Operation proceeds to step 2016 from step 2014, wherein content playback system
System receives the one or more additional images for corresponding to second rear view section of the environment, including at least corresponds to institute
State the second image of second rear view section of environment.Operation proceeds to step 2018 from step 2016, and wherein content is returned
Place system storage corresponds to the one or more additional images received of second rear view section of the environment.
Operation proceeds to step 2020 from step 2018.
In step 2020, contents playback system receives the one or more for corresponding to the sky aerial view part of the environment
Image.Operation proceeds to step 2022 from step 2020, and wherein contents playback system storage corresponds to the day of the environment
The one or more images received of aerial view part.Operation proceeds to step 2024 from step 2022, wherein content
Playback system receives the one or more images for corresponding to the ground View component of the environment.Operation is proceeded to from step 2024
Step 2026, wherein contents playback system storage correspond to the environment the ground View component one received
A or multiple images.In some embodiments, sky aerial view and ground view are used to refer to the nominal head relative to viewer
The up direction and down direction at direction visual angle (perspective) and be applied to indoor environment and outdoor environment.
In some embodiments, such as specific embodiment is relied on, can receive corresponding to the first rear view section, second
Some in rear view section, sky aerial view part and ground View component but it is not necessarily whole images.
Operation proceeds to step 2030 via connecting node A 2028 from step 2026, reaches step 2034, and via
Connecting node B 2036 reaches step 2028,2040,2042,2044,2046,2048,2050 and 2052.Back to step
2030, in step 2030, contents playback system determines that the head position of viewer, the head position correspond to and work as forward sight
It is wild.Operation proceeds to step 2032 from step 2030, and wherein contents playback system is based on identified head position to determine
State the present viewing field of viewer.Operation proceeds to step 2030 from step 2032.Such as it on the basis of continuing, is repeatedly carried out
Step 2030 and 2032, and present viewing field is updated (for example, refreshing).Identified present viewing field can be used for generating output figure
It is used when picture.
In step 2034, contents playback system, which receives, provides the first part for corresponding to environment (for example, forward portion regards
Figure) content first content stream.
In step 2038, contents playback system receives control information, and control information instruction is answered during playback duration
When display corresponds to multiple which of images previously transmitted of first rear view section of the environment, the playback
Time is measured relative to the playback duration indicated in the first content stream.In step 2040, content playback
System receives image selection information, and image selection information instruction should use during a part of the event corresponds to ring
Which of the multiple images of first rear view section in border.
In step 2042, contents playback system receives control information, which indicates during playback duration
2236 should show multiple which of images previously transmitted of second rear view section corresponding to the environment,
The playback duration is measured relative to the playback duration indicated in the first content stream.It is interior in step 2044
Hold playback system receive image selection information, the image selection information instruction during a part of the event should use pair
It should be in which of the multiple images of second rear view section of environment.
In step 2046, content playback device receives control information, and control information instruction is answered during playback duration
When the multiple images which of that had previously transmitted of the display corresponding to the sky aerial view part of the environment, the playback
Between be to be measured relative to the playback duration indicated in the first content stream.In step 2048, content playback system
System receives image selection information, and image selection information instruction should use during a part of the event corresponds to environment
The sky aerial view part which of multiple images.
In step 2050, contents playback system receives control information, and control information instruction is answered during playback duration
When the multiple images which of that had previously transmitted of the display corresponding to the ground View component of the environment, the playback
Between be to be measured relative to the playback duration indicated in the first content stream.In step 2052, content playback system
System receives image selection information, and image selection information instruction should use during a part of the event corresponds to environment
The ground View component which of multiple images.
Operate from step 2032, step 2034 and step (can execute parallel 2038,2040,2042,2044,2046,
2048,2050 and 2052, step 2058 is proceeded to via connecting node D 2056 via connecting node C2054).
In some embodiments, step 2038,2040,2042,2044,2046,2048,2050 and 2052 control are come from
Information by for example one by one in a manner of slightly sent prior to corresponding first flow content of step 2034, for this it is first-class in
Hold, control information will be used.In some other embodiments, control block of information start receive first content stream before or with
It is received simultaneously.
In step 2058, contents playback system based on it is following at least one generating corresponding to one of present viewing field or
Multiple output images: the institute of the first content stream from the first part's view (for example, forward portion view) for corresponding to environment
Reception content, corresponding to environment the first rear view section storage received image, corresponding to the second rearview of environment
The received image of partial storage, corresponding to environment sky aerial view part storage received image, correspond to environment
Ground View component storage received image, or the composite diagram of not available a part of image corresponding to present viewing field
Picture.Step 2058 includes step 2060,2062,2064,2066,2068,2076 and 2078.
In step 2060, contents playback system will be in generating one or more output images based on present viewing field determination
The set of the View component (there are data availables for example, to it) used.The set of some exemplary determinations includes for example: { },
{ front view part }, { the first rear view section }, { the second rear view section }, { sky aerial view part }, { ground View component },
{ front view part, sky aerial view part }, { front view part, ground View component }, { front view part, the first rearview portion
Point }, { front view part, the second rear view section }, { front view part, the first rear view section, sky portion }, { front view
Partially, the second rear view section, sky aerial view part }, { front view part, the first rear view section, ground View component },
{ front view part, the second rear view section, ground View component }, { the first rear view section, sky aerial view part }, { first
Rear view section, ground View component }, { the first rear view section, the second rear view section }, { the first rear view section, second
Rear view section, sky aerial view part }, { the first rear view section, the second rear view section, ground View component }, { after second
View component, sky aerial view part } and { the second rear view section, ground View component }.
Operation proceeds to step 2062 from step 2060.In step 2062, contents playback system determine whether meet with
Lower two conditions: the set for the View component that (i) is determined from step 2060 only includes first view part and (ii) in the first view
There is no the part of present viewing field except figure part.If it is determined that identified set is only including first view part and first
The part of present viewing field is not present except View component, then operation proceeds to step 2064 from step 2062；Otherwise, it operates from step
Rapid 2062 proceed to step 2066.
In step 2064, contents playback system, which is based on generating from the received content of first content stream, corresponds to present viewing field
One or more output images.
In step 2066, contents playback system determines whether there is any portion of the not available present viewing field of image
Point.If contents playback system is determined there are at least one portion of the not available present viewing field of image, operate from step
2066 proceed to step 2076；Otherwise, operation proceeds to step 2068 from step 2066.
In step 2068, contents playback system is determined based on to be used when generating one or more output images
The set of View component generate one or more output images corresponding to present viewing field.Step 2068 (and can have
When it is certain) include step 2070, wherein contents playback system is based on being included at least some of first content stream and receiving
Content and the content stored of the second part corresponding to the environment generate one or more corresponding to present viewing field
A output image.In some embodiments, step 2070 includes one or two of step 2072 and 2074.In step 2072
In, contents playback system selects the image of the second part view corresponding to environment based on the image selection information received.
Operation proceeds to step 2074 from step 2072.In step 2074, contents playback system will be from capturing at the second time point
The content that the first content stream obtains is combined with the first image for corresponding to the first time point, the first time point and
Second time point is different.
In some embodiments, the first image is the first image of the second part of environment, and second part is environment
The first rear view section and one of the second rear view section of environment.In some such embodiments, first time point pair
It should be in the time before the second time point.In some such embodiments, first time point is before the time of live event,
The image in first content stream is captured during the live event.
Back to step 2076, each part not available to image executes step 2076.In step 2076, content is returned
Place system is the not available partially synthetic image of image of the present viewing field.Operation proceeds to step 2078 from step 2076.
In step 2078, contents playback system is based on generating one or more output images and/or one or more composographs
In the determined View component set of environment to be used generate one or more output images corresponding to present viewing field.By
The output image that step 2078 generates may include: the image of complete synthesis；Including the content from composograph and from the
The image of the content of one content stream；Including the content from composograph, the content from first content stream and from storage
The image of the content of image；It and include the image of the content from composograph and the content from storage image.Various
In embodiment, step 2078 can (and sometimes really) include one or two of step 2080 and 2082.
In step 2080, contents playback system is based on including the content received at least some of first content stream
It is defeated corresponding to the one or more of present viewing field to generate with the composograph of a part (for example, second part) of simulated environment
Image out.In step 2082, content playback device combines composograph at least part of the image received, with life
At the image for corresponding to present viewing field.
It should be appreciated that present viewing field can (and usually really) change over time.In response to the change of present viewing field,
Determination the different sets for exporting the View component of image can be used to generate in step 2060, different images may need
It wants and is synthesized in step 2076, for example, corresponding to the different piece of the not image in the visual field.In addition, in different time, base
In the control information received, different storage images can be identified for generating the combination output image in different time.
Operation proceeds to step 2086 from step 2058 via connecting node E 2084, and wherein contents playback system exports
And/or display one or more output images generated.Step 2086 includes step 2088, and wherein contents playback system exports
And/or display the first output image, the first output image are one or more one of output images generated.
In some embodiments, such as in step 2070 the output image of the generation corresponding to present viewing field generated can
It include the letter of the Part III of first part from environment, the second part of environment and environment with (and sometimes really)
Breath.In some embodiments, the first part corresponding to the environment of first content stream is front view part, the second part of environment
It is the first rear view section (for example, right rear corner pictorial view part) and the second rear view section (for example, left back View component) of environment
One of, the Part III of environment is one of sky aerial view part and ground View component of environment.In some such embodiments
In, the content corresponding to the first part includes capture and being streamed to the reality of the playback system when event is carrying out
When content, and correspond to described second and the content of Part III be non-real-time images.
In various embodiments, combined arrangement with generate correspond to present viewing field output image be included in one or more
Filtering, fuzzy, brightness change and/or color change, such as the frontier district between following any two are executed in borderline region
In domain: from correspond to environment front view part first content stream obtain image, corresponding to the first rearview portion of environment
Point storage image, corresponding to environment the second rear view section storage image, sky aerial view portion corresponding to environment
Point storage image, corresponding to environment ground View component storage image, and correspond to present viewing field in it
There is no the composographs in the region of image.
In some embodiments, there is first (for example, the front view part) covering for not corresponded to first content stream
Environment some parts, and for which stores one or more extentions of the image received.In various embodiments
In, the part uncovered for those of environment, image is synthesized.For example, in one embodiment, can be not present and correspond to
Storage image in sky aerial view part, and when present viewing field includes a part of sky aerial view, image is synthesized.Another
In one example, may exist dead angle, such as uncovered area between the first rear view section and the second rear view section
Domain.In some embodiments, composograph includes a part for repeating the image of the adjacent area corresponded in environment, such as is deposited
A part of a part of the image of storage or the image obtained from the first content stream received.
In some embodiments, the first picture material received in step 2034 be include left-eye image and right eye figure
The stereoscopic image content of picture, for example, left eye and eye image are to being received.In some such embodiments, it is received and deposits
That stores up, which corresponds to the image of the first rear view section, the second rear view section, sky aerial view part and ground View component, includes
Such as pairs of left-eye image and eye image.Therefore, correspond to one of present viewing field when generating for example in step 2070
Or when multiple output images, the left eye figure from the first content stream corresponding to first view part (for example, front view part)
As being combined with the left-eye image of the storage for the one or more of the other part for corresponding to environment, from corresponding to first view part
The storage of the eye image of the first content stream of (for example, front view part) and the one or more of the other part for corresponding to environment
Eye image combination.
In some other embodiments, be received and stored correspond to the first rear view section, the second rear view section,
The image of sky aerial view part and ground View component includes left-eye image from original image pair or eye image or packet
Single (monochrome) image is included, for example, capturing the single camera of View component from individually operated.Therefore, in such implementation
In example, when generating the one or more output images for corresponding to present viewing field for example in step 2070, first content is come from
The left-eye image of (for example, corresponding to front view part) and the corresponding eye image from first content stream are flowed all and from another
The identical storage image of one View component combines.
In yet another embodiment, the image of some storages includes left eye and eye image pair, and other images include
The image individually stored.For example, the reception image stored for corresponding to the first rear view section may include left eye and right eye
Image pair, and corresponding to the reception image that sky aerial view part is stored may include single image, such as monochrome image, without
It is pairs of.
In some embodiments, first part's (for example, corresponding to the received first content stream of institute) of environment is front view
Part, the second part of environment are rear view sections (for example, corresponding to the first rearview portion of the right back corresponding to viewer
Point or the left back corresponding to viewer the second rear view section), the Part III of environment is sky aerial view part or ground
Face View component.In some such embodiments, image is connect with the different rates for corresponding to the first, second, and third part
It receives, more images are corresponding to the first part rather than the event of the second part is received.In some such realities
It applies in example, receives and correspond to the second part rather than more images of the Part III.
According to some embodiments, the illustrative methods of operation content playback system comprise determining that the head position of viewer,
The head position corresponds to present viewing field；It receives and the first content stream of the content for the first part for corresponding to environment is provided；Base
In including the content received at least some of described first content stream and i) correspond to the second part of the environment
Storage interior perhaps ii) the simulation environment second part composograph, generate one for corresponding to present viewing field or
Multiple output images；And exporting or show the first output image, the first output image is the defeated of one or more generations
One of image out.In some embodiments, contents playback system is content playback device.In some embodiments, content playback
System is coupled to the computer system of display.
In various embodiments, this method further include: receive the first figure for corresponding to the second part of the environment
Picture；And storage corresponds to the first image of the second part of the environment.
In some embodiments, the first image of the second part of the environment corresponds to first time point；
Generating corresponding to one or more output images of present viewing field includes combining out of, capture at the second time point described first
Hold the content and the first image for corresponding to the first time point, the first time point and the second time point that stream obtains
It is different.In some such embodiments, the first time point corresponds to the time before second time point.Some
In such embodiment, the first time point captured during the live event and is included in front of the time of live event
Image in the first content stream.
In various embodiments, this method further includes one or more for receiving the second part for corresponding to the environment
A additional image, one or more of additional images correspond to the second part of the environment, include at least the second figure
Picture.
In some embodiments, this method includes receiving control information, and control information instruction corresponds to the institute of the environment
Which of the multiple images previously transmitted for stating second part should be shown during playback duration, which is
It is measured relative to the playback duration indicated in the first content stream.
In some embodiments, the second part of the environment is the first rear view section, the second rear view section, sky
One of View component or ground View component.In some such embodiments, this method further include: receive and correspond to the ring
One or more images of the Part III in border.
In various embodiments, the first part of the environment is front view part；The Part III is sky
One of view or ground View component；And image is connect with the different rates for corresponding to first, second, and third part
It receives, more images are corresponding to the first part rather than the event of the second part receives.
In various embodiments, it is included in when event is carrying out corresponding to the content of the first part and is captured
And it is streamed to the real time content of the playback apparatus；Content corresponding with the described image for corresponding to described second and Part III
It is non-real-time images.In some such embodiments, the multiple images that instruction corresponds to the second part of environment are received
Which of the image selection information that should be used during a part of the event；And
Being generated based at least some contents received corresponding to one or more output images of present viewing field includes: base
Correspond to the image of the second part of environment in the image selection information selection received.
In various embodiments, illustrative methods include determining image to a part of unavailable of the present viewing field；It closes
At image of the image to its not available part that be used for the present viewing field；And combinatorial compound image with receive
Image at least part, with generate correspond to present viewing field image.
In various embodiments, the first image content is to include in the stereo-picture of left-eye image and eye image
Hold.
Figure 21 shows the example content playback system 2100 realized according to the present invention, can be used to receive, decode,
Storage is handled and is shown from content delivery system (content delivery system shown in such as Fig. 1 and 7) received image content.
System 2100 can be implemented as include display 2102 single playback apparatus 2100', or be embodied as such as being coupled to computer
The combination of the element of the external display (for example, head-mounted display 2105) of system 2100'.
In at least some embodiments, contents playback system 2100 includes 3D head-mounted display.Head-mounted display can
To utilize the OCULUSRIFT that may include head-mounted display 2105TMVR (virtual reality) earphone is realized.In various implementations
In example, head-mounted display 2105 is identical as head-mounted display 805.Other head-mounted displays also can be used.In some realities
It applies in example, the wear-type helmet or other headset equipments, wherein one or more display screens are used to left eye and the right side to user
Eye display content.By showing different images to left eye and right eye on single screen, wherein headset equipment is configured as
The different piece of single screen is exposed to different eyes, individual monitor can be used show will be by the left eye of viewer
The left eye and eye image perceived respectively with right eye.In some embodiments, mobile phone screen is used as head-mounted display apparatus
Display.In at least some such embodiments, mobile phone is inserted into headset equipment and mobile phone is used to display image.
Contents playback system 2100, which has, to be decoded the coded image data received and generates for showing to consumer
3D rendering content ability, wherein coded image data such as left eye and eye image and/or corresponding to environment or scene
The monophonic (single image) of different piece, wherein display is for example to be regarded by the different left eyes and right eye that perceive user
Figure renders and is shown as 3D rendering.In some embodiments, contents playback system 2100 is located at consumer guard station position, such as family
Or office, but image capture place can also be located at.According to the present invention, contents playback system 2100 can be executed according to this hair
Bright signal is received, decoded, showing and/or other operations.
System 2100 connects including display 2102, display device interfaces 2103, input equipment 2104, input/output (I/O)
Mouth 2106, processor 2108, network interface 2110 and memory 2112.Memory 2112 includes modular assembly 2114, such as soft
Part modular assembly and data/information 2116.In some embodiments, system 2100 includes modular assembly 2115, such as hardware mould
Block (such as circuit) component.The various parts of system 2100 are via the bus for allowing data to communicate between the component of system 2100
It 2109 and/or is coupled by other connections or by wireless interface.Although display 2102 is made in some embodiments
Included for optional element, as shown in using dotted line frame, but in some embodiments, external display device 2105, such as
Wear-type stereoscopic display device can be coupled to playback apparatus via display device interfaces 2103.
For example, being used as processor 2108 in cell phone processor and mobile phone generates in headset equipment and show image
In the case where, system may include processor 2108, display 2102 and memory 2112, and one as headset equipment
Point.Processor 2108, display 2102 and memory 2112 can be a part of mobile phone.In other implementations of system 2100
In example, processor 2108 can be a part of the game system of such as XBOX or PS4, and wherein display 2105, which is mounted on, wears
In formula equipment and it is coupled to game system.Whether processor 2108 and/or memory 2112, which are located at, is worn overhead equipment
Although it is not crucial in also, as it would be appreciated, co-locates 2108 meeting of processor in headwear in some cases
It is convenient, but from the perspective of power, heat and weight, at least some cases, it may be desirable to make processor
2108 and memory 2112 be coupled to the headwear including display.
Although various embodiments contemplate head-mounted display 2105 or 2102, this method and device can also with can
The non-head-mounted display of 3D rendering is supported to be used together.Thus, although system 2100 includes wear-type in many examples
Display, but it can also be realized using non-head-mounted display.
Memory 2112 includes that various modules, such as routine control content playback system when it is executed by processor 2108
System 2100 executes operations according to the instant invention.Memory 2112 includes modular assembly 2114, such as software module components, and number
According to/information 2116.
Data information 2116 includes following one or more: the reception image, right corresponding to the first rear view section 2118
Image should be received, corresponding to the reception image of sky aerial view part 2122 and corresponding to ground in the second rear view section 2120
The reception image of face View component 2124.Exemplary receiver image corresponding to rear view section 2118 or 2120 includes for example seeing
The image that image that image, spectators or the crowd that many or crowd stands are seated, the image with different visible advertisements, crowd hail
Deng.Exemplary receiver image corresponding to sky aerial view include such as clear sky, different cloud atlas cases, corresponding to different time not
With darkness etc..Data information 2116 further includes following one or more or whole: what is received corresponds to the first rear view section
2130 control information, receive corresponding to the second rear view section 2132 control information, receive correspond to sky
The control information of View component 2134, and the control information corresponding to ground View component 2136 received.Data information
2116 further include determining current viewer's head position 2126, the present viewing field 2128 determined, received first content stream
2128 (e.g., including corresponding to front view part eye image and left-eye image to), for generating output image 2138
(for example, corresponding to present viewing field, for the present viewing field, there is can be used for combining at least one in determining View component set
A little reception contents), composograph 2140, and generate output image 2142.The output image 2142 of generation.The output figure of generation
As can (and sometimes really) include the output image generated, the output image of the generation includes from first content stream
It combines picture material (for example, corresponding to first, (such as front view) part), and the storage image including receiving (for example,
From rear view section, sky portion or above ground portion) a part, and/or one including composograph or composograph
Point.
Figure 22 is the example content playback system 2100 that can be included in Figure 21 or Fig. 8 accoding to exemplary embodiment
System 800 in modular assembly 2200 figure.Module in modular assembly 2200 can (and in some embodiments really
In fact) with hardware realization completely in processor 2108, such as individual circuit.Module in modular assembly 2200 can be with
(and certain in some embodiments) corresponds to disparate modules completely in modular assembly 2115 with hardware realization, such as conduct
Independent circuit.In other embodiments, some modules are implemented as such as circuit in processor 2108, and other module quilts
The circuit being embodied as in such as modular assembly 2115 in 2108 outside of processor and is coupled to processor 2108.It should such as manage
Solution, the level in the integrated horizontal of processor upper module and/or some modules portion outside the processor can be one of design alternative.
As an alternative, it is not implemented as circuit, but all or some modules can be implemented in software and be stored
In the memory 2112 of system 2100, wherein when module is executed by processor (for example, processor 2108), module control system
The operation of system 2100 corresponds to the function of module to realize.In some such embodiments, modular assembly 2200 is included in
In memory 2112, as modular assembly 2114.There are also in other embodiments, the various modules in modular assembly 2200 are by reality
It is now the combination of hardware and software, for example, outside the processor then another circuit in portion is handled to the offer input of processor 2108
Device 2108 operates under software control, to execute a part of execution module function.Although being illustrated as in the embodiment of figure 21
Single processor, such as computer, but it is to be understood that processor 2108 may be implemented as one or more processors, example
Such as computer.
When implemented in software, module includes code, and when code is executed by processor 2108, configuration processor 2108 is real
Now correspond to the function of the module.In the embodiment that modular assembly 2200 is stored in memory 2112, memory 2112 is
Computer program product including computer-readable medium is for realizing at least one computer (such as processor 2108)
The corresponding function of module, the computer-readable medium include code, such as the independent code of each module.
Completely hardware-based or fully software-based module can be used.It is understood, however, that software and hardware is appointed
What combination (such as module of circuit realization) can be used to realize function.As it should be appreciated, the module control shown in Figure 22
System and/or configuration system 2100 or in which element (such as processor 2108), Lai Zhihang is in the side of the flow chart 2000 of Figure 20
The function for the correspondence step for showing and/or describing in method.Therefore, modular assembly 2200 include execute corresponding to one of Figure 20 or
The various modules of the function of multiple steps.
Modular assembly 2200 includes observer's head position determining module 2202, present viewing field determining module 2204, content
It flows selecting module 2206, content stream receiving module 2208, image receiver module 2210, receive image storage module 2212, control
Information receiving module 2214, output image generation module 2216, output module 2242, display module 2244 and control routine
2246。
Viewer's head position determining module 2202 is configured to determine that the head position of viewer, the head position pair
It should be in present viewing field.Present viewing field determining module 2204 is configured as determining the observer based on identified head position
Present viewing field.
Content stream selecting module 2206 is configured as example inputting based on user (for example, viewer) come out of multiple substitutions
Rong Liuzhong selects content stream.Different content streams can correspond to different events.In various embodiments, corresponding to mutually working together
The different content stream of part corresponds to the different cameral being pointed in different directions, for example, to provide the front view view of substitution to viewer
Angle.In some embodiments, at least some of the image stream that can be selected include comprising left-eye image to and eye image pair
Stereoscopic image content.
Content stream receiving module 2208, which is configured as receiving, provides the first part for corresponding to environment (for example, forward portion
View) content first content stream.In various embodiments, received first picture material is to include in first content stream
The stereoscopic image content of left-eye image and eye image pair.
Image receiver module 2210, which is configured to receive, corresponds to one or more different views part (for example, after first
View component, the second rear view section, sky aerial view part and ground View component) image.2210 quilt of image receiver module
It is configured to receive the first image of the second part for corresponding to the environment.In some such embodiments, image receives mould
Block is additionally configured to receive the one or more additional images for the second part for corresponding to the environment, one or more
A additional image corresponds to the second part of the environment, including at least the second image.In some embodiments, the ring
The second part in border is one of the first rear view section, the second rear view section, sky aerial view part or ground View component.?
In some embodiments, the second part of the environment is one of the first rear view section or the second rear view section.In some realities
It applies in example, image receiver module 2210 is configured as receiving the one or more images for the Part III for corresponding to the environment.
In some embodiments, the first part of environment is front view part, and the Part III of environment is sky aerial view or ground view
One of part.
In some embodiments, image receiver module 2210 is configured as receiving the first rearview for corresponding to the environment
The first partial image, and it is additionally configured to one or more that reception corresponds to first rear view section of the environment
A additional image, one or more of additional images corresponding to first rear view section include corresponding to described first
At least the second image of rear view section.In some embodiments, image receiver module 2210, which is configured as receiving, corresponds to institute
State the first image of the second rear view section of environment, and be additionally configured to receive correspond to the environment described second after
One or more additional images of View component, one or more of additional images corresponding to second rear view section
Including at least the second image for corresponding to second rear view section.In some embodiments, 2210 quilt of image receiver module
It is configured to receive the one or more images for the sky aerial view part for corresponding to the environment.In some embodiments, image connects
Module 2210 is received to be configured as receiving the one or more images for the ground View component for corresponding to the environment.
Received image storage module 2212 is configured as storage by the received image of image receiver module 2212.It is received
Image storage module 2212 is configured as the first image that storage corresponds to the second part of the environment.Received image
Memory module 2212 is configured as one or more additional images that storage corresponds to the second part of environment.Received image is deposited
Storage module 2212 is configured as the one or more images for the Part III corresponding to the environment that storage receives.Various
In embodiment, received image storage module 2212 is configured as first rear view section that storage corresponds to the environment
The first image, and one or more additional images of first rear view section corresponding to the environment.Various
In embodiment, receives image storage module 2212 and be configured as storing second rear view section for corresponding to the environment
First image, and one or more additional images of second rear view section corresponding to the environment.In some realities
It applies in example, received image storage module 2212 is configured as one that storage corresponds to the sky aerial view part of the environment
A or multiple images.In some embodiments, received image storage module 2212 is configured as storage corresponding to the environment
The ground View component one or more images.
Control information receiving module 2214 is configured as receiving control information, and control information instruction corresponds to the environment
Which of multiple images previously transmitted of second part should be shown during playback duration, which is
It is measured relative to the playback duration indicated in first content stream.In various embodiments, information receiving module is controlled
2214 are additionally configured to receive control information, and control information instruction corresponds to multiple first forward pass of the Part III of the environment
Which of image sent should be shown during playback duration, which referred to relative in first content stream
What the playback duration shown measured.In some embodiments, control information receiving module 2214 is configured as receiving image choosing
Information is selected, which of the multiple images of second part that image selection information instruction corresponds to environment should be in event
It is used during a part.In some embodiments, control information receiving module 2214 is configured as receiving image selection information,
Which of the multiple images of Part III that image selection information instruction corresponds to environment should be in a part of event
Period is used.
In some embodiments, control information receiving module 2214 is configured as receiving control information, which refers to
Showing which of multiple images previously transmitted of first rear view section corresponding to the environment should play back
It is shown during time, which measured relative to the playback duration in the first content stream.One
In a little embodiments, control information receiving module 2214 is configured as receiving image selection information, image selection information instruction pair
It should should be used during a part in event in which of multiple images of first rear view section of environment.
In some embodiments, control information receiving module 2214 is configured as receiving control information, and control information instruction corresponds to
Which of multiple images previously transmitted of second rear view section of the environment should be during playback duration
Shown, which measured relative to the playback duration in the first content stream.In some embodiments
In, control information receiving module 2214 is configured as receiving image selection information, and image selection information instruction corresponds to environment
Which of multiple images of second rear view section should be used during a part of event.
In some embodiments, control information receiving module 2214 is configured as receiving control information, which refers to
Show which of multiple images previously transmitted of the sky aerial view part corresponding to the environment should be in playback
Between during it is shown, which measured relative to the playback duration in the first content stream.Some
In embodiment, control information receiving module 2214 is configured as receiving image selection information, and image selection information instruction corresponds to
It should be used during a part in event in which of multiple images of the sky aerial view part of environment.One
In a little embodiments, control information receiving module 2214 is configured as receiving control information, and control information instruction corresponds to described
Which of multiple images previously transmitted of the ground View component of environment should be shown during playback duration,
The playback duration is measured relative to the playback duration in the first content stream.In some embodiments, it controls
Information receiving module 2214 be configured as receive image selection information, the image selection information instruction correspond to environment describedly
Which should be used during a part of event in the multiple images of face View component.
Output image generation module 2216 is configured to generate based at least one of the following corresponding to present viewing field
One or more output images: the reception content of the first content stream from the first part's view for corresponding to environment is deposited
The reception image of the first rear view section corresponding to environment of storage, the second rear view section corresponding to environment stored
Receive image, the sky aerial view part corresponding to environment stored reception image, the ground corresponding to environment that is stored
The reception image of View component, or the composograph of the not available part of image corresponding to present viewing field.Image is exported to generate
Module 2216 includes View component set determining module 2218, only content stream determining module 2220, lack part determining module
2222, image compositer module 2224, content stream output image generation module 2226, synthesis output image generation module 2228,
Based on the generation module 2230 of output picture material stream, and the generation module 2236 based on output image non-streaming.
View component set determining module 2218 is configured as being used to generate one or more based on present viewing field determination
The View component set of the environment of image is exported, such as there are the view portions of at least some available picture materials to it
Point.The set of some exemplary determinations includes for example: { }, { front view part }, { the first rear view section }, { the second rearview
Part }, { sky aerial view part }, { ground View component }, { front view part, sky aerial view part }, front view part,
Face View component }, { front view part, the first rear view section }, { front view part, the second rear view section }, { front view portion
Point, the first rear view section, sky portion, { front view part, the second rear view section, sky aerial view part }, { front view
Partially, the first rear view section, ground View component }, { front view part, the second rear view section, ground View component },
{ the first rear view section, sky aerial view part }, { the first rear view section, ground View component }, the first rear view section,
Second rear view section }, { the first rear view section, the second rear view section, sky aerial view part }, the first rear view section,
Second rear view section, ground View component }, { the second rear view section, sky aerial view part } and the second rear view section,
Ground View component }.
Only content stream determining module 2220 is configured as determining whether to generation one based on the content in first content stream
A or multiple output images, without relying on from other parts view (such as the first rear view section, the second rearview portion
Point, sky aerial view part or ground View component) the image for receiving image or synthesis stored.Only content stream determining module
2220 are configured to check for whether identified set includes as the first (for example, preceding) the view portion for corresponding to first content stream
The individual element divided, and check the visual field whether in first (for example, preceding) View component.
Lack part determining module 2222 be configured to determine that image for a part of unavailable of present viewing field, for example,
Image or from correspond to environment front view first content stream or from correspond to environment another part received storage
Image is unavailable.Image compositer module 2224 is configured as the not available part of image that synthesis will be used for present viewing field
Image.In various embodiments, image synthesis module 2224 is generated slightly larger than composograph needed for filling lack part, example
Such as, to allow some whole boundaries.
Content stream output image generation module 2226 is configured as, when identified set only includes first view part
(for example, front view part) and the present viewing field such as determined by only content stream determining module 2220 be not in first view part
Except part when, be based only upon reception content from first content stream and generate the one or more outputs for corresponding to present viewing field
Image.In some embodiments, content stream output image generation module 2226 executes sanction to the image obtained from first content stream
Cut operation.
Generation module 2230 based on output picture material stream is configured as being based on being included in first content stream (for example, right
Should be in the content stream of front view) at least some of received content and i) corresponding to environment second part storage it is interior
Hold (such as storage image corresponding to the first rearview, the second rearview, sky aerial view or ground view) or ii) simulated environment
The composograph (such as composograph of a part in the not available visual field of analog image) of second part correspond to generate
One or more output images of present viewing field.The storage content of second part corresponding to environment is for example to be deposited by reception image
Store up the storage image that module 2212 stores.Composograph is, for example, the image generated by module 2224.In some embodiments, base
It is configured as selecting to correspond to ring based on the received image selection information of institute in the generation module 2230 of output picture material stream
The image of the second part in border, one as the one or more output images being configurable to generate corresponding to present viewing field
Part.
Generation module 2230 based on output picture material stream includes being configured as combining one or more composographs
To the composograph binding modules 2232 in the output image generated.Composograph binding modules 2232 are configured as to close
At image and the image received (for example, come first (for example, preceding) View component obtained since the first content stream received
Reception image) at least part or correspond to the first rear view section, the second rear view section, sky aerial view part or
The reception image of one of ground View component stored is combined, to generate the image for corresponding to present viewing field.Storage figure
As binding modules 2134 are configured as a part of one or more storage images being integrated in the output image generated.
As a part for generating output image, module 2230, module 2232 and/or module 2234 execute mixing in borderline region.?
In various embodiments, mixing includes filtering, fuzzy, brightness change and/or color change.
Generation module 2236 based on output image non-streaming be configured as based on it is following at least one generate corresponding to working as
One or more output images of forward view: the i) storage content corresponding to a part of environment, such as corresponding to the first backsight
The storage image or ii of figure, the second rearview, sky aerial view or ground view) simulated environment a part composograph, example
Such as the composograph of a part in the not available visual field of analog image.In present viewing field corresponding to corresponding with first content stream
When except the region of first (for example, front view) part, module 2236 generates one or more images.In some embodiments,
Generation module 2236 based on output image non-content stream is configured as selecting to correspond to based on the received image selection information of institute
In the image of the second part of environment, as the one or more output images being configurable to generate corresponding to present viewing field
A part.
Generation module 2236 based on output image non-streaming includes being configured as one or more composographs being integrated to
The composograph binding modules 2238 in output image generated.Composograph binding modules 2238 are configured as to synthesize
Image and reception image are (for example, come from the first rear view section, the second rear view section, sky aerial view part or ground view
Partial reception image) at least part combination, with generate correspond to present viewing field image.The image binding modules of storage
2240 are configured as a part of the image of one or more storage being integrated in the output image generated.As generation
A part of image is exported, module 2236, module 2232 and/or module 2234 execute mixing in borderline region.In various implementations
In example, mixing includes filtering, fuzzy, brightness change and/or color change.
Output module 2242 is configured as output for example by defeated based on output picture material stream generation module 2130, content stream
Image generation module 2130 and the one or more generated based on output image non-content stream generation module 2236 are generated defeated out
Image out, one or more of output images include the first output image.Output module is configured as example connecing via network
Mouth 2110 and/or the output of display device interfaces 2,103 first output image.
Display module 2244 is configured as display for example by the generation module 2230 based on output picture material stream, content stream
Export one or more generations that image generation module 2230 and the generation module 2136 based on output image non-content stream generate
Output image, one or more of output images include first output image.Display module 2244 be configured as example through
By display 2102 and/or the display of display 2,105 first output image.
Control routine 2246 includes equipment control routine and Communications routines, with the operation of control system 2100.
According to some embodiments, example content playback system (such as system 2100 of Figure 21) includes: to be configured as really
Determine viewer's head position determining module 2202 of the head position of viewer, the head position corresponds to present viewing field；Quilt
It is configured to receive the content stream receiving module of the first content stream for the content that the first part for corresponding to environment is provided；Based on output
The generation module 2230 of picture material stream is configured as received based at least some of described first content stream is included in
Content and i) storage content or ii corresponding to the second part of the environment) the simulation environment second part synthesis
Image exports images corresponding to the one or more of present viewing field to generate；And it is following at least one: be configured as output the
The output module 2242 of one output image or the display module 2244 for being configured as display the first output image, described first is defeated
Image is one or more one of output images generated out.
In some embodiments, the contents playback system 2100 is content playback device 2100'.In some embodiments
In, the contents playback system 2100 is coupled to the computer system 2100' of display 2105.
In some embodiments, system further include: be configured as receiving the second part for corresponding to the environment
The first image image receiver module 2210；And it is configured as the institute that storage corresponds to the second part of the environment
State the reception image storage module 2212 of the first image.
In various embodiments, the first image of the second part of the environment corresponds to first time point；
The generation module 2230 based on output picture material stream is configured as will be out of, capture at the second time point described first
Hold the content that stream obtains to combine with the first image for corresponding to the first time point, first and second described time point is not
Together.In some such embodiments, the first time point corresponds to the time before second time point.In various realities
It applies in example, the first time point is before the time of live event, and capture is included in described first between the live event
Image in content stream.
In some embodiments, described image receiving module 2210, which is additionally configured to receive, corresponds to the described of the environment
One or more additional images of second part, one or more of additional images correspond to the institute including at least the second image
State the second part of environment.
In various embodiments, system further include: it is configured as receiving the control information receiving module 2214 of control information,
Multiple which of the images previously transmitted for the second part that control information instruction corresponds to the environment should
It is shown during playback duration, which surveyed relative to the playback duration indicated in the first content stream
Amount.
In some embodiments, the second part of the environment be the first rear view section, the second rear view section,
One of sky aerial view part or ground View component.In some such embodiments, described image receiving module 2210 goes back quilt
It is configured to receive the one or more images for the Part III for corresponding to the environment.
In some embodiments, the first part of the environment is front view part；The Part III is sky
One of view or ground View component；Image be with correspond to first, second, and third part different rates it is received,
More images are corresponding to the first part rather than the event of the second part receives.
In some embodiments, it is included in when event is carrying out corresponding to the content of the first part and is captured
And it is streamed to the real time content of the playback apparatus；And it is corresponding with the described image for corresponding to described second and Part III
The content is non-real-time images.In some such embodiments, the control information receiving module 2214 is additionally configured to
Image selection information is received, which indicates to correspond in the multiple images of the second part of the environment
Which should be used during a part of the event；2230 quilt of generation module based on output picture material stream
Be configured to received image selection information come select corresponding to the environment the second part image, as quilt
It is configured to generate a part for the one or more output images for corresponding to present viewing field.
In various embodiments, system further includes being configured to determine image to a part of not available scarce of the visual field
Lose portion determination module 2222；It is configured as synthesizing the figure of the image for the not available part of image that be used for the visual field
As Senthesizer module 2224；And it is configured as combining composograph to generate and correspond to at least part for receiving image
The composograph binding modules 2232 of the image of present viewing field.
In various embodiments, the first image content is to include in the stereo-picture of left-eye image and eye image
Hold.
In some embodiments, other than other modules shown in fig. 8, modular assembly 2200 is also included in Fig. 8
Contents playback system 800 in.Module in modular assembly 2200 may include in memory 812, in processor 808, and/
Or as the hardware module outside processor 808 for being coupled to bus 809 in system 800.
Figure 23 is to show the exemplary stream selecting module in accordance with some embodiments used in playback system 1900
1938 Figure 23 00.
Figure 24 is to show Figure 24 00 of exemplary stream prioritization module 2306, can be implemented as the stream selection mould of Figure 23
A part of block or as individual module.
Some embodiments are for including encoding and compressing the software of three-dimensional video-frequency for controlling computer or other equipment
Instruct the non-transient computer-readable medium of (for example, computer executable instructions) set.Other embodiments are to be directed to include
For controlling computer or other equipment in the software instruction of player end decoding and decompressed video (for example, computer can be held
Row instruction) set computer-readable medium.Although coding and compression as it is possible it is individually operated be mentioned, should manage
Solution, coding can be used to execute compression and therefore coding can include compression in some cases.Similarly, decoding can be with
It is related to decompressing.
The technology of various embodiments can use the combination of software, hardware and/or software and hardware to realize.Various implementations
Example is directed to device, such as image data processing system.Various embodiments are also directed to method, such as the method for processing image data.
Various embodiments are also directed to non-transient machine, the machine for example including the one or more steps for controlling machine implemented method
Computer-readable medium of device readable instruction, such as ROM, RAM, CD, hard disk etc..
Various features of the invention are realized using module.This module can (and in some embodiments really
It is real) it is implemented as software module.In other embodiments, module is with hardware realization.There are also in other embodiments, module is benefit
It is realized with the combination of software and hardware.In some embodiments, module is implemented as individual circuit, wherein each module
It is implemented as the circuit for the corresponding function of execution module.Various embodiments are it is contemplated that including wherein different moulds
Some embodiments that block is realized in different ways, for example, it is some within hardware, it is some in software, it is some use hardware and software
Combination.It should also be noted that with the software that executes on aageneral-purposeaprocessor on the contrary, routine and/or subroutine or by this example
Some steps of Cheng Zhihang can be realized in specialized hardware.This embodiment is kept within the scope of the invention.It is many above-mentioned
Method or method and step can use the machine being included in machine readable media (such as memory devices, such as RAM, floppy disk etc.)
Device executable instruction (such as software) Lai Shixian, to control machine (such as general purpose computer with or without additional firmware)
Realize all or part of the above method.Thus, in addition to others, present invention is more particularly directed towards include machine-executable instruction
Machine readable media, the machine-executable instruction is for making machine (for example, processor and associated hardware) to execute above-mentioned side
The one or more steps of method.
In view of above description, numerous additional variations of the method and apparatus of above-mentioned various embodiments are to those skilled in the art
Member will be apparent.These variations are considered in range.
Claims (41)
1. a kind of method for operating playback system, comprising:
Operation includes the content playback device of the interface comprising receiver, includes in the information for identify multiple content streams to receive
Hold stream information, the multiple content stream, which includes at least, provides the first of content corresponding with the capture first part of environment of image
Content stream, and the second content stream corresponding with the second part of the environment, the second part of the environment is described
The environment part different from the first part；
The content playback device is operated, to store the content stream information in memory；
The processor in the content playback device is operated, to execute the head position based on user, selection will receive described more
Use when which to play back at the first time in a content stream；And
The content playback device is operated, to receive one or more selected content streams, is used for playback.
2. the method as described in claim 1, wherein the selection includes:
The content playback device is operated, to execute the head position based on user, is prioritized content stream.
3. method according to claim 2, wherein the priorization includes:
Identification transmits one or more content streams of content corresponding with user's present viewing field, and the present viewing field is the environment
In the visual field；And
The size of the part of environment based on the offer of each content stream, is prioritized and provides scene corresponding with present viewing field portion
Point one or more content streams, the different size of part of some offer environment in each content stream.
4. method as claimed in claim 3, wherein being prioritized content stream further include:
The degree of approach or end rotation direction based on the image section and present viewing field transmitted at least one of are worked as, and are prioritized
Transmit one or more additional content streams of content corresponding with the part except the present viewing field.
5. method as claimed in claim 4,
It is wherein described to be prioritized the one or more additional contents for transmitting content corresponding with the part except the present viewing field
Stream includes:
It is prioritized the user for not being shown to the content playback device with the environment and the portion except the present viewing field
Divide at least one corresponding additional content stream；And
Transmission and the portion except the present viewing field are prioritized based on the degree of approach of the picture material and present viewing field that are transmitted
Divide one or more additional content streams of corresponding content, transmits the picture material close to the region of the present viewing field
Content stream is than simultaneously the content stream far from the present viewing field is assigned higher priority except the present viewing field.
6. method as claimed in claim 4, wherein priorization one or more additional content stream is the head based on user
Portion direction of rotation provides the content stream of the picture material except the present viewing field but on end rotation direction than providing
Other content streams of picture material on the direction in the external separate end rotation direction of the present viewing field are assigned higher
Priority.
7. method according to claim 2, wherein at least some of the multiple content stream transmits the difference of 360 degree of environment
It is partially separated version of code, the multiple content stream includes at least one content corresponding with each part of 360 degree of environment
Stream；And
Wherein the selection content stream includes selection multiple content streams corresponding with the different piece of 360 degree of environment.
8. method according to claim 2, wherein the selected step further include:
At the content playback device, the available bandwidth for being used for reception content stream is determined；And
Based on identified amount of bandwidth available, list corresponding with the first part of environment is selected from the multiple content stream
A content stream and single content stream corresponding with the second part of environment, the multiple content stream include the institute with environment
State the corresponding multiple content streams of first part and multiple content streams corresponding with the second part of environment.
9. method according to claim 8, further includes:
At the content playback device, based on the bandwidth constraint for being transmitted to the playback system, determine at least one
Hold the bandwidth of stream；And
Wherein it is based upon the bandwidth that at least one described content stream determines from the selection that the multiple content stream carries out.
10. method as claimed in claim 9, further includes:
Receive the bandwidth constraint for being directed to different view directions, each bandwidth constraint is specified to be used to receive offer correspond to it is described
The maximum bandwidth of one or more content streams of the content of the corresponding view direction of each bandwidth constraint.
11. method according to claim 8, wherein the selection carried out from multiple content streams with equal priority
Including selecting a content stream from the multiple content streams for be assigned highest priority, it is assigned each content of highest priority
Stream provides the content for corresponding to identical view direction.
12. method as claimed in claim 11, wherein the choosing carried out from multiple content streams with equal priority
It selects including selecting a content stream from the multiple content streams for be assigned the second high priority, is assigned the every of the second high priority
A content stream provides the content for corresponding to identical view direction.
13. the method as described in claim 1,
Wherein receiving includes identifying that the content stream information of multiple content streams is included at the content playback device, and receiving includes closing
The guide information of the information of received content stream can be selected in playback system.
14. method as claimed in claim 13, wherein the guide information includes the content listed for multiple and different events
The information of stream, the information include the flow identifier for multiple content streams corresponding with first event, the flow identifier packet
The first content flow identifier for including identification first content stream corresponding with the first part of environment of the generation first event, knows
Second content flow identifier of the second content stream not corresponding with the second part of the environment that the first event occurs, with
And the corresponding third content traffic identifier of corresponding with the second part of the environment that the first event occurs other content streams
Symbol, the information further include the data of instruction the second content stream corresponding with the second part of environment that the first event occurs
The information of rate and other data transfer rates, other data transfer rates are by the second part pair with the environment that first event occurs
The data transfer rate of the other content streams for the third content flow identifier identification answered, the data transfer rate of content stream and other data
Rate is different.
15. method as claimed in claim 14, wherein for first content stream, the guide information includes one of following: energy
It is added into receive the multicast address of the multicast group of first content stream, can be used to request access to for providing first content stream
The information of switched digital video channel or can be used to control playback system tuner be tuned to described the is broadcasted above
The channel tuning information of the broadcast channel of one content stream.
16. method as claimed in claim 15, further includes:
Start the delivery of selected content stream, the delivery of the selected content stream of starting includes sending signal to be added and correspond to described be chosen
The multicast group of content stream.
17. method as claimed in claim 15, further includes:
The delivery for starting selected content stream, the delivery for starting selected content stream includes the equipment transmission request into network, thus
The delivery for the exchange digital channel for requesting the selected content stream to be transmitted above.
18. the method as described in claim 1, wherein the environment is that camera capture is encoded to generate the multiple content stream
Image environment.
19. the method as described in claim 1, wherein the content playback device is wear-type content playback device.
20. the method as described in claim 1,
Wherein the environment is 360 degree of environment；And
Wherein the method also includes:
The received content of decoding institute, to generate the image of each part for 360 degree of environment；
Image of the storage for each part of 360 degree of environment in a buffer；And
Use buffered image final to render as needed come the present viewing field of the user based on the content playback device
Output image, the final output image includes less than complete 360 degree of environment.
21. method as claimed in claim 20, wherein the image of final rendering is left-eye image, this method further include:
Render different eye images；And
Both the left-eye image and the eye image are shown to the user of the content playback device.
22. the method as described in claim 1, further includes:
The content playback device is operated, set by the head position for the user that will test in the initial phase of operation
To zero degree environment viewing location before the viewing location of user is initialized as to zero degree viewing location before being set to.
23. method according to claim 2, wherein the content playback device includes head-mounted display, the wear-type
The position of display and the head position of user are corresponding.
24. a kind of playback system, comprising:
Content playback device, comprising:
Interface comprising receiver includes the content stream information for identifying the information of multiple content streams for receiving, the multiple interior
Hold stream and the first content stream for providing and capturing the corresponding content of the first part of environment of image be provided, and with the ring
Corresponding second content stream of the second part in border, the second part of the environment be the environment and the first part not
Same part；
Memory, for storing the content stream information；
Flow selecting module, be configured as the head position based on user select to receive in the multiple content stream which with
It is used when playing back at the first time；And
Interface including receiver, the receiver are configured as receiving one or more selected content streams, use for playback.
25. playback system as claimed in claim 24, wherein the stream selecting module includes:
Prioritization module is flowed, is configured as the head position based on user to be prioritized content stream.
26. playback system as claimed in claim 25, wherein the stream prioritization module includes:
Identification module is configured as one or more content streams that identification transmits content corresponding with user's present viewing field, described
Present viewing field is the visual field in the environment；And
Priority assignment module is configured as the size of the environment division provided based on each content stream to be prioritized offer and institute
One or more content streams of the corresponding scene parts of present viewing field are stated, the difference of some offer environment in each content stream is big
Small part.
27. playback system as claimed in claim 26, wherein the stream prioritization module further includes additional flow priority distribution
Module, the additional streams priority assignment module be configured as the degree of approach based on the image section and present viewing field transmitted or
End rotation direction at least one of is worked as, to be prioritized the one of transmission content corresponding with the part except the present viewing field
A or multiple additional content streams.
28. playback system as claimed in claim 27, wherein as transmission and the part except the present viewing field is prioritized
The part of one or more additional content streams of corresponding content, the additional streams priority assignment module are prioritized and the ring
Border is not shown to the user of the content playback device and part except the present viewing field, and corresponding at least one is additional
Content stream；And
Wherein the additional streams priority assignment module is configured as based on the picture material and the present viewing field transmitted
The degree of approach transmits one or more additional content streams of content corresponding with the part except the present viewing field to be prioritized, and passes
Send content stream close to the picture material in the region of the present viewing field than except present viewing field and far from present viewing field
Content stream is assigned higher priority.
29. playback system as claimed in claim 27, wherein the head of the additional streams priority assignment module based on user
Direction of rotation is prioritized one or more additional content streams, provides except the present viewing field but on end rotation direction
Picture material content stream than providing picture material on the direction in the external separate end rotation direction of present viewing field
Other content streams are assigned higher priority.
30. playback system as claimed in claim 25 has been assigned most wherein the stream selecting module is additionally configured to selection
One or more content streams of high priority.
31. playback system as claimed in claim 25, further includes:
Available bandwidth and data rate determination module are configured to determine that the available bandwidth for reception content stream；And
Wherein the stream selecting module includes selecting module, and the selecting module is configured as based on identified amount of bandwidth available
It is selected from multiple content streams with equal priority.
32. playback system as claimed in claim 31,
Wherein the stream selecting module further includes stream bandwidth determination module, and the stream bandwidth determination module is configured as based on transmission
The bandwidth at least one content stream is determined to the bandwidth constraint of the playback system；And
Wherein the selecting module is configured as being based further on as the determining bandwidth of at least one described content stream from phase
It is selected in the multiple content stream of same priority.
33. playback system as claimed in claim 32, further includes:
Wherein the receiver is additionally configured to receive the bandwidth constraint for being directed to different view directions, and each bandwidth constraint is specified to be wanted
For receiving the maximum for corresponding to one or more content streams of content of view direction corresponding with each bandwidth constraint
Bandwidth.
34. playback system as claimed in claim 31, wherein the selecting module is configured as from being assigned highest priority
Multiple content streams in select a content stream, as being configured as being selected from multiple content streams with equal priority
The a part selected, each content stream for being assigned highest priority provide the content for corresponding to identical view direction.
35. playback system as claimed in claim 34, wherein the selecting module is configured as, it is high preferential from being assigned second
A content stream is selected in multiple content streams of grade, as being configured as carrying out from multiple content streams with equal priority
A part of selection, each content stream for being assigned the second high priority provide the content for corresponding to identical view direction.
36. playback system as claimed in claim 24 provides wherein the receiver is additionally configured to receive about playback system
System can select the guide information of the information of received content stream.
37. playback system as claimed in claim 36, wherein the guide information includes that can be used to access to provide finger for it
The information of the content stream of southern information.
38. playback system as claimed in claim 37, wherein for first content stream, the guide information include it is following it
One: can be added into receive the multicast address of the multicast group of first content stream, can be used to request access to for providing in first
Hold stream switched digital video channel information or can be used to control playback system tuner be tuned to broadcasted above
The channel tuning information of the broadcast channel of the first content stream.
39. playback system as claimed in claim 38, further includes:
Content delivery starting module is configured as starting the delivery of selected content stream, and the content delivery starting module is also matched
It is set to and sends signal so that multicast group corresponding with the selected content stream is added.
40. playback system as claimed in claim 38, further includes:
Content delivery starting module is configured as starting the delivery of selected content stream, and the content delivery starting module is also matched
It is set to the equipment generated and into network and sends request, thus the exchange number for requesting the selected content stream to be transmitted above
The delivery of channel.
41. a kind of non-transient computer-readable medium including processor-executable instruction, described instruction is worked as to be held by processor
When row, playback system is controlled:
Receiving includes the content stream information for identifying the information of multiple content streams, and the multiple content stream includes at least offer and capture
The first content stream of the corresponding content of the first part of the environment of image, and corresponding with the second part of the environment second
Content stream, the second part of the environment are the environment parts different from the first part；
The content stream information is stored in memory；
Head position based on user, selection make when which receiving in the multiple content stream to play back at the first time
With；And
One or more selected content streams are received, are used for playback.
Applications Claiming Priority (3)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US201462004547P | 2014-05-29 | 2014-05-29 | |
| US62/004,547 | 2014-05-29 | ||
| PCT/US2015/033420 WO2015184416A1 (en) | 2014-05-29 | 2015-05-29 | Methods and apparatus for delivering content and/or playing back content |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106416239A CN106416239A (en) | 2017-02-15 |
| CN106416239B true CN106416239B (en) | 2019-04-09 |
Family
ID=54699946
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201580028645.9A Active CN106416239B (en) | 2014-05-29 | 2015-05-29 | Method and apparatus for delivering content and/or playing back content |
Country Status (6)
| Country | Link |
|---|---|
| EP (1) | EP3149937A4 (en) |
| JP (1) | JP2017527230A (en) |
| KR (2) | KR102407283B1 (en) |
| CN (1) | CN106416239B (en) |
| CA (1) | CA2948642A1 (en) |
| WO (1) | WO2015184416A1 (en) |
Families Citing this family (49)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10204658B2 (en) | 2014-07-14 | 2019-02-12 | Sony Interactive Entertainment Inc. | System and method for use in playing back panorama video content |
| CN106937128A (en) * | 2015-12-31 | 2017-07-07 | 幸福在线（北京）网络技术有限公司 | A kind of net cast method, server and system and associated uses |
| CN105791882B (en) * | 2016-03-22 | 2018-09-18 | 腾讯科技（深圳）有限公司 | Method for video coding and device |
| US9986221B2 (en) * | 2016-04-08 | 2018-05-29 | Visbit Inc. | View-aware 360 degree video streaming |
| US10587934B2 (en) | 2016-05-24 | 2020-03-10 | Qualcomm Incorporated | Virtual reality video signaling in dynamic adaptive streaming over HTTP |
| US10219014B2 (en) | 2016-06-02 | 2019-02-26 | Biamp Systems, LLC | Systems and methods for bandwidth-limited video transport |
| WO2018004934A1 (en) * | 2016-06-30 | 2018-01-04 | Sony Interactive Entertainment Inc. | Apparatus and method for capturing and displaying segmented content |
| US11089280B2 (en) * | 2016-06-30 | 2021-08-10 | Sony Interactive Entertainment Inc. | Apparatus and method for capturing and displaying segmented content |
| KR20180025797A (en) * | 2016-09-01 | 2018-03-09 | 삼성전자주식회사 | Method for Streaming Image and the Electronic Device supporting the same |
| EP4084480A1 (en) * | 2016-09-09 | 2022-11-02 | Vid Scale, Inc. | Methods and apparatus to reduce latency for 360-degree viewport adaptive streaming |
| WO2018050529A1 (en) * | 2016-09-13 | 2018-03-22 | Thomson Licensing | Method, apparatus and stream for immersive video format |
| CN109792537B (en) * | 2016-09-29 | 2023-06-23 | 皇家飞利浦有限公司 | Apparatus and method for providing image, and image processing apparatus and system |
| WO2018063957A1 (en) * | 2016-09-30 | 2018-04-05 | Silver VR Technologies, Inc. | Methods and systems for virtual reality streaming and replay of computer video games |
| KR102633595B1 (en) | 2016-11-21 | 2024-02-05 | 삼성전자주식회사 | Display apparatus and the control method thereof |
| KR20180059210A (en) * | 2016-11-25 | 2018-06-04 | 삼성전자주식회사 | Image processing apparatus and method for image processing thereof |
| US10244200B2 (en) | 2016-11-29 | 2019-03-26 | Microsoft Technology Licensing, Llc | View-dependent operations during playback of panoramic video |
| US10244215B2 (en) | 2016-11-29 | 2019-03-26 | Microsoft Technology Licensing, Llc | Re-projecting flat projections of pictures of panoramic video for rendering by application |
| CN108156484B (en) * | 2016-12-05 | 2022-01-14 | 奥多比公司 | Prioritizing tile-based virtual reality video streams with adaptive rate allocation |
| FI20165925L (en) * | 2016-12-05 | 2018-06-06 | Rolls Royce Oy Ab | Optimizing data stream transmissions from marine vessel |
| US10595069B2 (en) | 2016-12-05 | 2020-03-17 | Adobe Inc. | Prioritizing tile-based virtual reality video streaming using adaptive rate allocation |
| US10242714B2 (en) | 2016-12-19 | 2019-03-26 | Microsoft Technology Licensing, Llc | Interface for application-specified playback of panoramic video |
| US10999602B2 (en) | 2016-12-23 | 2021-05-04 | Apple Inc. | Sphere projected motion estimation/compensation and mode decision |
| US11259046B2 (en) * | 2017-02-15 | 2022-02-22 | Apple Inc. | Processing of equirectangular object data to compensate for distortion by spherical projections |
| US10924747B2 (en) | 2017-02-27 | 2021-02-16 | Apple Inc. | Video coding techniques for multi-view video |
| CN108668138B (en) * | 2017-03-28 | 2021-01-29 | 华为技术有限公司 | Video downloading method and user terminal |
| US10979663B2 (en) * | 2017-03-30 | 2021-04-13 | Yerba Buena Vr, Inc. | Methods and apparatuses for image processing to optimize image resolution and for optimizing video streaming bandwidth for VR videos |
| US10311630B2 (en) * | 2017-05-31 | 2019-06-04 | Verizon Patent And Licensing Inc. | Methods and systems for rendering frames of a virtual scene from different vantage points based on a virtual entity description frame of the virtual scene |
| US11093752B2 (en) | 2017-06-02 | 2021-08-17 | Apple Inc. | Object tracking in multi-view video |
| US20210204019A1 (en) * | 2017-07-18 | 2021-07-01 | Hewlett-Packard Development Company, L.P. | Virtual reality buffering |
| KR20200064998A (en) * | 2017-10-20 | 2020-06-08 | 소니 주식회사 | Playback apparatus and method, and generating apparatus and method |
| TWI653882B (en) | 2017-11-23 | 2019-03-11 | 宏碁股份有限公司 | Video device and encoding/decoding method for 3d objects thereof |
| US10990831B2 (en) | 2018-01-05 | 2021-04-27 | Pcms Holdings, Inc. | Method to create a VR event by evaluating third party information and re-providing the processed information in real-time |
| US11328456B2 (en) * | 2018-01-22 | 2022-05-10 | Apple Inc. | Method and device for presenting synthesized reality companion content |
| JP7059662B2 (en) * | 2018-02-02 | 2022-04-26 | トヨタ自動車株式会社 | Remote control system and its communication method |
| ES2955538T3 (en) * | 2018-02-06 | 2023-12-04 | Phenix Real Time Solutions Inc | Simulating a local experience by live streaming shareable views of a live event |
| CN110198457B (en) * | 2018-02-26 | 2022-09-02 | 腾讯科技（深圳）有限公司 | Video playing method and device, system, storage medium, terminal and server thereof |
| KR102638415B1 (en) | 2018-03-22 | 2024-02-19 | 브이아이디 스케일, 인크. | Viewport dependent video streaming events |
| CN110519652B (en) * | 2018-05-22 | 2021-05-18 | 华为软件技术有限公司 | VR video playing method, terminal and server |
| US10764494B2 (en) | 2018-05-25 | 2020-09-01 | Microsoft Technology Licensing, Llc | Adaptive panoramic video streaming using composite pictures |
| US11917127B2 (en) | 2018-05-25 | 2024-02-27 | Interdigital Madison Patent Holdings, Sas | Monitoring of video streaming events |
| US10666863B2 (en) | 2018-05-25 | 2020-05-26 | Microsoft Technology Licensing, Llc | Adaptive panoramic video streaming using overlapping partitioned sections |
| KR20190136417A (en) * | 2018-05-30 | 2019-12-10 | 삼성전자주식회사 | Method for tramsmitting stereoscopic 360 degree video data, display apparatus thereof, and storing device of video data thereof |
| KR102435519B1 (en) * | 2018-06-20 | 2022-08-24 | 삼성전자주식회사 | Method and apparatus for processing 360 degree image |
| EP3588970A1 (en) * | 2018-06-22 | 2020-01-01 | Koninklijke Philips N.V. | Apparatus and method for generating an image data stream |
| CN111614974B (en) * | 2020-04-07 | 2021-11-30 | 上海推乐信息技术服务有限公司 | Video image restoration method and system |
| KR20220039113A (en) * | 2020-09-21 | 2022-03-29 | 삼성전자주식회사 | Method and apparatus for transmitting video content using edge computing service |
| US20220103655A1 (en) * | 2020-09-29 | 2022-03-31 | International Business Machines Corporation | Proactively selecting virtual reality content contexts |
| US11632531B1 (en) * | 2021-05-03 | 2023-04-18 | Amazon Technologies, Inc. | Synchronization and presentation of multiple 3D content streams |
| CN115250363A (en) * | 2022-09-22 | 2022-10-28 | 广州市千钧网络科技有限公司 | Multi-view live broadcast system |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102740154A (en) * | 2011-04-14 | 2012-10-17 | 联发科技股份有限公司 | Method for adjusting playback of multimedia content according to detection result of user status and related apparatus thereof |
| CN103533340A (en) * | 2013-10-25 | 2014-01-22 | 深圳市汉普电子技术开发有限公司 | Naked eye 3D (three-dimensional) playing method of mobile terminal and mobile terminal |
Family Cites Families (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6466254B1 (en) * | 1997-05-08 | 2002-10-15 | Be Here Corporation | Method and apparatus for electronically distributing motion panoramic images |
| JPH1127577A (en) * | 1997-06-30 | 1999-01-29 | Hitachi Ltd | Image system with virtual visual point |
| US20040104935A1 (en) * | 2001-01-26 | 2004-06-03 | Todd Williamson | Virtual reality immersion system |
| JP4461739B2 (en) * | 2003-08-18 | 2010-05-12 | ソニー株式会社 | Imaging device |
| US8130260B2 (en) * | 2005-11-09 | 2012-03-06 | Johns Hopkins University | System and method for 3-dimensional display of image data |
| JP2009017064A (en) * | 2007-07-03 | 2009-01-22 | Hitachi Ltd | Video receiver and multicast distribution content reception control method |
| KR101313797B1 (en) * | 2009-12-18 | 2013-10-01 | 한국전자통신연구원 | Apparatus and method for presenting display of 3D image using head tracking |
| US20110181601A1 (en) * | 2010-01-22 | 2011-07-28 | Sony Computer Entertainment America Inc. | Capturing views and movements of actors performing within generated scenes |
| CN102783170A (en) * | 2010-03-05 | 2012-11-14 | 汤姆森特许公司 | Bit rate adjustment in an adaptive streaming system |
| US9423994B2 (en) * | 2012-02-22 | 2016-08-23 | Citrix Systems, Inc. | Hierarchical display |
| FR2988964A1 (en) * | 2012-03-30 | 2013-10-04 | France Telecom | Method for receiving immersive video content by client entity i.e. smartphone, involves receiving elementary video stream, and returning video content to smartphone from elementary video stream associated with portion of plan |
-
2015
- 2015-05-29 EP EP15798986.4A patent/EP3149937A4/en not_active Withdrawn
- 2015-05-29 KR KR1020167036714A patent/KR102407283B1/en active IP Right Grant
- 2015-05-29 JP JP2017515038A patent/JP2017527230A/en not_active Withdrawn
- 2015-05-29 CN CN201580028645.9A patent/CN106416239B/en active Active
- 2015-05-29 CA CA2948642A patent/CA2948642A1/en active Pending
- 2015-05-29 KR KR1020227019042A patent/KR102611448B1/en active IP Right Grant
- 2015-05-29 WO PCT/US2015/033420 patent/WO2015184416A1/en active Application Filing
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102740154A (en) * | 2011-04-14 | 2012-10-17 | 联发科技股份有限公司 | Method for adjusting playback of multimedia content according to detection result of user status and related apparatus thereof |
| CN103533340A (en) * | 2013-10-25 | 2014-01-22 | 深圳市汉普电子技术开发有限公司 | Naked eye 3D (three-dimensional) playing method of mobile terminal and mobile terminal |
Also Published As
| Publication number | Publication date |
|---|---|
| KR20220081390A (en) | 2022-06-15 |
| CA2948642A1 (en) | 2015-12-03 |
| KR102611448B1 (en) | 2023-12-07 |
| KR20170015938A (en) | 2017-02-10 |
| EP3149937A4 (en) | 2018-01-10 |
| KR102407283B1 (en) | 2022-06-10 |
| CN106416239A (en) | 2017-02-15 |
| JP2017527230A (en) | 2017-09-14 |
| EP3149937A1 (en) | 2017-04-05 |
| WO2015184416A1 (en) | 2015-12-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106416239B (en) | Method and apparatus for delivering content and/or playing back content | |
| US11871085B2 (en) | Methods and apparatus for delivering content and/or playing back content | |
| JP7029562B2 (en) | Equipment and methods for providing and displaying content | |
| US20210409672A1 (en) | Methods and apparatus for receiving and/or playing back content | |
| RU2665872C2 (en) | Stereo image viewing | |
| KR20170120631A (en) | Methods and apparatus for generating and utilizing reduced resolution images and / or for communicating such images to a playback or content distribution device | |
| US20060244831A1 (en) | System and method for supplying and receiving a custom image | |
| US11461871B2 (en) | Virtual reality cinema-immersive movie watching for headmounted displays | |
| EP3316247B1 (en) | Information processing device, information processing method, and program | |
| CN110100435A (en) | Generating means, identification information generation method, transcriber and imaging reconstruction method | |
| US11010923B2 (en) | Image encoding method and technical equipment for the same | |
| US20220329886A1 (en) | Methods and devices for handling media data streams |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |