WO2018232016A1 - 360 degree television system - Google Patents

Abstract

Disclosed is a video system including a video display and a remote control. The video display is capable of playing traditional videos and non-traditional videos. The system plays the non-traditional video in a view window, where the view window provides a view of a portion of the non-traditional video, and the remote control can adjust the portion of the non-traditional video that the view window is showing. The non-traditional video may be a 360 degree video or a multi camera video.

Description

360 DEGREE TELEVISION SYSTEM

Cross-Reference to Related Application

[0001] This application claims the benefit of co-pending United States Provisional Patent Application Serial No. 62/518,998, filed June 13, 2017, which is incorporated herein in its entirety.

Technical Field

[0002] The present invention relates generally to video display devices, and more particularly, to an improved system for watching and interacting with a 360-degree video.

Background Art

[0003] Traditionally, videos display a single track of moving visual media. But, in recent years non-traditional video such as 360-degree video content and multi-camera videos, have become more available. 360-degree videos are commonly recorded via very wide camera views, and they may capture a view in many if not all directions concurrently from one camera location. When playing a 360-degree video the viewer may move a field of view of a 360 degree video while the 360 degree video is playing.

[0004] There are various ways to provide an input to a video playback system for choosing or moving a field of view, depending on a playback device. For example, a user of a personal computer viewing a 360 degree video can use a mouse to direct a field of view. In a dedicated head mounted 360 degree playback system the field of view may be controlled by a user turning their head. A smartphone can be a display for a 360 degree playback system such as a head mounted display, where the smartphone may be held in front of a user's face, and where gyros in the smartphone may be configured to change a field of view as the user moves their head. An example of such a head mounted system is Google Cardboard.

[0005] Another example of non-traditional video content is multi camera video, where multiple cameras are placed at different locations around a scene to simultaneously record content. During playback a user may select preferred camera views of the multiple cameras.

[0006] There are video systems such as smart TVs and other entertainment systems that provide a big screen displaying standard video content, from various digital sources such as HBO, Showtime, Netflix, Hulu, YouTube and others. Commercially available smart TVs have a computing system running an operating system, for example, LG running webOS, Samsung running Tizen, or Panasonic running Firefox, but these systems lack the ability to play non-traditional video content like 360 degree videos.

[0007] Accordingly, there is a need for a television system that allows a user to view and easily control non-traditional video playback.

Summary of the Invention

[0008] This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features of essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any or all disadvantages noted in any part of this disclosure.

[0009] Disclosed is a video system, comprising one or more storage machines holding instructions executable by one or more logic machines to actualize the operations of: at a video display, playing video content through a view window, said video content being non- traditional video, where the view window displays a field of view in a non-traditional video, and at a remote control, receiving a field of view changing input where the field of view changing input causes a field of view of the non-traditional video to change.

[0010] In another aspect, the view window may pan left, pan right, pan up, pan down, zoom in and zoom out based on commands issued from the remote control.

[0011] In another aspect, the video content may be organized at the highest level in themes, themes having a collection of channels, and where a channel has a collection of video capsules, where the theme can be sports, entertainment, foodie, and music.

[0012] In another aspect, the non-traditional content may be a 360 degree video, where the view window provides a view of a portion of the 360 degree video.

[0013] In another aspect, the 360 degree video may show an item available for sale and the item can be selected and put in a shopping cart, where the user may make the selection by clicking a button on the remote control.

[0014] In another aspect, a root program running on a computing system may control the video display.

[0015] In another aspect, the remote control can adjust the portion of the non- traditional video that the view window is showing as the non-traditional video is being played.

[0016] In another aspect, the remote control can be a handheld remote control.

[0017] In another aspect, the remote control controls an infrared pointer on a display for selecting items on the display.

[0018] These and other objects, features, and advantages of the present invention will become more readily apparent from the attached drawings and the detailed description of the preferred embodiments, which follow.

Brief Description of the Drawings

[0019] The preferred embodiments of the claimed subject matter will hereinafter be described in conjunction with the appended drawings provided to illustrate and not to limit the scope of the claimed subject matter, where like designations denote like elements, and in which:

[0020] FIG. 1 schematically shows a view window into a 360 degree video depicted as a panorama and shows an exemplary way that the view window may be adjusted, in accordance with aspects of the present disclosure;

[0021] FIG. 2 schematically shows a block diagram of a video system showing the view window on a display subsystem and a handheld remote control, in accordance with aspects of the present disclosure;

[0022] FIG. 3 schematically shows a flow chart of how a user may interact with the video system, in accordance with aspects of the present disclosure;

[0023] FIGS. 4A and 4B schematically show example view window displays that a user may interact with, in accordance with aspects of the present disclosure;

[0024] FIG. 5 schematically shows an organization of images that the video system may use to help manage pre-loading of images, in accordance with aspects of the present disclosure;

[0025] FIG. 6 schematically shows an exemplary flow chart of a way to manage preloading images in the video system, in accordance with aspects of the present disclosure; and

[0026] FIG. 7 schematically shows a block diagram of a computing system, in accordance with aspects of the present disclosure.

[0027] It is to be understood that like reference numerals refer to like parts throughout the several views of the drawings.

Description of Embodiments

[0028] The following detailed description is merely exemplary in nature and is not intended to limit the described embodiments or the application and uses of the described embodiments. As used herein, the word "exemplary" or "illustrative" means "serving as an example, instance, or illustration." Any implementation described herein as "exemplary" or "illustrative" is not necessarily to be construed as preferred or advantageous over other implementations. All of the implementations described below are exemplary implementations provided to enable persons skilled in the art to make or use the embodiments of the disclosure and are not intended to limit the scope of the disclosure, which is defined by the claims. For purposes of description herein, the terms "upper", "lower", "left", "rear", "right", "front", "vertical", "horizontal", and derivatives thereof shall relate to the invention as oriented in FIG. 1. Furthermore, there is no intention to be bound by any expressed or implied theory presented in the preceding technical field, background, brief summary or the following detailed description. It is also to be understood that the specific devices and processes illustrated in the attached drawings, and described in the following specification, are simply exemplary embodiments of the inventive concepts defined in the appended claims. Hence, specific dimensions and other physical characteristics relating to the embodiments disclosed herein are not to be considered as limiting, unless the claims expressly state otherwise.

[0029] FIG. 1 depicts a 360 degree video as a panorama 100 and a view window 102 (e.g. field of view) which looks into the panorama 100. The panorama 100 is shown in a simplified form as a cylinder in FIG. 1. However panorama 100 may be a complete or partial sphere. The panorama 100 may have a front view 104, to the right may be the right view 106, continuing around may be the back view 108 that is not directly visible in FIG. 1, and finally, the left view 1 10 is to the left of the front view 104. The front, left, right, and back views may be virtual views relative to an initial field of view which starts with an initialization of a 360 degree video. The front, left, right, and back views may be moved or rotated into a field of view, or alternatively, a field of view may be moved or rotated to the front, left, right and back views. Since panorama 100 may be a sphere, an upper view 101 and a lower view 103 may also be included, and the view window 102 may display portions of the upper and lower views. For example, the upper view and the lower view may be upper and lower hemispheres of a sphere. The upper, lower, front, left, right, and back may combine to be a full sphere, with a user's view originating from within the sphere.

[0030] When viewing the 360 degree video a pan left action 112 may rotate the view window 102 to the left and display more of what is in the left view 1 10. When viewing the 360 degree video a pan right action 1 14 may rotate the view window 102 to the right and show more of the right view 106. When viewing the 360 degree video a pan up action 1 16 may move the view window 102 up on the panorama 100. When viewing the 360 degree video a pan down action 1 18 may move the view window 102 down on the panorama 100. When viewing the 360 degree video a zoom in action 120 may change the view window 102 to enlarge portions of the panorama within the view window 102, or zoom into portions of the panorama within the view window 102. When viewing the 360 degree video a zoom out action 122 may change the view window 102 to shrink portions of the panorama within the view window 102, or zoom out portions of the panorama within the view window 102.

[0031] Each of these described panning and zooming actions (pan left action 112, pan right action 114, pan up action 1 16, pan down action 118, zoom in action 120 and zoom out action 122) may have different panning speeds. For example, panning speed may depend on pressure applied to a button, or a speed in which a user moves an infrared cursor. In addition, an action may be a combination, for example panning both up and left executing the panning action at some angle in-between horizontal and vertical. Another combined action may be to pan and zoom at the same time. It is to be understood that panning may be achieved via any appropriate user input, such as via a pointing device, mouse, or keyboard.

[0032] FIG. 2 is a block diagram of a video system 200 showing a display screen 202 and a handheld remote control 204. The display screen 202 shows a view window 102. The display screen 202 may be a flat screen Liquid Crystal Display (LCD), Light Emitting Diode (LED) display, Organic LED display, a projected image onto a surface such as a wall, or other display devices that can display a series of images for playing video. A computing system, as described below, may control what the display screen 202 and the view window 102 show and display.

[0033] The handheld remote control 204 may be in communication with the display screen 202 and may send commands to the display screen 202 to adjust the view window 102. The commands may be sent to adjust the view window 102 in any appropriate way, such as directly or indirectly through an internet network. The handheld remote control 204 may have a left-button 206 and pressing it may send a command to execute the pan left action 112. The handheld remote control 204 may have a right-button 208 and pressing it may send a command to execute the pan right action 1 14. The handheld remote control 204 may have an up-button 210 and pressing it may send a command to execute the pan up action 1 16. The handheld remote control 204 may have a down-button 212 and pressing it may send a command to execute the pan down action 1 18. The handheld remote control 204 may have an in-button 214 and pressing it may send a command to execute the zoom in action 120. The handheld remote control 204 may have an out-button 216 and pressing it may send a command to execute the zoom out action 122. It is to be understood that the remote may be operable to execute the actions in any appropriate way without departing from scope of this disclosure, such as swiping on a touch-screen display or moving an infrared cursor. Further, the buttons may be virtual buttons instead of hardware buttons.

[0034] There are many ways that the handheld remote control 204 may indicate the panning and zooming actions (i.e. pan left action 112, pan right action 1 14, pan up action 116, pan down action 118, zoom in action 120 and zoom out action 122). For example, in place of a left-button 206, the left side of a circle could be pressed, a small joystick could be push left, handheld remote control 204 could be physically pointed left, a gesture on the handheld remote control 204 such as moving the handheld remote control 204 to the left may be used, or any number of other methods that could be associated to the pan left action 112. Similarly, the other panning and zooming activities (i.e. pan right action 114, pan up action 116, pan down action 1 18, zoom in action 120 and zoom out action 122) could indicate commands in alternative analogous to what was described for the left-button 206.

[0035] The handheld remote control 204 could provide more refined control over the panning and zooming activities, for example the speed of the panning and zooming could be variable, increasing for example with pressure on a button, or increasing with an angle of the tilt of the remote control or other operable feature of the handheld remote control 204.

[0036] A traditional video is a series of images that provide a single track of video content that is not adjustable by a user. A non-traditional video may be 360 degree video content or may be a multi camera video. A multi camera video may have multiple cameras all watching a same scene from different vantage points. If the non-traditional video is a multi camera video then panning the view window 102 may move from one camera view to a different camera view. For example, there could be a series of cameras around a perimeter of a stadium looking down at action on a football field. The pan right action 114 could cause the view window 102 to show a camera view to a right of a current camera view. The video system 200 may provide analogous functionality for the pan left action 112. There may be analogous pan up action 116 and pan down action 118 depending if there are any camera feeds available that are available above or below the current camera. For multi camera video, the zoom in action 120 and zoom out action 122 may be the same as described above. Therefore, the system allows a user to switch between various camera views using a remote controller.

[0037] FIG. 3 illustrates a process flow 300 of how a user may interact with the video system 200. The process flow 300 starts in box 302 where the process flow 300 presents a main menu with various themes and channels related to a currently selected theme. Next, the process flow 300 responds to a user selecting a theme by proceeding to box 304 where the process flow 300 displays channels associated with a selected theme. The process flow 300 may reach box 306 from box 302 that presented the main menu or from box 304 that displayed the channels associated with the selected theme. At box 306, the process flow 300 may present a set of video capsules related to the selected theme.

[0038] A video capsule may cover a particular subject or event, for example how to prepare a cooking recipe, a new line of dresses for sale, or a football game. The video capsule may have a video that is either traditional or non-traditional. The video capsule may have additional non-video information related to the capsule's subject or event for example for a cooking recipe, a capsule may provide an ingredients list or step-by-step instructions to accompany the video. The step-by-step instructions may include clickable links that jump to the corresponding portion of the capsule's video.

[0039] Next, the process flow 300 may respond to the user selecting a video capsule by proceeding to decision diamond 310.

[0040] At decision diamond 310, the process flow 300 checks to see what type of video the user selected. If the user selected a traditional video then the process flow 300 continues to box 312. At box 312, the process flow 300 plays the traditional video.

[0041] If the selected video capsule is a non-traditional video like a 360 degree video then the process flow plays the 360 degree video through the view window 102 with handheld remote control 204 sending commands as described above.

[0042] The non-traditional video may show an item available for sale and a user may select the item and put it in a shopping cart. The user may make the selection by clicking a button on the remote control. A user may purchase the items in the shopping cart by going through a checkout procedure. During the checkout procedure payment information may be gathered for completing a payment. During the checkout procedure, an address may be gathered and later the user may receive the purchased items at that address.

[0043] When the video capsule playback is complete (either box 312 for traditional video or box 314 for non-traditional video), then the process flow 300 returns to box 302 and displays the main menu.

[0044] FIGS. 4A and 4B show an exemplary main display 400 that the video system 200 may show on the display screen 202. A user may interact with the main display 400 through the handheld remote control 204. The main display 400 shows how the display screen may look after the user has selected a theme 402 and channel 404. The user could then select on the main display 400 the video capsule 406 and the video system 200 may respond by displaying a video capsule playback display 410 with a view window 102.

[0045] The video may be a stream, in which case the images making up the video (be it traditional or non-traditional) may not be located on the video system 200. Instead, the images may reside at another location and they may be transmitted from that other location and received by the video system 200. The stream may be live in which case the video may be transmitted shortly after the camera records the image.

[0046] FIG. 5 depicts an organization of images 500 that make up a video. The video system 200 may use the organization of the images 500 to help control the pre-loading of images to provide improved video playback within memory limitations of the video system 200.

[0047] A video may have an audio track and a series of images. For a 360 degree video, the view window 102 provides a view in the 360 degree video. For a proper video viewing experience, the video system 200 needs to maintain synchronization of the audio track and the images. To accomplish this, the audio track can be broken into various time- segments 502A, 502B, 502C, and 502D (hereinafter 502), for example one-second time- segments. Each time-segment 502, may be associated to packs of images 504A, 504B, 504C, and 504D (hereinafter 504). Packs 504 may each be grouped in any appropriate fashion, such as into a pack group 506. In one example, each pack includes five images in each time-segment 502. For example, respectively pack 504A may correspond to segment 502A, and 504B may correspond to segment 502B.

[0048] Each pack of images 504 may be a subset of all the available images from an original video. For example, in the original video there may be sixty images in one second (e.g. 60 frames per second) but the video system 200 may assign twenty images in one second, and the twenty images may be distributed into packs or pack groups as described herein.

[0049] A set of images per second may be assigned in various ways, for example picking twenty images at random, selecting every 3^rd image, selecting images that have the most pixel changes or any other selection criteria that provide an improved viewing experience while working within a maximum count of images allowed by the system. For example, a maximum count of images per second may be 20. While playing back video the video system 200 may merge the audio to the images to provide audio-visual synchronization.

[0050] FIG. 6 is an exemplary process flow 600 of a way to manage pre-loading of images in the video system 200. The process starts at box 602 with a request to play a video capsule. Next, at box 604 the process flow 600 determines the time-segment 502 for the soundtrack of the video capsule. Then at box 606, the process flow 600 identifies the pack of images 504 associated to the time-segment 502. Next at box 608, the process flow 600 determines the pack group 506 and starts pre-loading the images of the pack group 506, for example loading the images into memory. Then the process flow 600 continues on to decision diamond 610. As a non-limiting example, a pack group 506 may be predetermined according to a maximum number of frames per second, or an optimal preloading process.

[0051] At decision diamond 610, the process flow 600 determines if the image count has reached a pre-load threshold. For example, the pre-load threshold could be fifty percent of the pack group 506 size, so if the pack group 506 were one hundred images then the preload threshold may be fifty images. If the pre-load image count is under the pre-load threshold, then the process flow 600 continues to box 612. At box 612 the process flow 600 waits for more images to be pre-loaded, then returns to decision diamond 610 to check again if the pre-load threshold is met. If the pre-load image count exceeds the pre-load threshold then the process flow 600 proceeds to box 614. At box 614, the process flow 600 starts or continues playing the video if it is not already playing. Next, the process flow 600 continues to the decision diamond 616.

[0052] At decision diamond 616, the process flow 600 checks to see if the number of images pre-loaded exceeds the maximum pre-load image threshold. For example, the maximum pre-load image threshold could be set to two hundred images, and if the number of pre-loaded image is two-hundred-and-one then the process flow 600 continues to box 618. At box 618, the process flow 600 waits for images to be removed from the pre-load memory as they are played then proceeds back to decision diamond 616. If the number of images pre-loaded is under the maximum pre-load image threshold then the process flow 600 continues back to box 604.

[0053] In some embodiments the methods, tasks, and/or operations described above may be executed, actualized, effected, or carried out by a computing system including a tangible computer-readable storage medium, also described herein as a storage machine, that holds machine-readable instructions executable by a logic machine (i.e. a processor or programmable control device) to provide, implement, perform, actualize, effect, and/or enact the above described methods, processes, operations, and/or tasks. For example, a suitable computing system may be computing system 700 shown in FIG. 7. When such methods and processes are implemented, the state of the storage machine 704 may change to hold different data. For example, the storage machine 704 may include memory devices such as various hard disk drives, CD, or DVD devices. The logic machine 702 may execute machine-readable instructions via one or more physical information and/or logic processing devices. For example, the logic machine 702 may be configured to execute instructions to perform tasks for a computer program. The logic machine 702 may include one or more processors to execute the machine-readable instructions. The computing system 700 may include a display subsystem 706 to display a graphical user interface (GUI) or any visual element presentable in the methods or processes described above for example controlling the display screen 202. For example, the display subsystem 706, storage machine 704, and logic machine 702 may be integrated such that the above method may be executed while visual elements of the disclosed system and/or method are displayed on a display screen 202 for user consumption. The computing system 700 may include an input subsystem 708 that receives user input. The input subsystem 708 may be configured to connect to and receive input from devices such as a mouse, keyboard, gaming controller, a remote control or handheld remote control 204. For example, a user input may indicate a request that a certain task is to be executed by the computing system 700, such as requesting the computing system 700 to display any of the above described .information, or requesting that the user input updates or modifies existing stored information for subsequent processing. A communication subsystem 710 may allow the methods described above to be executed or provided over a computer network. For example, the communication subsystem 710 may be configured to enable the computing system 700 to communicate with a plurality of personal computing devices or one or more servers. The communication subsystem 710 may include wired and/or wireless communication devices to facilitate networked communication. The described methods or processes may be executed, provided, or implemented for a user on one or more computing devices via a computer-program product such as via an application programming interface (API).

[0054] The instructions that the logical machine 702 runs when the video system 200 starts may be a root program. The root program may allow small applications (i.e. apps) to load separately and run. The root program may load apps from the storage machine 704 or via the communication subsystem 710. The functionality described above may reside in the root program.

[0055] Since many modifications, variations, and changes in detail can be made to the described preferred embodiments of the invention, it is intended that all matters in the foregoing description and shown in the accompanying drawings be interpreted as illustrative and not in a limiting sense. Thus, the scope of the invention should be determined by the appended claims and their legal equivalents.

Claims

What is claimed is;

1. A video playing system, the system comprising one or more storage machines holding instructions executable by one or more logic machines to actualize the operations of: at a video display, playing non-traditional video through a view window that displays a field of view of the non-traditional video; and at a controller, receiving a field of view changing input where the field of view changing input causes a field of view of the non-traditional video to change.

2. The system of claim 1, wherein the non-traditional video is 360 degree video.

3. The system of claim 1, wherein the field of view changing input causes the field of view to pan and zoom in, concurrently.

4. The system of claim 1, wherein the field of view changing input is a gesture.

5. The system of claim 1, wherein the non-traditional video is multi-camera video and the field of view changing input causes a displayed field of view to switch between at least two camera fields of view.

6. The system of claim 1, wherein the controller is operable by a user to select a link.

7. The system of claim 1, wherein the controller is operable by a user to buy items.

8. The system of claim 1, wherein the non-traditional video is a video stream received online.

9. The system of claim 1, wherein the non-traditional video is a live video feed.

10. The system of claim 1, wherein the non-traditional video includes an audio track and images, and a segment of time audio track is automatically synchronized with a subset image pack of the images.

11. The system of claim 1 , wherein the system pre loads a subset image pack of images of the non-traditional video, and upon a pre load threshold being reached, plays the non- traditional video.

12. The system of claim 1 , wherein the system pre loads a subset image pack of images of the non-traditional video, and upon a pre load threshold being reached, plays the non- traditional video, and wherein upon a maximum threshold of pre loaded images being exceeded, the system waits for images to be removed from a pre-load memory as the images are displayed before pre loading more images.

13. A video playing system, the system comprising one or more storage machines holding instructions executable by one or more logic machines to actualize the operations of: at a video display, playing non-traditional video through a view window that displays a field of view of the non-traditional video; at a controller, receiving a field of view changing input where the field of view changing input causes a field of view of the non-traditional video to change; wherein the non-traditional video includes an audio track and images, and a segment of the audio track is automatically synchronized with a subset image pack of the images; wherein the system pre loads the subset image pack of images of the non-traditional video, and upon a pre load threshold being reached, the system plays the non-traditional video; wherein upon a maximum threshold of pre loaded images being exceeded, the system waits for images to be removed from a pre-load memory as the images are displayed before pre loading more images; and wherein the non-traditional video is 360 degree video.

14. The system of claim 13, wherein the field of view changing input causes the field of view to pan and zoom in, concurrently.

15. The system of claim 13 wherein the field of view changing input is a gesture.

16. The system of claim 13, wherein the non-traditional video is multi-camera video and the field of view changing input causes a displayed field of view to switch between at least two camera fields of view.

17. The system of claim 13, wherein the controller is operable by a user to select a link.

18. The system of claim 13, wherein the controller is operable by a user to buy items.

19. The system of claim 13, wherein the non-traditional video is a video stream received online.

20. A video playing system, the system comprising one or more storage machines holding instructions executable by one or more logic machines to actualize the operations of: at a video display, playing non-traditional video through a view window that displays a field of view of the non-traditional video; at a controller, receiving a field of view changing input where the field of view changing input causes a field of view of the non-traditional video to change; wherein the non-traditional video includes an audio track and images, and a segment of the audio track is automatically synchronized with a subset image pack of the images; wherein the system pre loads the subset image pack of images of the non-traditional video, and upon a pre load threshold being reached, plays the non-traditional video; wherein upon a maximum threshold of pre loaded images being exceeded, the system waits for images to be removed from a pre-load memory as the images are displayed before pre loading more images; wherein the non-traditional video is 360 degree video; wherein the field of view changing input causes the field of view to pan and zoom in, concurrently; wherein the field of view changing input is a gesture; wherein the controller is operable by a user to select a link; wherein the controller is operable by a user to buy items; and wherein the non-traditional video is a video stream received online.