WO2014162239A2 - Interface mechanism for massive resolution displays - Google Patents

Interface mechanism for massive resolution displays Download PDF

Info

Publication number
WO2014162239A2
WO2014162239A2 PCT/IB2014/060127 IB2014060127W WO2014162239A2 WO 2014162239 A2 WO2014162239 A2 WO 2014162239A2 IB 2014060127 W IB2014060127 W IB 2014060127W WO 2014162239 A2 WO2014162239 A2 WO 2014162239A2
Authority
WO
WIPO (PCT)
Prior art keywords
user interface
layout
display panel
display
widget
Prior art date
Application number
PCT/IB2014/060127
Other languages
French (fr)
Other versions
WO2014162239A3 (en
Inventor
Paul Harding
Original Assignee
Nds Limited
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nds Limited filed Critical Nds Limited
Publication of WO2014162239A2 publication Critical patent/WO2014162239A2/en
Publication of WO2014162239A3 publication Critical patent/WO2014162239A3/en

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/41—Structure of client; Structure of client peripherals
    • H04N21/4104—Peripherals receiving signals from specially adapted client devices
    • H04N21/4126—The peripheral being portable, e.g. PDAs or mobile phones
    • H04N21/41265—The peripheral being portable, e.g. PDAs or mobile phones having a remote control device for bidirectional communication between the remote control device and client device
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0487—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
    • G06F3/0488—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
    • G06F3/04886—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
    • G06F3/1423—Digital output to display device ; Cooperation and interconnection of the display device with other functional units controlling a plurality of local displays, e.g. CRT and flat panel display
    • G06F3/1438—Digital output to display device ; Cooperation and interconnection of the display device with other functional units controlling a plurality of local displays, e.g. CRT and flat panel display using more than one graphics controller
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
    • G06F3/1423—Digital output to display device ; Cooperation and interconnection of the display device with other functional units controlling a plurality of local displays, e.g. CRT and flat panel display
    • G06F3/1446—Digital output to display device ; Cooperation and interconnection of the display device with other functional units controlling a plurality of local displays, e.g. CRT and flat panel display display composed of modules, e.g. video walls
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G5/00—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators
    • G09G5/14—Display of multiple viewports
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/41—Structure of client; Structure of client peripherals
    • H04N21/422—Input-only peripherals, i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS]
    • H04N21/42204—User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor
    • H04N21/42206—User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor characterized by hardware details
    • H04N21/42208—Display device provided on the remote control
    • H04N21/42209—Display device provided on the remote control for displaying non-command information, e.g. electronic program guide [EPG], e-mail, messages or a second television channel
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/41—Structure of client; Structure of client peripherals
    • H04N21/422—Input-only peripherals, i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS]
    • H04N21/42204—User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor
    • H04N21/42206—User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor characterized by hardware details
    • H04N21/4222—Remote control device emulator integrated into a non-television apparatus, e.g. a PDA, media center or smart toy
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/41—Structure of client; Structure of client peripherals
    • H04N21/422—Input-only peripherals, i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS]
    • H04N21/42204—User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor
    • H04N21/42206—User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor characterized by hardware details
    • H04N21/42224—Touch pad or touch panel provided on the remote control
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/431—Generation of visual interfaces for content selection or interaction; Content or additional data rendering
    • H04N21/4312—Generation of visual interfaces for content selection or interaction; Content or additional data rendering involving specific graphical features, e.g. screen layout, special fonts or colors, blinking icons, highlights or animations
    • H04N21/4314—Generation of visual interfaces for content selection or interaction; Content or additional data rendering involving specific graphical features, e.g. screen layout, special fonts or colors, blinking icons, highlights or animations for fitting data in a restricted space on the screen, e.g. EPG data in a rectangular grid
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/45—Management operations performed by the client for facilitating the reception of or the interaction with the content or administrating data related to the end-user or to the client device itself, e.g. learning user preferences for recommending movies, resolving scheduling conflicts
    • H04N21/462—Content or additional data management, e.g. creating a master electronic program guide from data received from the Internet and a Head-end, controlling the complexity of a video stream by scaling the resolution or bit-rate based on the client capabilities
    • H04N21/4622—Retrieving content or additional data from different sources, e.g. from a broadcast channel and the Internet
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/47—End-user applications
    • H04N21/478—Supplemental services, e.g. displaying phone caller identification, shopping application
    • H04N21/4788—Supplemental services, e.g. displaying phone caller identification, shopping application communicating with other users, e.g. chatting
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/47—End-user applications
    • H04N21/482—End-user interface for program selection
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/47—End-user applications
    • H04N21/488—Data services, e.g. news ticker
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
    • G06F3/1454—Digital output to display device ; Cooperation and interconnection of the display device with other functional units involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G2320/00—Control of display operating conditions
    • G09G2320/06—Adjustment of display parameters
    • G09G2320/0606—Manual adjustment
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G2340/00—Aspects of display data processing
    • G09G2340/04—Changes in size, position or resolution of an image
    • G09G2340/0464—Positioning
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G2340/00—Aspects of display data processing
    • G09G2340/14—Solving problems related to the presentation of information to be displayed
    • G09G2340/145—Solving problems related to the presentation of information to be displayed related to small screens
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G2354/00—Aspects of interface with display user
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G2370/00—Aspects of data communication
    • G09G2370/06—Consumer Electronics Control, i.e. control of another device by a display or vice versa
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G2370/00—Aspects of data communication
    • G09G2370/20—Details of the management of multiple sources of image data

Definitions

  • Widget in all of its grammatical forms, as used in the present specification and claims is understood to refer to an element of a graphical user interface (i.e. the display referred to, above) that displays an information arrangement changeable by the user (see the list of exemplary widgets, above).
  • the defining characteristic of a widget is to provide a single interaction point for the manipulation of a given kind of data. Widgets may be thought of as basic visual building blocks which, combined in an application, hold all the data processed by the application and the available interactions on this data.
  • the present invention in certain embodiments thereof, seeks to provide an improved interface mechanism for massive resolution displays.
  • the embodiments of the present invention described herein enable mirroring of widget positions from a massive resolution display to an app running on a companion device. Mirroring of widget positions enables fast locating and selection of objects for users by leveraging spatial awareness and recognition.
  • a user's local interactions and selections are being tracked and logically duplicated on the massive resolution display, which thus enables some basic user interface functions to be operated without looking down at the control app, by users familiar with the control interface. For example, without looking down at the control app , the user is able to perform widget selection, widget movement, and trigging a refresh or page scroll of widget data.
  • a system for controlling a display panel array including a plurality of display panels and being operative to display a plurality of user interface widgets
  • the system including a plurality of display adaptors, each one of the plurality of display adaptors controlling a layout of an output on at least one display panel of the display panel array, a layout engine for arranging each of the plurality of user interface widgets on the display panel array, the layout engine arranging each of the plurality of user interface widgets on the display panel array by utilizing at least metadata published by each of the plurality of user interface widgets, the metadata including constraints on a user interface widget aspect ratio or orientation, a layout of the plurality of user interface widgets presently in use on the display panel array, and an area of the display panel array, the layout engine being operative to split a plurality of outputs among each one of the display adaptors, the plurality of outputs including data which each of the display adaptors requires in order to display the layout, thereby
  • the area and configuration of the display panel array is stored by the layout engine during one of installation of the system, and a system update.
  • the management application provides visual cues to the user of the companion device, thereby enabling fast locating and selection of objects.
  • the plurality of user interface widgets presently in use on the display panel array is mirrored by a layout of thumbnail representations of the plurality of user interface widgets in the management application.
  • each of the thumbnail representations appear proportionally in the same size and position as its corresponding user interface widget as the corresponding user interface widget appears on the display panel array.
  • each of the thumbnail representations appears in the management application with a lower graphical resolution than each corresponding user interface widget appears on the display panel array.
  • each of the thumbnail representations displays headline status information within its boundaries.
  • the user of the companion device interacts with the management application using standard gestures.
  • the standard gestures include a pinch and zoom gesture, and a drag gesture.
  • the management application is operative to enter a tracking mode, causing a selected one of the plurality of user interface widgets to be highlighted on the display panel array.
  • a method for controlling a display panel array including a plurality of display panels and being operative to display a plurality of user interface widgets
  • the method including controlling a layout of an output on at least one display panel of the display panel array at each one of a plurality of display adaptors, arranging each of the plurality of user interface widgets on the display panel array by a layout engine, by utilizing at least metadata published by each of the plurality of user interface widgets, the metadata including constraints on a user interface widget aspect ratio or orientation, a layout of the plurality of user interface widgets presently in use on the display panel array, and an area of the display panel array, the layout engine being operative to split a plurality of outputs among each one of the display adaptors, the plurality of outputs including data which each of the display adaptors requires in order to display the layout, thereby producing the desired layout on the display panel array, receiving at a system controller a multimedia input from at least one network adapter,
  • Fig. 1 is a simplified pictorial illustration of a massive resolution display controlled by a mechanism constructed and operative in accordance with an embodiment of the present invention
  • Fig. 2 is a depiction showing widget types identified for the display configuration of Fig. 1;
  • Fig. 3 A is a depiction of a thumbnail layout, referred to as a viewport, on a companion device for the display configuration shown in Figure 1 ;
  • Fig. 3B is a depiction of a large scale widget and its corresponding thumbnail
  • Fig. 4 is a depiction of the use of a "pinch and zoom" gesture in order to zoom in to the viewport of Fig. 3 A;
  • Fig. 5 is a depiction of a method for using a drag gesture in order to scroll the viewport of Fig. 3 A;
  • Fig. 6 is a depiction of the massive resolution display of Fig. 1 with the system of Fig. 1 in tracking mode;
  • Fig. 2 is a depiction showing widget types identified for the display configuration of Fig. 1.
  • the massive resolution display 110 is depicted as displaying a weather widget 120, 220; a home security widget 130, 230; a TV sidebar widget 140, 240; a live TV widget 150, 250; and a social networking widget 160, 260.
  • the massive resolution display 110 typically comprises a plurality of display panels which are arranged to appear as a single massive display.
  • the plurality of user interface widgets 120, 130, 140, 150, 160, 220, 230, 240, 250, 260 depicted in Figs. 1 and 2 are depicted by way of example. Any other appropriate user interface widget may appear, or, alternatively, one of the user interface widgets 120, 130, 140, 150, 160, 220, 230, 240, 250, 260 depicted in Figs. 1 and 2 may not appear in actual usage.
  • Companion device in all of its grammatical forms, is understood to refer to an additional electronic device (for example, and without limiting the generality of the foregoing, a tablet computing device, or a smartphone) that allows a viewer of the massive resolution display 110 to interact with any of the massive resolution display 110 itself or with any of the widgets displayed thereon.
  • the companion device 300 of Fig. 3 A is depicted running an app which displays the viewport 310 on the screen of the companion device 300.
  • the viewport 310 comprises a thumbnail layout on which a representation of one of a complete or a partial section of the massive resolution display 110 appears.
  • the thumbnail images appearing on the viewport 310 i.e. the weather thumbnail 320; the home security thumbnail 330, TV sidebar thumbnail 340, the live TV thumbnail 350, and the social networking thumbnail 360
  • the thumbnail images on the viewport 310 are rendered such that all objects appear proportionally having the same size and in the same position as the corresponding objects appear on the massive resolution display 110.
  • graphical detail which appears on the massive resolution display 110 is not shown on the viewport 310.
  • a widget displayed on the massive resolution display might show a cluster of images in full, while the corresponding thumbnail representation of the widget might be displayed as an outline of the cluster.
  • each of the thumbnails images appearing on the viewport 310 corresponding to each widget 120, 130, 140, 150, 160, 220, 230, 240, 250, 260 depicted in Figs. 1 and 2 displays headline status information within its boundaries, which information is specific and pertinent to the type of widget represented.
  • the live TV thumbnail 350 corresponding to a live TV widget 150 could show the current channel to which the live TV widget is tuned.
  • the headline status information displayed comprises a presentation of the elements considered by the system to be the most important elements from a user's perspective of the widget's current state.
  • the headline status information displayed comprises a subset of and in some cases complement to the information displayed on the main widget.
  • the live TV thumbnail 350 could also display now and next programming information. Also, the live TV thumbnail 350 may be branded or styled in such a way as to be visually similar at a glance to its larger counterpart on the remote display, thus to provide a further associational link.
  • the home security thumbnail 330 corresponding to the home security widget 130 may display a list of currently displayed home security cameras currently displayed in the home security widget 130, as well as a list of other home security cameras which are available to display in the home security widget 130.
  • a social networking widget may display the logged-in user's news feed, the date/time data last refreshed, and a list of available sources which can be switched on/off to filter the widget data (e.g. Facebook, Twitter, Instagram, and so forth).
  • Fig. 3B is a depiction of a large scale widget 370 and its corresponding thumbnail 410.
  • Large scale widgets 370 on the massive resolution display 110 may be comprised of a number of panels 380, 385, 390, which have logically different operations available to be performed on them. In such cases, its corresponding thumbnail 410 will paint the outline of the panels 480, 485, 490 in proportionally the same positions as in the full scale widget 370, to facilitate quick recognition and selection by the user 100A, 100B.
  • Fig. 3B depicts a social network widget 370 as it may appear on the massive resolution display 110.
  • the social network widget 370 is depicted, in a non-limiting fashion, as comprising three panels: a panel comprising a feed of socially shared photos 380, a panel comprising a socially shared status update feed 385, and a panel comprising a view of linked content 390.
  • the corresponding thumbnail 410 also comprises three corresponding panels, a photo stream thumbnail 480, a status stream thumbnail 485, and a linked content thumbnail 490.
  • Fig. 4 is a depiction of the use of a "pinch and zoom” gesture 500 in order to zoom in to the viewport 310 of Fig. 3 A. (It is appreciated that the "opposite" gesture would result in a zoom out.)
  • the viewport 310 is zoomed in or out using a two finger pinch and drag, as is known in the art, and often referred to as a pinch and zoom gesture 500.
  • the live TV thumbnail 350 which is depicted in Fig. 4 as being the widget which is zoomed in on appears in the center of Fig. 4, enlarged in accordance with the zoom.
  • the TV sidebar thumbnail 340 appears on the right side of the viewport 310.
  • Fig. 5 is a depiction of a method for using a drag gesture 530 in order to scroll the viewport 310 of Fig. 3A.
  • the viewport 310 is scrolled in the x-direction, the y-direction, or both directions using a single finger flick or drag gesture 530, as is known in the art.
  • Selection of a thumbnail object, for example, the social networking thumbnail 360, as depicted in Fig. 5, in order to perform operations on the selected thumbnail object is accomplished by a single tap on the thumbnail while the viewport 310 is static (i.e. not still rendering scrolling motion as a result of a swipe or pinch and zoom gesture 500).
  • thumbnail when a thumbnail corresponding to a "simple" widget is selected, the thumbnail will render options available to the user 100A, 100B within the thumbnail boundary, thereby enabling quick selection.
  • simple widget is intended as a verbal construct, referring to a widget which meets this requirement).
  • thumbnail selected will render a button enabling the user 100A, 100B to access a full-detail view of the thumbnail.
  • the full-detail view of the thumbnail may fill the entire screen on the companion device. Navigating back to the massive resolution display 110 display viewport 310 view is achieved with the standard back button on the companion device 310, or, alternatively, with a swipe right gesture.
  • the user 100A, 100B has access to the full-detail view of the thumbnail, the user 100A, 100B has access to the complete feature set or configuration (if relevant) of the selected widget.
  • a live TV widget would be deemed a complex widget because the live TV widget needs a full screen to show an EPG grid.
  • the live TV widget might simply display now/next info for favorite channels.
  • Fig. 6, is a depiction of the massive resolution display of Fig. 1 with the system of Fig. 1 in tracking mode.
  • the user 100 A, 100B may be preferable for the user 100 A, 100B to interact with the app while remaining focused on the massive resolution display 110. (Having to move eyes between the screens of the massive resolution display 110 and companion device 300 excessively would be inefficient, and would likely detract from the user experience).
  • the widget covering the center of the viewport 310 is highlighted (using a thicker border or other visual cue) on the massive resolution display 110.
  • the live TV widget 150 is highlighted and the live TV thumbnail 350 covers the center of the viewport 310. Widget selection is implicit (i.e.
  • thumbnails are centered on the viewport, and the corresponding widget is selected based on no other user action), based on viewport 310 position and zoom - no tap is required.
  • full screen widgets mirror the selection on the viewport 310. This allows users to swipe and scroll on the companion device 330 while looking up at the massive resolution display 110, seeing the current widget selection moving as it is being tracked on massive resolution display 110.
  • This "tracking" feature is typically not the default system behavior because of the potential distraction to other individuals in the room watching elements of which appear on the massive resolution display 110, as the user 100A, HOB of the viewport 310 app navigates around the larger surface (i.e. the massive resolution display 110). Hence, it is a feature optionally switched on by the user 100A, 100B or when deemed appropriate by a display controller based (see the discussion below, with reference to Fig. 7) on the current widget layout.
  • the massive resolution display 110 typically comprises an array of display panels 710.
  • the array of display panels 710 is operative to display a plurality of user interface widgets.
  • the weather widget 120; the home security widget 130; the TV sidebar widget 140; the live TV widget 150; and the social networking widget 160 are examples of typical user interface widgets which might be displayed on the array of display panels 710.
  • a layout engine 730 implemented on one of a computer or processing device arranges each of the plurality of user interface widgets 120, 130, 140, 150, 160 on the array of display panels 710.
  • the layout engine 730 utilizes at least the following information in determining the layout of the plurality of user interface widgets 120, 130, 140, 150, 160 on the array of display panels 710:
  • the layout engine 730 is operative to split a plurality of outputs among each one of the display adaptors 720.
  • the plurality of outputs comprise data which each of the display adaptors 720 requires in order to display the layout, thereby producing the desired layout on the display panel array 710.
  • a system controller 740 implemented on one of a computer or processing device receives a multimedia input from at least one network adapter 750.
  • the network adapter 750 may input content for the various widgets, the content being provided via TCP/IP from various Internet services 755.
  • the system controller 740 manages runtime context (i.e. a sandboxed environment in which each widget process executes, that is to say, at least, the widget's code, data, its input/output, and current state ) for each of the plurality of user interface widgets 120, 130, 140, 150, 160.
  • the system controller 740 is further operative to produce an output comprising at least a portion of the multimedia input for each one of the plurality of user interface widgets 120, 130, 140, 150, 160, the system controller being operative to provide the produced output to the layout engine 730.
  • the system controller 740 via the network adaptor 750 is further in bi-directional WIFI communication with via a WTFI module 760, utilizing either a wireless IEEE 802.11 home network; a Bluetooth connection, or other appropriate communication protocol, with the companion device 770.
  • the companion device 770 is running a management application 780, such as the viewport app 310, described above.
  • the system controller 740 sends information on the layout and metadata about layout constraints for each of the plurality of user interface widgets to the management application 780 on the companion device 770.
  • the management application 780 sends at least one of: layout changes; widget selection notifications; and widget commands to the system controller 740 for relay to the layout engine 730, thereby causing the least one of: layout changes; widget selection notifications; and widget commands to be rendered on the display panel array 710.
  • Fig. 8 is a simplified flowchart diagram of preferred methods of operation of the system of Fig. 1. The method of Fig. 8 is believed to be self-explanatory in light of the above discussion.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • Computer Graphics (AREA)
  • Databases & Information Systems (AREA)
  • Computer Hardware Design (AREA)
  • User Interface Of Digital Computer (AREA)
  • Digital Computer Display Output (AREA)
  • Controls And Circuits For Display Device (AREA)

Abstract

A system for controlling a display panel array, the display panel array comprising a plurality of display panels and being operative to display a plurality of user interface widgets is described. The system includes a plurality of display adaptors controlling a layout of output on at least one display panel of the array, a layout engine for arranging each of the plurality of user interface widgets on the display panel array, the layout engine utilizing at least metadata published by each of the plurality of user interface widgets, a layout of the plurality of user interface widgets presently in use, and an area of the display panel array, the layout engine being operative to split a plurality of outputs among each one of the display adaptors, receiving at a system controller a multimedia input from at least one network adapter, the system controller managing runtime context for each of the plurality of user interface widgets, the system controller operative to produce an output comprising at least a portion of the multimedia input for each one of the plurality of user interface widgets, communicating bi-directionally between the system controller and a companion device external to the system, wherein the system controller sends information on the layout and metadata about layout constraints for each of the plurality of user interface widgets to the management application. Related apparatus, systems, and methods are also described.

Description

INTERFACE MECHANISM FOR MASSIVE RESOLUTION DISPLAYS
FIELD OF THE INVENTION
The present invention relates to controllers for video displays, and particularly to controllers for massive resolution video displays.
BACKGROUND OF THE INVENTION
Massive resolution displays, typically comprised of either a single or multiple physical panels, are decreasing in cost each year. It is envisaged that displays of a magnitude to cover the entire wall of a room - or a significant proportion of the entire wall of a room - will become commonplace in homes.
In such an environment, the display would offer the potential to be configured to show a number of "widgets" in different areas, such as (but not limited to): live TV, social networking feeds, a Flickr photostream, and upcoming reminders/appointments.
It is appreciated that the term "widget", in all of its grammatical forms, as used in the present specification and claims is understood to refer to an element of a graphical user interface (i.e. the display referred to, above) that displays an information arrangement changeable by the user (see the list of exemplary widgets, above). The defining characteristic of a widget is to provide a single interaction point for the manipulation of a given kind of data. Widgets may be thought of as basic visual building blocks which, combined in an application, hold all the data processed by the application and the available interactions on this data.
As such, the display becomes a shared resource which can be used and enjoyed by all members of the household as an "entertainment hub", simultaneously.
However, having a graphical user interface of such scale renders existing interaction solutions (enumerated below) impractical, or inefficient.
· Conventional IR remote control: the interaction possible is too limited for many tasks such as typing text. • Direct touch & Mobile touch (using near field communication [NFC]): the display covers too large a surface area for this to be practical without inducing human fatigue; the solution needs to be "lean-back". Also screen touch would be a distraction or obstruction for other users in the room.
• Motion tracking / gestures: too much of a distraction for other users in the room.
• Wand / pointing devices: not immediate enough to get results, and not flexible enough to support all the different interactions and operations required by multitude of different widgets.
• Voice control: Too much of a distraction for other users in the room.
The following non-patent and patent literature is believed to reflect the state of the art:
A Remote Control Interface for Large Displays, Azam Khan, et al., available at www.autodeskresearch.com/pdf/2004UISTFrisbee.pdf;
A Comparative Study of Interaction Metaphors for Large-Scale Displays, Ricardo Jota, et al, CHI 2009, Spotlight on Works in Progress, Session 2;
Literature Survey on Interaction Techniques for Large Displays, Brian Badillo, et al. available at people.cs.vt.edu/~bowman/cs6724/large_display_survey.pdf;
Drag-and-Pop and Drag-and-Pick: Techniques for Accessing Remote Screen Content on Touch- and Pen-Operated Systems, Patrick Baudisch, et al, available at research.microsoft.com/en- us/um/people/cutrell/DragAndPop_Interact2003.pdf;
Exploring Pen and Paper Interaction with High-Resolution Wall Displays, Nadir Weibel, et al, available at ucsd. academia. edu/weibel/Papers/227587/Exploring_Pen_and_Paper_Interaction_ with High-Resolution Wall Displays;
A Survey of Interaction Techniques and Devices for Large High Resolution Displays, Taimur K. Khan, available at drops, dagstuhl.de/opus/volltexte/2011 /3094/pdf/5.pdf ; Natural throw and tilt interaction between mobile phones and distant displays, Raimund Dachselt, et al, available at dl.acm.org/citation. cfm?id=l 520340.1520467;
Touch & Interact: Touch-based Interaction of Mobile Phones with Displays, Robert Hardy, et al, available at eprints.lancs.ac.uk/13090/l/mobilehci2008_hardy.pdf;
Fluid interaction with high-resolution wall-size displays, Francois Guimbretiere, et al. In Proceedings of the 14th annual ACM symposium on User interface software and technology (UIST Ό1). ACM, New York, NY, USA, 21-30;
Courtyard: integrating shared overview on a large screen and peruser detail on individual screens, Masayuki Tani, et al. In Proceedings of the SIGCHI conference on Human factors in computing systems: celebrating interdependence (CHI '94), Beth Adelson, Susan Dumais, and Judith Olson (Eds.). ACM, New York, NY, USA, 44-50; and
A toolset for creating iconic interfaces for interactive workspaces,
Jacob T. Biehl, et al, In Proceedings of the 2005 IFIP TC13 international conference on Human-Computer Interaction (INTERACT'05), Maria Francesca Costabile and Fabio Patemo (Eds.). Springer- Verlag, Berlin, Heidelberg, 699-712.
US 6,832,353 to Itavaara et al;
US 7,634,780 to Rhoten et al;
US 2009/247222 of Bonnat; and
WO 2012/153290 of Ashley at al,
SUMMARY OF THE INVENTION
The present invention, in certain embodiments thereof, seeks to provide an improved interface mechanism for massive resolution displays.
The embodiments of the present invention described herein enable mirroring of widget positions from a massive resolution display to an app running on a companion device. Mirroring of widget positions enables fast locating and selection of objects for users by leveraging spatial awareness and recognition.
Additionally, mirroring of widget branding and the display of headline status information enables fast locating and selection of objects for users by leveraging visual cues.
Furthermore, interaction using standard gestures on multitouch interfaces ensures a minimal user learning curve to operate the system.
Moreover, a user's local interactions and selections are being tracked and logically duplicated on the massive resolution display, which thus enables some basic user interface functions to be operated without looking down at the control app, by users familiar with the control interface. For example, without looking down at the control app , the user is able to perform widget selection, widget movement, and trigging a refresh or page scroll of widget data.
There is thus provided in accordance with an embodiment of the present invention a system for controlling a display panel array, the display panel array including a plurality of display panels and being operative to display a plurality of user interface widgets, the system including a plurality of display adaptors, each one of the plurality of display adaptors controlling a layout of an output on at least one display panel of the display panel array, a layout engine for arranging each of the plurality of user interface widgets on the display panel array, the layout engine arranging each of the plurality of user interface widgets on the display panel array by utilizing at least metadata published by each of the plurality of user interface widgets, the metadata including constraints on a user interface widget aspect ratio or orientation, a layout of the plurality of user interface widgets presently in use on the display panel array, and an area of the display panel array, the layout engine being operative to split a plurality of outputs among each one of the display adaptors, the plurality of outputs including data which each of the display adaptors requires in order to display the layout, thereby producing the desired layout on the display panel array, a system controller which receives a multimedia input from at least one network adapter, the system controller managing runtime context for each of the plurality of user interface widgets, the system controller operative to produce an output including at least a portion of the multimedia input for each one of the plurality of user interface widgets, the system controller being operative to provide the produced output, as the layout of the plurality of user interface widgets presently in use, to the layout engine, the system controller being in bi-directional communication with a companion device external to the system, the companion device including one of a computer or processing device running a management application in bi-directional communication with the system controller, wherein the system controller sends information on the layout and metadata about layout constraints for each of the plurality of user interface widgets to the management application, such that when a user of the companion device interacts with the management application, the management application sends at least one of layout changes, widget selection notifications, and widget commands to the system controller for relay to the system controller, thereby causing at least one of layout changes, widget selection notifications, and widget commands to be rendered on the display panel array, and the plurality of display adaptors, the layout engine, and the system controller are each implemented on at least one of a computer or processing device.
Further in accordance with an embodiment of the present invention the area and configuration of the display panel array is stored by the layout engine during one of installation of the system, and a system update.
Still further in accordance with an embodiment of the present invention the management application provides visual cues to the user of the companion device, thereby enabling fast locating and selection of objects.
Additionally in accordance with an embodiment of the present invention the plurality of user interface widgets presently in use on the display panel array is mirrored by a layout of thumbnail representations of the plurality of user interface widgets in the management application. Moreover in accordance with an embodiment of the present invention each of the thumbnail representations appear proportionally in the same size and position as its corresponding user interface widget as the corresponding user interface widget appears on the display panel array.
Further in accordance with an embodiment of the present invention each of the thumbnail representations appears in the management application with a lower graphical resolution than each corresponding user interface widget appears on the display panel array.
Still further in accordance with an embodiment of the present invention each of the thumbnail representations displays headline status information within its boundaries.
Additionally in accordance with an embodiment of the present invention the user of the companion device interacts with the management application using standard gestures.
Moreover in accordance with an embodiment of the present invention the standard gestures include a pinch and zoom gesture, and a drag gesture.
Further in accordance with an embodiment of the present invention the metadata published by the user interface widget is published via a common application programming interface.
Still further in accordance with an embodiment of the present invention the management application is operative to enter a tracking mode, causing a selected one of the plurality of user interface widgets to be highlighted on the display panel array.
There is also provided in accordance with another embodiment of the present invention a method for controlling a display panel array, the display panel array including a plurality of display panels and being operative to display a plurality of user interface widgets, the method including controlling a layout of an output on at least one display panel of the display panel array at each one of a plurality of display adaptors, arranging each of the plurality of user interface widgets on the display panel array by a layout engine, by utilizing at least metadata published by each of the plurality of user interface widgets, the metadata including constraints on a user interface widget aspect ratio or orientation, a layout of the plurality of user interface widgets presently in use on the display panel array, and an area of the display panel array, the layout engine being operative to split a plurality of outputs among each one of the display adaptors, the plurality of outputs including data which each of the display adaptors requires in order to display the layout, thereby producing the desired layout on the display panel array, receiving at a system controller a multimedia input from at least one network adapter, the system controller managing runtime context for each of the plurality of user interface widgets, the system controller operative to produce an output including at least a portion of the multimedia input for each one of the plurality of user interface widgets, the system controller being operative to provide the produced output, as the layout of the plurality of user interface widgets presently in use, to the layout engine, communicating bi-directionally between the system controller and a companion device external to the system, the companion device including one of a computer or processing device running a management application in bi-directional communication with the system controller, wherein the system controller sends information on the layout and metadata about layout constraints for each of the plurality of user interface widgets to the management application, such that when a user of the companion device interacts with the management application, the management application sends at least one of layout changes, widget selection notifications, and widget commands to the system controller for relay to the system controller, thereby causing at least one of layout changes, widget selection notifications, and widget commands to be rendered on the display panel array, and the plurality of display adaptors, the layout engine, and the system controller are each implemented on at least one of a computer or processing device. BRIEF DESCRIPTION OF THE DRAWINGS
The present invention will be understood and appreciated more fully from the following detailed description, taken in conjunction with the drawings in which:
Fig. 1 is a simplified pictorial illustration of a massive resolution display controlled by a mechanism constructed and operative in accordance with an embodiment of the present invention;
Fig. 2 is a depiction showing widget types identified for the display configuration of Fig. 1;
Fig. 3 A is a depiction of a thumbnail layout, referred to as a viewport, on a companion device for the display configuration shown in Figure 1 ;
Fig. 3B is a depiction of a large scale widget and its corresponding thumbnail;
Fig. 4 is a depiction of the use of a "pinch and zoom" gesture in order to zoom in to the viewport of Fig. 3 A;
Fig. 5 is a depiction of a method for using a drag gesture in order to scroll the viewport of Fig. 3 A;
Fig. 6 is a depiction of the massive resolution display of Fig. 1 with the system of Fig. 1 in tracking mode;
Fig. 7 is a system diagram depicting an embodiment of an implementation of the interface mechanism of Fig. 1; and
Fig. 8 is a simplified flowchart diagrams of methods of operation of the system of Fig. 1.
DETAILED DESCRIPTION OF AN EMBODIMENT
Reference is now made to Fig. 1, which is a simplified pictorial illustration of a massive resolution display controlled by a mechanism constructed and operative in accordance with an embodiment of the present invention.
Fig. 1 depicts two users 100A, 100B seated before a massive resolution display 110. The massive resolution display 110 is depicted as filling an entire wall of a room, however, the massive resolution display 110 may be smaller than such a wall, or, in principle, may "bend" around the corner of a room, occupying at least a portion of two separate walls (although, in principle there is no reason to limit the massive resolution display 110, which might also be on the ceiling, or even, in some structures, on at least a portion of a floor).
Various widgets appear on the massive resolution display 110. Reference is now additionally made to Fig. 2, which is a depiction showing widget types identified for the display configuration of Fig. 1. Thus, from right to left, the massive resolution display 110 is depicted as displaying a weather widget 120, 220; a home security widget 130, 230; a TV sidebar widget 140, 240; a live TV widget 150, 250; and a social networking widget 160, 260.
Those of skill in the art will appreciate that the massive resolution display 110 typically comprises a plurality of display panels which are arranged to appear as a single massive display. The plurality of user interface widgets 120, 130, 140, 150, 160, 220, 230, 240, 250, 260 depicted in Figs. 1 and 2 are depicted by way of example. Any other appropriate user interface widget may appear, or, alternatively, one of the user interface widgets 120, 130, 140, 150, 160, 220, 230, 240, 250, 260 depicted in Figs. 1 and 2 may not appear in actual usage.
The components comprising the system of Figs. 1 and 2 are described below with reference to Fig. 7.
Reference is now made to Fig. 3A, which is a depiction of a thumbnail layout, referred to as a viewport 310, on a companion device 300 for the display configuration shown in Figure 1. The viewport 310 comprises an app depicted as appearing on the companion device 300 (the term "app" is a common short hand term for an "application", and is commonly used to refer to a specialized program downloaded onto a device, often, but not limited to a handheld or mobile device - for example, and without limiting the generality of the foregoing, a tablet computing device, or a smartphone).
The term "thumbnail" as used herein, in all of its grammatical forms, is understood to refer to a reduced-size image or representation of a corresponding widget on the massive resolution display. A number of thumbnails are depicted on the viewport 310 on the companion device 300. (For instance, a calendar widget on the massive resolution display will have a corresponding calendar thumbnail on the companion device.)
The term "companion device" as used herein, in all of its grammatical forms, is understood to refer to an additional electronic device (for example, and without limiting the generality of the foregoing, a tablet computing device, or a smartphone) that allows a viewer of the massive resolution display 110 to interact with any of the massive resolution display 110 itself or with any of the widgets displayed thereon.
The companion device 300 of Fig. 3 A is depicted running an app which displays the viewport 310 on the screen of the companion device 300. The viewport 310 comprises a thumbnail layout on which a representation of one of a complete or a partial section of the massive resolution display 110 appears. For example, the thumbnail images appearing on the viewport 310 (i.e. the weather thumbnail 320; the home security thumbnail 330, TV sidebar thumbnail 340, the live TV thumbnail 350, and the social networking thumbnail 360) correspond, respectively, to the weather widget 120, 220; a home security widget 130, 230; a TV sidebar widget 140, 240; a live TV widget 150, 250; and a social networking widget 160, 260 of Figs. 1 and 2.
The thumbnail images on the viewport 310 are rendered such that all objects appear proportionally having the same size and in the same position as the corresponding objects appear on the massive resolution display 110. However, due to the substantially lower resolution available on the companion device, graphical detail which appears on the massive resolution display 110 is not shown on the viewport 310. For example, and without limiting the generality of the foregoing, a widget displayed on the massive resolution display might show a cluster of images in full, while the corresponding thumbnail representation of the widget might be displayed as an outline of the cluster.
As depicted in Fig. 3 A, each of the thumbnails images appearing on the viewport 310 corresponding to each widget 120, 130, 140, 150, 160, 220, 230, 240, 250, 260 depicted in Figs. 1 and 2 displays headline status information within its boundaries, which information is specific and pertinent to the type of widget represented. For example and without limiting the generality of the foregoing, the live TV thumbnail 350 corresponding to a live TV widget 150 could show the current channel to which the live TV widget is tuned. The headline status information displayed comprises a presentation of the elements considered by the system to be the most important elements from a user's perspective of the widget's current state. For example, and without limiting the generality of the foregoing, the headline status information displayed comprises a subset of and in some cases complement to the information displayed on the main widget.
The live TV thumbnail 350 could also display now and next programming information. Also, the live TV thumbnail 350 may be branded or styled in such a way as to be visually similar at a glance to its larger counterpart on the remote display, thus to provide a further associational link.
The home security thumbnail 330 corresponding to the home security widget 130 may display a list of currently displayed home security cameras currently displayed in the home security widget 130, as well as a list of other home security cameras which are available to display in the home security widget 130.
Other exemplary widget types might include:
a calendar widget. The thumbnail (not depicted) of the calendar widget may display the current day displayed on the widget and details of upcoming appointments.
a social networking widget. The thumbnail 360 of the social networking widget may display the logged-in user's news feed, the date/time data last refreshed, and a list of available sources which can be switched on/off to filter the widget data (e.g. Facebook, Twitter, Instagram, and so forth). Reference is now made to Fig. 3B, which is a depiction of a large scale widget 370 and its corresponding thumbnail 410. Large scale widgets 370 on the massive resolution display 110 may be comprised of a number of panels 380, 385, 390, which have logically different operations available to be performed on them. In such cases, its corresponding thumbnail 410 will paint the outline of the panels 480, 485, 490 in proportionally the same positions as in the full scale widget 370, to facilitate quick recognition and selection by the user 100A, 100B.
Specifically, Fig. 3B depicts a social network widget 370 as it may appear on the massive resolution display 110. The social network widget 370 is depicted, in a non-limiting fashion, as comprising three panels: a panel comprising a feed of socially shared photos 380, a panel comprising a socially shared status update feed 385, and a panel comprising a view of linked content 390. The corresponding thumbnail 410 also comprises three corresponding panels, a photo stream thumbnail 480, a status stream thumbnail 485, and a linked content thumbnail 490.
Reference is now made to Fig. 4, which is a depiction of the use of a "pinch and zoom" gesture 500 in order to zoom in to the viewport 310 of Fig. 3 A. (It is appreciated that the "opposite" gesture would result in a zoom out.) The viewport 310 is zoomed in or out using a two finger pinch and drag, as is known in the art, and often referred to as a pinch and zoom gesture 500. The live TV thumbnail 350, which is depicted in Fig. 4 as being the widget which is zoomed in on appears in the center of Fig. 4, enlarged in accordance with the zoom. The TV sidebar thumbnail 340 appears on the right side of the viewport 310. Since the TV sidebar thumbnail 340 is also zoomed, the text (i.e. the headline status information as described above) which is displayed in the TV sidebar thumbnail 340 appears cut off. A fragment of the social networking thumbnail 360 also appears, on the left side of the viewport 310.
Reference is now made to Fig. 5, which is a depiction of a method for using a drag gesture 530 in order to scroll the viewport 310 of Fig. 3A. The viewport 310 is scrolled in the x-direction, the y-direction, or both directions using a single finger flick or drag gesture 530, as is known in the art. Selection of a thumbnail object, for example, the social networking thumbnail 360, as depicted in Fig. 5, in order to perform operations on the selected thumbnail object is accomplished by a single tap on the thumbnail while the viewport 310 is static (i.e. not still rendering scrolling motion as a result of a swipe or pinch and zoom gesture 500).
In some embodiments of the present invention, upon selection of one of the thumbnails 320, 330, 340, 350, 360, the viewport 310 will be automatically zoomed in to an appropriate detail level to display the selected one of the thumbnails 320, 330, 340, 350, 360, in order to ensure that the headline status information for the selected one of the thumbnails 320, 330, 340, 350, 360, is fully visible and legible. Selection of one of the thumbnails 320, 330, 340, 350, 360, will also render a "highlight" around the selected one of the thumbnails 320, 330, 340, 350, 360, in order to clearly indicate that the selected one of the thumbnails 320, 330, 340, 350, 360, is in the selected state.
In some embodiments of the present invention, when a thumbnail corresponding to a "simple" widget is selected, the thumbnail will render options available to the user 100A, 100B within the thumbnail boundary, thereby enabling quick selection. (It is appreciated that the term "simple" widget is intended as a verbal construct, referring to a widget which meets this requirement).
On the other hand, when a thumbnail corresponding to a "complex" widget is selected (refer to Fig. 3B, and the description thereof), the thumbnail selected will render a button enabling the user 100A, 100B to access a full-detail view of the thumbnail. The full-detail view of the thumbnail may fill the entire screen on the companion device. Navigating back to the massive resolution display 110 display viewport 310 view is achieved with the standard back button on the companion device 310, or, alternatively, with a swipe right gesture. When the user 100A, 100B has access to the full-detail view of the thumbnail, the user 100A, 100B has access to the complete feature set or configuration (if relevant) of the selected widget. By way of a non-limiting example, a live TV widget would be deemed a complex widget because the live TV widget needs a full screen to show an EPG grid. In the thumbnail view, the live TV widget might simply display now/next info for favorite channels. Reference is now made to Fig. 6, which is a depiction of the massive resolution display of Fig. 1 with the system of Fig. 1 in tracking mode.
Those skilled in the art will appreciate that for some tasks, it may be preferable for the user 100 A, 100B to interact with the app while remaining focused on the massive resolution display 110. (Having to move eyes between the screens of the massive resolution display 110 and companion device 300 excessively would be inefficient, and would likely detract from the user experience). Hence, to facilitate effective navigation in this mode, as the user 100A, 100B scrolls around the viewport 310 on companion device 330, the widget covering the center of the viewport 310 is highlighted (using a thicker border or other visual cue) on the massive resolution display 110. In this case, the live TV widget 150 is highlighted and the live TV thumbnail 350 covers the center of the viewport 310. Widget selection is implicit (i.e. a thumbnail is centered on the viewport, and the corresponding widget is selected based on no other user action), based on viewport 310 position and zoom - no tap is required. Hence, the full screen widgets mirror the selection on the viewport 310. This allows users to swipe and scroll on the companion device 330 while looking up at the massive resolution display 110, seeing the current widget selection moving as it is being tracked on massive resolution display 110.
This "tracking" feature is typically not the default system behavior because of the potential distraction to other individuals in the room watching elements of which appear on the massive resolution display 110, as the user 100A, HOB of the viewport 310 app navigates around the larger surface (i.e. the massive resolution display 110). Hence, it is a feature optionally switched on by the user 100A, 100B or when deemed appropriate by a display controller based (see the discussion below, with reference to Fig. 7) on the current widget layout.
It is appreciated, however, that when operating using this "tracking feature" the system allows basic manipulation of the selected widget without looking at the companion device, via further gestures on the companion device app, including, for example, but not limited to:
- Hold two fingers and move: to move the widget location and have the layout engine reflow the layout; - Hold two fingers and drag to very top of screen and hold there: make the widget it's maximal size across the display panel array and hide all other widgets;
- Hold two fingers and drag to very bottom of screen and hold there: to remove the widget from the display; and
- Fast swipe down then up with one or two fingers: to refresh the widget content immediately.
Reference is now made to Fig. 7, which is a system diagram depicting an embodiment of an implementation of the interface mechanism of Fig. 1. The components comprising the system 700 of Figs. 1 and 2 are now described. The massive resolution display 110, as was noted above, typically comprises an array of display panels 710. The array of display panels 710 is operative to display a plurality of user interface widgets. For example, and without limiting the generality of the foregoing, the weather widget 120; the home security widget 130; the TV sidebar widget 140; the live TV widget 150; and the social networking widget 160 are examples of typical user interface widgets which might be displayed on the array of display panels 710.
A plurality of display adaptors 720 implemented on at least one of a computer or processing device are in wired HDMI communication (or other appropriate communication protocol) with the array of display panels 710. Each one of the display adaptors of the plurality of display adaptors 720 control the display of an output on at least one display panel of the display panel array 710.
A layout engine 730 implemented on one of a computer or processing device arranges each of the plurality of user interface widgets 120, 130, 140, 150, 160 on the array of display panels 710. The layout engine 730 utilizes at least the following information in determining the layout of the plurality of user interface widgets 120, 130, 140, 150, 160 on the array of display panels 710:
metadata published by each of the plurality of user interface widgets 120, 130, 140, 150, 160 via a common interface, the common interface comprising a programmatic interface or a static manifest file, the metadata including, but not limited to display sizes and/or aspect ratios, and orientations which are supported on the output display array. These constraints must be respected and adhered to by the layout engine 730 in determining the layout of the overall interface;
a layout of the plurality of user interface widgets 120, 130, 140, 150, 160 presently in use on the display panel array 710; and
an area (i.e. the length x width) of the display panel array
710.
The layout engine 730 is operative to split a plurality of outputs among each one of the display adaptors 720. The plurality of outputs comprise data which each of the display adaptors 720 requires in order to display the layout, thereby producing the desired layout on the display panel array 710.
A system controller 740 implemented on one of a computer or processing device receives a multimedia input from at least one network adapter 750. For example and without limiting the generality of the foregoing, the network adapter 750 may input content for the various widgets, the content being provided via TCP/IP from various Internet services 755.
The system controller 740 manages runtime context (i.e. a sandboxed environment in which each widget process executes, that is to say, at least, the widget's code, data, its input/output, and current state ) for each of the plurality of user interface widgets 120, 130, 140, 150, 160. The system controller 740 is further operative to produce an output comprising at least a portion of the multimedia input for each one of the plurality of user interface widgets 120, 130, 140, 150, 160, the system controller being operative to provide the produced output to the layout engine 730.
The system controller 740 via the network adaptor 750 is further in bi-directional WIFI communication with via a WTFI module 760, utilizing either a wireless IEEE 802.11 home network; a Bluetooth connection, or other appropriate communication protocol, with the companion device 770. The companion device 770 is running a management application 780, such as the viewport app 310, described above.
The system controller 740 sends information on the layout and metadata about layout constraints for each of the plurality of user interface widgets to the management application 780 on the companion device 770. When a user 100A, HOB of the companion device 770 interacts with the management application 780 (via an appropriate user interface, such as the viewport app 310 described above), the management application 780 sends at least one of: layout changes; widget selection notifications; and widget commands to the system controller 740 for relay to the layout engine 730, thereby causing the least one of: layout changes; widget selection notifications; and widget commands to be rendered on the display panel array 710.
The area and configuration of the display panel array 710 is stored by the layout engine 730 during one of: installation of the system 700; and during a system 700 update. The system 700 update may be a software update in which any of the software components which are described herein as a part of the system 700 are updated, or alternatively, the system 700 update may be a hardware update, in which new hardware (for example, and without limiting the generality of the foregoing, new display panels may be added to the display panel array 710) is added, or already existing hardware is either replaced or upgraded.
Reference is now made to Fig. 8, which is a simplified flowchart diagram of preferred methods of operation of the system of Fig. 1. The method of Fig. 8 is believed to be self-explanatory in light of the above discussion.
It is appreciated that software components of the present invention may, if desired, be implemented in ROM (read only memory) form. The software components may, generally, be implemented in hardware, if desired, using conventional techniques. It is further appreciated that the software components may be instantiated, for example: as a computer program product or on a tangible medium. In some cases, it may be possible to instantiate the software components as a signal interpretable by an appropriate computer, although such an instantiation may be excluded in certain embodiments of the present invention.
It is appreciated that various features of the invention which are, for clarity, described in the contexts of separate embodiments may also be provided in combination in a single embodiment. Conversely, various features of the invention which are, for brevity, described in the context of a single embodiment may also be provided separately or in any suitable subcombination. It will be appreciated by persons skilled in the art that the present invention is not limited by what has been particularly shown and described hereinabove. Rather the scope of the invention is defined by the appended claims and equivalents thereof:

Claims

What is claimed is: CLAIMS
1. A system for controlling a display panel array, the display panel array comprising a plurality of display panels and being operative to display a plurality of user interface widgets, the system comprising: a plurality of display adaptors, each one of the plurality of display adaptors controlling a layout of an output on at least one display panel of the display panel array; a layout engine for arranging each of the plurality of user interface widgets on the display panel array, the layout engine arranging each of the plurality of user interface widgets on the display panel array by utilizing at least: metadata published by each of the plurality of user interface widgets, the metadata comprising constraints on a user interface widget aspect ratio or orientation; a layout of the plurality of user interface widgets presently in use on the display panel array; and an area of the display panel array, the layout engine being operative to split a plurality of outputs among each one of the display adaptors, the plurality of outputs comprising data which each of the display adaptors requires in order to display the layout, thereby producing the desired layout on the display panel array; a system controller which receives a multimedia input from at least one network adapter, the system controller managing runtime context for each of the plurality of user interface widgets, the system controller operative to produce an output comprising at least a portion of the multimedia input for each one of the plurality of user interface widgets, the system controller being operative to provide the produced output, as the layout of the plurality of user interface widgets presently in use, to the layout engine; the system controller being in bi-directional communication with a companion device external to the system, the companion device comprising one of a computer or processing device running a management application in bidirectional communication with the system controller; wherein the system controller sends information on the layout and metadata about layout constraints for each of the plurality of user interface widgets to the management application, such that when a user of the companion device interacts with the management application, the management application sends at least one of: layout changes; widget selection notifications; and widget commands to the system controller for relay to the system controller, thereby causing at least one of: layout changes; widget selection notifications; and widget commands to be rendered on the display panel array, and the plurality of display adaptors, the layout engine, and the system controller are each implemented on at least one of a computer or processing device.
2. The system of claim 1 wherein the area and configuration of the display panel array is stored by the layout engine during one of:
installation of the system; and
a system update.
3. The system of either claim 1 or claim 2 wherein the management application provides visual cues to the user of the companion device, thereby enabling fast locating and selection of objects.
4. The system of any of claims 1 - 3 wherein the plurality of user interface widgets presently in use on the display panel array is mirrored by a layout of thumbnail representations of the plurality of user interface widgets in the management application.
5. The system of claim 4 wherein each of the thumbnail representations appear proportionally in the same size and position as its corresponding user interface widget as the corresponding user interface widget appears on the display panel array.
6. The system of either claim 4 or claim 5 wherein each of the thumbnail representations appears in the management application with a lower graphical resolution than each corresponding user interface widget appears on the display panel array.
7. The system of any of claims 4 - 6 wherein each of the thumbnail representations displays headline status information within its boundaries.
8. The system of any of claims 1 - 6 wherein the user of the companion device interacts with the management application using standard gestures.
9. The system of claim 8 wherein the standard gestures include a pinch and zoom gesture; and a drag gesture.
10. The system of any of claims 1 - 9 wherein the metadata published by the user interface widget is published via a common application programming interface.
11. The system of any of claims 1 - 10 wherein the management application is operative to enter a tracking mode, causing a selected one of the plurality of user interface widgets to be highlighted on the display panel array.
The companion device of claim 1.
13. The management software of claim 1.
14. A method for controlling a display panel array, the display panel array comprising a plurality of display panels and being operative to display a plurality of user interface widgets, the method comprising: controlling a layout of an output on at least one display panel of the display panel array at each one of a plurality of display adaptors; arranging each of the plurality of user interface widgets on the display panel array by a layout engine, by utilizing at least: metadata published by each of the plurality of user interface widgets, the metadata comprising constraints on a user interface widget aspect ratio or orientation; a layout of the plurality of user interface widgets presently in use on the display panel array; and an area of the display panel array, the layout engine being operative to split a plurality of outputs among each one of the display adaptors, the plurality of outputs comprising data which each of the display adaptors requires in order to display the layout, thereby producing the desired layout on the display panel array; receiving at a system controller a multimedia input from at least one network adapter, the system controller managing runtime context for each of the plurality of user interface widgets, the system controller operative to produce an output comprising at least a portion of the multimedia input for each one of the plurality of user interface widgets, the system controller being operative to provide the produced output, as the layout of the plurality of user interface widgets presently in use, to the layout engine; communicating bi-directionally between the system controller and a companion device external to the system, the companion device comprising one of a computer or processing device running a management application in bidirectional communication with the system controller; wherein the system controller sends information on the layout and metadata about layout constraints for each of the plurality of user interface widgets to the management application, such that when a user of the companion device interacts with the management application, the management application sends at least one of: layout changes; widget selection notifications; and widget commands to the system controller for relay to the system controller, thereby causing at least one of: layout changes; widget selection notifications; and widget commands to be rendered on the display panel array, and the plurality of display adaptors, the layout engine, and the system controller are each implemented on at least one of a computer or processing device.
PCT/IB2014/060127 2013-04-04 2014-03-25 Interface mechanism for massive resolution displays WO2014162239A2 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
GB201306077A GB2512626B (en) 2013-04-04 2013-04-04 Interface mechanism for massive resolution displays
GB1306077.7 2013-04-04

Publications (2)

Publication Number Publication Date
WO2014162239A2 true WO2014162239A2 (en) 2014-10-09
WO2014162239A3 WO2014162239A3 (en) 2015-01-29

Family

ID=48483311

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/IB2014/060127 WO2014162239A2 (en) 2013-04-04 2014-03-25 Interface mechanism for massive resolution displays

Country Status (2)

Country Link
GB (1) GB2512626B (en)
WO (1) WO2014162239A2 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2017004707A1 (en) * 2015-07-09 2017-01-12 Genetec Inc. Security video monitoring client

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
FR3069676B1 (en) * 2017-07-28 2019-09-20 Psa Automobiles Sa DEVICE FOR PROVIDING A GRAPHICAL INTERFACE COMPRISING WIDGETS IN A VEHICLE.
CN111048016B (en) * 2018-10-15 2021-05-14 广东美的白色家电技术创新中心有限公司 Product display method, device and system

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20020163597A1 (en) * 2001-05-03 2002-11-07 Kevin Kelleher Digital multi-tile video display system
WO2005076993A2 (en) * 2004-02-09 2005-08-25 Regis Development, L.L.C. Computer presentation and command integration apparatus and method
EP2428947A2 (en) * 2010-09-08 2012-03-14 Lg Electronics Inc. Terminal and contents sharing method for terminal
US20120084663A1 (en) * 2010-10-05 2012-04-05 Citrix Systems, Inc. Display Management for Native User Experiences
WO2012153290A1 (en) * 2011-05-10 2012-11-15 Nds Limited Adaptive presentation of content

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7812786B2 (en) * 2005-01-18 2010-10-12 Nokia Corporation User interface for different displays
US20120319927A1 (en) * 2008-12-02 2012-12-20 Nvidia Corporation Remote management of a simultaneous display of multimedia content in display devices

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20020163597A1 (en) * 2001-05-03 2002-11-07 Kevin Kelleher Digital multi-tile video display system
WO2005076993A2 (en) * 2004-02-09 2005-08-25 Regis Development, L.L.C. Computer presentation and command integration apparatus and method
EP2428947A2 (en) * 2010-09-08 2012-03-14 Lg Electronics Inc. Terminal and contents sharing method for terminal
US20120084663A1 (en) * 2010-10-05 2012-04-05 Citrix Systems, Inc. Display Management for Native User Experiences
WO2012153290A1 (en) * 2011-05-10 2012-11-15 Nds Limited Adaptive presentation of content

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
PHILIP I S LEI ET AL: "The Multiple-Touch User Interface Revolution", IT PROFESSIONAL, IEEE SERVICE CENTER, LOS ALAMITOS, CA, US, vol. 11, no. 1, 1 January 2009 (2009-01-01), pages 42-49, XP011241580, ISSN: 1520-9202, DOI: 10.1109/MITP.2009.19 *

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2017004707A1 (en) * 2015-07-09 2017-01-12 Genetec Inc. Security video monitoring client

Also Published As

Publication number Publication date
GB2512626B (en) 2015-05-20
GB2512626A (en) 2014-10-08
GB201306077D0 (en) 2013-05-22
WO2014162239A3 (en) 2015-01-29

Similar Documents

Publication Publication Date Title
AU2019216686B2 (en) System for organizing and displaying information on a display device
KR101867644B1 (en) Multi-application environment
US20220230603A1 (en) Adaptable user interface with dual screen device
WO2019140976A1 (en) Method and device for operating smart interactive tablet, and smart interactive tablet
US9268423B2 (en) Definition and use of node-based shapes, areas and windows on touch screen devices
EP3859501B1 (en) Application execution method by display device and display device thereof
KR102391699B1 (en) Dynamic joint dividers for application windows
KR102027612B1 (en) Thumbnail-image selection of applications
US8302026B2 (en) Multi-panel user interface
US10261741B2 (en) Content sharing with consistent aspect ratios
US20100293501A1 (en) Grid Windows
EP2972735B1 (en) User interface for toolbar navigation
JP2008276776A (en) Touch-type tab navigation method and related device
US10656790B2 (en) Display apparatus and method for displaying a screen in display apparatus
US9513795B2 (en) System and method for graphic object management in a large-display area computing device
US8872813B2 (en) Parallax image authoring and viewing in digital media
JP5905783B2 (en) Image display system
WO2014162239A2 (en) Interface mechanism for massive resolution displays
JP6392573B2 (en) Multi display system
US20140184535A1 (en) Method for Displaying Children Components in a Display that Operates at Different Screen Parameters and Orientations
KR20140074490A (en) Image display apparatus and contents control method of the same
US10768803B2 (en) User interface system with active and passive display spaces
JP6327834B2 (en) Operation display device, operation display method and program
JP6068428B2 (en) Image display system control method and control apparatus
US20160227151A1 (en) Display apparatus, remote control apparatus, remote control system and control method thereof

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 14718171

Country of ref document: EP

Kind code of ref document: A2

122 Ep: pct application non-entry in european phase

Ref document number: 14718171

Country of ref document: EP

Kind code of ref document: A2