WO2014008438A1

WO2014008438A1 - Systems and methods for tracking user postures and motions to control display of and navigate panoramas

Info

Publication number: WO2014008438A1
Application number: PCT/US2013/049390
Authority: WO
Inventors: Alexander I. GORSTAN; Charles Robert ARMSTRONG; Kang S. Lim
Original assignee: Tourwrist, Inc
Priority date: 2012-07-03
Filing date: 2013-07-03
Publication date: 2014-01-09

Abstract

A panoramic display system includes a camera, a processor and a display device for displaying images for a user. The camera recognizes a facial location and a facial orientation of a user relative to the display device, and tracks the pupil orientation of the user relative to the display device. The processor derives an object of interest base on the facial location and the pupil orientation of the user. The processor can also derive a field of view of the user based on the facial location and the facial orientation of the user. Also, a mobile device is configured to teleshift from a first lateral viewing perspective to a second lateral viewing perspective of a virtual tour object. The mobile device includes a sensor, a processor and a display. The sensor detects a teleshifting motion of the mobile device caused by a user, and the processor determines if a magnitude of the teleshifting motion is greater than a threshold. If the magnitude of the teleshifting motion is greater than the threshold, then the display teleshifts by transitioning from a first lateral viewing perspective to a second lateral viewing perspective of the virtual tour.

Description

SYSTEMS AND METHODS FOR TRACKING USER POSTURES AND

MOTIONS TO CONTROL DISPLAY OF AND NAVIGATE PANORAMAS

BACKGROUND

[0001] The present invention relates to systems and methods for efficiently storing, displaying and navigating panoramas. More particularly, the present invention relates to storing panoramic image data with focal metadata thereby enabling users to subsequently experience pseudo three-dimensional panoramas. The present invention also relates to offering, retrieving and presenting panoramas with supplemental data thereby enabling users to view enhanced panoramic images.

[0002] The increasing wideband capabilities of wide area networks and proliferation of smart devices has been accompanied by the increasing expectation of users to be able to experience three-dimensional (3D) viewing in real-time during a panoramic tour.

[0003] However, conventional techniques for storing and transmitting three- dimensional images in high resolution images require a lot of memory and bandwidth, respectively. Further, attempts at "shoot first and focus later" still images have been made, but require specialized photography equipment (for example, light field cameras having a proprietary micro-lens array coupled to an image sensor such as those from Lytro, Inc. of Mountain View, California).

[0004] It is therefore apparent that an urgent need exists for efficiently storing and displaying in real-time 3-D-like panoramic images without substantially increasing storage or transmission requirements.

[0005] Presently, the increasing wideband capabilities of wide area networks and proliferation of smart devices has been accompanied by the increasing expectation of users to be able to experience viewing of panoramas in real-time with supplemental information on-demand. However, conventional techniques for storing and retrieving panoramas with supplemental data are generally unintuitive and/or cumbersome.

[0006] Further, in many viewing circumstances, it may be preferable for the user to control their viewing experience, for example, affecting which supplemental information is displayed, through physical movement of their mobile device. [0007] It is also apparent that an urgent need exists for efficiently offering, retrieving and presenting panoramas with supplemental data thereby enabling users to view enhanced panoramic images with optional intuitive user motion controls.

SUMMARY

[0008] To achieve the foregoing and in accordance with the present invention, systems and methods for efficiently storing, displaying and navigating panoramas are provided. In particular, these systems store panoramic image data with focal metadata thereby enabling users to be able to navigate and to experience pseudo three- dimensional panoramas.

[0009] In one embodiment, a display system includes a camera, a processor and a display device for displaying images for a user. The camera is configured to recognize a current facial location and a current facial orientation of a user relative to the display device, and to track the current pupil orientation of the user relative to the display device.

[0010] The processor can be configured to derive a current object of interest based on the facial location and the pupil orientation of the user. The processor can also be configured to derive a current field of view (FOV) of the user based on the current facial location and the current facial orientation of the user.

[0011] In some embodiments, the processor is further configured to retrieve image data associated with a panorama, and to retrieve flex-focal metadata associated with the panorama for at least two focal distances. The processor can process the image data and flex-focal metadata in accordance with the computed current user FOV of the user and generate a current image of the panorama for the display device.

[0012] In another embodiment, a mobile device is configured to teleshift from a first lateral viewing perspective to a second lateral viewing perspective of a virtual tour object. The mobile device includes a sensor, a processor and a display. The sensor is configured to detect a teleshifting motion of the mobile device caused by a user, and the processor is configured to determine if a magnitude of the teleshifting motion is greater than a threshold. If the magnitude of the teleshifting motion is greater than the threshold, then the display teleshifts by transitioning from a first lateral viewing perspective to a second lateral viewing perspective. The first lateral viewing perspective and the second video lateral viewing perspective may be adjacent lateral viewing perspectives of the virtual tour.

[0013] In some embodiments, the teleshifting includes teleturning from the first lateral viewing perspective to the second lateral viewing perspective located around an object of interest of the virtual tour.

[0014] Note that the various features of the present invention described above may be practiced alone or in combination. These and other features of the present invention will be described in more detail below in the detailed description of the invention and in conjunction with the following figures.

BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order that the present invention may be more clearly ascertained, some embodiments will now be described, by way of example, with reference to the accompanying drawings, in which:

[0016] Figure 1 is an exemplary flow diagram illustrating the capture of flex- focal images for pseudo three-dimensional viewing in accordance with one embodiment of the present invention;

[0017] Figures 2A and 2B illustrate in greater detail the capture of flex-focal images for the embodiment of Figure 1;

[0018] Figure 3 A is a top view of a variety of exemplary objects (subjects) at a range of focal distances from the camera;

[0019] Figure 3B is an exemplary embodiment of a depth map relating to the objects of Figure 3 A;

[0020] Figure 4 is a top view of a user with one embodiment of a panoramic display system capable of detecting the user's field of view, perspective and/or gaze, and also capable of displaying pseudo 3-D panoramas in accordance with the present invention;

[0021] Figure 5 is an exemplary flow diagram illustrating field of view, perspective and/or gaze detection for the embodiment of Figure 4;

[0022] Figure 6 is an exemplary flow diagram illustrating the display of pseudo

3-D panoramas for the embodiment of Figure 4; [0023] Figures 7-11 are top views of the user with the embodiment of Figure 4, and illustrate field of view, perspective and/or gaze detection and also illustrates generating pseudo 3-D panoramas;

[0024] Figures 12 and 13 illustrate two related front view perspectives corresponding to a field of view for the embodiment of Figure 4;

[0025] Figures 14 and 15 are exemplary flow diagrams illustrating the selection, retrieval and presentation of panoramas with supplemental data in accordance with one embodiment of the present invention;

[0026] Figure 16 is a mobile device screenshot with an exemplary menu of user selectable panoramic images for the embodiment of Figure 14;

[0027] Figure 17 is a mobile device screenshot with an exemplary menu of user selectable supplemental data for the embodiment of Figure 14;

[0028] Figures 18 to 22 are screenshots of exemplary panoramas with and without supplemental data for the embodiment of Figure 14;

[0029] Figure 23 is a perspective view showing the three exemplary rotational axes for the mobile device of Figure 16;

[0030] Figure 24 is a front view illustrating the Y-axis rotation useful for navigational control of the mobile device of Figure 16; and

[0031] Figure 25 is a top view illustrating a plurality of exemplary user viewing perspectives associated with navigating virtual tours using the mobile device of Figure 16.

DETAILED DESCRIPTION

[0032] The present invention will now be described in detail with reference to several embodiments thereof as illustrated in the accompanying drawings. In the following description, numerous specific details are set forth in order to provide a thorough understanding of embodiments of the present invention. It will be apparent, however, to one skilled in the art, that embodiments may be practiced without some or all of these specific details. In other instances, well known process steps and/or structures have not been described in detail in order to not unnecessarily obscure the present invention. The features and advantages of embodiments may be better understood with reference to the drawings and discussions that follow.

[0033] The present invention relates to systems and methods for efficiently storing panoramic image data with flex-focal metadata for subsequent display, thereby enabling a user to experience pseudo three-dimensional panoramas derived from two- dimensional image sources.

[0034] To facilitate discussion, Figure 1 is an exemplary flow diagram 100 illustrating the capture of panoramic images for pseudo three-dimensional viewing in accordance with one embodiment of the present invention. Note that the term

"perspective" is used to describe as a particular composition of an image with a defined field of view ("FOV"), wherein the FOV can be defined by one or more FOV

boundaries. For example, a user's right eye and left eye see two slightly different perspectives of the same FOV, enabling the user to experience stereography. Note also that "gaze" is defined as a user's perceived region(s)/object(s) of interest.

[0035] Flow diagram 100 includes capturing and storing flex-focal image(s) with associated depth map(s) (step 110), recognizing a user's FOV, perspective, and/or gaze (step 120), and then formulating and displaying the processed image(s) for composing a panorama (step 130).

[0036] Figures 2 A and 2B are flow diagrams detailing step 110 and illustrating the capture of flex-focal image(s) and associated depth map(s) with flex-focal metadata, while Figure 3 A is a top view of a variety of exemplary objects (also referred by photographers and videographers as "subjects"), person 330, rock 350, bush 360, tree 370 at their respective focal distances 320d, 320g, 320j, 3201 from a camera 310.

[0037] Figure 3B shows an exemplary depth map relating to the objects 330,

350, 360 and 370. Depth map 390 includes characteristics for each identified object, such as region/object ID, region/object vector, distance, opacity, color information and other metadata. Useful color information can include saturation and contrast (darkness).

[0038] In this embodiment, since most objects of interest are solid and opaque, the respective front surfaces of objects can be used for computing focal distances.

Conversely, for translucent or partially transparent objects, the respective back surfaces can be used for computing focal distances. It is also possible to average focal distances of two or more appropriate surfaces, e.g., average between the front and back surfaces for objects having large, multiple and/or complex surface areas.

[0039] As illustrated by the exemplary flow diagrams of Figures 2A and 2B, an image is composed using camera 310 and the image capture process is initiated (steps 210, 220). In this embodiment, the focal distance (sometimes referred to as focal plane or focal field) of camera 230 is initially set to the nearest one or more regions/objects, e.g., person 330, at that initial focal distance (step 230). In step 240, the image data and/or corresponding flex-focal metadata can be captured at appropriate settings, e.g., exposure setting appropriate to the color(s) of the objects.

[0040] As shown in step 250, the flex-focal metadata is derived for a depth map associated with the image. Figure 2B illustrates step 250 in greater detail. Potential objects (of interest) within the captured image are identified by, for example, using edge and region detection (step 252). Region(s) and object(s) can now be enumerated and hence separately identified (step 254). Pertinent region/object data such as location (e.g., coordinates), region/object size, region/object depth and/or associated

region/object focal distance(s), collectively, flex-focus metadata can be appended into the depth map (step 256).

[0041] Referring back to Figure 2A, in steps 260 and 270, if the focal distance of camera 310 is not yet set to the maximum focal distance, i.e., set to "infinity", and then the camera focal distance is set to the next farther/farthest increment or next farther region or object, e.g., shrub 340. The process of capturing pertinent region/object data, i.e., flex-focal metadata is repeated for shrub 340 (steps 240 and 250).

[0042] This iterative cycle comprising of steps 240, 250, 260 and 270 continues until the focal distance of camera 310 is set at infinity or the region(s)/object(s) and corresponding flex-focal metadata of any remaining potential region(s)/object(s) of interest, e.g., rock 350, bush 360 and tree 370, have been captured. It should be appreciated that the number of increments for the focal distance is a function of the location and/or density of region(s)/object(s), and also the depth of field of camera 310.

[0043] Figure 4 is a top view of a user 480 with one embodiment of a panoramic display system 400 having a camera 420 capable of detecting a user's field of view ("FOV"), perspective and/or gaze, and also capable of displaying pseudo 3-D panoramas in accordance with the present invention. Figure 5 is an exemplary flow diagram illustrating FOV, perspective and/or gaze detection for display system 400, while Figure 6 is an exemplary flow diagram illustrating the display of pseudo 3-D panoramas for display system 400.

[0044] Referring to both the top view of Figure 4 and the flow diagram of Figure

5, camera 420 has an angle of view ("AOV") capable for detecting user 480 between AOV boundaries 426 and 428. Note that AOV of camera 420 can be fixed or adjustable depending on the implementation.

[0045] Using facial recognition techniques known to one skilled in the art, camera 420 identifies facial features of user 480 (step 510). The location and/or orientation of user's head 481 relative to a neutral position can now be determined, for example, by measuring the relative distances between facial features and/or orientation of protruding facial features such as nose and ears 486, 487 (step 520).

[0046] In this embodiment, in addition to measuring the absolute and/or relative locations and/or orientations of user's eyes with respect to the user's head 481, the camera 420 can also measure the absolute and/or relative locations and/or orientations of user's pupils with respect to the user's head 481 and/or user's eye sockets (step 530).

[0047] Having determined the location and/or orientation of the user's head and/or eyes as described above, display system 400 can now compute the user's expected field of view 412 ("FOV"), as defined by FOV boundaries 422, 424 of Figure 4 (step 540).

[0048] In this embodiment, having determined the location and/or orientation of the user's head, eyes, and/or pupils, display system 400 can also compute the user's gaze 488 (see also step 540). The user's gaze 488 can in turn be used to derive the user's perceived region(s)/object(s) of interest by, for example, triangulating the pupils' perceived lines of sight.

[0049] Referring now to the top view of Figure 4 and the flow diagram of Figure

6, the user's expected FOV 412 (defined by boundaries 422, 424), perspective and/or perceived region(s)/object(s) of interest have (derived from gaze 488) have been determined in the manner described above. Accordingly, the displayed image(s) for the panorama can be modified to accommodate the user's current FOV 412, current perspective and/or current gaze 488, thereby providing the user with a pseudo 3-D viewing experience as the user 480 moves his head 481 and/or eye pupils 482, 484. [0050] In step 610, the display system 400 adjust the user's FOV 412 of the displayed panorama an appropriate amount in the appropriate, e.g., opposite, direction relative to the movement of user's head 481 and eyes.

[0051] If the to-be-displayed panoramic image(s) are associated with flex-focal metadata (step 620), then system 400 provides user 480 with the pseudo 3-D experience by inferring e.g., using interpolation, extrapolation, imputation and/or duplication, any previously obscured image data exposed by any shift in the user's perspective (step 630).

[0052] In some embodiments, display system 400 may also emphasize region(s) and/or object(s) of interest derived from the user's gaze by, for example, focusing the region(s) and/or object(s), increasing the intensity and/or the resolution of the region(s) and/or object(s), and/or decreasing the intensity and/or the resolution of the region(s) and/or object(s), and/or defocusing the foreground /background of the image (step 640).

[0053] Figures 7-11 are top views of the user 480 with display system 400, and illustrate FOV, perspective and/or gaze detection for generating pseudo 3-D panoramas. Referring first to Figure 7, camera 340 determines that the user's head 481 and nose are both facing straight ahead. However the user's pupils 482, 484 are rotated rightwards within their respective eye sockets. Accordingly, the user's resulting gaze 788 is offset towards the right of the user's neutral position.

[0054] In Figure 8, the user's head 481 is facing leftwards, while the user's pupils 782, 784 are a neutral position relative to their respective eye sockets. Hence, the user's resulting gaze 888 is offset toward the left of the user's neutral position.

[0055] Figures 9 and 10 illustrate the respective transitions of the field of view

(FOV) provided by display 430 whenever the user 480 moves towards and away from display 430. For example, when user 480 moves closer to display 430 as shown in Figure 9, the FOV 912 increases (see arrows 961, 918) along with the angle of view as illustrated by the viewing boundaries 922, 924. Conversely, as shown in Figure 10 when user 480 moves further away from display 430, the FOV 1012 decreases (see arrows 1016, 1018) along with the angle of view as illustrated by the viewing boundaries 1022, 1024. In both examples, user gazes 988, 1088 are in the neutral position. [0056] It is also possible for user 480 to move laterally relative to display 430.

Referring to exemplary Figure 11, as user 480 moves laterally toward the user's right shoulder and turns head 418 towards the left shoulder. As a result, the FOV 1112 is shifted towards the left (see arrows 1116, 1118) as illustrated by viewing boundaries 1122, 1124. In this example, user gaze 1188 is also in the neutral position.

[0057] Figures 12 and 13 show an exemplary pair of related front view perspectives 1200, 1300 corresponding to a user's field of view, thereby substantially increasing the perception of 3-D viewing of a panorama including objects of interest, person 330, rock 350, bush 360, tree 370 (see Figure 3A). In this example, as illustrated by Figure 11, when viewing user 480 moves laterally towards the user's right shoulder, the change in perspective (and/or FOV) can result in the exposure of a portion 1355 of rock 350 as shown in Figure 13, which had been previously obscured by person 330 as shown in Figure 12. The exposed portion 1355 of rock 350 can be inferred in the manner described above.

[0058] Many modifications and additions are also possible. For example, instead of a single camera 420, system 400 may have two or more strategically located cameras which should increase to accuracy and possibly speed of determining FOV, perspective and/or gaze of user 480.

[0059] It is also possible to determine FOV, perspective and/or gaze using other methods such as using the user's finger(s) as a joystick, or using a pointer as a joystick. It should be appreciated that various representations of flex-focal metadata are also possible, including different data structures such as dynamic or static tables, and vectors.

[0060] In sum, the present invention provides systems and methods for capturing flex-focal imagery for pseudo three-dimensional panoramic viewing. The advantages of such systems and methods include enriching the user viewing experience without the need to also substantially increasing bandwidth capability and storage capacity.

[0061] The present invention also relates to systems and methods for offering, retrieving and presenting panoramas with optional supplemental data, and navigating the viewing experience with, for example, user motion controls. To facilitate discussion, Figures 14 and 15 are exemplary flow diagrams illustrating the selection, retrieval and presentation of panoramas with supplemental data for mobile devices in accordance with one embodiment of the present invention. Figure 16 is a screenshot showing an exemplary menu of user selectable panoramic images for a mobile device 1300, while Figure 17 is a screenshot showing an exemplary menu of user selectable supplemental data for mobile device 1300. Note that the term "mobile device" is used to describe a variety of portable electronic appliances including cellular phones, tablets, laptops and cameras. Note also that panoramic images (also referred to as panoramas) are used to describe a variety of images including both static and moving images and also virtual tours.

[0062] In this embodiment, mobile device 1300 receives a user request for a panorama which may be selected by the user (not shown) from a customizable menu of choices as shown in Figure 16 (step 1110). As shown in the exemplary screenshot 1310 of Figure 16, mobile device 1300 offers choices of panoramic icons, for example, geographical locations such as "Pebble Beach" 1321, "Paris" 1322, "Cape Cod" 1323, "New York" 1324 ... "Las Vegas" 1328 and "San Francisco" 1329.

[0063] The mobile device 1300 may respond to the panorama request by offering the user one or more customizable optional forms of supplemental data from menu (step 1120). Supplemental data may be based on, for example, metadata such as visual data from the panorama itself or any objects or individuals displayed within the panorama, the known location of the environment shown in the panorama, the known weather at the location displayed within the panorama, the seasonal or daily time at which the panorama is being viewed, or personal data known to pertain to the user. In Figure 17, exemplary screenshot 1410 of mobile device 1300 provides the user with a plurality of supplemental data choices such as "weather" 1421, "geographical distance and/or direction" 1422, "proximate contacts" 1423, "favorite restaurants" 1424 and "lodging choices" 1429, described in greater detail below. Other examples of supplemental data include targeted messages including advertisements and/or announcements for products, services, and/or events.

[0064] In steps 1130 and 1140, if the user elects to display one or more supplemental data, then the mobile device 1300 retrieves and displays the optional supplemental data together with the requested panorama.

[0065] Referring now to Figure 15 which illustrated step 1140 in greater detail, mobile device 1300 sends a request for supplemental data, e.g., by sending reference metadata, to a (real-time) datasource server(s) via for example a wide area network such as the Internet (step 1241). The datasource server(s) can be one or more of other mobile devices up to large stationary dedicated data storage facilities.

[0066] In step 1242, if the requested supplemental data is associated with placement data, then the server provides both supplemental data and associated placement data to be presented by mobile device 1300 to the user (steps 1243, 1244). Conversely, in step 1242, if the requested supplemental data does not require placement, then the server provides supplemental data be presented by mobile device 1300 to the user (steps 1245, 1246).

[0067] In some embodiments, the mobile device 1300 is pre-loaded with and/or caches the supplemental data, and hence only requires periodic updates from the datasource server(s). It may also possible to share and update supplemental data amongst groups of users.

[0068] As discussed above and illustrated by the screenshot 1550 of Figure 18, if the user selects supplemental data choice 1421 which is the "weather", then the default current local weather may be overlaid onto the scenery of the original screenshot 1510.

[0069] Supplemental geographical data may also be displayed as shown in screenshot 1650 of Figure 19, wherein the distance from the user's location is shown in the top right of the original scenery 1610.

[0070] Referring now to the screenshot 1750 of Figure 20, it is also possible for the user to select the display of contact(s), such as friend(s), business associate(s) and/or favorite restaurant(s) or hotel(s) together with the original scenery 1710. The server may also provide associated placement data for these contact(s) so that the contact(s) may be displayed proximate to their respective locations within the scenery. It is also possible for the server to provide mobile device 1300 with contact information associated with these contacts for display.

[0071 ] In the exemplary screenshot 1850 of Figure 21 , targeted notices such as wrinkle cream advertisement 1856 and/or shoe advertisement 1858 may also be displayed together with the original scenery 1810.

[0072] As exemplified by the daytime screenshot 1910 and nighttime screenshot

1950 of Figure 22, supplemental data can include temporal data such as current date and/or time. Accordingly, a different panoramic image may be selected to correspond with the current or specified time and/or date. [0073] In some embodiments, supplemental data choices may also be combined by the user. For example, choosing both "weather" 4121 and "lodging" 1429 may result in the overlaying of current weather and also lodging locations that have vacancies at the displayed geographic location.

[0074] Alternatively, if the user chooses "weather" 1421 and "current time or season" (not shown), the resulting display on mobile device 1300 may include temporal weather, i.e., the local weather at a specific season, date and/or time. Other exemplary combinations include hotel room availability and dinner reservation availability, and travel time estimates, each of which require an understanding of the location and date/time. In the case of travel time, other data sources such as weather and traffic conditions can also be combined.

[0075] Figure 23 is a perspective view showing the three exemplary rotational axes for the mobile device 1300, while Figure 24 is a front view illustrating the Y-Axis rotation useful for menu navigational control of the mobile device 1300.

[0076] In some embodiments, mobile device 1300 includes one or more accelerometer(s), magnetometer(s), gyroscope(s) and/or imaging sensor(s) (not shown) for measuring the angular rotations along the X-Axis 2002, Y-Axis 2003, and Z-Axis 2004. Suitable accelerometers, magnetometers, gyroscopes, and imaging sensors for mobile device 1100 are commercially available from a variety of manufacturers including ST Electronics Ltd of Berkshire, United Kingdom, AKM Semiconductor Inc. of San Jose, California, InvenSense Inc. of Sunnyvale, California, and Sony Electronics of San Diego, California.

[0077] In order to enable the user's hand-held mobile device 1300 to navigate the supplemental data menu without the need to use touch-screen or physical buttons of mobile device 1300, translational planar and/or angular acceleration may be measured using, for example, the mobile device 1300's accelerometer, magnetometer, gyroscope and/or image sensor.

[0078] Accordingly, rotational angular acceleration can be used as a menu navigational control of mobile device 1300, namely, a quick rotation in the Y-Axis rotation 2003 to "flick" mobile device 1300 in the "clockwise" or "counter-clockwise" axially. This somewhat "abrupt" rotation in the Y-Axis 2003 may be performed in a short, finite period of time to better discern the user's desire to flick mobile device 1300, rather than a relatively slower rotation intended to, for example, adjusting the horizon of the scenery.

[0079] To successfully register a valid "clockwise" flick, mobile device 1300 should for example achieve between approximately 20° to approximately 45° in relative Y-Axis rotation within approximately 500 milliseconds. Conversely, to successfully register a "counter-clockwise" flick, mobile device 1100 should for example achieve between approximately -20° to approximately -45° in relative Y-Axis rotation within approximately 500 milliseconds.

[0080] In this embodiment as shown in Figure 17, flicking "clockwise" causes the mobile device 1300 to advance to the next menu choice to the "right" of the current menu choice. Conversely, flicking "counter-clockwise" causes the mobile device 1300 to advance to the next menu choice to the "left" of the current menu choice. For example, a "clockwise" flick of mobile device 1300 may cause mobile device 1300 to transition from displaying the contact location(s) to displaying the dining choice(s), i.e., transition from icon 1423 to icon 1424.

[0081] The above described menu navigational control for mobile device 1300 can be implemented in place of or in addition to a touchscreen based menu navigational control. It is also possible to use the above described Y-Axis flick(s) to scroll the menu choice(s) in combination with X-Axis flick(s) to select specific menu choice(s).

[0082] The above described detection of flicking motion(s) of mobile device

1300, in one or more of the X-Axis, Y-Axis and/or Z-Axis, can also be used to navigate panoramas and/or virtual tours.

[0083] For example, as illustrated by Figure 25, a top view illustrating a plurality of user viewing perspectives 2280a, 2280b, 2280c, 2280d, 2280e and 2280f, a user can use "right" flicks and/or "left" flicks of mobile device 1300 in the Z-Axis, i.e.,

"teleshift" motions to laterally navigating during a virtual tour. In this example, teleshifting includes "teleturning" from a first lateral viewing perspective to a second lateral viewing perspective around an object of interest, e.g., from perspective 2280a to perspective 2280b positioned around car 2210.

[0084] In this exemplary embodiment, to successfully register a valid "right" flick, mobile device 1300 should for example achieve between approximately 20° to approximately 45° in relative Z-Axis rotation within approximately 500 milliseconds. Conversely, to successfully register a "left" flick, mobile device 1100 should for example achieve between approximately -20° to approximately -45° in relative Z-Axis rotation within approximately 500 milliseconds. Accordingly, the user viewing car 2210 can use a "right" flick to transition from viewing perspective 2280c to viewing perspective 2280d, and/or use a "left" flick to transition from viewing perspective 2280c to viewing perspective 2280b.

[0085] The user may also use double "right" or "left" flicks of mobile device

1300 to continually view around car 2210 in the right or left directions, respectively. In this continually laterally "moving" viewing mode, a flick of mobile device 1300 in the opposite direction can be used to freeze the user's viewing perspective.

[0086] It is also possible to use the above described Z-Axis flick(s) to laterally transition viewing perspective in combination with X-Axis flick(s) to cause the user's viewpoint to advance and/or to retreat. For example, a "forward" flick can be accomplished by quickly rotating the top of mobile device 1300 away from the user, thereby causing the user viewpoint to advance from the exterior of car 2210 into the interior of car 2210. Conversely, a "backward" flick can be accomplished by quickly rotating the top of mobile device 1300 toward the user, thereby causing the user viewpoint to retreat from the interior of car 2210 back to viewing the exterior of car 2210.

[0087] In sum, the present invention provides systems and methods for offering, retrieving and presenting panoramas with optional supplemental data. The advantages of such systems and methods include providing contextually relevant details which may not be readily apparent or available through panoramic imagery alone, more fully immersing a user in a panoramic environment, and allowing a user to affect their view or the data presented through more natural, tactile methods than afforded by conventional virtual or physical button pressing.

[0088] While this invention has been described in terms of several embodiments, there are alterations, modifications, permutations, and substitute equivalents, which fall within the scope of this invention. It should also be noted that there are many alternative ways of implementing the methods and apparatuses of the present invention. It is therefore intended that the following appended claims be interpreted as including all such alterations, modifications, permutations, and substitute equivalents as fall within the true spirit and scope of the present invention.

Claims

CLAIMS What is claimed is:

1. A computerized method for recognizing a current user field of view and a current gaze, useful in association with a display device, the method comprising: while a user is viewing an image on a display device: determining a current facial location of the user relative to the display device; determining a current facial orientation of the user relative to the display device; deriving a current field of view (FOV) of the user based on the facial location and the facial orientation of the user. tracking at least one current pupil orientation of the user relative to the display device; and deriving a current object of interest based on the facial location and the pupil orientation of the user.

2. The method of claim 1 wherein the image is a panorama.

3. The method of claim 2 further comprising: retrieving image data associated with the panorama; retrieving flex-focal metadata associated with the panorama for the at least two focal distances; and wherein while the user is viewing the panorama on the display device: processing the image data and flex-focal metadata in accordance with the computed current user FOV of the user and generating a current panoramic image; and displaying the current panoramic image on the display device.

4. The method of claim 1 further comprising determining the current perspective and wherein generating the current panoramic image includes inferring obscured image data derived from the current perspective.

5. A computerized method for recognizing a current object of interest, useful in association with displaying panoramas, the method comprising: while a user is viewing an image on a display device: determining a current facial location of the user relative to the display device; tracking at least one current pupil orientation of the user relative to the display device; and deriving a current object of interest based on the facial location and the pupil orientation of the user.

6. The method of claim 5 wherein the image is a panorama.

7. The method of claim 6 further comprising: retrieving image data associated with the panorama; retrieving flex-focal metadata associated with the panorama for the at least two focal distances; and wherein while the user is viewing the panorama on the display device: processing the image data and flex-focal metadata in accordance with the computed current user FOV of the user and generating a current panoramic image; and displaying the current panoramic image on the display device.

8. The method of claim 1 further comprising emphasizing the object of interest.

9. The method of claim 8 further comprising deemphasizing at least one background object.

10. A panoramic display system configured to display panoramas for a user, the display system comprising: a camera configured to: recognize a current facial location and a current facial orientation of a user relative to a display device; and track at least one current pupil orientation of the user relative to the display device; and a processor configured to deriving a current object of interest based on the facial location and the pupil orientation of the user.

11. The panoramic display device of claim 10 wherein the processor is further configured to derive a current field of view (FOV) of the user based on the current facial location and the current facial orientation of the user.

12. The display system of claim 11 further comprises a display device configured to display a panorama, and wherein the processor is further configured to: retrieving image data associated with the panorama; retrieving flex-focal metadata associated with the panorama for at least two focal distances; and processing the image data and flex-focal metadata in accordance with the computed current user FOV of the user and generating a current image of the panorama for the display device.

13. The display system of claim 10 wherein the processor is further configured to emphasize the object of interest.

14. The display system of claim 13 wherein the processor is further configured to deemphasize at least one background object.

15. The panoramic display device of claim 10 wherein the processor is further configured to derive a current field of view (FOV) of the user based on a finger location of the user.

16. The panoramic display device of claim 10 wherein the processor is further configured to derive a current field of view (FOV) of the user based on a finger orientation of the user.

17. A computerized method for teleshifting from a first lateral viewing perspective to a second lateral viewing perspective of a virtual tour object, the method useful in association with a mobile device configured to be hand-held by a user, the teleshifting method comprising: detecting a teleshifting motion of a mobile device configured to conduct a virtual tour for a user; evaluating a magnitude of the teleshifting motion; and if the magnitude of the teleshifting motion is greater than a threshold, then teleshifting from a first lateral viewing perspective to a second lateral viewing perspective, and wherein the first lateral viewing perspective and the second lateral viewing perspective are adjacent lateral viewing perspectives of the virtual tour.

18. The teleshifting method of claim 17 wherein the first lateral viewing perspective and the second lateral viewing perspective are adjacent lateral viewing perspectives of a virtual tour object.

19. The teleshifting method of claim 18 wherein the teleshifting includes teleturning from the first lateral viewing perspective to the second lateral viewing perspective, and wherein the first lateral viewing perspective and the second lateral viewing perspective are both positioned around the virtual tour object.

20. The teleshifting method of claim 17 wherein the teleshifting motion includes a flick.

21. The teleshifting method of claim 20 wherein detecting the flick includes detecting angular acceleration along a substantially vertical axis of the mobile device.

22. The teleshifting method of claim 21 wherein the flick is one of a left flick and a right flick.

23. The teleshifting method of claim 17 wherein the teleshifting motion includes a double flick.

24. The teleshifting method of claim 23 wherein detecting the double flick includes detecting angular acceleration along a substantially vertical axis of the mobile device.

25. The teleshifting method of claim 24 wherein the double flick is one of a left double flick and a right double flick.

26. The teleshifting method of claim 17 wherein the threshold is user adjustable.

27. A mobile device configured to teleshift from a first lateral viewing perspective to a second lateral viewing perspective of a virtual tour object, the mobile device comprising: a sensor configured to detect a teleshifting motion of the mobile device caused by a user; a processor configured to determine if a magnitude of the teleshifting motion is greater than a threshold; and a display configured to teleshift if the magnitude of the teleshifting motion is greater than the threshold, wherein the teleshifting causes the display to transition from a first lateral viewing perspective to a second lateral viewing perspective, and wherein the first lateral viewing perspective and the second video lateral viewing perspective are adjacent lateral viewing perspectives of the virtual tour.

28. The mobile device of claim 27 wherein the first lateral viewing perspective and the second lateral viewing perspective are adjacent lateral viewing perspectives of a virtual tour object.

29. The mobile device of claim 28 wherein the teleshifting includes teleturning from the first lateral viewing perspective to the second lateral viewing perspective, and wherein the first lateral viewing perspective and the second lateral viewing perspective are both positioned around the virtual tour object.

30. The mobile device of claim 27 wherein the teleshifting motion includes a flick.

31. The mobile device of claim 30 wherein detecting the flick includes detecting angular acceleration along a substantially vertical axis of the mobile device.

32. The mobile device of claim 31 wherein the flick is one of a left flick and a right flick.

33. The mobile device of claim 27 wherein the teleshifting motion includes a double flick.

34. The mobile device of claim 33 wherein detecting the double flick includes detecting angular acceleration along a substantially vertical axis of the mobile device.

35. The mobile device of claim 34 wherein the double flick is one of a left double flick and a right double flick.

36. The mobile device of claim 27 wherein the threshold is user adjustable.

37. A computerized method for navigating a menu of a panorama, useful in association with a mobile device configured to be handheld by a user, the method comprising: detecting a flicking motion of a mobile device configured to navigate a supplemental panoramic data menu for a user, and wherein the flicking motion is substantially around an axis substantially perpendicular to a display of the mobile device; and evaluating a magnitude and a direction of the flicking motion, wherein: if the magnitude of the flicking motion is greater than a threshold and is clockwise, then traversing along a first direction of the menu; and if the magnitude of the flicking motion is greater than a threshold and is counterclockwise, then traversing along a second direction of the menu.

38. A mobile device configured to navigate a menu of a panorama, the mobile device comprising: a sensor configured to detect a flicking motion of a user holding the mobile device, wherein the flicking motion is intended to navigate a supplemental panoramic data menu for the user, and wherein the flicking motion is substantially around an axis substantially perpendicular to a display of the mobile device; and a processor configured to evaluate a magnitude and a direction of the flicking motion, wherein: if the magnitude of the flicking motion is greater than a threshold and is clockwise, then traversing along a first direction of the menu; and if the magnitude of the flicking motion is greater than a threshold and is counterclockwise, then traversing along a second direction of the menu.