Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
  • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
  • H04N21/436—Interfacing a local distribution network, e.g. communicating with another STB or one or more peripheral devices inside the home
  • H04N21/4363—Adapting the video stream to a specific local network, e.g. a Bluetooth® network
  • H04N21/43637—Adapting the video stream to a specific local network, e.g. a Bluetooth® network involving a wireless protocol, e.g. Bluetooth, RF or wireless LAN [IEEE 802.11]
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
  • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
  • G06F3/0304—Detection arrangements using opto-electronic means
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
  • H04N21/41—Structure of client; Structure of client peripherals
  • H04N21/4104—Peripherals receiving signals from specially adapted client devices
  • H04N21/4122—Peripherals receiving signals from specially adapted client devices additional display device, e.g. video projector
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
  • H04N21/41—Structure of client; Structure of client peripherals
  • H04N21/4104—Peripherals receiving signals from specially adapted client devices
  • H04N21/4126—The peripheral being portable, e.g. PDAs or mobile phones
  • H04N21/41265—The peripheral being portable, e.g. PDAs or mobile phones having a remote control device for bidirectional communication between the remote control device and client device
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
  • H04N21/41—Structure of client; Structure of client peripherals
  • H04N21/422—Input-only peripherals, i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS]
  • H04N21/42204—User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor
  • H04N21/42206—User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor characterized by hardware details
  • H04N21/42222—Additional components integrated in the remote control device, e.g. timer, speaker, sensors for detecting position, direction or movement of the remote control, microphone or battery charging device
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
  • H04N21/41—Structure of client; Structure of client peripherals
  • H04N21/422—Input-only peripherals, i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS]
  • H04N21/42204—User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor
  • H04N21/42206—User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor characterized by hardware details
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N7/00—Television systems
  • H04N7/01—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level
  • H04N7/0117—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level involving conversion of the spatial resolution of the incoming video signal
  • H04N7/0122—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level involving conversion of the spatial resolution of the incoming video signal the input and the output signals having different aspect ratios

Definitions

• This disclosure relates to interactive pointing devices and systems for implementing the same useful in controlling media driven pointers on displays such as televisions, computer monitors, and other viewing devices.
• An interactive display recognition tool uses imaging devices to determine the location of a display and position an icon on the display via a computer.
• the tool comprises various input devices, which can be used to control the icon and interact with a computer driven multimedia platform.
• a device comprising a display recognizer to recognize a display in a space, at least one input device, and a communicator for sending data to a computer.
• a system comprising a display recognition device having at least a display recognizer and a input device, a computer providing a multimedia platform and receiving data from the display recognition device, and a sender and a receiver connected to the computer, wherein the receiver is connected to a display and wherein the sender and receiver facilitate transfer of a signal from the computer to a display.
• a method comprising capturing at least one image of a space with a display recognition device having display recognizer, the space having within it a display, identifying the quadrilaterals in the space to obtain a set of identified quadrilaterals, determining the length to width ratio of each identified quadrilateral to identify a set of qualifying quadrilaterals substantially having a predetermined ratio, resolving the qualifying quadrilaterals to select a selected quadrilateral, and determining a location on the selected quadrilateral where the display recognition device is pointed.
• a method comprising providing a system that at least: captures at least one image of a space with a display recognition device having display recognizer, the space having within it a display, identifies the quadrilaterals in the space to obtain a set of identified quadrilaterals, determines the length to width ratio of each identified quadrilateral to identify a set of qualifying quadrilaterals substantially having a predetermined ratio, resolves the qualifying quadrilaterals to select a selected quadrilateral, and determines a location on the selected quadrilateral where the display recognition device is pointed.
• FIG. 1 is a block diagram of an embodiment of the interrelationship of a display recognition device and a display;
• FIG. 2 is a perspective view of an embodiment of a display recognition device of the present disclosure
• FIG. 3 is a block diagram of an embodiment of a method of recognizing a display using a display recognition device
• Fig. 4 is an illustration an embodiment of an image captured by display recognition device and used to identify a display; and [0013] Fig. 5 is a view of an embodiment of a display recognition device of the present disclosure.
• the term "computer” shall be understood to have the meaning commonly associated and known by artisans including, but not limited to, personal computers and personal digital assistants, portable devices having a microprocessor or capable of performing functions traditionally associated with computers, console-type multimedia devices, and set-top devices used to communicate multimedia content to displays.
• the present disclosure is directed to novel methods of interactive communication with displays, such as televisions, large stadium displays, arena displays, and billboards. Such interactivity allows users viewing the displays to interactively point and select items shown on a display, much like computer users interact with a computer graphical user interface (GUI) with a mouse.
• GUI computer graphical user interface
• the present disclosure comprises a display recognition device (DRD) 100, computer 200, sender 300 and receiver 350, and a software application that are integrated with display 400, as illustrated in Fig. 1.
• DRD 100 is configured to recognize display 400. By determining where DRD 100 is pointed relative to display 400, an icon is shown on display 400 and a user is permitted to interact with computer 200 having multimedia content.
• DRD 100 communicates information retrieved to computer 200.
• Computer 200 positions an icon on display 400 by sending data between sender 300 and receiver 350, according to embodiments.
• Receiver 350 outputs a signal to display 400, including an icon having a position determined by the position at which DRD 100 is pointing.
• a computer operating system which operates as a program within a third party operating system and which is designed to facilitate communication with DRD 100, computer 200, sender 300 and receiver 350.
• the operating system is designed to run programs compiled and optimized for the operating system and is further optimized for output to the display.
• the operating system provides a signed environment, whereby only signed programs may run within the overlaid operating system's environment.
• DRD 100 comprises display recognizer 102 and controls 110, 112.
• Display recognizer 102 comprises an imaging device, which images the space in which display is positioned and identifies display.
• display recognizer 102 may be a video camera, charge-coupled device (CCD) or acoustic imaging device.
• CCD charge-coupled device
• display recognizer 102 may be any device that is able to form an image of the environment in which display is located, whereby the position of display 400 is able to be deduced as explained in greater detail below.
• Controls 110, 112 comprise input devices that allow for users to communicate with computer 200, including but not limited to, buttons, dials, touch pads, joysticks, voice activated output devices, microphones, etc. Artisans will recognize the many variations of display recognizer 102 and controls 110, 112 that are possible according to the general principles disclosed herein.
• display recognizer 102 may be a CCD, CMOS digital sensor, or other digital or analog video imaging device useful for capturing video or still images.
• display recognition device 100 As display recognition device 100 is aimed at display 400, display recognizer 102 captures images of the space where display 400 is located. From the captured images, display location is determined as disclosed in detail below, for example based on aspect ratios (i.e., the width divided by the height) of common displays. As used herein, the term "captured image" may mean complete images captured much like a photograph or may mean other imaging techniques, for example scanning of lines.
• an icon on display is shown via computer 200, sender 300, and receive 350, whereby the icon corresponds to the location on display 400 where display recognition device 100 is pointed.
• Display recognizer 102 captures images in the visual electromagnetic (EM) range. According to embodiments, display recognizer 102 captures images in the infrared EM range. According to embodiments, display recognizer 102 may capture images in other EM ranges.
• EM visual electromagnetic
• display recognizer 102 may be an acoustic signaler, such as one that images a space based on echolocation-type technologies.
• display recognizer 102 may be a light detection and ranging (LIDAR) device, such as one that measures properties of scattered light to find range or other information of a distant target.
• An output from display recognizer 102 may comprise a point cloud corresponding to the results of the operation of display recognizer 102.
• LIDAR light detection and ranging
• DRD 100 is a handheld, remote control-like device.
• DRD 100 may comprise a handheld device that has greater processing power, for example having an ARM processor as used in personal digital assistant type devices.
• DRD 100 may also comprise controllers that are designed specifically for use together with a video game platform.
• DRD 100 is pointable by a user, whereby it may be variably oriented with respect to display 400.
• DRD 100 is battery powered. Low power embodiments of DRD 100 may be powered by alkaline battery sources, for example. Higher power requirements may require specialized power sources, for example nickel metal hydride batteries or lithium ion batteries. Artisans will clearly understand the requisite power requirements necessary for each embodiment on a case by case basis and provide a suitable power source without undue experimentation.
• controllers 110, 112 provide an input mechanism for manipulation of display content, which is displayed via computer 200, sender 300, and receiver 350.
• controller may comprise button 112, and touch receptive dial 110.
• These input mechanisms are mapped to at least one command on computer 200, thereby allowing users of DRD 100 to interface with and communicate inputs to computer 200, which inputs are correspondingly shown by icons or events on display 400.
• the configuration of input devices may take on various forms and include many variable components.
• keyboards for example, keyboards, keypads, additional buttons, touch screens, touch pads, scrolling wheels, accelerometers, vibrating devices, gravity sensing devices, audio- responsive devices, and other input devices installable on remote controls, gaming controls, or computers are appropriate and are expressly contemplated by the present disclosure.
• DRD 100 further comprises, according to embodiments, a transmitter and receiver for wireless communication between DRD 100 and computer 200.
• wireless protocols for transfer of data may be proprietary and optimized for the transfer of specific data from DRD 100 to computer 200, or may be transferred according to well established protocols such as IEEE 802.11 or Bluetooth (e.g., IEEE 802.15.x).
• IEEE 802.11 or Bluetooth (e.g., IEEE 802.15.x).
• Bluetooth e.g., IEEE 802.15.x
• a similar type of connection may exist between computer 200 and display 400.
• an icon for example an arrow or a crosshair, on display 400 is moved or actuated by moving DRD 100.
• the icon substantially tracks where DRD 100 is pointed relative to display 400. If DRD 100 is pointed at the center of display 400, then the icon is centered in the center of display 400; if DRD 100 is moved from pointing at the center of display 400 to the upper left of display 400, then the icon is repositioned to track the movement whereby the icon always substantially points to where DRD 100 is pointed until DRD 100 and the icon are pointing to the upper left of display 400.
• movement is polled by computer 200 or communicated from DRD 100 to computer 200 on a relatively rapid basis whereby the movement of the icon across the screen appears to be smooth as it tracks the movement of DRD 100.
• the orientation, location, and motion of DRD 100 relative to display 400 may have a 1:1 correlation with the location of the icon displayed on display 400, as described above.
• the relative motion of DRD 100 may be multiplied by a coefficient (e.g., a "factor"), such that the relative motion of DRD 100 may translate into a suppressed or exaggerated motion of the icon displayed on display 400.
• the location of an icon to be shown on display 400 may comprise a direction from a center of display 400 and a distance from the center of the display. The direction to the icon from the center of display 400 corresponds to the direction from the center of the quadrilateral to where DRD 100 is pointed.
• the distance of the icon from the center of display 400 is the distance from the center of the quadrilateral to where DRD 100 is pointed, multiplied by a coefficient greater than, less than, or equal to 1.
• the relationship between the distance of the icon from the center of display 400 and the distance from the center of the quadrilateral to where DRD 100 is pointed may be linear, exponential, or be of variable sensitivity.
• the factor may be a constant, a variable, or a function of other known or determinable conditions.
• DRD 100 is pointed at the center of display 400, then the icon is centered in the center of display 400; if DRD 100 is moved from pointing at the center of display 400 to the upper left of display 400, then the icon is repositioned according to the direction of the movement of DRD 100, but in a manner merely proportionate to the movement of DRD 100 (e.g., by a factor greater than or less than one).
• DRD 100 may be pointed at a location beyond the upper left boundary of display 400, and an icon may be displayed in the upper left of display 400 (where the factor is less than 1), thereby causing the icon movement on display 400 to be perceived as "slower" and less responsive to minor movements of DRD 100.
• DRD 100 may be pointed at a location within the upper left boundary of display 400, and an icon may be displayed at the upper left boundary of display 400 (where the factor is greater than 1), thereby causing the icon movement on display 400 to be perceived as "faster” and more responsive to minor movements of DRD 100.
• DRD 100 captures at least one image of the space in which display 400 resides.
• this space may comprise a room or an outdoor venue, for example a stadium having a large display, in operation 302.
• Images may be captured in a variety of analog or digital formats, for example lossy compression (jpeg), bitmap, tiff compression, and others that would be appropriate for manipulation of the captured image or transfer of the image via a wireless signal.
• lossy compression jpeg
• bitmap bitmap
• tiff compression tiff compression
• the three dimensional images in the image(s) of the space may be mapped from three dimensions to two dimensions in operation 304.
• This process is performed by the camera lens, as it transforms the three dimensional image into a two dimensional image.
• operations 302 and 304 may be generally accomplished in the same process, that is during the capture of the image by display recognizer 102.
• DRD 100 further comprises an illumination source that illuminates a space in low-light conditions.
• the illumination source may provide infrared illumination.
• Display recognizer 102 may be configured to detect reflected infrared light. Because infrared is outside of the human visible spectrum, an infrared illumination source will not increase the illumination of a dark room as perceived by a human eye, for example.
• the image may be analyzed in operation 306 to determine edges of the objects in the space, including display 400.
• Edges are detected, according to embodiments, using both horizontal and vertical scan lines.
• Scan lines are horizontal or vertical "lines" of an image.
• the amplitude of a measurable signal for example black or white intensity in a black and white image.
• display recognizer 102 is positioned at 45°, whereby both vertical edges and horizontal edges are detected in the same step.
• the image captured by DRD 100 is passed through a Gaussian blur filter. Blurring removes high frequencies with low amplitude, which reduces noise that is naturally present in the input captured image. Other noise-reduction algorithms can be used as well.
• the Gaussian function may be expressed as:
• blurring may be achieved by the optical properties of DRD 100.
• the output blurred image may be passed through a Sobel operator to highlight edges.
• a Sobel operator highlights samples (pixels) with high frequencies (edges), and simultaneously inhibits low frequency pixels.
• the Sobel operator is applied in two passes, one to highlight horizontal edges, and one for vertical.
• A is the source image
• Gx and Gy are two images which at each point contain the horizontal and vertical derivative approximations.
• the output is a measurement for the horizontal and vertical gradient at each pixel, Gx and Gy. This output indicates how strong of a horizontal and vertical edge there is at a pixel.
• the two can be combined to compute an overall gradient G for the pixel:
• the output from this step indicates the strength of an edge at every given pixel (G) and the angle of that edge ( ⁇ ) (from 0 to 360 degrees).
• the output from the Sobel operation is provided for Canny edge detection. Where the Sobel operator leaves wide edges, a more precise location of each edge may be provided with Canny edge detection.
• a Canny algorithm finds the pixels where the gradient intensity G from the Sobel operation is a local minimum or a local maximum compared to its neighboring pixels, and discards all other pixels. For example, a pixel having a value of G and having neighbors on the left and right where G is smaller, the pixel in question is identified as a local maximum and that pixel is preserved, identifying a vertical edge.
• the same is repeated relative to pixels above and below the pixel in question, identifying a horizontal edge, and relative to diagonal pixels, identifying diagonal edges.
• the output is an edge map, in which each pixel indicates one of the four types of edges it might contain (horizontal, vertical, and the two diagonals) and its strength.
• the separate pixels may be combined to form continuous lines using hysteresis thresholding.
• a high threshold and a low threshold are established. Pixels are evaluated to determine whether they have a value that exceeds the high threshold. Pixels with such a value are traced in any direction the contains pixels that don't fall below the low threshold. When a pixel is encountered that has a value that falls below the low threshold, the tracing stops.
• the coordinates of the traced pixels are processed using a least squares method or singular value decomposition, to both check if the pixels are on a straight line, and deduce the line equation.
• a connection is assumed.
• four lines are connected by four vertices, then the four lines are assumed to form a quadrilateral.
• the common line may be inferred and applied for purposes of calculating and identifying quadrilaterals.
• each line segment may be extended to evaluate whether the lines would intersect if they were extended.
• quadrilaterals in the space may be identified in operation 308. Because DRD 100 will not always be positioned at a right angle to display 400, the quadrilaterals captured in the image may not resemble a rectangle having the aspect ratio of display 400, but rather some form of another quadrilateral. For example, if DRD 100 is positioned at an angle of 45° > the closest vertical edge of display 400 will appear to be longer than the farthest vertical edge of display 400 as imaged. Thus, quadrilaterals may be identified rather than rectangles, thereby making irrelevant the angle from which display recognizer 102 is to display 400. Artisans will readily understand that identifiers for qualifying quadrilaterals include apparent aspect ratios, as well as other identifiers.
• one or more operations may be performed in operation set 309 to qualify an identified quadrilateral or one of a set of identified quadrilaterals as display 400.
• one or more operations may be performed in operation set 309 to reduce the set of identified quadrilaterals to a set of quadrilaterals potentially qualifying as display 400.
• various operations are capable of being performed within operation set 309, in combination or alone, and in any order, wherein each operation has a possibility of qualifying an identified quadrilateral as display 400, eliminating an identified quadrilateral as not being display 400, or ranking an identified quadrilateral based on its probability of being display 400.
• DRD 100 may qualify or eliminate a quadrilateral as display 400 based on properties observed in an operation within operation set 309 and comparison of those properties with predetermined identifiers.
• properties observed may include: an apparent aspect ratio of a rectangle corresponding to an identified quadrilateral, a hypothetical location of DRD 100 based on the apparent aspect ratio of a rectangle corresponding to an identified quadrilateral, the location of an identified quadrilateral within an image captured by DRD 100, the amount of visible light emanating from an identified quadrilateral, the amount of infrared radiation emanating from an identified quadrilateral, and the location of an identified quadrilateral relative to other identified quadrilaterals. These properties may be compared with at least one identifier corresponding to analogous features of display 400.
• each identified quadrilateral may be qualified or eliminated based on whether it corresponds to a rectangle having an apparent aspect ratio comparable to the known aspect ratio of display 400.
• each identified quadrilateral may be split into a pair of triangles in operation 310.
• the triangles of each quadrilateral may be analyzed to calculate an apparent aspect ratio as an estimation of the actual aspect ratio of the rectangle to which each quadrilateral corresponds.
• each of the two triangles may have two sides corresponding to two sides of a given quadrilateral.
• the ratio of the lengths of the two sides from one triangle may be combined with the ratio of the lengths of the two sides from the other triangle to determine an apparent aspect ratio of the rectangle to which each quadrilateral corresponds.
• the two ratios may be averaged, summed, combined by a weighted average, or otherwise mathematically combined to estimate the apparent aspect ratio.
• the apparent aspect ratio may be compared to a desired or predetermined aspect ratio of the desired or predetermined display 400 in operation 312.
• 4:3 or 16:9 may be a desired or predetermined aspect ratio, although other ratios are also easily adaptable to the principles of the present disclosure. If the apparent aspect ratio of a rectangle corresponding to the given quadrilateral is determined not to fall within a predetermined range of the predetermined aspect ratio of display 400, then the quadrilateral may be ignored in operation 314 and the triangles of the next quadrilateral may be analyzed. If only a single quadrilateral has a requisite ratio, it may be selected and the process may be advanced to operation 322. Otherwise the set of potentially qualifying quadrilaterals may be passed to a resolution process, which may be performed in operation 316 or another operation.
• a determination of whether a quadrilateral corresponds to a rectangle having the correct ratio may be accomplished by process of elimination.
• the angle at which a rectangle is viewed affects the appearance of the quadrilateral corresponding to the rectangle when viewed in two-dimensions. Therefore, the process of determining whether a given quadrilateral corresponds to a rectangle having the correct ratio depends on the angle from which the quadrilateral is viewed.
• a transformation occurs for each quadrilateral to map it from a three dimensional shape to two dimensions. The transformation is based on the lens, for example, of the imaging device. For example, a wide angle lens will produce quadrilaterals in an image differently than a fish-eye lens.
• an analysis of the 2D image must be de- transformed to determine whether the quadrilateral could actually correspond to a rectangle having a qualifying ratio.
• Such analysis is based on the known characteristics of the lens, the focal length, and other known data. From these data, a hypothetical location of DRD 100 may be determined based on the quadrilateral that is determined to correspond to a rectangle appearing to have a qualifying aspect ratio and the known data in the de-transformation process. Once the hypothetical location of DRD 100 is known, it can be deemed plausible or eliminated— for example if it were determined that the rectangle would have to be located 20 feet off the floor or 90 feet away.
• a set of quadrilaterals may yield multiple hypothetical locations of DRD 100 when such an analysis is performed, thereby yielding multiple qualifying quadrilaterals.
• a selection process would then be performed for each of the quadrilaterals that might qualify to select the quadrilateral that is display 400.
• DRD 100 is primarily used with respect to a single space. Consequently, computer 200 may be initialized whereby the relative positions of the quadrilaterals in a space are known. Thus, once the relationship among the quadrilaterals is determined, the location of display 400 is known based on the interrelationship of the quadrilaterals in a given space. As the system detects changes in the quadrilateral relationships, the full method of determining which quadrilateral is display 400 is performed and the resulting relationship is redetermined. For example, a new framed photograph of a fireplace hearth may be placed on the wall in the space in which display resides after the system has been initialized. At least two additional quadrilaterals will be detected— the frame (including matting and the outline of the photograph itself). By way of example, a television may be shown on display 400 as part of the multimedia content, which would also be detected as an additional quadrilateral.
• a resolution process may be performed in operation 316 or in another operation within operation set 309.
• the resolution process comprises methods for determining which quadrilateral is display 400 among multiple quadrilaterals having a requisite ratio.
• various resolution techniques may be employed to determine which quadrilateral is display 400.
• resolution may be accomplished in operation 318 by selecting the potentially qualifying quadrilateral that is closest to the center of an image captured by DRD 100.
• resolution may be accomplished in operation 320 by selecting one of a plurality of nested quadrilaterals.
• the outermost one of a plurality of nested quadrilaterals may be selected as display 400.
• the largest one of a plurality of nested quadrilaterals may be selected as display 400.
• DRD 100 may gather data regarding the amount of visible light a given identified quadrilateral is emanating. Knowing that display 400 is emanating light in the visible spectrum, DRD 100 may qualify or eliminate an identified quadrilateral based on the amount or relative amount of light it is emanating.
• DRD 100 may gather data regarding the amount of infrared radiation a given identified quadrilateral is emanating. Knowing that display 400 may generate heat, and thereby infrared radiation, DRD 100 may qualify or eliminate an identified quadrilateral based on the amount or relative amount of infrared radiation it is emanating.
• DRD 100 may determine a known image to be shown on display 400 in an operation within operation set 309.
• the known image may be an image caused to be shown by DRD 100, or the known image may be an image shown on display 400 regardless of DRD 100, wherein DRD 100 is simultaneously aware of the image being shown.
• DRD 100 may capture a view of the image as it is shown on display 400, and compare the captured view with the known image. Comparison of the captured view with the known image may yield information regarding the direction, orientation, and location of the display 400 and features of the known image shown on display 400 relative to the direction, orientation, and location of DRD loo at the time it captured the view of the image.
• display 400 may show an image with features recognizable by DRD 100.
• the features may be recognizable based on their known characteristics, such as shape, size, color, location relative to other features, etc.
• the features as captured by DRD 100 demonstrate the location of display 400 within the captured view as well as the location of the features relative to the direction in which DRD 100 is pointed.
• the captured view may be compared to the known image independent on the viewpoint of DRD 100, based on scale and rotation invariant feature detection.
• the image shown on display 400 may be fill display 400 or be a partial overlay imposed on at least a portion of display 400.
• computer vision algorithms such as SIFT or SURF may be used to recognize the known image as it is shown on display 400 independent of scale or rotation of DRD 100 relative to display 400.
• This analysis facilitates the determination of where an icon corresponding to the direction of DRD 100 should be displayed on display 400.
• Such a method may be performed as a single calibration step or repeatedly during operation of DRD 100.
• display recognizer 102 may be used in conjunction with DRD 100 or computer 200 to determine generally the direction in which DRD 100 moves over time, much like the algorithms that control an optical mouse. These methods can be used alone or in conjunction with those or others disclosed herein.
• DRD 100 may have an override button or command sequence whereby the user can tell computer 200 that the wrong quadrilateral was selected.
• Computer 200 may "remember" the relationship between the quadrilaterals and thereby avoid choosing the quadrilateral that is incorrect in subsequent icon location determination iterations.
• computer 400 selects the screen housing of display 400 rather than the screen itself, the user can instruct computer to select a nested quadrilateral.
• DRD 100 is pointed to a position on the qualifying quadrilateral that isn't display 400, it may select the non-display qualifying quadrilateral and output the icon to the corresponding part of display 400.
• the location of an icon on display 400 may be determined in operation 322.
• the location of the icon on display 400 may be determined by positioning the icon at substantially the center of each captured image. More specifically, according to embodiments, the icon may be placed at the pixel position intersected by the imaginary longitudinal parallel line extending through the center of display recognizer 102 of DRD 100. According to embodiments, the location of the icon on display 400 may correspond to any selected location on the captured image or may be related to any selected location on the captured image, as discussed further herein.
• Image 150 captured by display recognizer 102.
• Image 150 comprises an image of space having potted plant 501, a television having display 400, a television stand 503, and wall hanging 505.
• quadrilaterals 502 including frame of wall hanging 505, photograph of wall hanging 505, cabinet doors of television stand 503, front and side walls of television stand 503, display 400, and rectangular icons shown on display 400, house door, window, and chimney shown on display 400 as part of the multimedia content.
• chimney is an irregular quadrilateral, wherein not every angle within the quadrilateral is a right angle. According to embodiments, the methods herein may rule out such quadrilaterals as soon as they are determined to be substantially irregular.
• Center 510 of image 150 may be represented by a dot.
• image 150 may also be subject to one or more operations (for example, in operation set 309) to qualify or eliminate an identified quadrilateral within image 150 as display 400, as discussed herein. According to embodiments, image 150 may be subject to one or more operations (for example, in operation 322) to determine the location of an icon on display 400.
• each quadrilateral identified in image 150 may be divided into two triangles and the apparent aspect ratio of the rectangle corresponding to each quadrilateral is determined. For example, if the apparent aspect ratio is not 4:3, 16:9, or another predetermined aspect ratio corresponding to the known characteristics of display 400, then the quadrilateral is not determined to be display 400.
• the quadrilateral closest to the center 510 of image is selected, which this time corresponds to display 400, wherein center 510 indicates the location toward which DRD 100 is pointed.
• the icon may be placed corresponding to the position in the selected quadrilateral closest to center 510.
• the icon would be positioned corresponding to the same position of center 510.
• this position can be represented by an X%, Y% values with respect to display 400 (selected quadrilateral).
• position can be represented as a horizontal and vertical pixel coordinates, provided the resolution of display 400 is known. Artisans will known and understand the various other methods of representing the coordinates where icon should be placed.
• the selected location on image 150 corresponds to the location of the icon on display 400.
• a quadrilateral selected as display 400 appears on image 150.
• the selected location on image 150 may be within the quadrilateral.
• the quadrilateral may be divided into two triangles, and the selected location may be determined to be within one of these two triangles.
• Barycentric or areal coordinates may be determined based on the position of the selected location relative to the vertices of the triangle within which the selected location appears on image 150. Because the vertices of the triangle correspond to three known vertices of display 400, the coordinates may be mapped onto display 400 to determine the appropriate location for the icon to be shown on display 400.
• icon is output to display 400 in operation 324, corresponding with the position where DRD 100 is pointed.
• the above-described process may be repeated over a rapid interval, for example io times per second, whereby the icon will appear to be responsive to even small movements of DRD 100 with respect to where DRD ioo is aimed.
• an apparent aspect ratio of a rectangle corresponding to a quadrilateral identified in image 150 may be determined based on the four vertices of each quadrilateral.
• the actual position (including location and orientation) of DRD 100 with respect to each identified quadrilateral and the actual aspect ratio corresponding to each identified quadrilateral may be unknown.
• a hypothetical position of DRD 100 relative to each identified quadrilateral may be calculated, and each hypothetical position may be evaluated for its likelihood to be the actual position of DRD 100.
• a hypothetical position corresponding to each identified quadrilateral may be calculated.
• the hypothetical position for a given quadrilateral is the position that DRD 100 would need to assume for the given quadrilateral to correspond to a rectangle having an actual aspect ratio equal to that of display 400.
• the actual aspect ratio is calculable based on the apparent aspect ratio, the optical characteristics of DRD 100, and the assumed hypothetical position of DRD 100.
• an assumption for the purpose of calculation may be made for each identified quadrilateral that it is or may be display 400.
• the first hypothetical position of DRD 100 (relative to the first quadrilateral) at the time image 150 was captured would be a position orthogonal to the first quadrilateral.
• the first hypothetical position may not be the actual position of DRD 100 relative to the first quadrilateral.
• the second hypothetical position of DRD 100 (relative to the second quadrilateral) at the time image 150 was captured would be a position that would transform an actual aspect ratio of 4:3 into an apparent aspect ratio of 5:3 (such as a position not orthogonal to the second quadrilateral).
• the second hypothetical position may not be the actual position of DRD 100 relative to the second quadrilateral.
• DRD 100 may compare each hypothetical position and determine which is most likely the actual position of DRD 100. For example, where the first hypothetical position is orthogonal to the first quadrilateral, and the second hypothetical position is not orthogonal to the second quadrilateral, DRD 100 may conclude that the first hypothetical position is preferred, based on an assumption that a user is more likely to position DRD 100 orthogonally to display 400. Therefore, DRD 100 may assume that the first quadrilateral is display 400, and select it accordingly. Any assumption regarding preferred positions of DRD 100 relative to display 400 may be pre-selected, user-defined, or based on usage data relating to the historical position of DRD 100. These principles are also applicable for determination of the location of DRD 100 in three-dimensional space. This information may be used as a source of input, for example in a gaming context to position a flashlight beam based on the direction DRD 100 is pointed and the origin of DRD 100 relative to display 400.
• At least one of the operations of Fig. 3 are performed on DRD 100 and at least one of the operations of Fig. 3 are performed on computer 200.
• Variations or permutations as to what device (DRD 100 or computer 200) performs what operation are possible and are determined on a case-by-case basis depending on the design factors and other characteristics of DRD 100 and computer 200.
• DRD 100 may solely capture an image display space in operation 302 and transfer each image as a low bandwidth lossy compressed file for processing on computer 200 for completion of the other steps.
• DRD 100 may perform capture of the display space in operation 302 followed by mapping of the three dimensional image to a 2D image. The 2D image is then transmitted to computer 200 for further processing.
• a third example may comprise a larger DRD 100 that is capable of performing more complex computations and therefore performs all of the computations through operations 318 and 320 and sends only a set of icon coordinates to computer 200 for display on display 400.
• computer 200 outputs to display 400 a graphical user interface (GUI) that displays the location of icon, as is well known and understood by artisans.
• GUI graphical user interface
• computer 200 outputs a signal that comprises the entire content of display 400, thereby facilitating a viewing experience much like a console video game system or digital video recording system complete with multimedia content or games, and an operating interface that allows users to manipulate items within the GUI to interface with computer 200.
• computer 200 outputs an overlay having at least one of an icon and a set of menu items.
• computer may run programs in the native operating system, except allowing DRD 100 to control the computer.
• DRD 100 may run programs in the native operating system, except allowing DRD 100 to control the computer.
• users may surf the internet using the DRD 100, which runs in a proprietary operating system and communicates with an internet browser that runs in the native operating system, for example one of the Microsoft Windows suites, Macintosh Operating System, or unix/linux-based operating systems.
• sender 300 and receiver 350 are specialized components for sending a signal to output to display 400 wirelessly.
• computer 200 preformats and outputs signal to sender 300.
• Sender 300 transmits wirelessly the signal, which is received by receiver 350.
• Receiver 350 is equipped with at least one output connector for connection to display 400, for example coaxial cable connector, RCA connectors, S-video connectors, component connectors, or HDMI connector.
• Specialized audio connectors are expressly contemplated as well, including digital output, for example PCM audio output, etc.
• the wireless signal by which sender 300 and receiver 350 connect may comprise industry standard signals, including IEEE 802.11, IEEE 802.15.x, or low power UHF, VHF, or common EM signals with sufficient bandwidth to communicate the signal output from computer 200.
• a low-level HD formatted broadcast may be transmitted by sender 300 and received by receiver 350, which, according to embodiments, comprises a stand-alone ATSC, DVB, ISDB, DTMB, DMB, or other standard broadcast tuner tuned to the frequency that the sender is broadcasting.
• receiver may comprise an ATSC, DVB, ISDB, DTMB, DMB, or other standard broadcast tuner built into display 400.
• sender 300 may communicate with receiver 350 according to Conditional Access System for Terrestrial Broadcast protocols, allowing an over-the-air broadcast signal to be "scrambled" so it can be interpreted and displayed only by an authorized and corresponding receiver 350. Accordingly, unauthorized receivers would be unable to interpret a signal, thus avoiding interference between non-corresponding sets of senders and receivers.
• receiver 350 may have built-in networking capability (even in cases where receiver is built-in to display 400), such as WiFi or Bluetooth. Naturally, for any standard signal technology used by receiver is also possible for used in sender, and visa versa. Sender 300 and receiver 350 also expressly contemplate use of non-standard or proprietary wired and wireless signals used to shared data between sender 300 and receiver 350.
• sender 300 may communicate with receiver 350 through a variety of wireless and wired communication pathways, such as a direct wired connection or through the Internet or other known protocols. Indeed, sender 300 may communicate with receiver 350 via WiFi, USB, Bluetooth, or other networking stardard. Sender 300 may also communicate wirelessly using a low power standard definition (e.g., NTSC, PAL, SECAM) or high definition (e.g., ATSC, DVB, ISDB, DTMB, DMB) broadcast signals, or a proprietary wireless signal.
• a low power standard definition e.g., NTSC, PAL, SECAM
• high definition e.g., ATSC, DVB, ISDB, DTMB, DMB
• sender 300 may send an HD signal to receiver 350 on an open frequency that is not tunable by built in tuners in display 400, etc.
• Receiver 350 receives the signal on the open frequency and either outputs the signal via well-known video connections (e.g., HDMI, component, S-video, composite, coaxial, etc.) or shifts the frequency to a frequency required by ATSC tuners, for example one built into display 400, and feed the signal to the ATSC tuner via a wire, according to embodiments.
• DRD 100 may optionally allow users to specify the HD frequency that is transmitted, thereby allowing users to avoid conflicting over the air or DRD 100 broadcast signals.
• the functions of computer 200, sender 300, and receiver 350 may be integrated into a single hardware unit that connects with display 400 via a hard-wired connection, as described previously and well understood and known to artisans.
• Other combinations are similarly possible, for example where display 400, computer 200, sender 300, and receiver 350 are integrated into a single device.
• sender 300 and receiver 350 may be omitted in embodiments, wherein computer 200 is connected directly to display 400 and outputs a preformatted signal to display 400.
• the functions of sender 300 and receiver 350 may be accomplished with a variety of devices or sub-devices that function similarly.
• these devices and methods are operational in a multimedia infrastructure or with numerous other general purpose or special purpose computing system environments or configurations or other complex systems.
• Examples of well known computing systems, environments, or configurations that may be suitable for use with the invention include, but are not limited to, personal computers, server computers, hand-held or laptop devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, telephony systems, distributed computing environments that include any of the above systems or devices, and the like.
• the devices and methods may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer.
• program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types.
• the system may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network.
• program modules may be located in both local and remote computer storage media including memory storage devices.
• the computer programs are stored in a memory medium or storage medium or they may be provided to a processing unit through a network or I/O bus.
• the DRD ioo includes at least one central processing unit (CPU) or processor.
• the CPU can be coupled to a memory, ROM or computer readable media containing the computer-executable instructions.
• Computer readable media can be any available media that can be accessed by the system and includes both volatile and nonvolatile media, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.
• Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory, portable memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by the devices of this disclosure.
• Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media.
• communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of any of the above should also be included within the scope of computer readable media.
• the computer readable media may store instructions or data which implement all or part of the system described herein.
• the process described above can be stored in a memory of a computer system as a set of instructions to be executed.
• the instructions to perform the processes described above could alternatively be stored on other forms of machine-readable media, including magnetic and optical disks.
• the processes described could be stored on machine-readable media, such as magnetic disks or optical disks, which are accessible via a disk drive (or computer-readable medium drive).
• the instructions can be downloaded into a computing device over a data network in a form of compiled and linked version.
• the logic to perform the processes as discussed above could be implemented in additional computer or machine readable media, such as discrete hardware components as large-scale integrated circuits (LSI's), application-specific integrated circuits (ASIC's), firmware such as electrically erasable programmable read only memory (EEPROM's); and electrical, optical, acoustical and other forms of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.).
• LSI's large-scale integrated circuits
• ASIC's application-specific integrated circuits
• firmware such as electrically erasable programmable read only memory (EEPROM's)
• electrical, optical, acoustical and other forms of propagated signals e.g., carrier waves, infrared signals, digital signals, etc.

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Human Computer Interaction (AREA)
• General Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Computer Networks & Wireless Communication (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• User Interface Of Digital Computer (AREA)
• Controls And Circuits For Display Device (AREA)

Priority Applications (1)

Applications Claiming Priority (4)

Publications (2)

Family

ID=41199740

Family Applications (1)

Country Status (3)

Families Citing this family (7)

Citations (4)

Family Cites Families (32)

• 2009
  • 2009-04-16 WO PCT/US2009/040873 patent/WO2009129419A2/en active Application Filing
  • 2009-04-16 US US12/425,260 patent/US8682023B2/en not_active Expired - Fee Related
  • 2009-04-16 EP EP09731911A patent/EP2277307A2/de not_active Withdrawn

Patent Citations (4)

Also Published As

Similar Documents

Legal Events