Classifications

• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
  • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
  • G06F3/0304—Detection arrangements using opto-electronic means
  • G06F3/0325—Detection arrangements using opto-electronic means using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image

Definitions

• the field of the invention relates to the electronic system arts, and, more specifically to a handheld vision based absolute pointing system.
• Figures 1 and 2 show the most pertinent types of handheld devices used for controlling electronic equipment.
• Figure 1 shows a mouse 101 that controls the position of a cursor 102 on the display 100 of a computer.
• the mouse 101 typically has a track-ball mounted on its underside that is used to translate the position of the mouse on the horizontal surface 103 that it "rolls-on" into the position on the computer display 100 that the cursor 102 is displayed at.
• a typical mouse will transmit (e.g., wirelessly through a radio frequency (RF) transmitter or electronically through a cable) data that describes the rolling activity of the mouse's track ball to the computer.
• the computer in turn, translates this information into an appropriate screen position for the cursor 102.
• the mouse 101 also includes buttons that can trigger certain actions on the computer when the cursor 102 overlaps or points to a "button” or “menu” or other feature that is displayed on the computer screen 200.
• Figure 2 relates to handheld devices used for controlling entertainment related electronic equipment.
• the typical system includes a television that is coupled to an external "box" 203 having circuitry designed to provide a certain type of "signal" to the television.
• Examples of such circuitry include cable TV receiver circuitry, satellite TV receiver circuitry, Video Cassette Recorder (VCR) circuitry, Digital Video Disk (DVD) player circuitry, gaming device circuitry, a computer's circuitry, music player circuitry (e.g., CD player circuitry, stereo receiver circuitry, etc.).
• VCR Video Cassette Recorder
• DVD Digital Video Disk
• gaming device circuitry e.g., a computer's circuitry
• music player circuitry e.g., CD player circuitry, stereo receiver circuitry, etc.
• a remote control 201 is often used to make user selections on a television display 200.
• the opportunity to make selections is typically presented as some kind of software generated user interface that is superimposed over (or integrated with) a television signal.
• the user interface often includes bordered regions and/or icons (e.g., regions/icons 204, 205). Typically one of these regions/icons 204, 205 is "highlighted” if the user is deemed to have implicated the particular region/icon.
• Figure 3 shows that digital cameras have also been integrated into handheld devices. Besides standard "digital handheld cameras", digital cameras have also been integrated in equipment having a traditional or primary purpose different than taking pictures. Cell phones, for instance, are now available having an integrated digital camera. Figure 3 shows a cell phone 301 in the process of taking a digital picture 302.
• a method involves identifying one or more images of respective one or more fixed markers. Each marker is positioned on or proximate to a display. The images appear on a pixilated sensor within a handheld device. The method also involves determining a location on, or proximate to, the display where the handheld device was pointed during the identifying. The method also involves sending from the handheld device information derived from the identifying of the one or more images of respective one or more fixed markers. The method also involves triggering action taken by electronic equipment circuitry in response to the handheld device's sending of a signal to indicate the action is desired.
• Figure 1 shows a computer mouse (prior art).
• Figure 2 shows a remote control (prior art).
• Figure 3 shows a cell phone with an integrated digital camera (prior art).
• Figures 4a through 4c show different depictions of absolute pointing
• Figure 5a show an embodiment of a handheld device having an integrated camera for purposes of effecting absolute pointing of the handheld device relative to an electronic equipment display;
• Figure 5b through 5e show various layouts of marker(s) used for calculating an absolute pointing position;
• Figures 6a through 6d show different images captured by the handheld device's camera in relation to the manner in which a handheld device faces an electronic equipment display;
• Figures 7 a through 7c show different processes that can be executed to effect absolute pointing with a handheld device such as the handheld device of
• Figure 8 shows three dimensional coordinate systems used for determining a handheld device's absolute pointing position on a display
• Figure 9 shows a process for determining a handheld device's absolute pointing position on a display
• Figures 10a and 10b shows a "virtual dial" function capable of being implemented if a handheld device's absolute pointing vector and roll position is known;
• Figure 11 shows an embodiment of a design for a handheld device capable of effecting absolute pointing control relative to an electronic equipment display
• Figure 12 shows an exemplary optical transfer function that can be implemented with the design presented in Figure 11;
• Figures 13a and 13b show different marker sizes that can be formed and directed to a handheld device's sensor
• Figure 14 shows a method for determining marker positions observed through a handheld device's sensor
• Figure 15 shows an exemplary collection of marker images observed through a handheld device's sensor
• Figure 16 shows sensor field of view for two different pointing directions
• Figure 17 shows a display and corresponding sensor image of the display for a one marker system
• Figure 18 shows calibration and "actual use” hand held device pointing situations
• Figure 19 shows markers radiating in two different directions
• Figure 20 shows marker images on a sensor for rolling hand held device
• Figure 21a and 21b show a three marker system
• Figure 22 shows a four marker system
• Figures 23a through 23e show different architectures that contain a processor.
• Absolute pointing from a handheld device may involve: 1) determining a location on, or proximate to, a display where a handheld device is being pointed; and, 2) rendering some kind of image (e.g., a cursor, icon, highlighted region, menu choice) at that location, and/or, triggering some kind of action from a user interface feature presented at that location.
• a cursor e.g., a cursor, icon, highlighted region, menu choice
• Absolute pointing from a handheld device may also, or by itself, involve: 1) determining a location proximate to a display where the handheld device is being pointed (e.g., the location of specific electronic equipment placed proximate to the display); and, 2) rendering some kind of image (e.g., a cursor, icon, highlighted region, menu choice) on the display, and/or, triggering some kind of action from that location (e.g., causing the specific electronic equipment to render an image or take some specific action as a consequence of its being pointed to).
• a location proximate to a display where the handheld device is being pointed e.g., the location of specific electronic equipment placed proximate to the display
• rendering some kind of image e.g., a cursor, icon, highlighted region, menu choice
• Absolute pointing from a handheld device may also, or by itself, involve: 1) determining that a specific appliance (e.g., a lamp) or other type of electrical equipment is being pointed to; and, 2) triggering some kind of action at that appliance or other type of electrical equipment.
• a specific appliance e.g., a lamp
• triggering some kind of action at that appliance or other type of electrical equipment e.g., a lamp
• the handheld device may point to a feature that is part of a user interface (e.g., an icon or menu choice) that is displayed on the display.
• the feature is understood to mean that some kind of action will be taken by electronic equipment controlled through the user interface (e.g., the selection of a television signal, the opening of a file, the startup of a software routine, etc.).
• a user holding the handheld device takes some kind of action (e.g., presses a button on the handheld device or "jerks" his/her wrist), a signal is sent from the handheld device that is ultimately interpreted to mean that the action suggested by the feature is desired by the user.
• Figures 4a through 4c demonstrate examples of absolute pointing between a handheld device 401 and a display 400', where, the handheld device 401 is located in a different position and points to a different display location ("X") across each of Figures 4a through 4c.
• Each of Figures 4a through 4c show a facing view (i), a top view (ii), and a side view (iii) for their respective handheld device 401 and display 400' orientation.
• the facing views 4a(i), 4b(i), 4c(i) show their respective handheld device 401 and display 400' orientation from a perspective that is directly in front of and facing the display 400'.
• the top views 4a(ii), 4b(ii), 4c(ii) show their respective handheld device 401 and display 400' orientation from a perspective that is above the display 400' looking directly down to the top of the display 400' and the handheld device 401.
• the side views 4a(iii), 4b(iii), 4c(iii) show their respective handheld device 401 and display 400' orientation from a perspective that is off the display's "left" hand side looking directly at the display's side and the handheld device's side 401.
• the term "left” is taken from the perspective of viewer standing in front of and facing the display 400' .
• the +y' direction is a direction that moves vertically relative to a viewer of the display 400' surface
• the +x' direction is a direction that moves horizontally to the right 400' relative to a viewer of the display surface
• the +z' direction is a direction that moves directly away from the front of the display 400' surface.
• Figure 4a shows a first set of perspectives when the handheld device 401 is beneath the display's left hand side and points to a location near the display's upper right hand corner.
• Figure 4b shows a second set of perspectives when the handheld device 401 is beneath the display's right hand side and points to a location near the display's lower left hand corner.
• Figure 4c shows a third set of perspectives when the handheld device 401 is above the display's left hand side and points to a location near the display's lower right hand corner.
• a cursor would be made to appear at each "X" location observed in Figures 4a(i), 4b(i) and 4c(i).
• some kind of user interface or icon may be presented at each "X" location observed in Figures 4a(i), 4b(i) and 4c(i).
• FIG. 5a shows a more detailed embodiment of a handheld device 501 , display 500' and electronic equipment 503 capable of effecting absolute pointing as described just above with respect to Figures 4a through 4c.
• the handheld device 501 includes a camera and wireless transmitter 505. Information taken or processed from an image captured by the handheld device's camera is transmitted by the handheld device's wireless transmitter to a wireless receiver 508.
• the communication between the handheld device and the electronic equipment 503 maybe achieved using a wired connection, such as Universal Serial Bus (USB) or RS-232 serial cabling.
• USB Universal Serial Bus
• Electronic circuitry (such as a processor 507, or a controller, or circuitry that does not execute program code) that is responsible for controlling the presentation of a cursor on display 500 and/or presenting a user interface on the display 500', receives the information sent by the handheld device 501.
• the electronic circuitry may be integrated in various types of electronic equipment.
• electronic equipment 503 may be any of: a) a television; b) a satellite TV receiver; c) a cable TV receiver; d) a VCR; e) a DVD player; f) a computer; g) a CD player; h) a music receiver and/or player; i) a video game box or some other type of equipment containing gaming circuitry; j) an IPTV receiver and/or television; k) a "receiver" that receives signals from one or more of items of electronic equipment such as those listed in a) through j) above; 1) home and/or office automation equipment (e.g., equipment including circuitry for turning household switches "on" and "off).
• home and/or office automation equipment e.g., equipment including circuitry for turning household switches "on" and "off.
• the electronic circuitry in response to the receipt of this information, causes a cursor (or other image) to be presented at the display location where the handheld device 501 is pointing and/or causes a user interface feature that is at the display location where the handheld device 501 is pointing to be highlighted in some fashion.
• a cursor or other image
• 5O4'_l and 5O4'_2 which are detectable by an sensor, are positioned proximate to the display 500'.
• a cursor By capturing the positions of these fixed markers 5O4'_l through 5O4'_2 with the handheld device's camera, and, through the automated execution of mathematical relationships (at the handheld device, with the aforementioned electronic circuitry, a combination of both, etc.) that relate the observed marker positions to the pointed to location on the display surface, a cursor can be made to appear at the pointed to display location, and/or, some kind of action can be triggered from a user interface feature presented at the pointed to display location.
• the fixed markers in Figure 5a are one possible two-marker arrangement. Other arrangements that comprise one, two, three, four, or more markers placed in other locations relative to both each other as well as the display 500' are possible.
• Figures 5b through 5e explore some possible marker layout arrangements for single marker (Figure 5b), two marker (Figure 5c), three marker ( Figure 5d) and four marker (Figure 5e) embodiments.
• the distance between the marker and display may vary (of course, the distance should not be such that the marker can not be seen by the handheld device).
• Figure 5b shows an embodiment where a single marker 514 is placed along the side of the display. In alternate approaches the marker may be placed in or about the corner of the display.
• Figure 5c(i) shows a two marker embodiment where both markers 515 1 , 516_1 are positioned along the same side of the display.
• Figure 5c(ii) shows another embodiment where each of the pair of markers 515_2, 516_2 are positioned along different display sides.
• one or both markers of a two marker embodiment may be positioned in or about one or more corners of the display (e.g., by placing the pair of markers around the same corner, or putting a first marker at a first display corner and the second marker at another display corner), hi two marker systems the pair of markers may be placed at the same or in different z' positions.
• Figures 5d(i) through 5d(iv) show various three marker embodiments.
• Figure 5d(i) shows each one of three markers positioned along a different display side.
• Figure 5d(ii) shows each one of three markers positioned along a same display side.
• Figure 5d(iii) shows three markers along a same display side where two of the markers have a same y' axis location.
• Figure 5d(iv) shows three markers along a same display side where one marker has a different z' location than the other two markers.
• Various other arrangements are also possible.
• Figures 5e(i) through 5e(iii) show various four marker embodiments. According to the approach of Figure 5e(i) each one of the markers is positioned along a different side of the display. According to the approach of Figure 5e(ii) each of the markers is positioned in a different corner of the display. According to the approach of Figure 5e(i ⁇ ) more than one marker is positioned along a same side of the display. Like the two and three marker systems, as long as the three markers can be seen by the handheld device and they each possess a unique x',y',z' location a workable system can be implemented.
• Figures 6a through 6d show images of markers for a two marker system, as observed through the handheld device's camera, for different handheld device positions and pointing orientations relative to the display surface (similar to Figures 4a through 4c).
• Each of Figures 6a through 6d include four sub-figures (i) through (iv).
• Each of Figures 6a(i), 6b(i), 6c(i) and 6d(i) include exemplary images 604_l, 604_2 of the display markers 604 '_1 and 604 '_2 as observed on a pixilated sensor 610 that corresponds to the "image capture" portion of the handheld device's camera.
• FIG. 6b( ⁇ ), 6c(ii) and 6d(ii) show a front view of the respective handheld device position and pointing direction
• FIG. 6b( ⁇ i), 6c(iii) and 6d(iii) show a top view of the respective handheld device position and pointing direction
• FIG. 6b(iv), 6c(iv) and 6d(iv) show a side view of the respective handheld device position and pointing direction.
• Figures 6a(i ⁇ ) through 6a(iv) indicate that the handheld device 601 is directly facing the center of the display 600'.
• the location 602' on the display 600' that is pointed to by the handheld device 601 corresponds to the center of the display 600'.
• the markers 6O4'_l and 6O4'_4 will appear as images 604_l and 604_2 on the handheld device's sensor 610.
• an initial calibration procedure is needed in order to determine values for variables that appear within mathematical relationship(s) that convert a detected marker position on the sensor 610 into the position on the display 600' where the handheld device is pointing.
• the calibration may be performed and any specific procedure may depend on the specific embodiment of the system being used.
• the user points at one or more defined locations (generally, on or near the screen) and the handheld device records the marker image 604_l, 604_2 positions on the sensor array 610 (a more detailed description of specific calibration procedures is given in section 4.0 below).
• the calibration procedure may involve pointing sequentially at visible markers (typically two) displayed temporarily on the screen (e.g., in opposite corners).
• changes in observed marker image 604_l, 604_2 position on the sensor 610 surface in response to changes in the handheld device's location and pointing direction can be defined in reference to their original calibration positions; which, in turn, can be related to a position on the display 600' where the handheld device 601 is pointing.
• the observed changes in the marker image positions of Figure 6b(i) can be mathematically related to the handheld device's correct pointing location on the display 600'; which, in turn, can be used to generate a cursor at the pointing location or trigger an action from a user interface feature that appears at the pointing location.
• a thorough discussion of such mathematical relationships are presented in more detail further below in section 4.0 entitled "Mathematical Relationships For Absolute Handheld Device Pointing".
• the observed changes in the marker image positions of Figure 6c(i) can be mathematically related to the handheld device's correct pointing location on the display 600'; which, in turn, can be used to generate a cursor at the pointing location or trigger an action from a user interface feature that appears at the pointing location.
• Figure 6d(i) shows observed sensor images when the handheld device 601 moves directly to its left from its position in Figure 6c, but, adjusts its pointing direction so as to keep pointing to the same display location 602' that is pointed to in Figure 6c. Because the handheld device does not make any changes along the y axis, the y' coordinate values of the marker images 604_l and 604_2 remain unchanged in Figure 6d(i) when compared against their values in Figure 6c(i). Moreover, because the face of the handheld device has made substantial changes along both the x' axis and the z' axis, the x coordinate values for the marker images change substantially from their values in Figure 6c(i).
• the observed changes in the marker image positions of Figure 6d(i) can be mathematically related to the handheld device's correct pointing location on the display 600'; which, in turn, can be used to generate a cursor at the pointing location or trigger an action from a user interface feature that appears at the pointing location.
• Figures 7a through 7c shows methodologies that can be executed by absolute pointing handheld device systems having different system designs. Most notably, the difference between the various designs involves the degree of processing that is performed by the handheld device.
• Figure 7a depicts a method in which the handheld device 501 not only captures the marker image data, but also performs all of the mathematical relationship calculations needed to translate the observed marker image positions into a specific pointed to location on the display 500.
• the handheld device 501 transmits 714 the coordinates of the pointed to location on the display to the wireless receiver 508.
• the overall process includes the handheld device 501: 1) capturing 711 marker image data with its camera; 2) identifying 712 the positions of the respective marker images on the camera's pixilated sensor; 3) determining 713 the handheld device's pointed to location on the display through calculations with mathematical relationships; and 4) sending 714 the pointed to location to a wireless receiver 508.
• Electronic circuitry responsible for controlling displayed content on the display 500 may then cause some effect to appear at the pointed to location 715 (e.g., a cursor, highlighting a menu feature, etc.).
• Figure 7b depicts a method in which the handheld device 501 does not determine the handheld device's pointed to location on the display through calculations with mathematical relationships, but, besides capturing 721 marker image data with its camera also identifies 722 the positions of the respective marker images on the camera's pixilated sensor. As such, the handheld device 501 transmits 723 the marker positions to the wireless receiver 508.
• Electronic circuitry responsible for controlling displayed content on the display 500 receives the marker positions and determines 724 the handheld device's pointed to location on the display through calculations with mathematical relationships; and, may then cause some effect to appear at the pointed to location 725 (e.g., a cursor, highlighting a menu feature, etc.).
• Figure 7c depicts a method in which the handheld device 501 does not determine the handheld device's pointed to location on the display nor identify the positions of the respective marker images on the camera's pixilated sensor. Instead, the handheld device 501 simply captures 731 the marker images with its camera and transmits 732 information describing these images to the wireless receiver 508.
• Electronic circuitry responsible for controlling displayed content on the display 500 receives the marker image data and: 1) identifies 733 the positions of the respective marker images on the camera's pixilated sensor; 2) determines 734 the handheld device's pointed to location on the display through calculations with mathematical relationships; and, 4) may then cause some effect to appear at the pointed to location 735 (e.g., a cursor, highlighting a menu feature, etc.).
• each of Figures 7a through 7c note that the image data is repeatedly captured 711 , 721 , 731 over time so that new pointed to locations on the display can be determined with rapid regularity (e.g., so that a cursor's movement can be at least quasi-fluidly tracked on the display 500). Moreover, note that each of Figures 7a through 7c demonstrate a process in which an effect is made to appear on the display at the pointed-to location. Alternatively or in combination, additional information in the form of a "button press" or "roll action" of the handheld device may be sent from the handheld device in order to trigger electronic equipment to take some kind of action.
• circuitry responsible for determining displayed content may interpret the collection of information sent from the handheld device to mean that the user has selected the icon or menu item.
• Figures 8a through 8d and 9 provide some higher level details that can be applied, at least in certain embodiments, to useable mathematical relationships that translate observed marker image positions to a specific pointed to location 802' on or proximate to the surface of the display 800' .
• Figures 8a through 8d illustrate various three-dimensional concepts that may be defined and applied in a certain set of applicable mathematical relationships.
• determination of the pointed to location 802 on the display 800' involves: 1) determining 901 a scaling factor (M); and, 2) executing 902 mathematical relationship(s) that use the scaling factor M to translate an sensor axis location where a marker is observed to a corresponding axis location on the display where the handheld device is pointing, hi an approach, a first scaling factor Mx is calculated for one device/display coordinate axis pair (x, x') and a second scaling factor My is calculated for a second device/display coordinate axis (y, y').
• the process observed in Figure 9 is essentially executed twice, once for the x,x' axis pair and the other for the y,y' axis pair.
• the scaling factor Mx is essentially a ratio between the observed change in position of a marker on the sensor along the x axis, relative to a specific sensor position along the x axis, and the change in position of the pointed to location along the x' axis on the display, relative to a specific position on the display along the x' axis; where, the specific sensor and display positions are determined during calibration. As described in more detail below in section 4.0, the accuracy of the scaling factor can be made to improve if additional details concerning the handheld device are gleaned from the observed marker images.
• the accuracy of the scaling factor can be enhanced if the position of the handheld device relative to the display (e.g., the distance L 820 between the center of the display 802 and the front of the handheld device) and/or the orientation of the handheld device relative to the display (e.g., the viewing angle ⁇ of the handheld device 901) is determined.
• the position of the handheld device relative to the display e.g., the distance L 820 between the center of the display 802 and the front of the handheld device
• the orientation of the handheld device relative to the display e.g., the viewing angle ⁇ of the handheld device 901
• the relative separation of marker image positions is tracked.
• the separation between all marker image positions scales inversely with distance from the display and the relative separations of different pairs of marker image positions depends on the handheld device angle with respect to the display surface.
• Figures 8a and 8b also indicate that another characteristic of the handheld device's orientation, referred to as the handheld device's "roll" ⁇ , can be tracked from the marker image positions.
• the handheld device's roll corresponds to the handheld device's rotation about the axis of its pointing vector (i.e., the vector protruding from the front of the handheld device whose intercept with the display corresponds to the pointed-to location on the display).
• the handheld device's roll ⁇ the collective rotation of the marker image positions is detected.
• the pointing vector is assumed originate from the center of the handheld device's sensor array.
• Figures 8a and 8d show the pitch ⁇ of the handheld device. The effective pitch or change in effective pitch of the handheld device can also be determined so as to more accurately define the pointed to position on the display.
• the handheld device's roll can be used to trigger some kind of action taken by electronic equipment.
• a "twist of the wrist" of the user can be interpreted as a "button press” (e.g., the user points the handheld device to an icon or user interface menu item and then twists his/her wrist to "select” that icon or menu item).
• Figures 10a and 10b demonstrate another useful application of tracking a handheld device's roll action.
• Figures 10a and 10 relate to a "virtual dial” function. According to a virtual dial implementation, a depiction of a dial 1010 is displayed on the display. In Figure 10a the dial is illustrated as being set to a value of LOW 1011.
• the dial may be displayed so as to rotate in conjunction with the user's wrist rotation(s).
• the user has pointed the handheld device to the dial and twisted his/her wrist clockwise 1002.
• the displayed virtual dial 1010 rotated clockwise until the user's wrist rotation was deemed “stopped”; which, according to Figure 10b, was at a setting of MID 1011.
• the user is expected to press a button on the hand held device during the rotation of the handheld device in order to activate the rotation of the dial 1010 (i.e., the user must press a button in combination with the wrist rotation).
• the handheld device sends: 1) the pointed to location on the display (or information from which the pointed to location on the display can be determined); 2) the roll position(s) of the handheld device (or information from which the handheld device's roll action can be determined); 3) information that signifies the user is pressing a button.
• the markers themselves act as the source of the light that is detected by the handheld device's sensor (e.g., the markers are implemented as LEDs).
• the handheld device acts as the source of light that is detected by the handheld device's sensor.
• the handheld device contains an infrared (IR) "flashlight", which shines IR light into the region of the display and each marker is implemented as a "passive" reflector. Additional image processing may be used to enhance the signal-to-noise ratio. For example, characteristics of the emitted light that are detectable to the handheld device could be subtracted from the reflected images to essentially improve the signal to noise ratio of the detected marker images.
• the flashlight is "strobed" such that a representation of the emitted light to be subtracted from each detected marker image is taken just prior to the detecting of a marker image.
• an IR filter may be placed over the camera to aid in the rejection of unwanted light.
• reflective markers include the application of a single handheld device to multiple appliances through the use of, for instance, home and/or office automation circuitry.
• the reflective reference markers are located on or near the appliance to be controlled, and, for each device, the reflective markers are arranged in a predetermined spatial pattern.
• the handheld device can then recognize a particular appliance by recognizing its specific pattern.
• one or more of the appliances may not even have a display.
• the appliances themselves and/or their remote use may be simplistic (e.g., having only a few states such as "on” and "off) such that the simple act of pointing to the appliance and making one or a few button presses on the handheld device is sufficient to enable use of the device.
• Example of such appliances include lamps, radios and televisions.
• Various formats can be used for the spatial marker patterns that identify a specific device/appliance. Examples include ones similar to those used in barcodes (i.e., universal product codes), and simple on/off binary codes.
• the number of required markers per appliance will depend on the number of different appliances that need to be identified and/or the pattern format used. For most household applications it is likely that this number will be small (less than five).
• the handheld device When the handheld device is pointed at an appliance (e.g., a lamp) and the appropriate button(s) on the handheld device is pressed (and/or an appropriate gesture is made with the device), the handheld device is programmed to send the appropriate command by wireless transmission (e.g., RF or IR) to the appliance, either directly or through a central command unit (e.g., electronic equipment 503 such as an XlO) that actually sends the command to the pointed to appliance directly.
• RF or IR wireless transmission
• a central command unit e.g., electronic equipment 503 such as an XlO
• the command is sent in a packet that includes the identity of the appliance being pointed to (i.e., the handheld device is programmed with the pattern recognition capability to identify each specific appliance).
• the handheld device sends the image pattern and handheld device action (e.g., button press, rolling action, etc.) or command to another device (e.g., the aforementioned central command unit).
• the other device then: 1) recognizes the pointed to appliance; and 2) sends/forwards a command to the pointed to appliance.
• a variety of commands may be communicated to an appliance including simple power on/off and increase/decrease of different appliance properties (e.g., light intensity, oven temperature, sound volume, etc.).
• the appliance will need to be configured with some sort of command detection and functional effectuation circuitry that may be fairly regarded as home and/or office automation circuitry (e.g., a integrated circuit having a wireless receiver coupled to a processor/controller (with executable code) and/or logic state machine circuitry (no executable code) that is coupled to a switch or other functional effectuation circuitry (such as a "selection" or "setting” circuitry (e.g., channel setting/selection, temperature setting/selection, volume setting/selection, etc.)).
• a "selection" or "setting” circuitry e.g., channel setting/selection, temperature setting/selection, volume setting/selection, etc.
• Figure 11 and 12 relate to a handheld device's hardware design.
• a handheld device may include an aperture 1101 (positioned on the front face of the handheld device), an optical channel 1102 that include various optical component (e.g., lenses, diffusers, filters, etc.), and a pixilated sensor 1103.
• the aperture 1101, optical channel 1102 and sensor 1103 form a digital camera whose output 1109 is coupled to a processor or controller 1104 (alternatively, the digital camera's output values can be written into a memory 1105 from where they are read and provided to the processor or controller 1104).
• Program code 1106 is executed on the processor/controller 1104 so that various functions can be performed (e.g., marker image position detection, mathematical calculations for relating detected marker image positions to a specific pointed to display location, mathematical calculations for relating detected marker image positions to the handheld device's roll position, wired or wireless communication reception and transmission protocols, etc.).
• a non- volatile memory device for storing the handheld device's program code, nor, one or more buttons that are coupled to the processor/controller 1104.
• memory 1105 is made of Dynamic Random Access Memory (DRAM) cells because of its low cost and low power consumption.
• DRAM Dynamic Random Access Memory
• a controller is understood to be a processor having a specially targeted instruction set.
• a processor is understood to be logic circuitry designed to execute program code 1106 instructions. Different forms of processors and controllers exist such as micro-processors, micro-controllers, embedded processors and embedded controllers.
• the processor or controller is also coupled to wireless transceiver circuitry 1107; which, in turn, is coupled to an antenna 1108 that sends/receives wireless signals.
• processor/controller 1104 could be replaced with dedicated logic circuitry designed to perform the applicable mathematical calculations and/or marker image position detection functions.
• data from the pixilated sensor 1103 is processed by the processor/controller 1104 to detect the marker image positions on the pixilated sensor 1103.
• the processor/controller 1104 then executes mathematical calculations to relate the marker image positions to a specific pointed-to display location.
• the processor/controller 1104 then prepares data that identifies the pointed-to location for wireless transmission and causes it to be forwarded to the wireless transceiver circuitry 1107 which in turn causes it to be sent from the wireless device.
• Wireless transceiver circuitry includes both transmitting and receiving circuitry.
• circuitry 1107 is wireless transmitter circuitry (i.e., the handheld device is not designed to receive and process wireless signals other than the electromagnetic radiation that enters the camera's aperture).
• wireless solutions exist such as the wireless Universal Serial Bus (USB), Z-Wave, IR, IEEE 802.15.1 (BLUETOOTH) 5 IEEE 802.15.4 (ZigBee), or IEEE 802.11 (WiFi) standards, as well as wired solutions such as the USB or RS-232 serial standards.
• the handheld device sends information to the electronic equipment through wiring rather than wirelessly.
• the wireless transceiver 1107 and antenna 1108 are replaced with driver circuitry 1107 and wiring that is coupled to the applicable electrical equipment.
• data from the pixilated sensor 1103 is processed by the processor/controller 1104 to detect the marker image positions on the pixilated sensor 1103.
• the processor/controller 1104 then prepares data that identifies the marker image positions for wireless transmission and causes it to be forwarded to the wireless transceiver circuitry 1107 which in turn causes it to be sent from the wireless device.
• data from the pixilated sensor 1103 is prepared for wireless transmission by the processor/controller 1104 and causes it to be forwarded to the wireless transceiver circuitry 1107 which in turn causes it to be sent from the wireless device.
• the processor/controller 1104 is replaced with circuitry that merely forwards pixilated sensor data to the wireless transceiver circuitry 1107.
• the wireless transceiver circuitry 1107 could conceivably be replaced with driver circuitry for driving electrical signals from the handheld device over cabling toward the display.
• Figure 12 shows an exemplary optical channel design for the handheld device's camera.
• a "bandpass filter” is essentially created with the optical transfer function of the pixilated sensor 1103 and the transfer function of an optical filter that is built into the handheld device's optical channel 1102.
• the pixilated sensor 1103 is a silicon-based CMOS device.
• Other sensor types such as CCD and photodiode arrays may also be used.
• CMOS devices have an optical transfer function 1203 that essentially corresponds to a low pass filter that rolls off on the high end of its passband above a peak response near 940nm.
• the optical channel 1102 is made to include an Infrared (IR) high pass filter 1202 that rolls off on the low end of its passband below 940nm.
• IR-pass filters such as a Kodak Wratten 87C, Hoya IR-85, or their equivalents are appropriate for this purpose. [0084] The combination of these two filters 1202, 1203 results in a narrow passband 1210 around 940nm.
• the pixilated sensor 1103 will present a strong detected "signal" at its output 1109 that corresponds mostly to the electromagnetic radiation received from the markers 504_l through 504_4 with little background from other electromagnetic radiation that is received through the handheld device's aperture 1101.
• LEDs Light Emitting Diodes
• the handheld device 501 may be designed to include its own IR transmitter and the markers 504_l through 504_4 may be implemented as mere mirrors (or, further, IR reflecting mirrors).
• the transfer function of the optical channel 1102 itself may be a specific passband (e.g., by including both a high pass filter and a low pass filter).
• the transfer function of the optical channel 1102 may be made to correspond to a low pass filter.
• the designed-for passband of the camera as a whole may be just a high pass filter or a low pass filter.
• little or no optical filtering may be performed in hardware, rather, filtering (if any) is performed in software by the processor/controller 1104; or, with dedicated logic circuitry.
• the wavelength of the electromagnetic radiation that is used to form the marker images is a matter of design choice. Besides a spectrum that resides within the IR spectrum (approximately 800 - 950 nm), other possible wavelength ranges for the marker images may be used.
• CMOS and CCD complementary metal-oxide-semiconductor
• the optical channel 1102 may be made to include a diffuser or an "out-of-focus" optical path to deliberately blur a marker image so that it is easier to detect.
• the larger marker size can permit more accurate determination of the true center position of the marker. This may be achieved using interpolation algorithms that result in sub-pixel resolution.
• Figures 13a,b, 14 and 15 relate to marker image position detection processing methods. According to the system methodology of Figures 7a and 7b, marker image position detection is performed by the handheld device. According to the system method of Figure 7c, marker image position detection is performed by circuitry associated with electronic equipment that is responsible for controlling the displayed content on the display (or at least circuitry that is coupled to such circuitry).
• Figure 13a shows a depiction of a pixilated sensor illuminated by a marker image's radiation.
• the passband of the handheld device's camera corresponds to a bandpass filter (e.g., as described with respect to Figure 12) such that, ideally, only electromagnetic radiation from a marker is responded to by the pixilated sensor, then, the amount of marker radiation that a pixel receives will determine that pixel's output value.
• each pixel has an 8 bit output allowing for 256 different intensity values.
• Pixels within the circle would therefore ideally have values toward 256, pixels at the periphery of the circle would have values around 128, and, pixels outside the circle would have values around 0.
• the collection of these values correlated to the two-dimensional (x,y) coordinates of their corresponding pixels corresponds to the marker's image (or, the pixilated sensor's output signal for this marker image).
• Figure 13b shows a representation 1302 of a marker image that is larger than the marker image of Figure 13a.
• Figure 13a represents a marker image when the optical processing in the optical channel 1102 is designed to focus a marker image to a small image.
• Figure 13b shows the same marker image if the optical channel 1102 includes a diffuser or is intentionally designed with the sensor 1103 positioned before or after the optical channel's output focal point so that the marker image is deliberately blurred. By deliberately blurring the marker image, the marker image is easier to detect by marker position detection algorithms.
• marker image size and signal strength that depends on several system factors such as the marker image signal strength, the sensor sensitivity, the noise and background signals on the sensor, the marker spacing, and the maximum distance from the markers to the user (the marker images should not be broadened further once they start to impinge on each other on the sensor), etc.
• Figures 14 and 15 relate to a marker position detection technique.
• Figure 14 shows a process for detecting the position of multiple marker images that are presented by the handheld device's sensor.
• Figure 15 provides an exemplary depiction of a sensor's output signal.
• a marker image in a simple case a "marker” such as marker 1505'_l of Figure 15
• a marker can be recognized, for instance by identifying a region of the sensor's pixel array having a sufficient number of neighboring pixels that are providing a "high" output value (above a pre-determined threshold).
• pixel data within a region 1402, 1520 around the found marker 15O4'_l is further processed to identify the "location" of the marker.
• the marker image 1302 of Figure 13b can be viewed as an exemplary illustration of the data that is associated with region 1504' 1. Recalling that each pixel location has an associated intensity value that increases as the amount of radiation from a marker that illuminates the pixel location increases, the marker image 1302 of Figure 13b can be viewed as an array of intensity values.
• the intensity values are summed "horizontally" 14O3'_l along the x axis to produce, for each row of data in the extracted data region 15O4'_l , a sum total intensity value for that row. Owing to the shape of the marker image, the distribution of these sum total intensity values across the vertical expanse of the extracted data region 15O4'_l should approximate that of a "bell-curve" 1305 (e.g., Gaussian or Gaussian-like distribution). Similarly, the intensity values are summed "vertically” 1403 '_2 along the y axis to produce, for each column of data in the extracted data region 15O4'_l, a sum total intensity value for that column.
• a "bell-curve" 1305 e.g., Gaussian or Gaussian-like distribution
• One such method essentially determines the width (e.g., full-width half- maximum - FWHM) of the fitted curves and takes the marker center to be the center position between the FWHM points, hi this case, the center may not be the same as the peak location. Other variations on this approach may also be used. Using these marker center determination algorithms sub-pixel resolution appears to be possible. The achievable resolution is limited by the noise in the signal and the number of bits of sampling in the digital sensor. For example, a noiseless signal sampled with 8-bit digitization would permit less than 0.005 pixel spatial resolution. Signal averaging may increase this resolution even further. In practice, the existence of system noise and background signals results in reduced resolutions that typically range between 0.01 and 0.1 pixels.
• the width e.g., full-width half- maximum - FWHM
• a primary task is to relate the pointing direction of a sensor located in the hand-held device to the desired cursor location on the display.
• This is generally a two-dimensional problem involving horizontal and vertical coordinates for both the display (x',y') and the sensor (x,y). Since the axes are orthogonal, the relevant equations are separable and the formulas can be derived for each axis independently. Accordingly, calculations for the horizontal axes (x' and x) are first presented. These are then extended to the vertical axes (y' and y).
• Figure 16 shows coordinate systems and sign convention. Two different pointing directions in the horizontal dimension are illustrated, from a "top view” perspective, to show how the variables are related. A similar picture may apply for the vertical dimension.
• the basic equation that relates the sensor 1603 and display 1600' coordinates is: (1) X CUR X ⁇ F - M x - [X 0 XREF ⁇
• x Q is the coordinate origin of the sensor 1603 or coordinate that corresponds to the desired pointing direction (in a preferred embodiment x 0 is near the center pixel on the sensor 1603, or, better said, the sensor's field of view 1604 is centered at the cursor x C ' UR );
• x ⁇ is the position on the sensor 1603 of the reference marker image (which moves as the sensor 1603 moves);
• x M ' F is the actual position of the reference marker with respect to the display 1600';
• x C ' UR is the desired cursor position on the display 1600' (which also moves as the sensor 1603 moves);
• M x is the magnification scale factor or proportionality constant that determines the mapping between the sensor 1603 and the display 1600' coordinates.
• M depends on several systems factors (e.g., sensor position with respect to the display, sensor field of view, display size, and orientation of the sensor with respect to the display), and, in general, x,x ' and y,y ' coordinates can have different values (magnitude and sign) for M (M x and M y ).
• Equation (1) indicates that the reference marker to display cursor distances 1605 1, 1605_2 and the reference marker image to sensor origin distances 1606 1 , 1606_2, in their respective local coordinate systems, are linearly related (i.e., proportional), and the proportionality constant, or scale factor, is M.
• the factors in (1) that are determined during calibration are M x , x 0 , and X R ' EF (where, x MF is the measured reference marker pixel number on the sensor 1603 and x c ' m is the output of the algorithm - the pixel number of the display cursor).
• x 0 could be any particular pixel number corresponding to the origin on the sensor 1603, however, the approximate center is the most natural to use.
• a sensor with an array size of 640 x 480 would have x o ⁇ 320 and y 0 ⁇ 240.
• these values do not need to be explicitly determined prior to calibration.
• X 0 and y 0 are automatically determined during calibration, as will be shown below.
• M x and x ⁇ ' p are the two factors that are explicitly determined during calibration.
• One method for determining x m ' F could be to physically measure its x ' position and corresponding relative pixel number on the display 1600'. Note that it is not necessary for the reference marker to be physically located within the display 1600' itself.
• a simpler and more accurate method to determine both x ⁇ ' and M x simultaneously is to use two visible calibration reference points displayed at known positions on the display 1600'.
• the calibration reference points will have different x and y coordinates, e.g., located at opposite corners of the screen.
• Ax ⁇ is the separation (in pixels) of the calibration reference points on the display (x C ' ALA — X' CALB ) > an( i &X REF ⁇ S ⁇ ne measured separation (in pixels) of the corresponding marker image positions on the sensor ( x ⁇ - x MFB ).
• both equations (2) and (3) maybe used to obtain x ⁇ ' p by summing them and dividing by 2: (6)
• XM' F - (x C ' ALA + xc' ALB )- M X - x o ⁇ - 2 ( V - X REFA " * " X REFB j
• Equation (7) represents the average pixel position of the marker images on the sensor (1/2(X R E F A+XR EFB )) 5 during calibration, relative to the actual pixel position of the reference marker image (X R E F ) for a given pointing direction, all scaled by M x . It essentially determines the amount that the cursor on the display 1600' should move as the reference image moves on the sensor 1603.
• the second term on the right hand side is the average pixel position of the calibration reference points on the screen and it represents any offsets that are applied to the cursor position in order to effect line-of-sight pointing.
• Figure 17 shows an example of the various (x and y) coordinates in the display 1700' and sensor 1703 reference frames.
• equation (7) is independent of x 0 since it cancels when substituting (6) into (1). This is based on the assumption the x 0 in equation (1) is the same as that in equations (2) and (3). This is generally true, though not necessarily the case. In other words, the effective x 0 will be determined by the pointing direction during the calibration procedure. Any offsets (intentional or not) in the pointing during calibration will result in an equal offset in the cursor position during use.
• M (both M x and M y ) is a function of the user's position with respect to the display 1600' during calibration ( ⁇ x ⁇ F depends on the user's position). Therefore, if the user's position changes, then M should change accordingly. To a very good approximation and under most circumstances, M varies linearly as the user's distance from the display, L, changes from its initial value, LQ
• Mo is the calibration scale factor that corresponds to Lo.
• Equation (10) is the exact expression for the effect of viewing angle and distance changes on the scale factor, M. However, under the majority of conditions, the last term in brackets is a small correction factor that can be dropped without introducing a substantial or noticeable error in M. Thus, an approximate and simplified expression for M may be used:
• the term in brackets is close to 1 and may be ignored under the following common conditions: when the viewing angle is relatively small, both during calibration ( Q 0 ) and during use ( ⁇ ) or when d is small compared with L and L 0 . Under typical operating conditions, the user's viewing angle changes less than ⁇ 25° and the distance from the screen is at least twice the calibration marker separation.
• equations (10) and (11) assume that the motion is in one plane only, or that ⁇ has only x or y components.
• the general case of arbitrary motion involves more complex equations and is covered in later sections.
• the geometry for equations (10) and (11) in the x ' dimension shown in Figure 18 indicates movement of the handheld device from an initial calibration position (corresponding to ⁇ 0 and Lo) to a final position (corresponding to ⁇ and L) with respect to the display 1800' and reference points 1810 1, 1810 2.
• Automatic calibration correction may be performed in the 2-marker embodiment in a manner similar to that described in section 4.1.
• equations (10) and (11) are not necessary. All that matters for calibration correction is the change in the separation of the images of the markers, regardless of whether it results from distance or viewing angle changes. Therefore, only equation (4) is needed to modify M as the user moves.
• Ax MF in equation (4) becomes the separation between the images of the two reference markers (instead of display calibration marker separation) on the sensor.
• the scale factor, M will automatically adjust as ⁇ x mF changes.
• the key difference between this dynamic calibration and the initial calibration described above is the fact that two markers used for calibration are sensed at all times during operation, and thus the user does not need to perform the calibration procedure described in 4.1 in order to update the scale factor, M, after having moved.
• the two reference markers are placed close enough to each other and the display such that they both remain within the field of view of the sensor under all operating conditions. On the other hand, they should be separated enough that they can be spatially resolved at the furthest operating distance from the display, or at the largest view angle.
• the reference markers There are a variety of arrangements that could be used for the reference markers as discussed above with respect to Figure 5c (e.g., one each on opposite corners of the screen, centered on adjacent sides, next to each other on the same side, etc). Each arrangement has its potential advantages; however, in practice, the two markers are normally located on the same side of the screen and several centimeters
• This arrangement is the most convenient in that it allows both markers to be placed in the same housing and wired together for powering from a single source. Although any side of the display can be used, it is usually most convenient to place the marker apparatus on top of the display and near the horizontal center. The exact location is not critical as long as the marker assembly does not move appreciably after calibration. Also note that if the two reference markers have the same x ' or y ' coordinate, then there is no additional spatial information about the orthogonal coordinate, and so dynamic calibration correction can only be performed along one axis in this case. However, for use as a remote control device, most of the user's movement will be along the horizontal (x 1 ) axis, and therefore, two reference markers with the same x coordinate, which permits dynamic calibration of M x , is sufficient for most applications.
• two reference markers provide additional spatial information about the position and orientation (i.e., degrees of freedom or DOFs) of the device.
• this information can be used for a variety of other functions that rely on measurement and tracking of the device's DOFs.
• the two additional DOFs that can be tracked using the 2-marker embodiment are device roll (i.e., rotation about the sensor's z axis, or the line between the device and the screen), and distance from the screen.
• angle and distance sensitivity are actually coupled in the 2-marker arrangement, as illustrated in equations (10) and (11).
• the 2- marker arrangement can be effectively used to monitor distance changes using a variation of equations (9) and (4): (12) £ _ ⁇ REFO _ £
• Ar mF is the measured separation of the images of the two reference markers on the sensor and Ar mF 0 is the separation of the images of the two reference markers that correspond to a known distance, Lo between the sensor and the midpoint between the actual markers themselves.
• a procedure for distance measurement using equation (12) is to record the separation between the marker images on the sensor, Ar ⁇ 0 , at a known distance between the sensor and the midpoint between the actual markers , Lo, which could be measured. Thereafter, equation (12) is used to track L. Also, since Ar x ⁇ 0 and
• ⁇ r mF is the reference marker image spacing on the sensor
• X s is the sensor size in the x dimension
• tan# 1/2 is the tangent of the sensor system's half angle field of view (half of the field of view).
• Equation (13) Either expression in equation (13) may be used to determine L, depending on which information about the system is known. For example, a typical arrangement might have a reference separation of 10 cm, a field of view of 50°, and a sensor with 640 total pixels in the x dimension. For a measured reference image separation of 40 pixels in the x dimension, equation (13) provides the distance to the screen of 1.71 m. As the user moves, will change accordingly and the user's distance is tracked.
• Equations (12) and (13) are essentially equivalent and either may be used. However, equation (12) incorporates the system parameters into the directly measured quantities, Lo and Ar MFQ , and is a way of determining the proper scale factor without prior knowledge of the system. Also, equation (13) assumes that the user's viewing angle is 0°. Otherwise, equations similar to (10) and (11), which account for oblique viewing, can be used. Equation (12) essentially accounts for the user's viewing angle through the direct calibration measurement procedure (as long as the viewing angle does not change after calibration).
• FIG. 19 shows one such arrangement in which the markers are pointed in different directions along the x' axis. Also shown is the intensity distribution 1901, 1902 of each marker as a function of angle relative to the screen normal. In this configuration, as the user moves to one side or the other of direct view, the relative signal strength from the two reference markers ⁇ i.e., ratio of the two measured signals) will change in proportion to the viewing angle, and the ratio will be different for each side of direct view.
• the angle may be determined and separated from the distance measurement described above.
• a direct method for implementing this configuration would be to calibrate the system by measuring with the sensor and recording the signal strength from each marker for a variety of known viewing angles on each side of the normal axis of the screen. The signal ratios are then computed for each angle and a function or look-up table may be generated from the data. Alternatively, if the spatial distributions of the reference signals are known, then an analytical expression for the signal ratio versus viewing angle may be generated or estimated.
• each reference source is Gaussian (a common form for optical sources)
• the signal ratio of the two sources versus viewing angle, R ⁇ ⁇ ) is a simple exponential function whose decay rate depends on the width of each distribution, A ⁇ , as well as the angular separation between the sources, 2 ⁇ Q , both of which may be measured or determined using a calibration procedure.
• Ai is the peak signal strength of first reference marker 1903 and A 2 is the peak signal strength of a second reference marker 1904.
• Equation (14) and Figure 19 assume that both markers have the same distribution width and tilt angle with respect to the screen normal.
• the expression can be modified for the general case of arbitrary tilting and different widths, although the symmetric arrangement of Figure 19 is the most natural.
• Other signal distributions for the reference markers are also possible using this approach, which results in a one-to-one relationship between the signal ratio and viewing angle.
• the main drawback of this approach is the necessity to offset the pointing direction of each reference marker such that the angle between them is a significant fraction of their angular distributions. The result of this is a reduced viewing angle for the system.
• the viewing angle reduction is approximately equal to the angular separation between the pointing axes of the markers, and thus it is desirable to minimize this separation.
• the angular sensitivity of R increases with angular separation, so there is an inherent tradeoff between viewing angle range and viewing angle resolution.
• the "angled-marker" configuration as described above, will work in one dimension unless the markers are angled in both x andy directions, the effect of which further reduces the effective viewing angle.
• the method of using the marker signal ratio works well as long as any variations in the optical power (Ai and A 2 ) of the two markers (e.g., LEDs) are correlated such that their ratio (Aj/A ⁇ in equation (14)) at any view angle remains constant over time (e.g., their powers both degrade by 10%) .
• the other DOF that can be measured using the 2- marker arrangement is "roll", which is rotation about the device's z axis (the axis perpendicular to the plane of the sensor array). Tracking of this DOF indicates the magnitude and direction in which the user has rotated the device (by twisting the wrist, for example) while pointing it at the screen.
• roll angle, ⁇ as a function of the reference marker image coordinates of the two markers, is:
• Equation (15) is the general expression for arbitrary (x', y ") coordinates of the reference markers with respect to the display.
• the sign convention for equation (15) is that clockwise rotation of the device (viewed from the back of the device while pointed at the screen) corresponds to positive roll angle.
• the effect of the device roll should be removed in the tracking algorithm so that the cursor position and motion correctly correspond to the device's true pointing direction.
• the first step in the roll compensation procedure is to accurately sense the roll angle using equations (15) - (17), whichever is appropriate.
• a new reference frame for the sensor is generated by rotating the original (x, y) coordinates by ⁇ about the sensor origin ( x 0 , y 0 ).
• the origin which corresponds to the pointing direction, is the center pixel in the array.
• the general expression for coordinate transformation involving rotation about point ( x 0 , y 0 ) by an angle ⁇ is:
• Equations (18) determine the rotated coordinate system. In order to compensate for the rotation and maintain proper cursor location, equation (18) must be transformed back to the original coordinate system:
• Equation (19) transforms the measured reference marker image coordinates back to the coordinate system prior to rotation. Applying this transformation has two effects: it maintains the correct screen cursor location and it generates the proper tracking direction for the screen cursor as the rotated device's pointing direction changes.
• the transformed (x, y) coordinates for each marker image are then used in the cursor tracking algorithm (e.g., XR E F and y REF in equation (7) and (8)).
• the scaling factors M x and M y used for cursor tracking may require modification according to the measured roll angle. In most cases, the magnification for the optical system is the same for x andj (spherical lenses are typically used). However, since M x and M y are defined in terms of pixel numbers, the pixel dimensions of the sensor should be accounted for in the transformation.
• each of the equations (1) - (8) has two versions - one for each marker.
• the markers are fixed with respect to each other, the two versions of each equation are simply spatially shifted with respect to each other and therefore contain redundant data for pointing and tracking.
• only one set of the equations can actually be used for pointing and tracking.
• the additional marker(s) in the multiple-marker embodiments are useable for the additional DOFs, such as view angle, distance, and roll, but basic pointing can be accomplished with only one marker.
• Another method for pointing in multiple-marker embodiments uses an equivalent of the equations for the 1 -marker arrangement.
• Several forms of these equations may be used - each with pros and cons.
• the simplest approach would be to use the equations corresponding to one of the markers (e.g., marker 1 in the 2-marker embodiment) for both calibration and tracking.
• a similar approach could be used for 3-markerand 4-marker arrangements.
• a reference image coordinate constructed from some or all of the actual reference images can be used.
• a simple example of this would be a "center of mass" coordinate or average coordinate of the two or more markers.
• the effective reference coordinate would be:
• x mFi is the reference marker image coordinate on the sensor for the 1 th reference marker.
• y coordinate is the reference marker image coordinate on the sensor for the 1 th reference marker.
• the effective reference coordinate is then used in the pointing and tracking algorithm described in above in section 4.1 for the one marker embodiment.
• Other weighted averages of the reference coordinates could also be used as the reference point for cursor tracking.
• Using an average coordinate as the reference in an n-marker embodiment has the potential benefit of producing a more accurate and stable reference point since any random fluctuations in the individual marker coordinates will tend to average out. Also, since there are situations where the identity of each marker is not clear, it may be best to take a mean position as the reference. For example, the device may be tilted or upside down in which case, the correspondence of the marker images to references 1 and
• a "single step" calibration procedure is used, where, the user points the hand-held device at a known position on the screen, ( x o ' , y o ' ).
• a single visible marker or icon, at which the user points, could be displayed on the display to aide in the calibration accuracy though this is not necessary, especially if an easily located position such as the screen center or corner is used.
• the user is pointing at ( x' o , y' o )
• the corresponding sensor coordinates of the two markers ( 8 ⁇ recorded.
• the screen cursor location is then determined from the measured quantities:
• equation (21) determines the correct cursor location relative to a known reference "point" (e.g., the center of the display) by tracking the movement of the average reference marker position on the sensor relative to the average reference point position; which, in turn, corresponds to the known display reference point.
• point e.g., the center of the display
• Equation (23) is the equivalent of equation (4) where the known separation of the two reference markers ( Ax M ' F ) is used in place of Ax C ' AL and Ax MF is the separation on the sensor of the images of the two reference markers. Note that since the marker image separation does not depend (to a very good approximation) on the pointing direction. Similar arguments apply to thej; coordinate. Note also the sign difference between equations (4) and (23). This is a consequence of the difference in the two calibration methods - pointing sequentially at the two calibration markers (equation (4)) versus a single step recording of the two reference markers (equation (23)).
• the ambiguity in viewing angle (left versus right or up versus down of the screen center) remains.
• the markers should be within the field of view of the sensor (i.e., the user is pointing somewhere in the vicinity of the screen) and the user should be within the field of view of the markers (e.g., the user cannot be standing too far to the side or behind the screen).
• a typical marker field of view is (+/-) 30 - 45 degrees, which is consistent with a user's own practical screen viewing angle. This section will focus on the latter, "out-of-plane", 3-marker arrangement. Other, "in-plane", 3 -marker arrangements follow by extension from the 2-marker analyses provided above in section 4.2.
• FIG. 21a and 21b An out-of-plane arrangement is shown in Figures 21a and 21b. With this arrangement, the marker image locations on the sensor permit measurement of both magnitude and sign of viewing angle in both x and y dimensions (other 3- marker and 2-marker arrangements permit only viewing angle magnitude detection).
• One arrangement shown in Figures 21a and 21b), has all three markers 2101 1, 2101_2 and 2101_3 on the same side of the display 2100' and in close proximity to each other. This has the practical advantage that the markers can be placed in a common enclosure and easily powered by a single supply. It may also reduce the required field of view of the sensor. However, any arrangement of the three markers where one is placed out of the x '-y ' plane is acceptable.
• d is the actual marker separation between the two outer markers (1 and 3) along the x ' dimension
• D is the offset of the middle marker (2) with respect to markers 2101 1 and 2101 3 in the z ' dimension
• Sr 1J Mx 1 — X j ) 2 + (y t - y j ) 2 is the separation on the sensor 2103 between marker images i andj.
• the second term in brackets represents a potential constant offset in the case of an asymmetric arrangement of markers in which the center marker (2) is not placed equidistant to each of the outer markers (1 and 3).
• the offset can be accounted for by adding or subtracting a constant "offset angle" from the measurement.
• the offset angle, ⁇ x0 can be simply determined from the geometry of the asymmetric arrangement:
• the markers 2101 1 through 2101_3 are placed symmetrically and the offset term vanishes or becomes negligibly small.
• the markers are arranged horizontally (along the x' dimension), so unless the hand held device is rotated, the y coordinates will typically be absent from the Q x calculation in which case equation (24) reduces to a function of x coordinates only.
• the marker image coordinates may have both JC and y components, in which case equation (24) is used.
• equation (24) does not depend on the user's distance from the screen even though the marker image coordinates and separations do. This is because the formula is normalized such that distance changes affect both numerator and denominator equally.
• a solution that avoids the compromise in resolution is to move one or more of the markers out of the x '- ⁇ ' plane in order to prevent the eclipse and ambiguity at all view angles.
• the simplest arrangement for this approach has the center marker 2101_2 at a different height from the outer markers (i.e., offset above or below the line connecting markers 2101_l and 2101 3).
• the eclipse described above is avoided and there is no ambiguity in the identity of the marker images — marker image 2 will always lie above, for example, marker images 1 and 3.
• D can then be made as large as is desired or practical in order to improve resolution.
• ⁇ r j can be positive or negative, depending on the relative positions of the marker images. Specifically, if % 2 is between X] and X 3 , then all 3- tJ in equation (24) are positive. However, if X 2 ⁇ xj, then
• the new coordinates for marker image 2 are the coordinates of the point of intersection between two lines - one that connects marker images 1 and 3 and one that passes through marker image 2 and is orthogonal to the first line, hi essence, these are the coordinates of marker image 2 in the absence of the offset. Specifically, the value of the coordinate (x2,yi) in equation (24) is replaced by (x° ,
• m — is the slope of the line connecting marker images 1 and 3 and is
• the main purpose for the offset arrangement is to permit high resolution for arbitrary view angles where the marker 2 image moves outside marker images 1 or 3.
• This situation presents a potential ambiguity in the marker identification when the device is rolled since the;; coordinate of marker image 2 may be larger or smaller than that of marker images 1 and 3 and it is the y coordinate of marker image 2 that is used to distinguish it from the other marker images.
• This situation is unlikely to arise under normal use since it occurs only for simultaneously large view angles (when marker is outside marker images 1 or 3) and large roll angles.
• the specific values of view and roll angle where this situation could arise depend on the details of the arrangement.
• One solution to this potential problem is to use some property of the marker image other than itsy position to distinguish it from the others.
• marker 2101_2 could be modulated (powered on and off repeatedly) or set to a different constant power level from markers 2101_l and 2101 3. In this way, the identity of marker image 2 is determined for any view or roll angle.
• Viewing angle in the y ' dimension is also tracked using the "out-of-plane" arrangement.
• the geometry is illustrated in Figure 21b.
• the center marker image is displaced in the sensor's y dimension with respect to the outer marker images. This displacement, properly normalized, is used to determine ⁇ y .
• the simplified equation for viewing angle in they dimension (assuming ⁇ x « 0) is: d ⁇ y
• ⁇ y o ' is the actual offset in y ' of the center marker 2101 2 with respect to they ' coordinates of the outer markers 2101_2, 2101 3.
• the error in the location tracking using these equations grows.
• the errors in angle can be tolerably small.
• the error introduced by using equation (24) can be 2° - 3° for view angles in the 20° - 45° range in the x ' direction.
• R 3l ⁇ ⁇ l - sm 2 0 x . cos 2 e y
• Lo is a known distance from the screen where the calibration is performed
• the angle, a is essentially the angle subtended by the lines connecting markers 1 and 3 and 1 (or 3) and 2, as viewed from the top of the marker arrangement.
• three additional assumptions are used in equations (29) - (30). These equations assume that the markers are arranged symmetrically (marker 2 midway between markers 1 and 3 along the x' axis) and that marker 2 is in line with markers 1 and 3 (as viewed from the front) as shown in Figure 21 (the preferred arrangement). While it is possible to relax these conditions, the corresponding basic equations become significantly more complex with addition of more constants.
• offset angles such as those in equations (25) and (28) can be determined either initially or during the calibration procedure and then added or subtracted from the view angles calculated from the equations provided below for arbitrary user location, hi the preferred arrangement, however, the offsets will be negligibly small or zero and the solutions given below can be used directly, hi addition, equations (29) - (30) assume that the markers 1 and 3 are aligned along the x' axis, which is the preferred arrangement.
• Equations (29) constitute a set of three independent equations and three unknowns that can be solved for L, ⁇ r , and ⁇ v .
• the solutions are:
• Equation (31) - (34) determine the location of the user for any arbitrary point provided that the markers are detectable by the sensor. However, several special cases exist and are addressed below. [0153] Because of the repeating nature of the tangent function in equation (31), there is the possibility of an ambiguity in both the magnitude and sign of O x for
• equation (31) reduces to an expression identical to equation (24) (without the offset terms).
• equations (31) and (32) reduce to:
• equation (32) breaks down and cannot be used, hi practice, this situation will virtually never occur since it is beyond the view of the screen and the markers.
• equation (32) breaks down and cannot be used, hi practice, this situation will virtually never occur since it is beyond the view of the screen and the markers.
• a separate expression for ⁇ y can be easily derived from the basic equations (29) —
• the 3 -marker out-of-plane arrangement is capable of determining all of the relevant DOFs for the device - pointing location on the screen, distance from the screen, roll angle, and view angle with respect to the screen.
• the combination of these parameters defines completely the hand held device's location and orientation.
• the general procedure for determining these parameters is essentially an integration of those described in the previous sections.
• the pointing direction of the device (specifically, the intersection of the device's pointing vector and the screen) is determined using the procedure described in section 4.1. As noted earlier, in the case of multiple markers, any one of them can be used as the display cursor reference.
• marker 2101_2 is typically located in the center and centered on the display , it would be natural to use it as the cursor reference point.
• the center, or mean, position of the three reference markers could be used (similar to the 2-marker arrangement). Note that there is a slight offset in the reference position as the user's view angle changes due to the out-of-plane geometry, but this offset is typically quite small and unlikely to be noticed in a typical pointing application. If necessary, a small view angle-dependent correction factor could be used to reduce any offset in the pointing. The view angle is then determined using the procedures described in this section.
• marker Distance may be determined using the same expressions used in the 2- marker arrangement (equations (12) and (13)) for direct view or equation (33) for the general case. markerAgain, any pair of markers may be used, but the preferred method would typically use markers 2101 1 and 2101_3 to determine distance, L.
• the 3-marker embodiment has the capability of detecting all of the relevant DOFs and thus it should be sufficient for most, if not all, applications that require both device pointing and location tracking.
• the preferred (out-of-plane) arrangement of the 3-marker embodiment is not possible, not practical, or undesirable.
• the main drawback to the preferred 3-marker geometry is that one of the markers must be set back or forward with respect to the other two, and the resolution of the view angle measurement is enhanced by increasing this offset. This can result in an arrangement where one of the markers is offset by an inconveniently large distance that may not be possible due to space constraints.
• the preferred 3- marker embodiment works well in most cases, there are possible situations where an "in-plane" geometry may be necessary or desirable.
• certain 4- marker arrangements can provide the required DOF sensitivity using a simple geometry that avoids some, or all, of the drawbacks of the 3 -marker embodiment.
• the 4-marker embodiment may have some additional advantages in some situations.
• the main benefit of the 4-marker arrangement is that it permits separation of the distance and view angle measurements in both x and y as well as determination of the angle sign (e.g., left or right of the screen). Recall that these variables are inherently coupled and cannot be separated based on marker image coordinates alone in the basic 2-marker embodiment described in section 4.2. As in the other embodiments, the basic measurement involves determination of the four marker image coordinates and, from those, their separations. The various image separations are used to calculate the distance and view angle.
• the 4- marker arrangement there are two fundamental modes or regimes of operation. One is referred to here as the "far field" and the other as the “near field". The solutions for the two regimes are complementary and rely on different measurements of the image separations.
• the far field is defined as the region where the user's distance to each of the markers is approximately the same and may be considered equal. This occurs when either the marker separation is much smaller than the user's distance to the screen (i.e., markers) or when the user is located at a small view angle (i.e., near screen center).
• the far field can be defined as when the following condition holds:
• the view angle is smaller and the far field condition is satisfied at even smaller distances.
• the far field begins at L ⁇ d, meaning that the user can be as close as the marker separation to the screen and still be in the far field.
• the marker separation, d is close to a screen dimension, the user could be about one screen width away and remain in the far field.
• the significance of the far field is that it permits useful approximations in the calculation of the distance and view angles.
• the near field regime exists in regions where the far field condition in equation (36) is invalid, or equivalently, when the distance from the user to each of the markers cannot be assumed to be equal. In this regime, the error in the far field solution may exceed acceptable limits and the near field solution, which relies on the variation in distance to each of the markers, may be used. As the user moves further from the markers, the near field solution becomes less sensitive and the far field solution can be used, hi most situations, the far field solution will be adequate for determining the distance and view angle. However, inherent in the far field solution is insensitivity to the view angle sign. While the near field solution (which is essentially exact within some assumptions) is fairly insensitive to view angle changes, it is sensitive to the sign of the angle. In many cases, therefore, the more sensitive far field solution is used to determine the magnitude of the view angle and the near field solution is used to determine the sign, or direction, of the angle.
• FIG. 22 The basic geometry of the 4-marker embodiment is shown in Figure 22. Conventional spherical coordinates are used to define the view angles, ⁇ x and ⁇ y , and distance, L, to the marker origin. The origin is taken as the center point of the markers, which is typically the screen center and the distance is measured to the origin. Referring to Figure 22, the primary measurable quantities are the various separations of the marker images on the sensor,
• the basic difference between the far field and near field approaches is that the far field uses combinations of adjacent sides (e.g., R 21 and R 31 ) and diagonals (R 41 and R 32 ), whereas the near field uses combinations of opposite sides (e.g., R 31 and R 42 ) to determine distance and view angles.
• the basic equations relating the measured marker image separations, R y , and L, G x , and ⁇ y , using the far field approximation are:
• Rn * 43 ⁇ J ⁇ l -sin ⁇ - cos 2 ⁇
• R n — - • ⁇ l - (sin G x • cos ⁇ y ⁇ cos a + sin ⁇ y • sin of
• ⁇ is a geometrical constant determined by the angle formed by the horizontal line connecting markers 1 and 2 and the diagonal line between markers 1 and 4 (see Figure 22).
• the geometrical angle is defined as:
• d x and d y are the actual marker separations in the x ' and y ' directions, respectively.
• the formulas for the two opposite sides are the same in equations (37) and that the values of L, Q x , and ⁇ y are the same in each expression. This is due to the assumption that the arrangement is rectangular and that the distances and angles are essentially the same with respect to all markers (i.e., the far field approximation).
• R 4- R combination is an average of the two sides.
• R x — —
• ⁇ s ⁇ -4 f y • Jfc . sin ⁇ j - sin ⁇ - cos ⁇ ,
• Equation (39) in conjunction with the formulas for adjacent sides (R21 and R 31 ) in equations (37) form a set of three equations and three unknowns that can now be solved for L, ⁇ x , and ⁇ y .
• the resulting formulas are:
• equations (40) - (42) are used to determine distance and the magnitudes of the view angles, and the following conditions can be used to determine the signs of the view angles: ⁇ x > 0 for R 42 - R 3 i > 0, Q x ⁇ 0 for R 42 - R 3 i ⁇ 0 and ⁇ y > 0 for Rn - R 42 > 0, ⁇ y ⁇ 0 for R 2I - R 4 2 ⁇ 0. Note, however, that these inequalities become inherently uncertain for positions near screen center, or far from the screen, i.e., the far field where the inequalities become approximate equalities.
• equation (40) becomes undefined and distance can not be determined.
• this situation will virtually never occur since it is outside of the typical range of operation of the system - the user cannot see the screen and the markers are generally not detectable by the sensor in this region. Therefore, this special case is almost never a problem.
• L y is the distance from the user to the center of the line joining markers i and j and Z°. is the corresponding distance during calibration.
• the far field equations can be used in general to determine, with high sensitivity, the user's distance and view angle magnitude.
• the conditions described earlier for the far field can be used (comparing the magnitudes of RiJ), or the sign of A x and ⁇ y can be used.
• the near field equations equations (45) — (52)
• pointing and tracking can be performed using one or more of the markers.
• the preferred embodiment uses equation (20) to determine an "average" coordinate for the marker images and tracks changes in this coordinate, as described in section 4.2.
• the remaining DOF that needs to be tracked is roll.
• a variation on the approach taken in equations (15) - (17) for the 2-marker embodiment can be used for the 4-marker embodiment as well.
• any two markers can be used as the reference for roll motion as long as their initial coordinates are known.
• the procedures for both roll detection and rotation compensation for the 4-marker embodiment are essentially the same as those for the 2- and 3-marker embodiments described in the previous sections.
• a potential issue for any embodiment that involves multiple markers is the correct identification of each marker, i.e., which is marker #1, #2, etc.
• the marker identity will be clear.
• marker image #1 will nearly always be in the upper left hand corner of the four marker arrangement on the image sensor, and the other markers will have image positions relative to marker #1 as shown in Figure 22.
• marker #1 may be in a position normally occupied by marker #2, for example, which could lead to errors in the position and orientation measurements, though pointing accuracy should not be significantly affected by any marker misidentif ⁇ cation.
• marker identity could be a problem
• One method is to require that the handheld device is initially calibrated at a known roll angle, and then subsequent motions are tracked relative to this initial orientation. In this way, the identity of each marker is always known after an initial calibration.
• a more robust method would be to uniquely identify at least one of the markers using a physical distinction.
• marker #1 in the 4-marker embodiment, or the center marker in the 3-marker embodiment may be temporally modulated (e.g., repeatedly turned on and off).
• the tracking algorithm could then be programmed to identify the modulated marker as #1, for example.
• the other markers would maintain their relative positions with respect to #1.
• the advantage of this approach is that no calibration is required to establish the identity of the markers.
• the disadvantage is that a more complex marker set up is required.
• absolute pointing involves a working system in which signals sent from a handheld device are processed, and, in response, certain effects are made to appear on a display and/or are triggered with electronic equipment.
• electronic equipment include electronic equipment containing the following types of circuitry: a) standard television circuitry such as the circuitry within a television used to process the signals received from the television's remote control; b) a satellite TV receiver circuitry; c) a cable TV receiver circuitry; d) VCR circuitry; e) DVD player circuitry; f) a computer's circuitry; g) music player circuitry (e.g., CD player circuitry, stereo receiver circuitry, etc.); h) gaming circuitry; i) an IPTV receiver and/or television circuitry; j) receiver circuitry for a "receiver" that receives signals from one or more of items of electronic equipment such as those listed in a) through i) above; k) home and/or office automation equipment (e.g.
• Each of these types of circuitry typically include some kind of processor or controller that executes program code that: 1) implements a user interface for the particular type of equipment at issue (e.g., a menu driven user interface in the case of a television, a TV receiver, a DVD player, a VCR player, etc,; or, a windows based interface in the case of a computer); and/or, 2) implements a cursor or icon (which may be associated with a user interface) or a highlighting mechanism for a menu based user interface in which specific menu options are highlighted.
• a user interface for the particular type of equipment at issue e.g., a menu driven user interface in the case of a television, a TV receiver, a DVD player, a VCR player, etc,; or, a windows based interface in the case of a computer
• a cursor or icon which may be associated with a user interface
• a highlighting mechanism for a menu based user interface in which specific menu options are highlighted.
• Figures 23a through 23e show different architectures that includes these processors for various types of electronic equipment such as those outlined above. Of course, other types of architectures are possible. For simplicity, none of Figures 23a through 23e show a non- volatile memory device typically used for storing program code (e.g., a Read Only Memory such as an EPROM, EEPROM, etc,). It should be understood that the architectures of Figures 23a through 23e are assumed to include them.
• Figure 23a shows an architecture that is often found in personal computer (PC) systems.
• a memory controller 2302 controls requests for access to a memory 2303_l by a processor 2307_l (single core or multi-core), a graphics controller 2304 and an I/O control hub 2305.
• the memory 2303_l typically contains instructions that can be executed by the processor 2307_l and data upon which these instructions operate.
• the display may be provided with content information prepared by the graphics controller 2304; or, if the display is further downstream in the processing chain, information for rendering content on the display may be sent from the computing system through an input/output (I/O) (such I/O 2301 1).
• I/O input/output
• An I/O is any kind of channel through which information processed by the processor 2307_l is sent and/or received by the system to which the processor 2307_l pertains. For instance, information sent by the handheld device would arrive through an I/O (e.g., this includes wireless transceiver circuitry).
• Figure 23b shows an architecture where the processor 2307J2 is directly coupled to the memory 2303_2, and where, the processor 2307_2 receives/provides its own I/O 2303_2.
• Figure 23c shows an architecture similar to that of Figure 23b except that a bus is used for the I/O 2306_3.
• Figure 23d shows an architecture where the memory 2303_4 shares the same bus as the I/O 2306_4.
• Figure 23e shows an architecture where the memory 2303_5 is accessed through a different bus than that of the I/O 2306_5.
• Other applications include the ability to support multiple users simultaneously (i.e., in parallel) or, in some cases, sequentially (i.e., serially), particularly in situations where it is desirable to have cursor/menu selection control shared among more than one user or to have multiple active cursors, hi the case of a single shared cursor or single shared menu, independent users, each with a remote pointing device, may possess control of the screen cursor (and thus function commands) and/or make menu selections using a prescribed protocol.
• One approach may be "centralized” wherein one of the remote devices may be designated as the master controller that may relinquish cursor control to any of the other users by sending appropriate commands to the control unit, hi other approaches control of the cursor may be "distributed” such as a token passing scheme (e.g., where a token is passed around by the handheld devices and the device having the token assumes control of the cursor), hi either centralized or distributed schemes, control priority may be determined based on a set of rules that account for activity level (e.g., an inactive remote control loses priority or an active one gains priority), a prescribed sequence (priority is rotated among the group of users), or a predetermined hierarchy of control (e.g., user #1 has control while active, user #2 has control while active and #1 is inactive, etc.). Many other sharing protocols that are encompassed by the spirit of this description may be readily implemented by those skilled in the art.
• a token passing scheme e.g., where a token is passed around by the handheld devices and the device having the
• Multiple users with multiple simultaneously active cursors and/or menu selection represents a more advanced form of a multiple-user environment.
• the control unit would display and move screen cursors corresponding to each user's pointing device based on the coordinate data for each device. Different cursor icons may be used to aid in distinguishing each user's actions (e.g., color, shape, etc.).
• multiple cursors maybe displayed and controlled simultaneously, priority of function commands has to be determined at least for situations where a single display feature is pointed to by multiple cursors and/or in systems capable of only executing one command at a time. The simplest form of priority determination could be based on which user is first to select a command. Other procedures for determining command priority will be apparent to someone skilled in the art. Multiple users with simultaneously active cursors may be particularly useful for gaming functions.
• another aspect of the multiple-reference or multiple-sensor form of the invention is the ability to sense rotational motion of the pointing device, as described in sections 1 and 4. For example, with two reference markers, relative rotation between the screen and the pointing device, along an axis approximately parallel to the line connecting the pointing device and screen, is detectable.
• This type of motion is not generally detectable with the single reference/single sensor preferred embodiment since it does not represent a change in pointing direction.
• This type of motion is typically referred to as roll, and, while it does not represent a pointing action, may be used to enable other commands and functions through simple and intuitive gestures.
• this rotational or twisting motion is the "virtual dial" discussed above for controlling various attributes of an appliance, e.g., volume, channel, song track, etc.
• Other basic motions or actions detectable by the present invention and described previously may be used for specific commands or functions.
• Another aspect of the operation of the handheld device is that it does not require the user to point within the screen boundary in order to be recognized by the overall system. This results from the location and orientation sensing capability of the system so that executable actions are not confined to the location of a screen cursor, as in conventional mouse-type pointing devices.
• the pointing range is determined by the device sensor's field of view and the user's distance from the screen instead of the screen size.
• the usable pointing range is larger than the screen size. Therefore, pointing actions beyond the screen boundary can be used for a variety of functions or commands. For example, screen menus can be activated or displayed upon the user directing the pointer beyond a specified boundary (e.g., the screen edge). Or motion up or down outside the screen boundaries may activate a scrolling function on a displayed screen menu. Many other "outside-the-screen" actions may be enabled using this property of the absolute pointing device.
• a related application of the absolute pointing device is its use as a video game interaction mechanism, or game controller.
• Conventional game controllers are based on mouse-type or joystick-type input devices and buttons, in which the game functions (e.g., scene navigation, weapon firing, etc.) are controlled by standard mouse or joystick movements (which are relative in nature) and button pressing.
• the navigation and motion control aspects of video games may be enhanced significantly by the absolute pointing and location sensing capabilities of the device.
• the sensitivity to screen distance, screen viewing angle, device roll angle, pitch, and yaw permit degrees of freedom for the player that are not found in conventional game controllers.
• These properties can be employed to control a variety of actions such as intuitive three dimensional motion (e.g., games using flight simulation) and location-sensitive interactions (e.g., scene perspective moving as player moves, or speed control using in-out motion).
• intuitive three dimensional motion e.g., games using flight simulation
• location-sensitive interactions e.g., scene perspective moving as player moves, or speed control using in-out motion.
• user interface should be construed to be broad enough to cover a displayed gaming interface.
• Some of the processes taught by the discussion above may be performed with program code such as machine-executable instructions which cause a machine (such as a processor disposed on a semiconductor chip, or an "interpreter” (e.g., a Java virtual machine) that converts abstract program code into processor-specific program code) to perform certain functions.
• program code such as machine-executable instructions which cause a machine (such as a processor disposed on a semiconductor chip, or an "interpreter” (e.g., a Java virtual machine) that converts abstract program code into processor-specific program code) to perform certain functions.
• a machine such as a processor disposed on a semiconductor chip, or an "interpreter” (e.g., a Java virtual machine) that converts abstract program code into processor-specific program code)
• these functions may be performed by specific hardware components that contain hardwired logic circuitry for performing the functions (e.g., a logic state machine), or by any combination of programmed computer components and hardwired logic components.
• An article of manufacture may be used to store program code.
• An article of manufacture that stores program code may be embodied as, but is not limited to, one or more memories (e.g., one or more flash memories, random access memories (static, dynamic or other)), optical disks, CD-ROMs, DVD ROMs, EPROMs, EEPROMs, magnetic or optical cards or other type of machine- readable media suitable for storing electronic instructions.
• Program code may also be downloaded from a remote computer (e.g., a server) to a requesting computer (e.g., a client) by way of data signals embodied in a propagation medium (e.g., via a communication link (e.g., a network connection)).

Applications Claiming Priority (5)

Publications (1)

Family

ID=36264057

Family Applications (1)

Country Status (3)

Families Citing this family (11)

Citations (5)

Family Cites Families (11)

• 2006
  • 2006-01-11 EP EP06718289A patent/EP1836549A2/en not_active Ceased
  • 2006-01-11 WO PCT/US2006/001198 patent/WO2006076557A2/en active Application Filing
  • 2006-01-11 JP JP2007551393A patent/JP5231809B2/ja not_active Expired - Fee Related

Patent Citations (5)

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events