CA2896357A1

CA2896357A1 - A method and laser pointing machine system for indicating items in a display case

Info

Publication number: CA2896357A1
Application number: CA2896357A
Authority: CA
Inventors: Kefim A. Nancoo
Original assignee: Individual
Current assignee: Individual
Priority date: 2015-07-03
Filing date: 2015-07-03
Publication date: 2017-01-03

Abstract

A method for customers to indicate items within display-cases is achieved by the control of various laser pointing mechanisms. These mechanisms comprise mechanical and sensory components that limit the spatial region within which the lasers operate.
Sensors determine the location and angular orientation of the laser and confine the laser to only illuminate points within a display case. A robotic machine with an end-effector is adapted to manipulate a laser diode. It is linked to a gesture recognition component which is comprised of a capture device and a computer that processes signals from the capture device. A customer points within the field of detection of the capture device and a vector corresponding to the direction of which the pointing member is pointing is computed. The robotic machine is oriented to illuminate a point that corresponds to the intersection of the vector with a topographically analyzed surface within the display-case.

Description

A Method and Laser Pointing Machine System for Indicating Items in a Display Case SUMMARY OF INVENTION
A method and technological system is provided for the indication of purchasable items situated within a display-case with a controllable laser pointing apparatus. Multiple embodiments of the invention are described wherein the claimed method of indication can be achieved.
A system for the control and manipulation of a laser pointing apparatus is described in one embodiment.
In another embodiment a plane equation of the display-case's product-placement surface is obtained through a technique of topography and is represented in world coordinates relative to a reference frame. A capture device is used to detect objects that have moved to within a predefined proximity of the display-case's transparent window. The detected objects are matched to probabilities that determine the object's eligibility as a detected pointing-object. In one embodiment detected pointing-objects are analyzed to compute a straight line that collinearly traverses the central moment of the most linear portion of the object that is most proximal to the display-case's transparent window surface. A 3-dimensional coordinate of the point of intersection of the straight line and the plane is computed in world-coordinates. These coordinates are outputted to a controllable robot with an end effector adapted to fit and orient a controllable laser emitting diode to illuminate the point of intersection. The pointing object's position is successively followed and the straight line, collinearly traversing the pointing-object's most linear portion, is continually calculated. The point of intersection is recalculated and the robotic mechanism continually adjusts the laser emitting diode to illuminate the point of intersection.
TECHNICAL FIELD
The present invention provides an improved method of indicating product items within a display-case and laser pointing systems that can be controlled by customers for the indication of these products. Laser pointing systems are comprised of displacement and rotational sensors, such as accelerometers and gyroscopes respectively, which are configured to track and limit the angular

- 2 -and spatial regions within which the laser pointing systems can operate. Other laser pointing systems are mechanically configured for lateral motion and pivotable motion about a display-case's transparent such that the direction of the lasers' emitted laser beams are only toward points within a display-case.
A robotic system is comprised of an RGB camera and an end-effector that manipulates a laser diode. The robotic system may also be configured for topographical analysis of a display-case surface using the orientation information of the end-effector and processed image data acquired from RGB camera. The robotic system is also communicably linked to a gesture recognition system comprising a central computing system and capture device or range imaging camera.
The gesture recognition system may use several object tracking techniques and depth data processing to detect pointing gestures and pointing members and also track the pointing members. A vector generated from pointing members within the capture device's field of vision is used to find points of intersection on a topographically analyzed surface within a display-case.
The central computing system computes the spatial coordinates of the point of intersection and manipulates the robotic system to illuminate the point of intersection with a laser beam emitted by the laser diode.
BACKGROUND AND PRIOR ART
In many retail stalls such as in groceries, bakeries and jewelry stores display cases generally hold a variety of products for customer viewing, selection, and choice. The customer service representative (CSR) in attendance will retrieve the specified selection for the customer.
These displays can hold a multitude of products allocated to some apportioned space. They can be organized in rows and columns, and sometimes randomly, but often in close proximity to each other so that a customer's ability to communicate specific items of interest to the CSR from among the many products can become a rather difficult prospect. Usually, there are tags that are appended to the end of the display case, or the products' apportioned spaces that distinguishes the corresponding products by name and price. Each displayed product may consist of a multitude of pieces or units. These pieces or units are often differentiated by variations in quality that are substantiated by different physical characteristics such as dimensions, weight and visual

- 3 -appearance. A customer will assess these units or pieces and decide on an item or items with the most desirable quality.
Display shelves are often equipped with transparent windows that separate the product on display from the customer for hygienic or security reasons, but they can impede the customer's ability to readily indicate to the CSR what specific item they want.
Currently, the process of a customer indicating their choice of a specific item or product within a display case involves two different methods or combinations thereof. The customer can verbally describe the location of their choice of a particular product to the CSR. This process begins with the customer using some arbitrary reference point that is mutually discernible to both the customer service representative and the customer, such as the first item closest to the customer on their left or right. With the customer identifying and relaying the reference point to the representative, they then direct the representative with a series of navigating instructions to their choice of items. This process also uses trial and error as the representative can wrongly interpret the customer's instructions and thus select the wrong item at which point the process begins again. This item is then used as a new reference point from which the customer can re-direct the representative to their right choice.
A customer may also attempt to physically point out the product from the transparent window barrier that separates the customer from the product. This involves the customer service representative attempting to determine the specific product being pointed to by approximating a straight line from the pointing finger tip to the intended item. This process often results in erroneous selections due to the CSR's inability to correctly estimate the projected line between the customer's pointing finger and the item in question.
These efforts can be tedious and time wasting, and result in customer frustration, longer waiting times for service, and occasional loss of sales.
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 illustrates a customer indicating a purchasable item within a display-case to a Customer Service Representative using a laser pointing apparatus FIG. 2 illustrates an example embodiment of a laser pointing apparatus that maybe used to indicate items within a display-case FIG. 3 illustrates the components that maybe used to create a laser pointing system of FIG. 2

- 4 -FIG. 4 illustrates a flow method of controlling the on-off powering of the laser pointing system of FIG. 2 FIG. 5 illustrates a flow method to set up a spatial region and range of angular rotation in which a laser pointing apparatus may be operable.
FIG. 6 illustrates a flow method by which the laser pointing apparatus maybe be used in accordance with the steps of FIG.5 FIG. 7 illustrates another example embodiment of a laser pointing apparatus that maybe used to indicate item within a display-case FIG.8 illustrates the components of a laser pointing apparatus that may be used to create a laser pointing apparatus of FIG. 7 FIG. 9 illustrates another example embodiment wherein a customer is indicating a purchasable item within a display-case to a Customer Service Representative with a laser pointing system FIG. 10 illustrates the components that maybe used to create a laser pointing system of FIG.9 FIG. 11 is a close up view of a pointing-object and ifs geometry FIG. 12 is a flowchart illustrating a method of establishing a coordinate system and the relative spatial distributions of components of a laser pointing system of FIG. 10 FIG. 13 is a flowchart illustrating a method of acquiring a plane equation and bounding region for use with the laser pointing system of FIG. 10 FIG. 14 is a flowchart illustrating a method of detecting a pointing object and illuminating a point being pointed to by the pointing object using the laser pointing system of FIG. 10 FIG. 15 illustrates the laser pointing system of FIG.10, an approximated plane and a bounding region that is collinear with the plane FIGS. 16a, 16b and 16c illustrate a sequence whereby a robotic end-effector with a laser diode maybe manipulated to illuminate a specific point on a plane FIG. 17 illustrates an alternate embodiment of the present invention adapted for use with a multi-shelved display case.
DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS
Turning now to the drawings in which like reference characters indicate corresponding elements throughout the several views, attention is first directed to FIG. 1 in which is seen a Customer 14

- 5 -indicating a specific purchasable item 30 disposed within a display-case 10 to a Customer Service Assistant, (CSA), 12. A customer service assistant maybe referred to by many different names including "retail clerk", "retail assistant", "customer service representative" and the like.
The term "customer service assistant" will henceforth be referred to as CSA, 12, in this Detailed Description of Preferred Embodiments. The customer 14 indicates a specific purchasable item 30 to the CSA, 12, by manipulating a laser pointing apparatus 26 to direct a coherent beam of light 24 across a transparent material 16 to illuminate at least one point 22 on the surface of the specific purchasable item 32. The transparent material 16 may also be referred to as a "display-case window" or "transparent window" and may be comprised of glass, plastic, acrylic or the like. The illuminated point 22 is visible to the customer 14 and the CSA 12.
Also disposed within the display-case 10 is a plurality of purchasable items 32 which the customer can view through the transparent material 16. The side of the display-case 10 on which the transparent material 16 is attached may be referred to as the "customer side". The side of the display-case 10 on which the CSA 12 has access to the purchasable items 32 and 30 may be referred to as the "clerk side". The purchasable items 32 and 30 within the display-case 10 are made accessible to the CSA through a door 12 attached to the display-case 10 and located on the "clerk side". The customer may wish to acquire this indicated item 30 and communicate this wish to the CSA 12 who in turn retrieves the indicated item 32 through the access provided by the door 12. The customer 14 may wish to indicate the item 30 to the CSA for a number or combination of reasons such as but not limited to: acquiring the item for purchase, inquiring the price of the item, inquiring the weight of the item, acquiring the item for closer inspection of the item and the like.
As illustrated in FIGS. 1 and 2 the laser pointing apparatus 26 may be connected to a connecting component 28 such as a chord, wire, cable and the like, which may be further attached to an anchoring component 20 which may be further secured to the display-case 10.
The connecting component 28 may serve to limit the distances through which the laser pointing apparatus 26 can be moved and manipulated. The connecting component 28 may also be comprised of a gooseneck or flexible neck or flexible tube. Other examples of connecting components that may be adapted to fit a laser pointing apparatus 26 include flexible arms, movable joints, telescopic or extendable tubes such as found in US Patent Number 1,010,335; 1,735,212;
3,381,122;
4,895,708, US 6803525 B1 for example.

- 6 -As shown in FIG. 1 the anchoring component 20 may also be comprised of a holstering component 21. In accordance with the current embodiment the holstering component 21 may serve to confine the laser pointing apparatus 26 when not in use and prevent undesirable movement as that caused by gravity and arbitrary vibrations. The holstering component 21 may comprise a rubberized band that provides sufficient friction to hold and prevent the laser apparatus 26 from unwanted movement.
In an alternate embodiment not here shown the anchoring unit 20 may be comprised of a magnetic surface to which a laser pointing apparatus 26 that may be comprised or partly comprised of a ferromagnetic material attracted to the magnetic surface with sufficient force to avoid unwanted movement.
In yet another embodiment not here shown the anchoring component 20 may be comprised of a holder adapted to hold the laser pointing apparatus 26 when not in use.
Suitable examples of holders that may be adapted to hold the laser pointing apparatus 26 when not in use include: US
patent 5405024A entitled "Pen Holder", patented on 11 April, 1996 and hereby fully incorporated herein by reference; US patent 6202862 B1 entitled "Tubular yielding holder for various size pens", patented on 20 March, 2001 and hereby fully incorporated herein by reference; US patent 5232103A entitled "Holder for Elongate Elements", patented on 3 August, 1993.
In an alternated embodiment of the invention, not here shown, the anchoring component 20 may be comprised of a retractable reel mechanism to which the connecting component 20 may be connected. In such an embodiment the connecting component 20, such as a chord, may be retracted into the reel mechanism when the laser pointing apparatus 26 is not in use. Suitable examples of retractable reel mechanisms include: US patent US 8387763 B2 entitled "Retractable cord reel" published 5 March 2013, and hereby fully incorporated herein by reference; US patent US 7661855 B2 entitled "Retractable reel assembly"
published 16 February 2010, and hereby fully incorporated herein by reference

- 7 -In an alternate embodiment of the invention, not here shown, the connecting component 28 may be directly secured to the display-case 10 via a fastener such as a screw, glue or the like. The connecting component 28 may also be connected to a part of which the display-case is comprised. Such a part may be a supporting member or a fastener such as a screw, bolt or the like.
Display-cases are commonly found in retail stores such as grocery stores, markets, warehouses, boutiques and the like. In such an environment the connecting component 28 may be connected to a surface that is a part of the retail store or retail environment. Such a surface may be in proximity to the display-case such that the laser pointing apparatus 26 may be used as described in the present embodiment of the invention. Examples of such surfaces in a retail environment to which a connecting component 28 may be fixed include walls, adjacent display cases, shelving unit, refrigerated coolers, doors and the like.
A retailer, in the interest of safety, may wish to constrain the operation of the laser pointing apparatus 26 such that the emitted laser beam 24 may only illuminate points within the display case 10. One example embodiment of a system by which to achieve such said constrained operation of a laser pointing apparatus 26 is illustrated in FIGS. 2 and 3.
FIGS. 2 and 3 illustrate a laser pointing apparatus 26 that maybe comprised of a pressure activated switch 57 communicably or electrically coupled to a controller 54. As shown in FIG. 2 the pressure activated switch 57 maybe fitted to the laser pointing apparatus 26 such that the laser beam 24 emitted from the laser diode 44 may traverse, unobtrusively, to the external environment. As shown in Fig. 3, the pressure sensor 57 maybe communicably linked to a controller 54 to control the on-off powering of the laser emitting diode 44. The pressure activated switch 57 may be fitted with the laser pointing apparatus 26 such that when the laser pointing apparatus 26 is manipulated to be in a sufficient force generating contact with the customer-side surface of the transparent medium 16, the resultant opposite force is transferred to the pressure activated sensor 57 which may output a signal to the controller 54 which switches the laser emitting diode 44 to a power-on state. The beam of laser 24 emitted by the laser emitting diode 44 may traverse the direction of the vector of the applied contact generating force where this said vector intersects with the transparent medium 16 at an angle of incidence that is less than the critical angle of

- 8 -reflection of the transparent medium 16 and the wavelength of light of the emitted laser beam 24.
"Critical angle" is defined as the angle of incidence above which total internal reflection of light Occurs.
An example embodiment of such a pressure activated laser pointing apparatus is shown in FIG. 3 which illustrates a pressure sensor 57 electrically couple to a controller 54.
Suitable examples of such pressure sensors 57 include a MEMS pressure sensor, silicon pressure sensors and piezoelectric pressure sensors. A suitable example of a signal generating pressure sensor that can be used in accordance with the present embodiment is US patent US
7290453B2, entitled "Composite MEMS pressure sensor configuration", patented on 6 November, 2007 and hereby fully incorporated herein by reference.
Another example embodiment of a system by which to achieve such constrained spatial operation of a laser pointing apparatus 26 is illustrated in FIGS. 1-3. As illustrated in FIG. 1 an external photo-optic component 18 may be affixed to the display-case 10 such that a beam of coherent light emitted by the photo-optic sensory unit 18 may traverse tangentially and vertically superior to the surface of the transparent material 16. As shown in FIG. 3, the external photo-optic component 18 may be comprised of a photoemitting component 50 which may further be comprised of a coherent infrared or visible light emitting device 51 such as an infrared line-laser or visible light line-laser. As shown in FIG. 2 a laser pointing apparatus 26 adapted for use with the external photo-optic component 18 may include a transparent covering 46 through which said emitted beam of coherent light may traverse. The laser pointing apparatus 26 may further include a photodetector 40 disposed within the transparent covering 46. The photodetector 40 may be sensitive to the wavelength of the light emitted by the infrared or visible light emitting device 51. Such an arrangement of components may be referred to, by one skilled in the art, as a photoelectric sensor. In accordance with the prior art of photoelectric sensors, the photodetector 40 may be used with a controller 57 to control the on-off powering of the laser emitting diode 44 in response to light signals received from a light emitting device 51.
FIG. 4 illustrates a method by which the control of the on-off powering of a laser emitting diode 44 may be achieved. At a first exposure of the photodetector 40 to a light signal emitted by a

- 9 -photoemitting device 51 the laser diode 44 is switched to a power-on state 62.
At a consecutive exposure to a light signal from a photoemitting device 51 the laser diode 44 is switched to a oower-off state 64.
In accordance with the present embodiment of the invention, FIG. 1 illustrates a customer 14 manipulating the laser pointing apparatus 26 to a first exposure of the photodetector 40 to a light signal emitted by a photoemitting device 51. This first exposure causes the laser emitting diode 44 to switch to a power-on state. The customer 14 may further manipulate the laser pointing apparatus 26 to direct a coherent beam of light 24 to illuminate at least one point 22 on product surface that is visible to the customer 14 and to the CSA 12. The illuminated point 22 may be used as a reference point by which the customer 14 can communicate the relative location of a indicated product item 30 to the CSA.
The customer 14 may wish to acquire additional purchasable items 32 and choose to further indicate additional purchasable items to the CSA according to the method described in the above paragraph. Upon completing the indication of the additional purchasable items to the CSA, 12, the customer 14 may chose to return the laser pointing apparatus 26 to the holstering unit, 20, whereby there is a second exposure of the photodetector 40 to light signal emitted by the photoemitting device 51. The second exposure may then cause the laser diode 44 to switch to a power-off state.
As illustrated in FIG. 3 the photoemitting component 50 may also be comprised of a two-photoemtitter device 52 such as two visible or infrared lighting components that are at a fixed and known distance relative to each other. The electromagnetic signal emitted 34 by these components may be received by the photodector, 40. The photodetector 40 may be at a known distance and orientation within the laser pointing apparatus 26. Upon reception of these electromagnetic signals 34, the photodetector, 40 may output a digital signal to a controller 54 which may use the signal for triangulation calculations to obtain 3-Dimensional distance and angular rotation measurements of the laser pointing apparatus 26 with respect to some reference point. Such a system may be known to those skilled in the art as a 'distance detection system' or 'distance detection circuit'.

- 10 -As illustrated in FIG. 3 that laser pointing apparatus 26 may also be comprised of a 2-photoemitter component 53 at a fixed and known distance relative to each other. The electromagnetic signal emitted 34 by these components 53 may be received by the photodetector, 41 which may output a digital signal to a controller 54. The controller may use the outputted signal for triangulation calculations to obtain a 3-dimensional distance and angular orientation measurement of the laser pointing apparatus 26 with respect to some reference point. Suitable examples of such distance detection systems include: US patent US 8873069, entitled "Motion sensing method for determining whether to perform motion sensing according to distance detection result and related apparatus thereof', published 28 October, 2014 and hereby fully incorporated herein by reference; US patent US 7193731 B2, entitled "Optical displacement sensor", published 20 March 2007 and hereby fully incorporated herein by reference.
As illustrated in FIG. 3, the laser pointing apparatus 26 may also be comprised of a 3-Dimensional gyroscope 56 used to obtain the angular orientation of the laser pointing apparatus 26 with respect to a fixed reference frame and a 3-Dimensional accelerometer 55 that may be used to obtain angular rotation values and displacement values in three dimensions with respect to some reference point. The gyroscope 56 and accelerometer 55 may output signals to a controller 54 that may be comprised of a clock and cpu that may process these signals to calculate and determine real time displacement, 3-Dimensional coordinates and orientation values.
FIG. 5 illustrates a method in accordance with the present embodiment by which the components illustrated in FIG. 3 may be used to set up a spatial region within which the laser pointing apparatus 26 may be constrained to operation and usability. According to one embodiment, at step 422, the laser pointing apparatus 26 is at a known orientation and distance relative to some reference point. An example of such a reference point may be a point disposed on the holstering unit 21. An example of such a known angular orientation could be a zenith of direction defined by the direction of the pull of gravity. According to step 424 the laser pointing apparatus 26 may be manipulated such that there is a first exposure of the photodetector 40 to the electromagnetic radiation emitted by the photoemitter 51 which switches the laser emitting diode 44 to a power-on state, 62. The point at which such a first exposure occurs may be the maximum perpendicular distance that is vertically superior to the transparent medium 16 and at which the laser emitting diode 44 may be switched to a power-on state or power-off state. According to steps 426 - 432, the laser pointing apparatus 26 is further manipulated to illuminate a vertex of a bounded region within the display case 10. As shown in step 426 four repetitions may be necessary to set up and store the maximum and minimum distance and angular rotation ranges with respect to the maximum and minimum distance within which the laser emitting diode 44 can be in a power-on state. As shown in step 430 the laser pointing apparatus 26 may be manipulated to illuminate a vertex point of a virtual quadrilateral or virtual bounded region within the display-case 10. At step 432 the 3-Dimensional distance coordinates and respective angular orientation values of the laser pointing apparatus 26 are calculated and stored in a memory unit such as which the controller 54 and, or, 58, may be comprised.
FIG.6 illustrates a method in accordance with the current embodiment by which the laser pointing apparatus 26 can be used, with accordance to the steps of FIG. 5, to illuminate points within the display-case. As shown in step 440 the laser emitting diode 44 is initially in a power-off state. At step 442 the laser pointing apparatus may be manipulated to a first exposure of the photodetectors to the photoemitters wherein such a first exposure may be at a maximum displacement vertically superior and perpendicular to the customer-side of the transparent material 16. At step 444 the distance coordinates in 3-dimensions and angular orientation of the laser pointing apparatus 26 is calculated. At step 446 the maximum and minimum angular orientation values are interpolated with respect to the current distance coordinates. The angular orientation values can be interpolated given the current distance coordinates and the maximum and minimum angular orientation values and distance coordinates obtained in steps 426, 430 and 432. Examples of such interpolation techniques include linear interpolation, polynomial interpolation, spline interpolation and the like. At step 448 the current distance coordinates and current angular orientation values of the laser pointing apparatus 26 are compared with the maximum and minimum angular orientation and distance coordinates interpolated in step 446.
As shown in step 452, the laser emitting diode 44 remains in the power-off state until the laser pointing apparatus is within the interpolated distance ranges and respective angular orientation ranges. As shown in step 450 if the current distance coordinates and current angular orientation values are within the interpolated bounds the laser emitting diode 44 is switched to a power-on state.
As shown by the steps in FIG. 6, a user may manipulate the laser pointing apparatus 26 to within the angular orientation bounds and distance coordinates to illuminate a point 22 that is visible to both the customer 14 and CSA 12. The illuminated point 22 may then be used as a reference point by the customer 14 to communicate the relative location of a indicated product item 30 to the CSA 12. The customer 14 may wish to acquire additional purchasable items 32 and choose to further indicate additional purchasable items to the CSA 12 according to the method described in FIG. 6.
Attention is now direction to FIGS.7 and 8 in which is seen an alternate embodiment of a laser pointing apparatus that maybe used in accordance with the current invention.
FIGS. 7 and 8 show a laser pointing machine 404 that exhibits pivotable motion along 2-axes 420, 408, 410 and linear motion 416 along a rail component 406. The rail component 406 can be adapted for use with the laser pointing machine 404 such that linear motion 416 can be achieved with the laser pointing machine 404. Suitable examples of such a rail component 406 that can be adapted for linear motion include US patent 7207432 B2 entitled "Linear motion drive system and rail holder" published 24 April, 2007 and hereby fully incorporated herein by reference; US patent 6499588 B1 entitled "Conveyor System" published 31 December, 2002 2007 and hereby fully incorporated herein by reference; US patent 6435719 B1 entitled "Linear guide device"
published 20 August, 2002 and hereby fully incorporated herein by reference.
As shown in FIGS. 7 and 8 an elongated component 402 maybe be adapted to fit and retain a laser emitting diode 44. The elongated component 402 may be fitted to the geometrical center of and orthogonally to the axes of rotation of two rotatable, slotted shafts 412, 414. The two rotatable, slotted shafts 412 and 414 are placed perpendicularly to each other such that the elongated component 402 may be pivoted along a y-axis 408 and x-axis 410 of the two rotatable slotted shafts 412, 414. The laser pointing machine 404 and railing 406 may be fixed to the 'customer-side' of the transparent material 26 where a customer 14 may manipulate the elongated component 402 with a manipulating member such as, but not limited to, a hand 418. As shown in FIGS. 8 and 9 the arrangement of components 44, 402, 406, 412 and 414 allows the emitted laser beam 24 to traverse unobtrusively and collinearly along the length of the elongated component 402 and orthogonally to the rotary axes of slotted shafts 412 and 414 to intersect and traverse through the transparent material 16 to illuminate points 22 within the display-case 10.
As one skilled in the art of machining would know, the laser pointing machine 404 can be so adapted to constrain the direction of the emitted laser beam 24 to within the display case by limiting the range of angular rotation of the rotatable slotted shafts 412,414 and the length of the rail component 406 to achieve such constraints.
Other suitable examples of multi-axial rotatable mechanisms for which may be adapted for use with a laser pointing apparatus in accordance with the present invention include: US Patent US
8770768 entitled "Spherical mechanical linkage and multi-axis trackers"
published 8 July 2014 and hereby fully incorporated herein by reference; US patent US 7905463 entitled "Multiple axis gimbal employing nested spherical shells" published 15 March 2011 and hereby fully incorporated herein by reference; US patent US 4628765 entitled 'Spherical robotic wrist joint"
published 16 December 1986 and hereby fully incorporated herein by reference.
In accordance with the present embodiment a customer 14 may manipulate the laser pointing machine 404 to illuminate a point 22 within the display case 10. The customer 14 may then use this point as a reference point by which to communicate the relative location of a specific product item 30 to the CSA 12. The customer 14 may wish to acquire additional purchasable items 32 and choose to further indicate additional purchasable items to the CSA 12 by further manipulating the laser pointing machine 404 to illuminate a point within the display-case 10 which may be used as a reference point by which to communicate the relative location of a specific product item 30.
Other suitable examples of laser pointing systems in which the emission of a laser beam is confined to illuminate points within a defined area are found in the following patents, all of which are hereby fully incorporated by reference: US patent 6,761,456 B2 entitled "Laser Presentation System Using A Laser Pointer", patented on 13 July, 2004 and hereby fully incorporated herein by reference; US patent 6,910,778 B2 entitled "Presentation System Using Laser Pointer", patented on 28 June, 2005 and hereby fully incorporated herein by reference; US

Patent 20020011987 entitled "Detection of pointed position using image processing", published on 20 July, 2004 and hereby fully incorporated herein by reference.
Attention is now directed to FIG. 9 which illustrates an alternate embodiment of the present invention and in which is seen a CSA 12 in attendance of a display-case 10.
Disposed within the display-case 10 is a plurality of purchasable items 32. A transparent material or 'display-case' window 16 separates the customer 14 from physically interacting with the purchasable items 32 and makes the purchasable items 32,30 visible to the customer 14.
A capture device 70 is affixed to the display-case 10 such that objects moving to within a pre-defined spatial region of the capture device and transparent window 16 may be detected and tracked. The capture device 70 is used for range imaging and extracting volumetric data, depth information and image features, such as textures, colors and the like, of objects moving within the said predefined spatial region.
According to an example embodiment, the capture device 70 may be a range camera that collects range imaging data via any suitable range imaging technique including stereo camera triangulation, sheet of light triangulation, structured light, time-of-flight, or the like. The capture device may organize the collected depth information into a depth image. Such a depth image may be comprised of pixel whose values correspond to distances along a z-axis that extends from the depth camera along its line of sight.
As shown in FIG. 9 customer 14 is shown making a pointing gesture within the capture device's 70 field of vision. The pointing gesture is made with the customer's 14 hand and the index finger may be referred to as a pointing member 76. This pointing member 76 may be used to indicate a specific purchasable item 30 to the customer service assistant 12.
In this document a pointing gesture may also consist of any motion toward an item with a pointing member. A pointing member shall refer to any object of sufficient visibility to the naked eye and with such a spatial property that when displaced toward an item to be indicated may only be in contact with said item if brought into point contact with said item along the direction of a vector formed between points of the item and points of said pointing object which are in closest, line-of-sight proximity.
The capture device 70 may be used for object recognition, object tracking, detecting gestures, such as pointing gestures, detecting pointing members and extracting volumetric and spatial data of objects like, for instance, pointing members. The capture device 70 may also be used to estimate a virtual line 78.
A laser-pointing machine 72, communicably linked to the capture device,70, is affixed to the display-case 10 such that it is vertically superior to the purchasable items 32. The laser pointing machine 72 is shown directing a coherent beam of light 24 to impinge upon a point 22 on the surface of a specific purchasable item 30 which intersects with the virtual line 78 extending from a linear portion of the customer's pointing member 76.
Customer 14 may manipulate a pointing member 76 within the capture device's 70 field of vision so to direct the laser pointing machine 72 to illuminate points within the display case 10, such as a point 22 on the surface of a specific purchasable item 30, to indicate this item 30 to the customer service representative 12. In an alternate embodiment a customer 14 may manipulate a pointing member 76 within the capture device's 70 field of vision to direct the laser pointing machine 72 to illuminate a point such as 22 wherein such point 22 is used as a point of reference by which a customer 14 may describe the relative location of a purchasable item 30 to a CSA 12.
FIG. 10 illustrates an example machine system that may be used in accordance with the present embodiment. FIG. 10 illustrates a system that comprises a capture device 70, a laser pointing machine 72, a computer module 170, a calibration switch 185 and an 110 controller interface 180.
As shown in FIG. 10 the Laser pointing machine 72 may be comprised of a robotic component 129 and a digital camera 82 which may be communicably linked with the robotic component 129.

As illustrated, the robotic component 129 comprises a spherically shaped robotic end effector 80 to which a cylindrically shaped laser emitting diode 154 maybe concentrically fitted. It is to be understood however, that in other embodiments, other robotic end effectors of a different design and constitution may be used and laser emitting diodes of a different design and construction maybe also be used. A switch 152 may be used to control the on-off powering of the laser diode 154. Such a switch 152 may be communicably linked to a controller or processor. As illustrated two drive shaft mechanisms are arranged perpendicularly to each other and on the same plane.
Each drive shaft mechanism is comprised of an electric motor 120 that provides torque and rotational movement 190 along a shaft 122, a rotatable contact mechanism 124, a coupling 126 and an incremental encoder 128 As illustrated in FIG. 10 the 2 motors 120 provide motive power or drive to the robotic component 129. The motor 120 may be commonly referred to, by one skilled in robotics as an "actuator" or "drive" component, and maybe powered by air, water pressure or electricity. The shaft 122 transmits the rotational movement of the motor to the rotating contact 124. The rotating contact 124 is in point contact with the spherically shaped robotic end effector 80 and causes counter-rotational movement to the end effector 80 about the same axis 190, 192. The rotatable contact mechanisms 124 maybe referred to, by one skilled in robotics, as a "manipulator".
The coupling device 126 is used to join the motor shaft to the incremental rotary encoder 138.
The incremental rotary encoder 138 may be calibrated to measure the total angular rotation provided by the motor 120. The rotary encoder may be referred to, by one skilled in the art of robotics, as a "robotic sensor".
A microcontroller 130 may process the angular rotation data from the incremental rotary encoder 138, control the rate and direction of rotation of the motor 130 and calculate the values of the angular rotation of the spherically shaped robotic end effector 80. The controller 130 may also be communicably linked 160 to the computing module 170 and may receive signals process by the computing module 170 or transmit processed signals to it. The communicable link 160 maybe by means of a wireless or wired transceiver.

As illustrated in FIG. 10, the laser pointing machine 72 also comprises a camera 82 that maybe used to acquire visual data and calibrated for use in topographical analysis.
The camera 82 is at a fixed and known position from the rotational center of the spherically shaped robotic end effector 80. Camera 82 may be, for example, a fixed-focus digital camera that collects and encodes image data in RGB color mode, a range-imaging camera and the like.
The laser pointing machine 72 may also include a 3-dimensional accelerometer 110 that determines the magnitude and direction of acceleration of the laser pointing machine 72 relative to a point or frame of reference. The laser pointing machine 72 may also include 3-dimensional gyroscope 112 that determines the magnitude of angular rotation and direction such as the pitch, heading and bank of the laser pointing machine 72 relative to a frame of reference.
FIG. 10 also illustrates an example embodiment of the capture device 70. As shown in FIG. 10 the capture device 70 may include an image camera component 106. The image camera component may be a range camera or depth camera that captures the depth image of a scene according to a number of different techniques. The depth image may include a two-dimensional (2-D) image whose pixel values correspond to the length of or distance extending along the image camera's line of sight to points in the scene. The depth value may be a length or distance measured in centimeters, meters, or the like of points of objects in the scene captured.
For example in structured light 3-D scanning, the illuminating unit 104 may project a striped or checkered pattern unto a scene using, for example, infrared or visible electromagnetic radiation.
The emitted striped and patterned electromagnetic radiation is reflected off the surfaces of objects in the scene and appears distorted or displaced from the perspective of a 3-D camera 105 or RGB camera 107 separated by a known distance from the illuminating unit 104. The distorted patterns and displaced lines are recorded by the 3-D camera 105 or RGB camera 107 and this data may be analyzed by a processing unit, such as a micro-controller 108 or microprocessor, and converted into 3-Dimensional coordinate data.
In another embodiment, a stereo camera system, not here shown, consisting of at least 2 cameras separated and at fixed and known distances from each other may be configured to determine the 3-Dimensional coordinates of points in a scene using stereo triangulation.
Examples of stereo-triangulation methods include mid-point method and direct linear transformation.
In yet another embodiment the standard time-of-flight range imaging technique may be used to obtain depth information from a captured scene. In such an example an illuminating component 104 may emit a short electromagnetic pulse to illuminate a scene and the camera component such as the 3-D camera 105 or RGB camera 106 may record the intensity of the reflected electromagnetic radiation which is further processed by a processing unit such as a microcontroller 108 to determine the physical distances from the capture device 70 to objects in the scene.
The capture device 70 may also include a 3-Dimensional (3-D) accelerometer 112 and 3-D
gyroscope 110. These components can be used to determine orientation and change is displacement relative to some rest frame. Changes in acceleration and orientation of the capture device 70 and the time at which these changes have taken place can be processed by the microcontroller 108 to determine the displacement and orientation of the capture device 70 relative to some rest frame.
As illustrated in FIG. 10 the capture device 70 may also include a microcontroller unit 108 that may store and execute instructions and store image data captured by the image components 106.
The microcontroller 108 may be comprised of a processor and memory component.
The microcontroller 108 may also be configured for object and target tracking, feature extraction, such as texture and blob analysis, and target recognition.
As illustrated in FIG.10 the capture device 70 and laser pointing machine 72 maybe communicably linked 160, 162 to a computing centre 170. The communicable link 160, 162 maybe by means of a wireless or wired transceiver such as an ethernet cable, wifi, bluetooth or the like. The computing center 170 may include a central processing unit (cpu), 176, a memory component, 172, a clock 174 and an I/O or "in-out" interface 177. The computing centre 170 may receive and exchange data and processed information with the capture device 70 and the laser pointing machine 72 via communicable links 160,162. Examples of such received and exchanged data include angular orientation data, displacement data, image data and depth image data. The computing centre 170 may also be used for more complex image processing operations such as advanced gesture recognition, multiple object tracking and advanced feature extraction. The computer center 170 may also be used to compute the angular orientation and and 3-Dimensional coordinates of the capture device 70 and laser pointing machine 72 from data retrieved from the accelerometer 112 and gyroscope 110.
The computing center may also be comprised of an 110 interface 177 to which a controller 180 may be communicably linked via, for instance, a USB input or the like.
Controller 180 may also be communicably linked 164 to the computing center 170. The communicable link 164 may be by means of a wireless or wired transceiver such as an ethernet cable, wifi, bluetooth or the like.
In one example embodiment the control device 180 may interface with the computing center 170 and be configured to transmit and receive data. The control device may transmit data to control the laser pointing machine 72 and, for example, manipulate the laser pointing robot 169 and control the on-off powering of the laser emitting diode 154. An example of such a controller 180 may be a pointing device such as a computer mouse or touch screen device that detects two-dimensional motion across a surface. This motion may be translated by the computing center 170 or laser pointing robot 169 into, for example, rotational movement of the end effector 80.
In yet another example embodiment, not here shown, the controlling device 180 may be a radio-control transmitter control column with multi-axis movement, such as a joystick. Each control column movement along an axis of movement may be translated by the computing center 170 or laser pointing robot, into, for example, rotational movement of the effector 80. Such a controller may also be configured to control the on-off powering of the laser emitting diode 154.
Yet another example embodiment of a controlling device 180 is a touch-screen device. Suitable examples of touch screen devices include, but are not limited to, smartphones, tablet computers and the like. Touch gestures detected by the touch screen device may be translated by computing center 170 or laser pointing robot 169, into, for example, rotational movement of the end effector 80 or like robotic devices. Furthermore, image data collected by the camera device 82 may be transmitted to said touch screen device via, for example, a communicable link 164 or I/O interface 177. This image data may be displayed on the touch-screen device and may be, for example, RGB images of a scene captured by the camera device 82.
In accordance with the example embodiment, FIG. 10 also illustrates a calibration switch 185 which may also be included to start the set up of a frame of reference for which the relative locations in 3-Dimensions and angular orientations of the capture device 70 and laser pointing machine 72 are measured. The calibration switch 185 may be a coded signal transmitted to computing center 170 via a communication link 166. Another example of the calibration switch may be, for example a hardwired electronic switch communicably linked to the computing center 170. An example of such an electronic switch may be a button-switch, pressure switch or the like which may be activated upon compression or decompression. hi such an example embodiment the laser pointing machine 72 and capture device 70 may be in a close proximity or contact with the switch to accomplish compression or decompression upon separation or increased displacement between the two devices.
FIG. 11 illustrates a sectional side-view of the current embodiment that illustrates a pointing object 76 and the spatial region 90 of the pointing object 76 that may be used to calculate a virtual line 78. The Spatial region 90 may be a group of pixelized depth values acquired by the capture device 70, from which a linear line 78 is approximated from the central moment of these depth values.
The virtual line 78 may also be calculated as the line of best fit through a series of centroids per unit length of the pointing object 76. Centroids of the pointing object 76 that are not within a certain threshold value may be discarded from the calculation of the line of best fit. Line fitting or 'linear line approximations" may be known to one skilled in the art as Linear Regression.
Techniques for the approximation of line of best fit include the "Least-Square Method", "Maximum Likelihood estimation", "Bayesian Linear Regression" and the like.

The system of the current embodiment, as illustrated in FIG. 9 -11 and discussed in the preceding paragraphs, may require a calibration process. Such a calibration process may determine the angular orientations and locations given in 3-Dimensional coordinates, of the individual components such as the capture device 70 and laser pointing machine 72, relative to some reference point. An example of such a calibration process or method is illustrated in FIG.12.
FIG. 12 illustrates an example flow method by which the coordinates and orientations of the capture device 70 and laser pointing machine 72 relative to one or more frames of reference or points of reference may be determined. The frame of reference or point of reference may be, for example, the laser pointing machine 72 or some point therein, the capture device 70 or some point therein, the computing center 170 or some point therein. However, it is known to one skilled in the art of Physics or mathematics, that spatial coordinates and angular orientation values relative to one frame of reference can be calculated or transformed with respect to another frame of reference or point of reference.
As illustrated in FIG. 12, at step 201, the capture device 70 and laser pointing machine 72 are at a known orientation and distance from each other. At step 203, the calibration switch 185 is triggered, for example, upon displacement of the capture device 70 and, or, laser pointing machine 72 from the point of known orientation and distance described in step 201. At step 205 data acquired from the accelerometer 112 and gyroscope 110 and the time the data was acquired is recorded. This data may be transmitted to and recorded by, for example, the computing center 170. At step 207 and 209 the coordinates and orientation of the laser pointing machine 72 and capture device 70 relative to the frame of reference is calculated using the data recorded in step 205. As illustrated in step 216 and 222, any change, or, further change, in the angular orientation and, or, position of the capture device and, or, laser pointing machine relative to the frame of reference or point of reference produces additional output data from the accelerometers and gyroscopes as shown in steps 213, 215, 218 and 220. This data and the time of the output of this data 205 is recorded and processed, for instance, by the computing center, 170, to calculate the new, modified coordinates and, or, orientation values of the capture device and, or, laser pointing machine as shown in steps 207 and 209. At step 211, the calculated coordinates and orientation values of the laser pointing machine 72 and capture device 70 is stored. This data may be stored, for example, within a read/write memory device such as 172.
FIG. 15 illustrates a display -case 10 wherein a virtual bounded region 94 lies on a 3-Dimensional surface 92, such a plane. As illustrated the 3-Dimensional surface 92 may be a plane that is tangential and, or, parallel to the surface within the display-case 10 on which purchasable items 32 may be placed for viewing. As shown in FIG. 15 the laser pointing machine 72 is configured to illuminate points within the virtual bounded region 94. The virtual bounded region may be of, for example, quadrilateral dimensions. The system of the current embodiment , as illustrated in FIG. 9 -12 and discussed in the preceding paragraphs, may also be configured such that the emitted laser beam 24 may only illuminate points within a specific region, such as, for example, the display case 10. Furthermore the laser pointing machine 72 may also be configured such that the laser emitting diode may illuminate points where the virtual line 78 intersects with a 3-Dimensional surface such as, for example, the 3-Dimensonal surface 92. Furthermore, the laser pointing machine 72 may also be configured such that the laser emitting diode may illuminate points only where the virtual line 78 intersects with the virtual bounded region 94. FIG. 13 illustrates an example flow method in accordance with the current embodiment, as illustrated in FIGS. 10 -12 and FIG. 15 wherein the laser pointing machine 72 may be configured to only illuminate points within a virtual bounded region 94.
Turning now to FIG. 13 we see that at step 230 the spherical end effector 80 is at a known angular orientation. This angular orientation may be, for example, some known angular displacement from the fixed zenith direction about one or more axes. As shown in step 232 a calibration mode may be started. This calibration mode may be started by, for example, the computing center 170 executing programmed code stored in the memory device 170. At step 234 there is a 'For Loop' consisting of at least four repetitions of steps 254. Steps 254 consists of 3 steps 236,238 and 240. A step 236 the laser pointing machine 72 is manipulated to illuminate a point within the display-case 70 that corresponds to a vertex of the virtual bounding region 94 or bounding quadrilateral. This illuminated point within the display case may be, for example, a point on the product placement surface. The product placement surface is the surface or surfaces of a display-case 70 wherein purchasable items are placed for customer viewing. The laser pointing machine 72 may be manipulated according to step 236 with an external controller such as 180. At step 238 the coordinates of the illuminated point of step 236 is calculated in 3-Dimensional Coordinates. One example, in accordance with step 238, by which the coordinates of the illuminated point may be calculated in 3-Dimensional coordinates is as follows:
1. Capture a grayscale image or an average of images before the illumination of the point of step 236, with, for example, camera 82. For the purposes of this example camera 82 may be a camera calibrated according to a camera resectioning process.
2. Capture a grayscale image at time of the illumination of point of step 236, with, for example, camera 82 3. Calculate the current angular orientation of spherical end effector 80 with data obtained from the incremental encoder 128 at time of illumination of point of step 236 4. Obtain an absolute difference image of the images of steps 1 and 2 above.
5. Binarize the absolute differenced image of step 4 above to isolate the pixels that correspond to the illuminated point of step 236 within the binarized image.
The technique of binarizing may also be referred to as 'gray image thresholding'.
6. Calculate the pixel coordinates of centroid of illuminated point from step 5, above 7. Convert centroid pixel coordinates into angular values. The angular values may be, for example, the visual angles or visual degrees between the illuminated point of step 236 and the focal point of a resectioned camera such as 82.
8. Calculate 3-Dimensional coordinates of the illuminated point of step 236 using the angular values from step 7, above, and angular orientation values of step 3, above.
At step 240 the 3-Dimensional coordinates of the point illuminated in step 236 is stored as bounding vertex point in a memory unit such as 172. At step 242 the process of indicating the vertices of the virtual bounding region 94 stops. At step 244 two vertices of the virtual bounding region are paired by proximate vertical displacements. In the current example, the laser pointing machine 72 is vertically superior to the surface in the display-case 10 on which the purchasable products are placed for customer viewing. In this example the term "points with proximate vertical displacements" is to be understood as points that share the closest vertical displacement from the laser pointing machine 72. At step 246 the midpoint of the paired points of step 244 is calculated in 3-Dimensional coordinates. At step 248 an equation of a plane 92 is calculated using the midpoint of step 246 and two unpaired bounding points of steps 234-242. At step 250 the equation of the plane 92 of step 249 is stored in a memory unit such as 172. At step 252 a virtual quadrilateral or the virtual bounding region 94 is approximated from the four points of steps 234-242. The virtual quadrilateral or bounding region 94 may be collinear with the surface of 92. At step 254 the equation representing the bounding quadrilateral or bounding region 94 is stored in a memory unit such as 172. Although not illustrated in the drawings, it can be inferred by one skilled in the art of topography or topology analysis, that a 3-Dimensional equation modeling the surface within the bounding region 94 may be calculated by acquiring a population of points within the bounding region 94 according the steps 236-240. A 3-Dimensional equation of a surface may then be fitted to this population of points using curve fitting techniques.
Furthermore, the coordinates of this population of points may be stored in a memory unit such as 172. Other examples of systems and methods by which such a surface such as 92 may be measure and mapped include: US patent US 6915243 entitled "Surface topology and geometry reconstruction from wire-frame models, published 25 August, 2000 and fully incorporated herein by reference; US patent US 6809803 entitled "Surface topology inspection", published 26 October, 2004 and fully incorporated herein by reference; US patent US 5311286 entitled "Apparatus and method for optically measuring a surface", published 10 May, 1994 and fully incorporated herein by reference.
Turning now to FIG. 14 we see an example flow method in accordance with the current embodiment by which a pointing object, such as 76, may be used to manipulate a laser pointing machine such as 72 to indicate a point within a display-case 10 by illumination with an emitted laser beam 24. At step 300 an object moves into the field of vision of the capture device 70 and within certain range parameters of the capture device 70. In one embodiment such range parameters may be a distance along the line of sight of the capture device 70 greater or shorter than certain distances. In another embodiment, the capture device 70 may be placed tangentially to the transparent material 16 and a range parameter may exclude points along the axis perpendicular to the transparent material 16 and vertically inferior to the transparent material 16 or capture device 70. Additionally, the range parameter may exclude points along the axis perpendicular to the transparent material 16 and which exceed a certain vertically superior distance from the transparent material 16 or capture device 70.

Step 300 may also include image segmentation of the depth image or depth images captured by the capture device 70 at step 300. Suitable examples of methods of image segmentation include:
US patent US 80009871 B2, entitled "Method and system to segment depth images and to detect shapes in three-dimensionally acquired data", published 30 August, 2011 and fully incorporated herein by reference; US patent US 8401225 B2, entitled "Moving object segmentation using depth images", published 19 March 2013 and fully incorporated herein by reference.
At step 302 the spatial measurements of the detected object or objects of step 300 are analyzed and calculated. Examples of such spatial measurements include, but are not limited to, the volumetric data of the objects, the volume of the object per unit distance along an axis or multiple axes, the surface gradient of object, and the like.
At step 304 the spatial measurements of the detected object or objects of step 300 are compared against a database of spatial parameter constraints. The spatial parameter constraints may contain data such as, but not limited to, a minimum and, or, maximum volume and a minimum and, or, maximum rate of change of volume per unit distance along an axis or multiple axes. Step 304 may also include an object classification of the detected object or objects of step 300 which may be compared against a database of objects. Suitable examples of such object classification include: US patent US 7646922 B2 entitled "Object classification in video images", published 12 January 2010 and fully incorporated herein by reference; US patent US 8934709 B2 entitled "Dynamic object classification", published 12 January 2015 and fully incorporated herein by reference.
Detected objects with spatial features that do not fall within the constraints of step 304 and remain with the range parameters of step 300 are passed onto an object tracking step 328 and are continually analyzed by the capture device according to steps 302 and 304.
Suitable examples of object tracking systems and methods include: US patent US 7907750 B2, entitled "System and method for autonomous object tracking", published 15 March 2011, and hereby fully incorporated herein by reference; US patent US 7590262 B2, entitled "Visual tracking using depth data", published 15 September 2009, and hereby fully incorporated herein by reference;

US patent US 20120327125, entitled "System and method for close-range movement tracking"
published 27 December 2012, and hereby fully incorporated herein by reference;
US patent 201200069143, entitled "Object tracking and highlighting in stereoscopic images", published 22 March 2012 and hereby fully incorporated herein by reference; US patent US
20120309532, entitled "System for finger recognition and tracking", published 6 December 2012, and hereby fully incorporated herein by reference.
Detected objects with spatial features that do fall within the constraints of step 304 are classified as pointing objects and passed unto step 306. At step 306, 3-Dimensional line equations, curved line equations or vectors are approximated from the pointing objects. For example, a curve may be fitted or interpolated from a group of centroids calculated per unit length of the pointing object 76.
At step 308 a derivative of the curved line equation of step 306 is calculated.
At step 310 the maxima and minima of the equation of step 308 is extracted and analyzed in step 312. The maxima and minima of the equation of step 308 may, for example, indicate the spatial points of the pointing object at which the greatest change in volume per unit length of the pointing object occurs, or the point at which significant bending or curvature of the detected pointing object occurs. At step 312 the maxima and minima values acquired at step 310 are filtered according to their proximity to the transparent medium.
At step 314 a line of constant slope or a vector is approximated from regions within the maxima and minima of step 312 wherein the rate of change of according to step 308 is closest to zero.
This approximated line equation or vector is the virtual line 78 of the detected pointing object.
At step 316 the line or vector or virtual line 78 of step 314 is used to calculate a point of intersection with a plane or curved surface such as 92.
At step 318 the calculated point of intersection of step 316 may be analyzed to determine if it is a point within the virtual bounding region or bounding quadrilateral, 94. If the point of intersection of step 316 is not a point within the bounding region 94, the detected object is tracked according to step 328. A point of intersection lying within the bounding region 94 or bounding quadrilateral is used in step 320.
Step 320 consists of a sequence of three steps 322, 324 and 326. At step 322 the current orientation values of the spherically shaped end effect 82 is acquired from the laser pointing machine 72. Orientation values of step 322 may correspond to a direction vector and line that is coincident and collinear with a line or vector along which a beam of light emitted from the laser emitting diode 44 may traverse. This line or vector intersects with the plane 92 or surface at a point that may be displaced from the point of intersection of step 318.
At step 324 new rotation values or angular orientation values of the robotic end effector 80 are computed such that the laser pointing machine 72 may orient the robotic end effector 80 and the laser emitting diode 154 to illuminate the point of intersection of step 318.
At step 326 the point of intersection of step 318 is illuminated by the laser emitting diode 154.
After step 326 we return to step 328 where the detected pointing object of step 320 is further tracked.
We now turn to FIGS. 16a-c, which illustrate a method, in accordance with the current embodiment, by which a spherically shaped robotic end effector 80 configured to fit and retain a laser emitting diode 154, may be manipulated to illuminate a point 342 on a plane or surface such as 92. The point 342 may, for example, be the point of intersection of the virtual line 78 with a plane of surface such as 92.
FIG. 16a-c illustrates a spherically shaped robotic end effector 80 adapted to fit and retain a laser emitting diode 154. The origin of the coordinate system may be the rotational center of the spherically shaped robotic end effector 80. As illustrated in FIGS. 16a-c a plane or curved surface such as 92 and the point 342 maybe vertically inferior to the rotational center of the spherically shaped robotic end effector 80. The robotic end effector 80 may also be comprised of a an aperture 340 through which a laser beam emitted from the laser emitting diode 154 may traverse unobtrusively to the external environment. FIGS. 16a-c also illustrate rotational contacts 124 which may be in point contact with the robotic end effector 80 and which rotate about a z-axis and x-axis. In accordance with the current embodiment the rotational contacts 124 may be cylindrically shaped with a radius of Rrotator= The spherically shaped robotic end effector may have a radius of Rsphere=
As illustrated in FIG.16a, the spherically shaped robotic end effector may be oriented to some angular orientation, such as polar angles ac and 13e. a, may be the angular displacement about the x-axis from the fixed zenith direction and f3, may be the angular displacement about the z-axis from the fixed zenith direction such as in a spherical coordinate system. The point to be illuminated 342 may have 3-Dimensional coordinates with respect to the origin:
Xpoinb, Ypoint, Zpoint and angular displacements ap from the fixed zenith direction about the x-axis and 13p from the fixed zenith direction about the z-axis, where:
f ap ¨ Lau k(Zpoint Ypoint), and, Pp ¨ sin-i kXpoint Ypotnt) As illustrated in FIG.16b, a rotational contact 124 rotates about the x-axis 346 which in turn generates a counter-rotational movement of the spherical end effector. For example, the contact rotator 124 rotating about the x-axis may complete an angular rotation of a, about the x-axis where:
R sphere ¨ ____________________________________________ x (ap-ac) R rotator As illustrated in FIG.16c, a rotational contact 124 rotates about the z-axis 348 which in turn generates a counter-rotational movement of the spherical end effector. For example, the contact rotator 124 rotating about the z-axis may complete an angular rotation of 13, about the z-axis where:
R sphere Or R rotator X (Op - 13) As FIG.16c illustrates the spherically shaped robotic end effector and laser emitting diode have been oriented to illuminate the point 342 which lies on the plane or surface 92. The laser emitting diode 154 illuminates the point 342 by emitting a laser beam 344 which traverses through the aperture 340 along a displacement vector from the rotation center of the spherically shaped robotic end effector to the point 342.
The system and methods as described above and as illustrated in FIGS. 10 -16 is an example embodiment and, as understood, is not meant to limit the scope of the invention. For example, the system and methods as illustrated in FIGS. 10-16 may be adapted for use with a multi-shelved display-case 500. FIG. 17 is an example embodiment wherein the system as described and illustrated in FIGS. 10-16 may be modified and configured for used with a multi-shelved display-case.
FIG. 17 illustrates a display-case 500 with multiple shelves 504 disposed within the display-case 500. Each shelf 504 may hold a plurality of purchasable items 32 which are separated from the environment external to the display-case 500 and made visible to customers by a transparent medium 16. FIG.17 also illustrates multiple capture devices 70 which may be affixed, for example, tangentially to the transparent medium 16. As shown in FIG.17 the laser pointing machine 72 may be modified and configured for linear displacement along a rail component 504.
The capture devices 70 and laser pointing machine 72 are calibrated such that a pointing object such as 76 may manipulate the laser pointing machine 72 to direct a coherent beam of light 24 to illuminate a point 22 on a purchasable item 30. For example, the flow method according to FIG.
13 maybe repeated for every shelf 504 within the display-case 500. The flow method according to FIG. 14 may be modified such that step 320 includes an additional step of acquiring the current linear displacement of the laser pointing machine relative to a reference point, calculating the necessary amount of displacement along the railing 504 required by the laser pointing machine 72 and displacing the laser pointing machine 72 along the rail component 504 such that it may illuminate the point being pointed to by the pointing object.
The methods and system described with reference to details illustrated by the drawings represented example embodiments are not intended to limit the scope of the invention and many alternate embodiments may become obvious to those skilled in the art. For example, the laser pointing machine 72 as described herein is a robotic device which has been adapted to fit, retain and manipulate a laser emitting device to illuminate points within a pre-defined area. The range and type of motion exhibited by a robot is commonly referred to by those skilled in the art as "degrees of motion" where each type of motion along an axis or about of axis is a degree of freedom. A suitable example of a robotic laser pointing machine which may be adapted for use with the current invention is: US patent US 6374158 entitled "Robotic Laser Pointer" published 16 April, 2002 and fully incorporated herein by reference. The system may also be configured such that multiple objects may be tracked by the capture device 70 and multiple points within a display-case may be simultaneously indicated and illuminated by a laser pointing mechanism.
For example, it may become apparent to one skilled in that art that a laser pointing machine may be configured to illuminate a series points within the display case such as to appear as a line whose length is proportional to the distance of separation between the tips of 2 pointing objects detected by a capture device. Additionally the system may also be configured such that movement gestures, such as swiping with a hand or finger, that are detected by the capture device are translated into movement vectors whose displacement magnitude and direction are translated into a proportional displacement in the direction of the vector of the emitted laser beam. Furthermore, a display-case as illustrated and described in the description is not meant to limit the interpretation of a display-case. A display-case includes any display arrangement of items whereby said items are viewable to a customer through a transparent material and said transparent material separates said customer from said items and wherein such an arrangement prevents said customer from physically interacting with said items. Even further, the various embodiments of the present invention as described refer to the use of a laser and laser pointing mechanism to illuminate a region of or a specific item, however, it will become apparent to one skilled in the art that a beam of non-coherent light may be focused, using optical techniques, to obtain a similar illuminating effect as the lasers and laser pointing mechanisms described. The scope of the present invention is defined by the claims in the appended claims.

Claims

What is claimed is:

1. A method of indicating a selected, purchasable item situated within a display-case wherein said display-case is located within a retail location and at least one controllable laser pointing apparatus is connected to a connecting component and said connecting component is attached to an anchoring component and said anchoring component is secured to said retail location comprising the steps of:
manipulating said at least one controllable laser pointing apparatus to direct a coherent beam of light to illuminate at least one point within the display's retail location and said illuminated point or points being visible to at least two persons and said illuminated point or points being used by at least one of said at least two individuals as a point of reference for the purpose of one or a combination of:
a. Describing the relative location of at least one said purchasable item b. Communicating the location of said at least one purchasable item relative to said illuminated point or points and, said purchasable item or items being located and discerned by said at least two individuals

2. A method according to claim 1 wherein said illuminated point or points is located within said display-case

3. A method according to claims 1-2 wherein said illuminated point or points comprise at least part of said located and discerned purchasable item or items.

4. A method according to claim 1-3 wherein said laser pointing apparatus directs a coherent beam of light to traverse across at least one transparent material to impinge upon and illuminate at least one of said points within the display's retail location

5. A laser pointing apparatus according to claims 1-4 wherein said laser pointing apparatus comprises:
a laser emitting diode, a pressure activated switch configured to control the on-off powering of said laser emitting diode.

6. A pressure switch according to claim 5 wherein said pressure switch is configured to switch said laser emitting diode to an on-power state and react to a sufficient resultant force of a force generated by said laser pointing apparatus upon contact with said transparent material and where the direction of the vector of said generated force is collinear with the direction of said coherent beam of light emitted by said laser emitting diode.

7. A pressure switch according to claim 6 adapted to said laser pointing apparatus such that the angle of intersection of said emitted coherent beam of light with said transparent material is less than the critical angle of reflection of said transparent material and the wavelength of light of said emitted coherent beam of light.

8. A system and method for indicating a selected, purchasable item situated within a display-case wherein, said display-case is located within a retail location and said system comprising:
a photoemitting component a photodector component sensitive to light signals from said photoemitting component a laser pointing apparatus comprising a laser emitting diode communicatively coupled to a microcontroller and said microcontroller configured to:
store data, process data, control the on-off powering of said laser emitting diode in response to signals received from said photodetector component, said method comprising the steps of:
manipulating said laser pointing apparatus to direct a coherent beam of light to illuminate at least one point within the display's retail location and, said illuminated point or points being visible to at least two persons and said illuminated point or points being used by at least one of said at least two individuals as a point of reference for the purpose of one or a combination of:
a) Describing the relative location of at least one said purchasable item b) Communicating the location of said at least one purchasable item relative to said illuminated point or points said purchasable item or items being located and discerned by said at least two individuals

9. A method according to claim 8 also comprising the step of manipulating said laser pointing apparatus such that there is a first exposure of said photodetector to said photoemitter and said first exposure causes said laser emitting diode to switch to a power-on state.

10. A laser pointing apparatus according to claim 8-9 wherein said laser pointing apparatus also comprises said photodetector or said photoemitter.

11. A laser pointing apparatus according to claim 8-10 wherein said laser pointing apparatus further comprises a gyroscope communicatively coupled to said microcontroller and said gyroscope being used to obtain angular orientations of said laser pointing apparatus

12. A laser pointing apparatus according to claim 8-11wherein said laser pointing apparatus further comprises an accelerometer communicatively coupled to said microcontroller and said accelerometer being used to determine spatial displacement of said laser pointing apparatus.

13. A laser pointing apparatus according to claims 8-12 wherein said microcontroller is additionally configured to:
compute the current spatial coordinates of said laser pointing apparatus relative to a point of reference, compute the current angular orientation of said laser pointing apparatus relative to a frame of reference, store computed spatial coordinates, store computed angular orientations, interpolate a range of angular orientation values with respect to computed current spatial coordinates using said stored spatial coordinates and said stored computed angular orientations, compare said current angular orientations and spatial coordinates to said interpolated angular orientation values and spatial coordinates, control the on-off powering of said laser emitting diode in response said comparison of current angular orientations and spatial coordinates with said interpolated angular orientation values and spatial coordinates.

14. A laser pointing apparatus according to Claims 8-13 wherein said laser pointing apparatus is connected by a connecting component to an anchoring component, wherein, said anchoring component is further secured to said retail location.

15)A method according to Claim 8-14, wherein, said coherent beam traverses across at least one solid transparent material to impinge upon and illuminate at least one of said points within the display's retail location.

16) A system and method for indicating a selected, purchasable item situated within a display-case wherein, said display-case is located within a retail location and said system comprising:
a laser pointing mechanism with at least 2 degrees of freedom adapted to fit and retain a laser emitting diode such that a laser beam emitted from said laser emitting diode may be directed to traverse unobtrusively in a direction resultant to the motion of said at least 2 degrees of freedom of said mechanism said method comprising the steps of:
manipulating said laser pointing mechanism to direct a coherent beam of light to illuminate at least one point within the display's retail location and, said illuminated point or points being visible to at least two persons and, said illuminated point or points being used by at least one of said at least two individuals as a point of reference for the purpose of one or a combination of:
a. Describing the relative location of at least one said purchasable item b. Communicating the location of said at least one purchasable item relative to said illuminated point or points said purchasable item or items being located and discerned by said at least two individuals

17. A method of indicating a selected, purchasable item situated within a display-case, wherein, said display-case is located within a retail location comprising the steps of:
capturing a scene within said retail location, tracking objects within a range parameter of said captured scene, detecting the occurrence of pointing gesture with a pointing member within said range parameter of said captured scene, calculating a virtual line in 3-dimensions formed by said pointing member, calculating the coordinates of a point of intersection of said virtual line with a surface located within said retail location, determining if said calculated coordinates is within a region of said surface and, manipulating a laser-pointing machine to direct a coherent beam of light to illuminate said point of intersection within said region of said surface located within said retail location and said illuminated point or points being visible to at least two persons and said illuminated point or points being used by at least one of said at least two individuals as a point of reference for the purpose of one or a combination of:
a) Describing the relative location of at least one said purchasable item b) Communicating the location of said at least one purchasable item relative to said illuminated point or points said purchasable item or items being located and discerned by said at least two individuals

18. A device for capturing a scene according to claim 17 that includes a range imaging camera and an RGB sensor.

19. A system for indicating a selected, purchasable item situated within a display-case, wherein, said display-case is located within a retail location comprising:
at least one capture device at a known distance and orientation comprising a range imaging camera and an RGB sensor at least one laser pointing machine at a known distance and orientation comprising:
at least one laser emitting diode at least one robotic component with at least 2 degrees of freedom configured to control the on-off powering of said laser emitting diode and manipulate the motion of said laser emitting diode a computing center communicably linked to said range imaging camera and said laser pointing machine configured for exchanging, storing and processing data and:
tracking multiple objects in a scene captured by said capture device, recognizing gestures captured by said capture device, extracting features of objects captured by said capture device, exchange data with a controller via an I/O interface or communicable link, calculating and storing the relative positions and angular orientations of said capture device and said laser pointing machine, calculating and storing topographical data, manipulating and controlling said robotic component to illuminate points within said retail location

20. A laser pointing machine of claim 19 wherein said laser pointing machine is also configured to perform topographical analysis of a surface within a display-case.

21. A laser pointing machine of claim 20 also comprising an image camera component and said image camera component is at a known distance and orientation from said robotic component.

22. A capture device according to claim 19 wherein said capture device includes a gyroscope and accelerometer communicably linked to said computing center.

23. A laser pointing machine according to claims 19-20 wherein said laser pointing machine includes a gyroscope and accelerometer communicably linked to said computing center.

24. A computing center of claim 19 additionally configured to:
calculate and store the relative positions and orientation of said capture device and laser pointing machine using data retrieved from said gyroscopes and accelerometers, transfer image data captured by a camera component of claim 21 to a controller via an I/O interface or communicable link, store instructions for the translation of data received from said controller to manipulate and control said robotic component , constrain said laser pointing machine to illuminate points within a region of a surface represented by stored topographical data wherein said points are points of intersection of said virtual lines with said surface.