GB2508840A

GB2508840A - An apparatus and method for tracking pointer devices within a scene

Info

Publication number: GB2508840A
Application number: GB1222335.0A
Authority: GB
Inventors: Ahmed Kharrufa
Original assignee: MODULAR SOFTWARE Ltd T A REFLECTIVE THINKING
Current assignee: MODULAR SOFTWARE Ltd T A REFLECTIVE THINKING
Priority date: 2012-12-12
Filing date: 2012-12-12
Publication date: 2014-06-18
Anticipated expiration: 2032-12-12
Also published as: GB2508840B; GB201222335D0; WO2014091225A1

Abstract

An apparatus for tracking the movement of a plurality of pointers (e.g. pens) is able to identify and differentiate between the pens. The apparatus includes an image capture device (e.g. IR camera 20) captures images of a scene and a transmitter for transmitting different trigger signals. Pointer devices for indicating (12, 14, 16) their location include a light emitting device (e.g. an infra-red light emitting device) and a receiver for receiving trigger signals (e.g. pen 1 on, pen 1 off etc). The light is captured allowing the pointer device to be tracked. The scene may include a projected or displayed image.

Description

METHOD AND APPARATUS FOR TRACKING THE MOVEMENT OF A

PLURALITY OF POINTER DEVICES WITHIN A SCENE

The present invention relates to a method and apparatus for tracking the movement of a plurality of pointer devices within a scene and relates particularly, but not exclusively, to a method and apparatus for utilising any existing table or wall display and using it as a multi-user digital table top or multi-user interactive board.

The use of interactive boards and table tops that use pens to allow a user to appear to write on a board and to interact with a board or screen are well known. One such system utilises a projector which displays images on to a board that contains apparatus to detect when a "pen" that is designed to work with the board is in close proximity to the board or touching it. The board produces data indicating where the pen is used on the board which can be fed back to a computer device controlling the projector. Some such systems allow for the use of multiple pens at the same time on a single board, which is extremely useful for collaborative work. However, the cost of the board is very high and it tends to be used in a fixed location, for example, attached to a wall or forming part of a table top.

In an alternative system, an infra-red camera is positioned to view a board or table top, on to which a projector is projecting an image. A pen that produces an infra-red light source when pressed into engagement with the board or table top and the infra-red camera is used to produce images that track the location of the pen. This apparatus has the advantage that it is significantly lower in cost than the board detectors mentioned above. However, if more than one pen is used the light emitted from the multiple pens is the same and cannot be distinguished by the processor processing the data from the infra-red camera. This significantly limits the uses to which such a system can be put.

In a further example of the prior art, pens are fitted with digital cameras which are able to read a matrix of faintly printed dots on a board, table top or sheet of paper and use these dots to determine a location which is wirelessly communicated to a processing device. This system allows the use of multiple pens. However, each pen is expensive and can only work on a surface that is printed with the special dot pattern.

Preferred embodiments of the present invention seek to overcome the above described disadvantages of the prior art.

According to an aspect of the present invention, there is provided an apparatus for tracking the movement of a plurality of pointer devices within a scene, the apparatus comprising: -at least one image capture device for capturing a plurality of images of a scene and producing an image signal including image data representing light at a predetermined frequency; at least one transmitter device for repeatedly transmitting a plurality of different trigger signals; and a plurality of pointer devices for indicating a location of said pointer devices in said scene, each pointer device including at least one light emitting device for emitting light substantially at said predetermined frequency and at least one receiver f or receiving said trigger signals, wherein plurality of said trigger signals are detected by respective said pointer devices causing said light emitter in the respective pointer device to produce said light that is captured by said image capture device thereby linking a plurality of said images with said pointer device and allowing the pointer devices to be tracked as they are moved within said scene.

By providing a system that utilises a processor connected to an infra-red camera and infra-red light emitting pens, together with a transmitter also connected to the processor and a receiver in each pen, the following advantages are provided. Multiple pens can be used and uniquely identified in a system that does not require expensive apparatus such as special boards or expensive pens.

Furthermore, any surface onto which an image can be projected can be used without the need for expensive boards or for specifically printed sheet materials. It is also the case that by only having the infra-red light on for a fraction of a second for each pen the battery consumption of each pen is reduced. Although additional circuitry and power consumption is required for the receiving device, which is permanently on whilst the pen in engaged with a surface via a micro-switch in the tip of the pen, the current required for receiving is less than that required to produce the infra-red light.

The apparatus may further comprise at least one processor for producing a transmitter control signal and an image capture device control signal thereby ensuring that said images produced by said image capture device are synchronised with the light produced by each respective pointer device in response to said transmitter control signal instructing said transmitter to produce said trigger signal.

In a preferred embodiment, the processor is directly connected to said image capture device.

Although a standard computer can be used to control the transmitter signal and synchronise the image capture device with the pens, by having a separate processor to control this, the advantage is provided that less communication with the computer, which is processing the data and displaying things via the projector or screen, is required.

Furthermore the direct connection between the processor and infra-red camera improves the synchronisation between camera and pens thereby allowing a large number of pens to be used simultaneously.

In another preferred embodiment, the predetermined frequency is infra-red light.

In a further preferred embodiment, the image capture device is an infra-red camera.

In a preferred embodiment, the transmitter device is a short range radio transmitter.

The pointer device may comprise a body formed substantially in the shape of a pen and including circuitry for processing said received trigger signal and controlling said light emitting device in response to said trigger signal.

The pointer device described above is a low cost item and as a result, provides significant advantages where collaborative working is required using multiple pens.

According to another aspect of the present invention, there is provided a method of tracking the movement of a plurality of pointer devices within a scene, comprising:-capturing a plurality of images of a scene with at least one image capture device and producing an image signal including image data representing light at a predetermined frequency; repeatedly transmitting from at least one transmitter device a plurality of different trigger signals; emitting light at substantially said predetermined frequency from at least one light emitting device in a respective pointer device being one of a plurality of devices in response to receiving a respective said trigger signal; and processing said image data to link a plurality of said images with said pointer device and allowing the pointer devices to be tracked as they are moved within said scene.

Preferred embodiments of the present invention will now be described by way of example only, and not in any limitative sense, with reference to the accompanying drawings in which: Figure 1 is a schematic representation of the apparatus of the present invention; Figure 2 is a schematic representation of a pen device used in the present invention; and Figure 3 is a flowchart showing the steps taken in the method of the present invention.

Referring to Figure 1, an apparatus 10 is used to track the movement of a plurality of pointer devices or pens 12, 14 and 16 within a scene 18. The apparatus 10 includes at least one image capture device, for example infra-red camera 20 that is used to capture a plurality of images of the scene 18 and to produce an image signal that includes image data representing light at a predetermined frequency, in this example in the infra-red range. In this example, the scene 18 is at least defined by the limits that the camera 20 is able to view. The apparatus also includes a control processor 22 that contains a transmitter in the form of a short range radio wave transmitter 24. The control processor 22 is directly connected to camera 20 via connecting cable 26 and is also connected to a controlling computer 28, which is typically a standard computing device such as a personal computer, desktop computer or laptop computer or the like, by a connecting cable, typically a USB connection 30. The infra-red camera 20 is also connected to the controlling computer 28 by a similar connecting cable 32.

In the example shown, the computer 28 is also connected to projector 34 via connecting cable 36 that projects an image that also defines scene 18 of the display output of the computer device 28. The projector 34 projects the image on to a surface 38 which in the example shown in Figure 1 is a horizontal table top surface. However, the surface 38 may equally be a vertical surface and could be a mounted board on a wall but could equally be simply any surface of a wall. In a further alternative, the surface 38 and projector 34 could be replaced by a display screen (for example a LCD or LED display screen or the like) which can also be formed as part of a table or mounted on a wall.

Turning to Figure 2, the pens 12, 14 and 16 include a body portion 40 and a tip portion 42 that together form a standard pen shape. Contained within the body portion is a battery power supply 44 which supplies power to a processor 46. Also attached to processor 46 are a receiver circuit 48 and an aerial 50 that together receive the data sent via radio transmitter 24. The radio receiver 48 and aerial 50 may be formed integrally as a single circuit.

Contained within the tip portion 42 and both connected to the processor 48 are a micro-switch 52 and infra-red light emitter 54. The infra-red light emitter 54 produces infra-red light in response to a signal from the processor 46 and the micro-switch 52 indicates to the processor whether the pen is engaged with the surface 38.

Operation of the apparatus will now be described with additional reference to Figure 3. The apparatus functions under the control of computer 28 which sends a signal to operate to processor 22 which in turn controls transmitter 24 and camera 20. The processor data capture occurs cyclically and repeats a number of times each second, according to a formula set out below. Following a start 60 in response to a signal from computer 28 (or alternatively from controller 22) an initial reference picture is taken by camera 20 at 62 with no pens emitting light. The pen counter n is set to 1 (at the step indicated at 64) and processor 22 causes transmitter 24 to send an "on" signal to pen n (in this instance pen 1 at step 66). At the same time processor 22 sends a signal, via connecting cable 26, to camera 20 to take a picture of scene 18 (at step 68) and this picture is timed to precisely synchronise with the light being emitted from the pen ri = 1.

The pen n = 1, for example pen 12, receives via aerial 20 and receiver 48 the signal from transmitter 24, This signal is processed by processor 48 which interprets the data to indicate that pen 1 should emit light and as a result the infra-red light emitter 54 emits infra-red light.

However, this only occurs if pen 12 is engaged with surface 38 such that micro-switch 52 is depressed thereby indicating that pen 12 is active. If the micro-switch 52 is not depressed, the light will not emit since the pen is not engaged with the surface 38.

In order to ensure that the camera captures the light when it is on, the light is switched on (step 66) Controller 22 waits by idling (at step 72) for an acknowledgment signal from the camera (at step 70) that a picture has been taken before sending an off signal to the pen (at step 74) . This is because the timing of the camera taking the image cannot be guaranteed for example due to variations in how long the lens will stay open. As a result it can be insufficient to simply transmit a pulse of light and assume that the picture will be taken while the light is one.

At step 76, the event reporting thread starts on computer 28 (preferably in parallel to further data gathering under the control of processor 22 or computer 28) with the computer 28 receiving image data from infra-red camera 20 and an indication from processor 22 (or computer 28) as to which pen this image relates to. Then at step 78 the computer 28 calculates the location of the pen within scene 18 (assuming that light was emitted from pen 12 because micro-switch 52 was in an "on" position as a result of the pen 12 being pressed against surface 38 and that the pen was within view of the infra-red camera and within the extremities defined for scene 18) At step 80 the status of the pen 12 is determined relative to its previous status, for example, is the pen now pressed into engagement with surface 38 if it previously was not, has it moved relative to its previous position, is it no longer in engagement with the surface 38? This allows the software running on the computer 28 to raise the following events which can be communicated to any application that registers to listen to these events, PenDown(X,Y,penlD), PenMove(X,Y,penlD) and Penup(X,Y,penlfl) at step 82.

In the meantime, processor 22 (or computer 28) increases the pen count number n by 1 (at step 84) If n has not reached the total number of pens as determined at step 86, the process returns to step 66 by sending an "on" signal to the pen n (now equal to 2) and the remaining steps are repeated for the second pen. This ioop continues until n reaches the total number of pens (i.e. is not c= total number of pens) at which point the cycle returns to step 62 where no pens are emitting light and a reference picture is taken.

The process, as run under the control of computer 8 or controller 22, can be summarised as follows: 1. All pens are off 2. The controller requests a frame taken by the camera as a reference frame 3. The controller sends an on' signal to Peni 4. The controller requests a frame taken by the camera to determine the X and Y location of Penl 5. Once an acknowledgement signal, that an image has been taken, is received from the camera the controller sends an of f' signal to Peal 6. Running as a parallel thread, the software calculates the correct X and Y of penl based on calibration information 7. while still in the parallel thread and based on the current and previous state of Penl the software triggers the corresponding pen event: PenJJown (for up- >down change), PenMove (down->down different xy), or PentJp (down->up change, send last recorded xy) 8. In the main thread the controller sends an on' signal to Pen2 9. Steps 4-7 are repeated for Pen2 The number of pens that can be supported simultaneously is limited by the frame rate of the camera, the number of times it is required to read the pen status per second, and the processing time to calculate the correct x and y for each pen. Given F = the camera frame rate, R = the number of times the pen status is read per second, and N = the number of possible pens, if it is assumed that the processing time is less than 1/k (which is usually the case), and if one reference frame is taken per each cycle, we will need N÷l frames per cycle.

As a result: N = IntegerOf (F/H) -1 For typical value of F of 30 frames per second, and for a refresh rate of 4 times per second1 N = IntegerOf (30/4)-i = 6 Therefore the apparatus can support a maximum of 6 pens for a camera with 30 frames per second and a refresh rate of 4 readings per second.

It will be appreciated by persons skilled in the art that the above embodiments have been described by way of example only and not in any limitative sense, and that the various alterations and modifications are possible without departure from the scope of the invention as defined by the appended claims. For example, the frequency that the camera detects at and the pens emit at can be any suitable frequency, although IR frequencies are easily used with low cost components readily available. In a further alternative, the camera may produce images at a constant rate (for example frames per second) and the processor and transmitter may operate to cause the pens to flash in synchronisation with the frame production of the camera and then provide the computer with the data indicating which frame relates to each pen. Furthermore, in another embodiment the receiver 48 in the pen and transmitter 24 can both be replaced with transceivers that allow data to be passed in both directions between pen and controller. This could be used to pass information about the status of the micro switch thereby indicating whether the pen is engaged with the board.

Claims

Claims 1. An apparatus for tracking the movement of a plurality of pointer devices within a scene, the apparatus comprising:-at least one image capture device for capturing a plurality of images of a scene and producing an image signal including image data representing light at a predetermined frequency; at least one transmitter device for repeatedly transmitting a plurality of different trigger signals; and a plurality of pointer devices for indicating a location of said pointer devices in said scene, each pointer device including at least one light emitting device for emitting light substantially at said predetermined frequency and at least one receiver for receiving said trigger signals, wherein plurality of said trigger signals are detected by respective said pointer devices causing said light emitter in the respective pointer device to produce said light that is captured by said image capture device thereby linking a plurality of said images with said pointer device and allowing the pointer devices to be tracked as they are moved within said scene.
2. An apparatus according to claim 1, further comprising at least one processor for producing a transmitter control signal and an image capture device control signal thereby ensuring that said images produced by said image capture device are synchronised with the light produced by each respective pointer device in response to said transmitter control signal instructing said transmitter to produce said trigger signal.
3. An apparatus according to claim 1 or 2, wherein said processor is directly connected to said image capture device.
4. An apparatus according to any of the preceding claims wherein said predetermined frequency is infra-red light.
5. An apparatus according to claim 4, wherein said image capture device is an infra-red camera.
6. An apparatus according to any of the preceding claims wherein said transmitter device is a short range radio transmitter.
7. An apparatus according to any of the preceding claims wherein said pointer device comprises a body formed substantially in the shape of a pen and including circuitry for processing said received trigger signal and controlling said light emitting device in response to said trigger signal.
8. A method of tracking the movement of a plurality of pointer devices within a scene, comprising;-capturing a plurality of images of a scene with at least one image capture device and producing an image signal including image data representing light at a predetermined frequency; repeatedly transmitting from at least one transmitter device a plurality of different trigger signals; emitting light at substantially said predetermined frequency from at least one light emitting device in a respective pointer device being one of a plurality of devices in response to receiving a respective said trigger signal; and processing said image data to link a plurality of said images with said pointer device and allowing the pointer devices to be tracked as they are moved within said scene.
9. An apparatus for displaying and for tracking the movement of a plurality of pointer devices, the apparatus comprising: -a display device for displaying images; an apparatus for tracking the movement of a plurality of pointer devices within a scene according to any of claims 1 to 7, wherein said scene at least partially overlapping with said displayed images.
10. An apparatus for tracking the movement of a plurality of pointer devices within a scene substantially as hereinbefore described with reference to the accompanying drawings.-
11. A method of tracking the movement of a plurality of pointer devices within a scene substantially as hereinbefore described with reference to the accompanying drawings.