WO2014091225A1

WO2014091225A1 - Method and apparatus for tracking the movement of a plurality of tracked devices within a scene

Info

Publication number: WO2014091225A1
Application number: PCT/GB2013/053254
Authority: WO
Inventors: Ahmed Kharrufa
Original assignee: Modular Software Limited T/A Reflective Thinking
Priority date: 2012-12-12
Filing date: 2013-12-11
Publication date: 2014-06-19
Also published as: GB2508840A; GB201222335D0; GB2508840B

Abstract

An apparatus for tracking the movement of a plurality of tracked devices or pens within a scene is disclosed. The apparatus is able to identify and differentiate between the pens. The apparatus includes a projector for projecting an image or screen for displaying an image, these images forming the scene. An IR camera captures images of the display and the pens and a transmitter transmits consecutive series of distinguishable trigger signals to the pens. The pens includes an IR light emitting device and a receiver for receiving the trigger signals so each pen emits its light only when instructed to do so. As a result each image only contains light from one pen allowing each pen to be tracked individually.

Description

METHOD AND APPARATUS FOR TRACKING THE MOVEMENT OF A PLURALITY OF TRACKED DEVICES WITHIN A SCENE

The present invention relates to a method and apparatus for tracking the movement of a plurality of tracked devices within a scene and relates particularly, but not exclusively, to a method and apparatus for utilising any existing table or wall display and using it as a multi-user digital table top or multi-user interactive board.

The use of interactive boards and table tops that use pens to allow a user to appear to write on a board and to interact with a board or screen are well known. One such system utilises a projector which displays images on to a board that contains apparatus to detect when a "pen" that is designed to work with the board is in close proximity to the board or touching it. The board produces data indicating where the pen is used on the board which can be fed back to a computer device controlling the projector. Some such systems allow for the use of multiple pens at the same time on a single board, which is extremely useful for collaborative work. However, the cost of the board is very high and it tends to be used in a fixed location, for example, attached to a wall or forming part of a table top.

In an alternative system, an infra-red camera is positioned to view a board or table top, on to which a projector is projecting an image. A pen that produces an infra-red light source when pressed into engagement with the board or table top and the infra-red camera is used to produce images that track the location of the pen. This apparatus has the advantage that it is significantly lower in cost than the board detectors mentioned above. However, if more than one pen is used the light emitted from the multiple pens is the same and cannot be distinguished by the processor processing the data from the infra-red camera. This significantly limits the uses to which such a system can be put. Existing solutions to this problem include each pen transmitting a certain unique on/off sequences or codes to identify the pen. Such sequences are either transmitted continuously and repeatedly when the pen is active requiring complicated sequences to guarantee synching and decoding. This in turn complicates the algorithm required to decipher the pens and reduces the number of possible pens due to rate constraints. In another example of the prior art (disclosed in US Patent application US2004/0239653) each device has a unique binary code (light on/light off) which is produced in response to a sequence of trigger signal and as a result the combination of pens the are turned on is never the same within the sequence of triggers. This approach has the problem that it significantly complicates the processing of the image data as it requires analysing a sequence of images with a location estimation algorithm to correctly identify a sequence of on/off signals as originating from the same pen and consequently track its location. The algorithm also needs to deal with special cases when pens are in close proximity to each other or when they leave the scene and return to it making this system complicated and unreliable.

In a further example of the prior art, pens are fitted with digital cameras which are able to read a matrix of faintly printed dots on a board, table top or sheet of paper and use these dots to determine a location which is wirelessly communicated to a processing device. This system allows the use of multiple pens. However, each pen is expensive and can only work on a surface that is printed with the special dot pattern.

Preferred embodiments of the present invention seek to overcome the above described disadvantages of the prior art.

According to an aspect of the present invention, there is provided an apparatus for tracking the movement of a plurality of tracked devices within a scene, the apparatus comprising : - at least one image capture device for capturing a plurality of images of a scene and producing an image signal including image data representing light at a predetermined frequency; at least one transmitter device for repeatedly transmitting a consecutive series of distinguishable trigger signals; and a plurality of tracked devices for indicating a location of said tracked devices in said scene, each tracked device including at least one light emitting device for emitting light substantially at said predetermined frequency and at least one receiver for receiving said trigger signals, wherein a said trigger signal is detected by a said tracked device causing said light emitter in the tracked devices to produce said light that is captured by said image capture device so that for each said trigger signal only one of said plurality of tracked devices produces said light when said image is captured thereby linking a plurality of said lights recorded in said images with respective said tracked device and allowing the tracked devices to be tracked as they are moved within said scene. By providing a system that utilises a processor connected to an infra-red camera and infra-red light emitting pens, together with a transmitter also connected to the processor and a receiver in each pen, the following advantages are provided. Multiple pens can be used and uniquely identified in a system that does not require expensive apparatus such as special boards or expensive pens. Furthermore, this is done reliably and without the need for any complicated algorithms. The further advantage is provided that any surface onto which an image can be projected can be used without the need for expensive boards or for specifically printed sheet materials. It is also the case that by only having the infra-red light on for a fraction of a second for each pen the battery consumption of each pen is reduced. Although additional circuitry and power consumption is required for the receiving device, which is permanently on whilst the pen in engaged with a surface via a micro-switch in the tip of the pen, the current required for receiving is less than that required to produce the infra-red light .

The apparatus may further comprise at least one processor for producing a transmitter control signal and an image capture device control signal thereby ensuring that said images produced by said image capture device are synchronised with the light produced by each respective tracked device in response to said transmitter control signal instructing said transmitter to produce said trigger signal.

In a preferred embodiment, the processor is directly connected to said image capture device. Although a standard computer can be used to control the transmitter signal and synchronise the image capture device with the pens, by having a separate processor to control this, the advantage is provided that less communication with the computer, which is processing the data and displaying things via the projector or screen, is required. Furthermore the direct connection between the processor and infra-red camera improves the synchronisation between camera and pens thereby allowing a large number of pens to be used simultaneously.

In another preferred embodiment, the predetermined frequency is infra-red light.

In a further preferred embodiment, the image capture device is an infra-red camera.

In a preferred embodiment, the transmitter device is a short range radio transmitter.

The tracked device may comprise a body formed substantially in the shape of a pen and including circuitry for processing said received trigger signal and controlling said light emitting device in response to said trigger signal .

The tracked device described above is a low cost item and as a result, provides significant advantages where collaborative working is required using multiple pens.

According to another aspect of the present invention, there is provided a method of tracking the movement of a plurality of tracked devices within a scene, comprising:- capturing a plurality of images of a scene with at least one image capture device and producing an image signal including image data representing light at a predetermined frequency; repeatedly transmitting from at least one transmitter device a consecutive series of distinguishable trigger signals; emitting light at substantially said predetermined frequency from at least one light emitting device in a respective tracked device being one of a plurality of devices in response to receiving a respective said trigger signal; and processing said image data to link a plurality of said images with said tracked device and allowing the tracked devices to be tracked as they are moved within said scene.

Preferred embodiments of the present invention will now be described by way of example only, and not in any limitative sense, with reference to the accompanying drawings in which:

Figure 1 is a schematic representation of the apparatus of the present invention;

Figure 2 is a schematic representation of a pen device used in the present invention; and

Figure 3 is a flowchart showing the steps taken in the method of the present invention.

Referring to Figure 1, an apparatus 10 is used to track the movement of a plurality of tracked devices or pens 12, 14 and 16 within a scene 18. The apparatus 10 includes at least one image capture device, for example infra-red camera 20 that is used to capture a plurality of images of the scene 18 and to produce an image signal that includes image data representing light at a predetermined frequency, in this example in the infra-red range. In this example, the scene 18 is at least defined by the limits that the camera 20 is able to view. The apparatus also includes a control processor 22 that contains a transmitter in the form of a short range radio wave transmitter 24. The control processor 22 is directly connected to camera 20 via connecting cable 26 and is also connected to a controlling computer 28, which is typically a standard computing device such as a personal computer, desktop computer or laptop computer or the like, by a connecting cable, typically a USB connection 30. The infra-red camera 20 is also connected to the controlling computer 28 by a similar connecting cable 32.

In the example shown, the computer 28 is also connected to projector 34 via connecting cable 36 that projects an image that also defines scene 18 of the display output of the computer device 28. The projector 34 projects the image on to a surface 38 which in the example shown in Figure 1 is a horizontal table top surface. However, the surface 38 may equally be a vertical surface and could be a mounted board on a wall but could equally be simply any surface of a wall. In a further alternative, the surface 38 and projector 34 could be replaced by a display screen (for example a LCD or LED display screen or the like) which can also be formed as part of a table or mounted on a wall.

Turning to Figure 2, the pens 12, 14 and 16 include a body portion 40 and a tip portion 42 that together form a standard pen shape. Contained within the body portion is a battery power supply 44 which supplies power to a processor 46. Also attached to processor 46 are a receiver circuit 48 and an aerial 50 that together receive the data sent via radio transmitter 24. The radio receiver 48 and aerial 50 may be formed integrally as a single circuit. Contained within the tip portion 42 and both connected to the processor 48 are a micro-switch 52 and infra-red light emitter 54. The infra-red light emitter 54 produces infrared light in response to a signal from the processor 46 and the micro-switch 52 indicates to the processor whether the pen is engaged with the surface 38.

Operation of the apparatus will now be described with additional reference to Figure 3. The apparatus functions under the control of computer 28 which sends a signal to operate to processor 22 which in turn controls transmitter 24 and camera 20. The processor data capture occurs cyclically and repeats a number of times each second, according to a formula set out below. Following a start 60 in response to a signal from computer 28 (or alternatively from controller 22) an initial reference picture is taken by camera 20 at 62 with no pens emitting light. The pen counter n is set to 1 (at the step indicated at 64) and processor 22 causes transmitter 24 to send an "on" signal to pen n (in this instance pen 1 at step 66) . At the same time processor 22 sends a signal, via connecting cable 26, to camera 20 to take a picture of scene 18 (at step 68) and this picture is timed to precisely synchronise with the light being emitted from the pen n = 1.

The pen n = 1, for example pen 12, receives via aerial 20 and receiver 48 the signal from transmitter 24. This signal is processed by processor 48 which interprets the -Si- data to indicate that pen 1 should emit light and as a result the infra-red light emitter 54 emits infra-red light. However, this only occurs if pen 12 is engaged with surface 38 such that micro-switch 52 is depressed thereby indicating that pen 12 is active. If the micro-switch 52 is not depressed, the light will not emit since the pen is not engaged with the surface 38.

In order to ensure that the camera captures the light when it is on, the light is switched on (step 66) . Controller 22 waits by idling (at step 72) for an acknowledgment signal from the camera (at step 70) that a picture has been taken before sending an off signal to the pen (at step 74) . This is because the timing of the camera taking the image cannot be guaranteed for example due to variations in how long the lens will stay open. As a result it can be insufficient to simply transmit a pulse of light and assume that the picture will be taken while the light is one .

At step 76, the event reporting thread starts on computer 28 (preferably in parallel to further data gathering under the control of processor 22 or computer 28) with the computer 28 receiving image data from infra-red camera 20 and an indication from processor 22 (or computer 28) as to which pen this image relates to. Then at step 78 the computer 28 calculates the location of the pen within scene 18 (assuming that light was emitted from pen 12 because micro-switch 52 was in an "on" position as a result of the pen 12 being pressed against surface 38 and that the pen was within view of the infra-red camera and within the extremities defined for scene 18) . At step 80 the status of the pen 12 is determined relative to its previous status, for example, is the pen now pressed into engagement with surface 38 if it previously was not, has it moved relative to its previous position, is it no longer in engagement with the surface 38? This allows the software running on the computer 28 to raise the following events which can be communicated to any application that registers to listen to these events, PenDown (X,Y,penID) , Pen ove (X, Y, penID) and PenUp (X, Y, penID) at step 82.

In the meantime, processor 22 (or computer 28) increases the pen count number n by 1 (at step 84) . If n has not reached the total number of pens as determined at step 86, the process returns to step 66 by sending an "on" signal to the pen n (now equal to 2) and the remaining steps are repeated for the second pen. This loop continues until n reaches the total number of pens (i.e. is not <= total number of pens) at which point the cycle returns to step 62 where no pens are emitting light and a reference picture is taken.

The process, as run under the control of computer 8 or controller 22, can be summarised as follows:

1. All pens are off

2. The controller requests a frame taken by the camera as a reference frame

3. The controller sends an οη' signal to Penl

4. The controller requests a frame taken by the camera to determine the X and Y location of Penl

5. Once an acknowledgement signal, that an image has been taken, is received from the camera the controller sends an 'off signal to Penl 6. Running as a parallel thread, the software calculates the correct X and Y of penl based on calibration information

7. While still in the parallel thread and based on the current and previous state of Penl the software triggers the corresponding pen event: PenDown (for up- >down change) , PenMove (down->down different xy) , or PenUp (down->up change, send last recorded xy)

8. In the main thread the controller sends an ^λοη' signal to Pen2

9. Steps 4-7 are repeated for Pen2

As a result, it can be seen that the transmitter device repeatedly transmits a consecutive series of distinguishable trigger signals. Each pen has a trigger signal assigned to it and it flashes when it receives its signal. Therefore, only one pen is emitting light in each image meaning that any one image can be identified as recoding the location of one pen. This in turn allows the series of images for each pen to be processed to track that pen's location within the scene without the need for any estimations or advanced algorithms.

The number of pens that can be supported simultaneously is limited by the frame rate of the camera, the number of times it is required to read the pen status per second, and the processing time to calculate the correct x and y for each pen. Given

F = the camera frame rate,

R = the number of times the pen status is read per second, and

N = the number of possible pens, if it is assumed that the processing time is less than 1/R (which is usually the case) , and if one reference frame is taken per each cycle, we will need N+l frames per cycle. As a result:

N = IntegerOf (F/R) - 1

For typical value of F of 30 frames per second, and for a refresh rate of 4 times per second,

N = IntegerOf (30/4) -1 = 6

Therefore the apparatus can support a maximum of 6 pens for a camera with 30 frames per second and a refresh rate of 4 readings per second.

It will be appreciated by persons skilled in the art that the above embodiments have been described by way of example only and not in any limitative sense, and that the various alterations and modifications are possible without departure from the scope of the invention as defined by the appended claims. For example, the frequency that the camera detects at and the pens emit at can be any suitable frequency, although IR frequencies are easily used with low cost components readily available. In a further alternative, the camera may produce images at a constant rate (for example 30 frames per second) and the processor and transmitter may operate to cause the pens to flash in synchronisation with the frame production of the camera and then provide the computer with the data indicating which frame relates to each pen. Furthermore, in another embodiment the receiver 48 in the pen and transmitter 24 can both be replaced with transceivers that allow data to be passed in both directions between pen and controller. This could be used to pass information about the status of the micro switch thereby indicating whether the pen is engaged with the board. In a further example, the tracked device, which could be referred to as a pointer device, could be a device other than the pen disclosed above. For example, the tracked device could be an eraser, a ring or any device that can contain the circuitry components described as present in the pen set out above.

Claims

1. An apparatus for tracking the movement of a plurality of tracked devices within a scene, the apparatus comprising: - at least one image capture device for capturing a plurality of images of a scene and producing an image signal including image data representing light at a predetermined frequency; at least one transmitter device for repeatedly transmitting a consecutive series of distinguishable trigger signals; and a plurality of tracked devices for indicating a location of said tracked devices in said scene, each tracked device including at least one light emitting device for emitting light substantially at said predetermined frequency and at least one receiver for receiving said trigger signals, wherein a said trigger signal is detected by a said tracked device causing said light emitter in the tracked device to produce said light that is captured by said image capture device so that only one of said plurality of tracked devices produces said light when said image is captured thereby linking a plurality of said lights recorded in said images with respective said tracked device and allowing the tracked devices to be tracked as they are moved within said scene.

2. An apparatus according to claim 1, further comprising at least one processor for producing a transmitter control signal and an image capture device control signal thereby ensuring that said images produced by said image capture device are synchronised with the light produced by each respective tracked device in response to said transmitter control signal instructing said transmitter to produce said trigger signal.

3. An apparatus according to claim 1 or 2, wherein said processor is directly connected to said image capture device.

4. An apparatus according to any of the preceding claims wherein said predetermined frequency is infra-red light.

5. An apparatus according to claim 4, wherein said image capture device is an infra-red camera.

6. An apparatus according to any of the preceding claims wherein said transmitter device is a short range radio transmitter .

7. An apparatus according to any of the preceding claims wherein said tracked device comprises a body formed substantially in the shape of a pen and including circuitry for processing said received trigger signal and controlling said light emitting device in response to said trigger signal .

8. A method of tracking the movement of a plurality of tracked devices within a scene, comprising :- capturing a plurality of images of a scene with at least one image capture device and producing an image signal including image data representing light at a predetermined frequency; repeatedly transmitting from at least one transmitter device a consecutive series of distinguishable trigger signals; emitting light at substantially said predetermined frequency from at least one light emitting device in a respective tracked device being one of a plurality of devices in response to receiving a respective said trigger signal; and processing said image data to link a plurality of said images with said tracked device and allowing the tracked devices to be tracked as they are moved within said scene.

9. An apparatus for displaying and for tracking the movement of a plurality of tracked devices, the apparatus comprising : - a display device for displaying images; an apparatus for tracking the movement of a plurality tracked devices within a scene according to any of claims to 7, wherein said scene at least partially overlapping wi said displayed images.

10. An apparatus for tracking the movement of a plurality of tracked devices within a scene substantially as hereinbefore described with reference to the accompanying drawings .

11. A method of tracking the movement of a plurality of tracked devices within a scene substantially as hereinbefore described with reference to the accompanying drawings.