WO2001099045A1

WO2001099045A1 - A system and method for use in determining a cursor position

Info

Publication number: WO2001099045A1
Application number: PCT/SG2000/000091
Authority: WO
Inventors: David Yeo Woo Cheow
Original assignee: Temasek Polytechnic
Priority date: 2000-06-19
Filing date: 2000-06-19
Publication date: 2001-12-27
Also published as: AU2000274680A1

Abstract

A computer (1) generates a series of slide image signals which are passed to a projector (6). The projector projects images from the signals onto a screen but during its flyback period projects no image. A laser pointer (3) is directed by a user at the image on the screen, even during the flyback period. A camera (7) is directed at the screen and captures images during the flyback period, when only the spot from the laser pointer (3) is on the screen. The captured image is sent to a processor to determine the position at which the laser pointer is pointing. This data is provided to the computer (1). It updates the position of its cursor on the image, the signal for which it is sending to the projector, to be at that position. If the position of the spot is a highlight function area, that area is automatically highlighted.

Description

A SYSTEM AND METHOD FOR USE IN DETERMINING A CURSOR POSITION

BACKGROUND OF THE INVENTION

The present invention relates to a system and method for use in determining a cursor position, by capturing the position of a lighted area on a display surface, to be used as the cursor position. More specifically, the lighted area can be one that is pointed at a projected image generated by a computer and the capturing apparatus and method allows the point on the image pointed at to be determined, with the information provided to a computer's system and display. It has a particular use in a visual or audio-visual presentation system for presenting visual or audio-visual items.

People often give presentations using prepared slides, generated by a computer, which are projected by a projector connected to the computer. Unfortunately, arrangements to date have not allowed the easy highlighting of a portion of a displayed slide, nor flexibility in the order in which slide presentations are given. Moreover, even using slides in a preordained order, most arrangements do not allow the presenter to deliver his presentation without disruption.

For highlighting alone, it is possible to use a laser pointer to direct attention to a particular area. However, that area is not highlighted beyond the light provided by the pointer.

The least disruptive system is to provide the presenter with a remote control connected to the computer by a cable. All he needs do is press a button to cause highlighting or change slides. However, this system can be awkward if the user wishes to move away from a set spot, due to entanglements of the wire and a limited length of cable. Moreover, it only allows highlighting and slide changing in a fixed, preprogrammed order.

If an action such as clicking a mouse has to be taken to highlight an area or change an image, although clicking the mouse itself is not a laborious task, it still involves the presenter in pausing to execute the clicking action. First he must look at the mouse and screen, and then move the cursor to the right position. Such a system also restricts the presenter to a small area close to the controlling computer for easy access. A longer pause in the presentation can be expected if the presenter drifts too far away from the controlling computer.

As an alternative, the above problems can be solved by engaging an assistant to control the image change. However, a new set of problems surfaces, such as the assistant misunderstanding the presenter's instruction, since instruction in this instant is mostly likely to be through hand signals or quiet verbal instructions. All it takes is a momentary lapse by the assistant and a wrong highlighting or image might be projected instead. Even if no error occurs, the presenter must still stop to give the instruction.

Finally, there is specialized presentation application software, which shows each highlight or slide for a predetermined time. However, such software requires perfect timing on the presenter's part. If there is an interruption, for instance by a member of the audience asking questions or seeking clarifications, or if the presenter loses his thread or gets his timing wrong, this can seriously disturb the flow and timing of the presentation relative to that programmed, thus ruining the presentation.

The object of this invention is to provide a method, system and/or apparatus, which allows a presenter to deliver a presentation with reduced disruption.

SUMMARY OF THE INVENTION

According to a first aspect of the present invention, there is provided apparatus for use in capturing the position of a lighted area appearing on a first image on a display surface, comprising: first image means for processing an image signal and for outputting a first image signal corresponding to a first image; image processing means for receiving a second image signal corresponding to a second image and for determining, from said second image signal, the position of a lighted area in said second image; wherein said first image means is operable to update a data position to correspond to the position of said lighted area superimposed on said first image, as determined by said image processing means.

This apparatus can for example be embodied by a programmed or dedicated micro-processor or by computer software.

The first image means is preferably a computer and the image processing means provided outside a housing of said computer and operable to supply the determined position of said lighted area to said first image means through a RS232 port of said computer.

According to a second aspect of the present invention there is provided a system for capturing the position of a lighted area on a display surface, comprising: apparatus as above; image providing means for providing said first image onto a display surface, based on said first image signal from said apparatus, and for providing a blanking image or no image during a blanking period, interspersing said first image on a periodic basis; and image capturing means for image capturing said second image, being an image of said display surface during said blanking period, and providing said second image signal to said apparatus; wherein said apparatus is operable to determine said position of said lighted area from said second image captured during said blanking period.

This aspect also preferably comprises a laser pointer for directing a beam of laser generated light onto said display surface to provide said lighted area. According to a third aspect of the present invention, there is provided a method of capturing the position of a lighted area on a image on a display surface, comprising the steps of: providing a first image corresponding to a first image signal on a display surface; interrupting said first image on said display surface on a periodic basis, each time with a blanking period; pointing a lighted area onto said display surface on said first image and during said blanking periods; image capturing a second image, comprising an image of said lighted area on said display surface during one of said blanking periods; determining the position said lighted area in said second image; and updating a data position relating to said first image to correspond to the determined position of the lighted area.

Preferably the second image is an image of a display surface during a blanking period, such as during the flyback period of a projector providing the first image. A lighted area from a pointer such as a laser pointer is then the only detail in the second image.

The invention may also be operable to determine when said data position corresponds to a function area of said first image and to operate one or more functions of a function area when predetermined conditions are met.

The said predetermined conditions may comprise the condition that the function area is at least a highlight area and said data position is determined to correspond to said function area, and said function is to highlight said function area.

The said predetermined conditions may comprise the condition that the function area is at least a further action area, said data position is determined to correspond to said function area and said lighted area changes in a predetermined manner, and said function is to perform said further action. In this instance, the predetermined manner may comprise said lighted area being detected in said second image at a position corresponding to said function area, then not being detected in said second image, then being detected in said second image again at a position corresponding to said function area, wherein the period of not being detected is no longer than a predetermined length of time.

The data position can be a cursor position, with the lighted area on the display surface being mirrored as a cursor appearing on a monitor, if one is present, and/or in the first image signal.

According to yet another aspect of the invention, there is provided apparatus for use with a slide projection system having a computer producing slides, a projector projecting said slides and an image capturing device for taking pictures of the projected slides, said apparatus comprising: determining means, operable to receive a signal from said image capturing means, said signal representing an image of a substantially blank display with an additional lighted area superimposed thereon, to determine the position of said lighted area and to output said position as a mouse signal.

In this instance the determining means is preferably also further operable to determine if said lighted area performs a predetermined action and to output a mouse double-click signal if such an action is determined to have occurred.

In another aspect, the invention provides software for processing a signal representing an image of a substantially blank display with an additional lighted area superimposed thereon, to determine the position of said lighted area, and to update an image signal to place a cursor at the determined position of said lighted area. BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will now be further described, by way of non-limitative example, reference being made to the accompanying Figures, in which:

Figure 1 is a diagram of a position capturing system in accordance with an embodiment of the invention;

Figure 2a is an example of a projection of a typical slide image during use of the present invention;

Figure 2b is a view of a projected image during the flyback period; and

Figure 2c is an example of a new slide image

DETAILED DESCRIPTION OF THE INVENTION

Figure 1 is a diagram of a capturing system in accordance with an embodiment of the invention.

As a very brief overview of the embodiment of Fig.1 , a computer (1) generates a series of slide image signals which are passed to a projector (6). The projector projects images from the signals onto a screen but during a blanking period, such as its flyback period, projects no image. A laser pointer (3) is directed by a user at the image on the screen, even during the flyback period. A camera (7) is directed at the screen and captures images during the flyback period, when only the spot from the laser pointer (3) is on the screen. The captured image is sent to a processor to determine the position at which the laser pointer is pointing. This data is provided to the computer (1). It updates the position of its cursor on the image, the signal for which it is sending to the projector, to be at that position. If the position of the spot is a highlight function area, that area is automatically highlighted. In a more detailed description, a presentation programme made up by the user and containing a number of slide images is pre-loaded into a first image means, such as a computer (1). There are a number of software packages that allow such a programme to be written easily. Upon activation of a command by the presenter to start the presentation, the computer (1) processes the programme into individual first image signals and transmits the images (frames), one at a time and as required, to an image providing means (6) such as an overhead projector or other projector. This takes the first image signals and converts them back into first images which are projected onto a display surface (5) such as a projection screen. An example of such a slide is shown in Fig.2a.

To ensure that the latest image signal from the computer is the one being projected, the projecting device is constantly refreshing the image that is projected. As with a cathode ray tube the image is scanned repeatedly from top to bottom.

The time period for refreshing the image, i.e. for the beam to go from the bottom back to the top is not instantaneous; in fact there is a time lag of, for example 1.5ms or a few milliseconds. This time lag is also known as the flyback period. During the flyback period, the screen within the projector and from which the image is projected onto the display surface is actually blanked out. However, since the flyback period is in milliseconds, the human eye is not fast enough to detect or register any change on the display surface.

When an image is projected onto the display surface (5), the presenter may wish to emphasize an important aspect or otherwise direct the attention of the audience. This can be done by illuminating a desired position on the displayed image (10) using a lighted area (11). The lighted area (11) is pointed at the display surface by the user using a pointer (3). Typically the pointer is a laser pointer, which produces only a spot as the lighted area, but other light emitting means can be used. Even so, the lighted area is likely to be small and directed only to a part of the aspect of interest. Light is emitted from the pointer (3) when an activation button or other means, usually located along the casing of the highlighter, is depressed. Such pointers do not tend to transmit light only periodically, or if they do, the period is very very short. Thus, during the flyback time, when no image is projected onto the display surface, the only detail is the point of light from the pointer. This is as is shown in Fig.2b. On the displayed image, the lighted area would be appearing in the middle of the head of the figurine in Fig.2a.

An image capturing device, such as a digital CCD or camera (7) is directed at the display surface. It is positioned to capture the whole scope of the display surface (5) onto which first images are displayed. The image capturing device is programmed to synchronize with the flyback period of the projector (6), so that it will be activated to capture the displayed image (10) only during the flyback period. This is done because the system is only, in fact, interested in the position of a point of light from the pointer and the only illuminated part on the display surface (5) during the flyback period is the lighted area (11) from the pointer. The obvious advantage of this practice is that no processing is required to remove unwanted material. Even assuming processing is not a problem, it also removes the possibility of a misreading of the correct position, as there might be multiple bright spots in a displayed image (10) which have the same radiation intensity as the correct lighted area.

A captured image is a second image and is processed into a digital signal. This digital signal is sent as a second image signal to a processing means (12), for instance in the form of a separate unit, which processes the signal to determine the X and Y coordinates of the centre of the lighted area (11). From the processing unit (12) a further signal, the X-Y signal is sent to a transmission device (8), which transmits that signal, via a wireless method, e.g. by radio or infra-red to a receiving device (9). The receiving device (9) is connected to the RS232 (mouse) port of the computer (1). Having received the transmitted X-Y signal, it sends these coordinates to the computer, as a normal mouse would.

The computer (1) treats the X-Y coordinates of the lighted area (11) as a data position. It is deemed to be the new position of a cursor as it would appear on the first image generated by a computer on its own monitor (screen). If a monitor is connected up, the processor updates the current cursor position data and that is displayed. The cursor can also be included in the first image signal sent to the projector (6), so that it appears in the projected first image. However, it is not necessary when the lighted area (11) is visible to at least the presenter with the pointer.

Using this system, the pointer can be used in a similar manner to a mouse. It allows the presenter to operate a function of a function area, for example in the form of a button displayed on a first image, without needing to use a mouse or otherwise directly instruct the computer. Thus he directs the pointer at one of at least one predetermined function area on the displayed first image (10). This may be an undefined area as it appears in the image, or a defined one. It may have writing or it may not. For instance it may indicate "next slide", "next page", "next program" or some other function.

When the lighted area is determined to be present on a position corresponding to a function area, it may initiate a function automatically, such as the computer changing the first image signal to highlight that function area and/or making a sound etc. Fig.2c. shows the slide of Fig.2a. with the head of the figurine highlighted. Alternatively a further action may be required to initiate a function. Some function areas may include both automatic functions and functions that require the further action. Such further action may be that the user clicks twice in quick succession on the activation button of the pointer (3). By clicking twice, the pointer is turned off for a brief period. The image capturing device captures flyback periods with the pointer on, then with it off, then with it on. The computer determines the time gap. If it falls within a predetermined maximum limit and, on both sides of the gap, the lighted area falls on the same "button", the computer treats it as a double click on that button by a mouse. No non-automatic activation of a function will occur in this instance if the two periods of light are directed to different "buttons" or if the lighted area simply wanders off the "button" for an instant. In a second embodiment of the invention the image capturing device (7) is programmed to capture all images on the display surface. However, the synchronization with the image providing means (6) ensures that it only converts the image captured during the flyback period to a digital signal to the processing means (12).

A third embodiment of the invention relates to the image capturing device (7) again being programmed to capture all images. This time it transmits all captured second images to the processing means (12). The processing means (12) in this instance is able to identify the image from the flyback period, either through software or synchronization with the projecting device and to compute the X-Y coordinates. Synchronization between the image capturing device (7) and the projecting device is not necessary in this embodiment, but could be useful to ensure that the captured images include the flyback period if images are otherwise captured too infrequently. Also synchronization ensures that the processing means (12) does not have to process a number of image signals in order to find those that correspond to the flyback period.

In terms of processing of the second image signal at least, the present invention allows various options. The processing means can be provided in the image capturing device, between the receiving device and the computer or in the computer, e.g. using the computer's main processor. The problem with providing it between the receiving device and the computer or in the computer is that it means that whole images are transmitted. This may require considerable radio bandwidth, i.e. about 5MHz. It is preferable not to use the computer's main processor as the processing means to determine the X-Y coordinates, so that it does not use too much processing resources. Of course it depends on the power of the processor and the hardware arrangements. The X-Y coordinates have been exemplified as entering the computer through the mouse port. Other ports can be used. Further the X-Y signal can be as a change in position from the last time rather than as an absolute position. As for the connexions between the various components, they are all, preferably by cable, except for transmission of the X-Y signal to the computer, but other options are available. For instance the receiving and transmission devices can be replaced with electrically conductive cable or fibre optic cable. This would be especially useful if the processing means cannot be provided in or near the image capturing device. It should be readily able to transmit the complete image signals (e.g. a 640 by 480 pixel array would provide approximately 2.4 Mbits per image). The components do not all have to be directly connected, but each can be connected through one or more of the others.

The image capturing device does not need to be digital. It can be analogue, although analogue-to-digital processing should be available at some stage to allow the signal to be processed by the processing means.

The above embodiments show use of a projector, such as an overhead projector. However, the type of image providing means is not so limited. Other types of projector can be used, including ones with no flyback period if the microprocessor provides blank images, as is discussed later. There is also no reason why the display surface could not be a CRT, LCD display or LED display, most likely incorporating the image providing means.

Preferably a laser pointer is used as the means of providing a lighted area on the display surface. However, other means, apparatus or method of providing a spot or lighted area on the display surface can be used, provided it can be directed manually by the presenter and can be captured by the image capturing device at the relevant timing.

A double click on an activation button is not the only possible method of activating a non-automatic function of a function area. Some other method can be used to stop the lighted area hitting the display surface for a while. Alternatively, activation of a non-automatic function of a function area could be achieved by changing the number, shape, size, colour or some other aspect of the lighted area, in a manner that can be detected by the computer. Because the non- automatic functions of function areas are usually of a type that are activated by a double-click of a computer mouse, the term "double-clicking" and derivative terms is used in this specification to denote any action used or necessary to activate such functions.

There is the problem that the computer may not be able to determine accurately the position of the pointed light relative to the images it is projecting, unless the image capturing device is set at a predetermined position relative to the projected image. An alternative is to have the computer generate a calibration slide. The light pointer is pointed at a very specific test portion of this calibration slide and double-clicked (or the return button of the computer is depressed, or some other method is used to inform the computer). The computer then determines the position of the lighted area from the signal incoming from the image capturing device and calibrates all future determined positions on the assumption that the lighted area is, at this time, pointed at the test portion. This would be a usual method employed when the system is ported around or tends to need recalibration on a regular basis.

Another option for calibration would be to use a calibration slide, having a grid array or the like, and calibrating without using a lighted area but during the non- flyback period.

The above embodiments assume that the position of the pointed light is determined using the flyback properties of the projecting device. In an alternative embodiment an alternative blanking period is used. In this instance, the computer generating the image signal can be used to generate a blank image signal producing a blank image at regular brief periods. For instance, this could be done once every 500ms and only appear for 40ms or so, i.e. for one frame period on the projecting device, assuming 25 frames per second and 2 fields per frame. However, the frequencies and periods would depend on the projector used. For instance, one that produces 70 frames per second, without interlacing, would display a blank frame for only about 14ms. In this embodiment the image capturing device can be synchronized but this time only to capture the image during these blank image periods. Additionally or alternatively, the processing means, whether it is the main computer processor or some other unit, can be synchronized or otherwise instructed only to consider those signals from the image capturing device during the blank image periods. Determination of the position of the pointed light and further processing can then take place as in the previous embodiments. The blank images could be plain white images or plain black images.

Many computers do not have the processing power to produce the blank images to be displayed for only a short while, such that they are not noticed by users. Therefore a special graphics card may be necessary.

Another possibility, using these blank images is for them not to be totally blank. Instead at the start of a presentation at least, and maybe occasionally, at least, thereafter, they can carry calibration data, for example in the form of one or more shapes or character patterns or strings. The position of the calibration data in the images provided by the image capturing device back to the computer would be determined, so that the position of pointed light on a projected image could be accurately determined.

Although the described embodiments use a light emitting device, there is no reason why the light cannot be generated on the display surface itself, for instance if the pointer emitted UV light to which the display surface reacts by luminescing. Alternatively, a non-visible beam can be used without a reaction by the display surface, or with a non-visible reaction with the display surface, provided it can be "seen", i.e. captured by the image capturing device. In which case the computer would need to include the cursor in the first image signal, so that the cursor appears in the image on the display surface, for instance as a red dot, and the presenter can see where he is directing the pointer. The term "lighted area" still covers such embodiments; they can still be distinguished by the image capturing device and computer. In this description, the term computer covers not just desk-top and lap-top computers. It at least also covers other apparatus that can provide the first image signals and incorporate the position of the lighted area into its processing.

It will be readily apparent to those skilled in the art that the present invention can, at least in its simplest form, be embodied just in a computer programme (software), with or without hardware. Further, the invention could readily work with equipment that is already available, in terms of a computer, a projector and a CCD.

Here we presented a preferred way to realize the method according to the invention. While we have considered primarily applications in a hall or lecture theatre, the method according to the invention is well suited for public address activities whereby some form of computing and projection equipment is involved.

Since certain changes may be made in the above method without departing from the scope of the invention herein involved, it is intended that all matter contained in the above description, as shown in the accompanying drawing, shall be interpreted both as illustrative as well as covering the specific that is mentioned.

Claims

1. Apparatus for use in capturing the position of a lighted area appearing on a first image on a display surface, comprising: first image means for processing an image signal and for outputting a first image signal corresponding to a first image; image processing means for receiving a second image signal corresponding to a second image and for determining, from said second image signal, the position of a lighted area in said second image; wherein said first image means is operable to update a data position to correspond to the position of said lighted area superimposed on said first image, as determined by said image processing means.

2. Apparatus according to claim 1 , wherein said second image does not include said first image.

3. Apparatus according to claim 1 or 2, wherein said second image is an image of a display surface during a blanking period.

4. Apparatus according to claim 1 , 2 or 3, operable to determine when said data position corresponds to a function area of said first image and to operate one or more functions of a function area when predetermined conditions are met.

5. Apparatus according to claim 4, wherein said predetermined conditions comprise the condition that the function area is at least a highlight area and said data position is determined to correspond to said function area, and said function is to highlight said function area.

6. Apparatus according to claim 4 or 5, wherein said predetermined conditions comprise the condition that the function area is at least a further action area, said data position is determined to correspond to said function area and said lighted area changes in a predetermined manner, and said function is to perform said further action.

7. Apparatus according to claim 6, wherein said predetermined manner comprises said lighted area being detected in said second image at a position corresponding to said function area, then not being detected in said second image, then being detected in said second image again at a position corresponding to said function area, wherein the period of not being detected is no longer than a predetermined length of time.

8. Apparatus according to any one of the preceding claims, wherein the said data position is a cursor position.

9. Apparatus according to claim 8, wherein a cursor is included in said first image signal at said data position.

10. Apparatus according to claim 8 or 9, operable to display said first image on a monitor, with a cursor displayed on the first image at said data position.

11. Apparatus according to any one of the preceding claims, wherein said first image means and said image processing means comprise software.

12. Apparatus according to any one of the preceding claims, wherein said first image means and/or said image processing means further comprise micro-processor means.

13. Apparatus according to any one of the preceding claims, wherein said first image means and said image processing means comprise a unitary processor.

14. Apparatus according to any one of the preceding claims, further comprising a monitor.

15. Apparatus according to any one of the preceding claims, wherein said lighted area is a point of light from a laser pointer.

16. Apparatus according to any one of the preceding claims, wherein: said first image means is a computer; and said image processing means is provided outside a housing of said computer and is operable to supply the determined position of said lighted area to said first image means through a RS232 port of said computer.

17. A system for capturing the position of a lighted area on a display surface, comprising: apparatus according to any one of the preceding claims; image providing means for providing said first image onto a display surface, based on said first image signal from said apparatus, and for providing a blanking image or no image during a blanking period, interspersing said first image on a periodic basis; image capturing means for image capturing said second image, being an image of said display surface during said blanking period, and providing said second image signal to said apparatus; wherein said apparatus is operable to determine said position of said lighted area from said second image captured during said blanking period.

18. A system according to claim 17, wherein said blanking period is the flyback period of said image providing means.

19. A system according to claim 17 or 18, further comprising a laser pointer for directing a beam of light onto said display surface to provide said lighted area.

20. A method of capturing the position of a lighted area on a image on a display surface, comprising the steps of: providing a first image corresponding to a first image signal on a display surface; interrupting said first image on said display surface on a periodic basis, each time with a blanking period; pointing a lighted area onto said display surface on said first image and during said blanking periods; image capturing a second image, comprising an image of said lighted area on said display surface during one of said blanking periods; determining the position of said lighted area in said second image; updating a data position relating to said first image to correspond to the determined position of the lighted area.

21. A method according to claim 20, wherein said first images are provided on said display surface by an image providing device and said blanking periods are during the flyback period of said image providing device.

22. A method according to claim 20 or 21 , wherein said first image contains at least one function area and further comprising the steps of: determining when said data position corresponds to one of said at least one function areas; determining when predetermined conditions are met; and operating one or more functions of a function area when said predetermined conditions are determined to have been met.

23. A method according to claim 22, wherein said predetermined conditions comprise the condition that the function area is at least a highlight area and said data position is determined to correspond to said function area, and said function is to highlight said function area.

24. A method according to claim 22 or 23, wherein said predetermined conditions comprise the condition that the function area is at least a further action area, said data position is determined to correspond to said function area and said lighted area changes in a predetermined manner, and said function is to perform said further action.

25. A method according to claim 24, wherein said predetermined manner comprises said lighted area being detected in said second image at a position corresponding to said function area, then not being detected in said second image, then being detected in said second image again, at a position corresponding to said same function area, wherein the period of not being detected is no longer than a predetermined length of time.

26. Apparatus for use with a slide projection system having a computer producing slides, a projector projecting said slides and an image capturing device for taking pictures of the projected slides, said apparatus comprising: determining means operable to receive a signal from said image capturing means, said signal representing an image of a substantially blank display with an additional lighted area superimposed thereon, to determine the position of said lighted area and to output said position as a mouse signal.

27. Apparatus according to claim 26, wherein said determining means is further operable to determine if said lighted area performs a predetermined action and to output a mouse double-click signal if such an action is determined to have occurred.

28. Software for processing a signal representing an image of a substantially blank display with an additional lighted area superimposed thereon, to determine the position of said lighted area, and to update an image signal to place a cursor at the determined position of said lighted area.

29. A system, apparatus or software for capturing the position of a lighted area on a display surface, arranged to operate substantially as hereinbefore described, with reference to and as illustrated in the accompanying drawings.

30. A method of capturing the position of a lighted area on a image on a display surface, substantially as hereinbefore described, with reference to and as illustrated in the accompanying drawings.