GB2350674A

GB2350674A - Control of computer using retroreflectors

Info

Publication number: GB2350674A
Application number: GB9919022A
Authority: GB
Inventors: Peter Mcduffie White
Original assignee: Individual
Current assignee: Individual
Priority date: 1999-06-02
Filing date: 1999-08-13
Publication date: 2000-12-06
Also published as: GB9919022D0; GB9912784D0

Abstract

A control arrangement eg for a computer cursor comprises an imaging device viewing an object space, a light source illuminating the space, a retroreflector capable of being moved within the space to reflect light from the source to enable a retroreflected image to be detected by the imaging device, and means to effect a control operation in dependence on the position and/or colour of the retroreflected image. One or more differently coloured retroreflectors (9,1) may be attached to the front and/or back of one or more fingers either with adhesive or using a ring or band, or may be attached to a hand held card or stick. A two way mirror may be used to present to a person standing in the object space with a combined an image of that person (3) and a computer screen (7) showing a program with interactive buttons (10, 11, 12).

Description

2350674 Device and method for human control of computers using

retroreflectors This invention relates to a device and method for human control of computer programs.

Various technologies have been developed to provide a means of wireless computer control. Remote mouse systems use infrared and radio waves to transmit commands to move a cursor on a computer screen and to make selections. Each of these systems requires an expensive device capable of transmitting signals to a receiver connected to the computer. Other systems have been developed to have sensors detect a stylus within a defined work area. These systems have limitations in terms of size and location of the work area.

Human computer interface systems have been developed using a video camera viewing the work area. Active zones are defined as blocks of pixels within the video image. When the pixels within these active zones are changed by movement in the video image area a computer response is triggered. Using this system a person in the view of the video image could reach for an active zone that is indicated as a visual displayed in the active zone. This system has the advantage of allowing interaction with the computer without requiring any devices in the work area. The limitation of this system is that any movement will trigger a response, which makes it difficult to control selections. A more fundamental limitation is that this system is not well suited to the standard human computer interface, which requires both a movement of the cursor and a selection typically made by pressing the mouse or a button.

The present invention provides an arrangement for human control of computer programs that is not subject to these limitations.

The invention comprises a control arrangement comprising:

an imaging device that forms an image of an object space; a light source that illuminates said object space; 2 a retroreflector capable of being moved within said object space and reflecting said light source such that a retroreflected image is detected by said imagined device; and means to control operation in dependence on the position and/or colour of said retroreflected image Tle imaging device may comprise a video camera.

The image formed by said imaging device may be displayed on a monitor or screen visible to the user within said object device.

Said retroreflector may be illuminated by configuration of said light source and said imaging device such that the centre line of the projection of light and the centre line of the receiving lens of said imaging device are positioned close enough together that the retroreflected image is received by the receiving lens of said imaging device or that both the centre line of the projection of light and the centre line of the receiving lens of said imaging device are aligned using a two-way mirror.

Said control operation may be mediated through a computer processor which is programmed to recognise and respond to the position and/or colour of said retroreflected image.

It may be arranged that a user can see a visualisation of said retroreflected image on a monitor or screen connected to said computer processor such that said user can move the position of said retroreflector in said object space to transpose the position of the retroreflected image within the programmed domain of said computer processor as displayed on the said monitor or screen.

A plurality of retroreflectors may be employed in the said object space such that the said imaging device captures the retroreflected images and the said computer processor recognises and responds to all of the retroreflected images in the programmed 3 domain to perform more than one discrete function simultaneously or to perform functions based on the quantity and movement of the said plurality of retroreflected images.

The said computer processor may recognise the colour of a retroreflected image and differentiate between two or more colours to perform different functions corresponding to the colour of the retroreflected image. For example, the image of a red retroreflector may control the movement of a cursor in the said programmed domain of the said computer processor and the image of a green retroreflector may control a selection function, such as a mouse click.

A retroreflector may be attached to a finger of a user, for example it may be adhered to the finger with an adhesive substance or device; it might be attached to a device such as a ring or thimble that fits on to a finger of a user; or it may be strapped to the finger of the user.

Any of the above methods for attaching a retroreflector to a finger can be used to attach both one reflector to the front tip of a firtger and a second reflector of a different colour on the back tip of the same finger. With this device it is possible for the user to move the cursor while exposing the retroreflector on the back of the finger and to make a selection by flicking the finger up to expose the second coloured reflector.

A retroreflector may comprise a device that is rotated to expose different coloured retroreflectors attached to different faces of the device to said imaging device or it may be a device that exposes different coloured retroreflectors according to movement of a sliding cover.

Within this present invention the user will be able to operate standard computer programs by moving a cursor with one retroreflector and making a selection with a second retroreflectors of a different colour. The obvious advantage of this system is that the cost of the retroreflectors can be extremely low. This human computer 4 interface is ideal for applications where the computer programs need to be made accessible to a large numbers of users, such as public exhibitions. The users will be able to navigate the computer programs and make selections even though all expensive computer equipment will be out of reach in a secured area.

This invention provides a device and method for human control of computer programs through the display of retroreflectors within thefield of view of a video camera and a computer to selectively detect the appearance and movement of the retroreflector within the video image.

This invention comprises an arrangement of a video camera, a light source, a computer system, a custom software program and one or more retroreflectors. The retroreflectors are illuminated by a configuration of a light source and the lens of a video camera that are close to each other and both pointing in the same direction. Even though the lens and the light of the light source do not exactly share the same centre line they are close enough that the light reflected from the retroreflector enters the lens of the video camera.

In a similar configuration of the invention a mirror is placed close to the receiving lens of the video camera. The mirror is placed at-45 degrees to the centre line of the receiving lens of the video camera so that the centre line of the projection of light from the light source facing the mirror is aligned at 90 degrees to the centre line of the receiving lens of the video camera and the centre line of the mirror reflection of the projection of light is closely aligned with the centre line of the receiving lens of the video camera so that the retroreflected light enters the receiving lens of the video camera.

In another configuration of the invention the centre line of the projection of light and the centre of the taking lens of the camera are matched by the use of a twoway mirror. This can be achieved with a video camera that views through a two-way mirror which is set at an angle, such as 45 degrees to the line of sight. A light source is placed at an angle, such as 90 degrees to the line of sight to the camera, and directly facing the two-way mirror. Conversely, the light source could project through the twoway mirror and the camera could view through the reflection on the two-way mirror.

A retroreflector within the said object space and will be exposed within the view of the video camera will appear illuminated by the light from the light source. This retroreflector could be especially bright compared to the other objects in the video scene. It would then be possible to set a threshold in the computer program analysis of the video image where only the light of the intensity of the retroreflector would be recognised for triggering a computer command.

With a colour video system it would be possible to recognise different colours, such as red, green and blue. The computer program could respond selectively to one colour, such as red, as the indicator for movement of the computer cursor. A second colour, such as green, could be used to trigger a selection in the same manner as a mouse click. A third colour, such as blue, could be used to trigger other functions.

A software program will be used to analysis the video images so that the groups of pixels illuminated by the retroreflectors are associated with standard computer commands, such as the movement of a cursor and a selection of a function.

In settings, such as an office meeting room or a family room in a home, the use of retroreflectors could allow the users to control a computer program or interactive television program from any location that is within the view of the video camera. One advantage of this system is that the retroreflective units can be inexpensively replaced compared to the cost of remote control devices. Also, the retroreflectors do not have the cost of batteries, which are required for most remote control devices.

A set up function can be established so that the user can define the desired work area. Upon the initial detection of a retroreflector within the view of the video camera, the computer program will display a set up function for the user. The user will move the retroreflector to define the area of movement to cover the full computer screen. This 6 work area could be changed at any time to allow for interaction from another user in a different part of the room.

The use of a retroreflector can compliment computer programs that are based on responses to movement within a video frame. Some computer programs generate predetermined computer responses to movement in certain parts of the video frame. As an example, interactive music programs have been developed that generate low notes from movement in the lower part of the video frame and high notes from movement in the upper parts of the video frame. The retroreflectors could be used to select specific functions which otherwise might have been too difficult to isolate from the general movement in the video image. These specific selections could be choices of audio effects, instruments or operational controls for the computer program.

Another aspect of this human computer interface is that the computer programs could be written to respond to more than one computer input simultaneously. Standard computer interfaces using a mouse or another form of navigation device are limited to controlling the position of a single cursor. While this works well for a single user it does not allow a second user to interact with the computer program at the same time. In many applications involving a team of people working on a project or a group of people interacting with a public exhibition, it would be an advantage for several people to interact with various aspects of the overall computer program. Since the video image could display more than one image of a retroreflector at one time, it is possible to have the computer program respond to each of the retroreflector images simultaneously. As an example, one user could make selections for creating music from one instrument displayed on the computer program while a second person could make selections to create music from a second instrument. The sound track would feature the music of both instruments. This same approach of multiple inputs can be applied to the creation of visual displays. In more industrial applications, a large control panel could receive simultaneous input from a number of operators.

All of the preceding embodiments of the invention are based on the user's movement of a retroreflector being transposed to a movement of a cursor on a video/computer 7 monitor or screen. In another embodiment of the invention, the user would see a video image of themselves on the video/computer monitor or screen so that they could more intuitively transpose their movement to the movement on the screen to make selections. The users would also see the images of the retroreflectors as they are used to trigger functions in the computer program.

To give the user a more involving experience of the computer program the display of the computer image can be on a large screen in front of the user. The users could see life-size images of themselves within the large computer display screen.

In a more advanced embodiment of the invention a large two-way mirror is positioned between the users and the rig containing the video camera and light source. The large two-way mirror is angled so that the users see a reflection of a large video screen displaying life-size images of themselves within the computer program. The configuration of the two-way mirror and the display screen is angled so that the reflected images appear to be directly in front of the users. In this way the users will see video images of themselves that appear to be mirror images. Since the video camera is behind the two-way mirror it can be positioned to be on a central axis with the display screen so that it can capture the images of the users without any perspective distortion. The large two-way mirror makes it possible for ' the camera to capture images of the users without blocking the centre of the screen.

An advantage of the system using the large two-way mirror is that the plane of the reflected video/computer image can appear to be closer to the viewer than the background plane that will be visible through the two- way mirror. In this configuration the users will see the images of themselves and any other video or computer generated images on a plane in front of a background. The images of the users superimposed with a background will give the perception that their reflected images are within a three dimensional setting where the computer program can be controlled.

8 In an more comprehensive computer programme the general movement of the users within the field of view can be analysed by the computer processor for responses that are in addition to the specific computer functions selected by the retroreflectors.

9 Specific embodiments of the invention will be described by way of example with reference to the accompanying drawings in which:

Fig. 1 shows the invention with a person in the field of view of a video camera and the coverage of a light with a close alignment

Fig. 2 shows the invention with a mirror close to the video camera to reflect light from a light source Fig. 3 shows a two-way mirror between the video camera and the light source so that they are aligned to share the same centre line Fig. 4 shows a retroreflector on a finger using an adhesive Fig. 5 shows a retroreflector on a finger using a thimble shaped device Fig. 6 shows a retroreflector on a finger using an adjustable strap Fig. 7 shows a clasp hand with two retroreflectors that are not within view of the video camera Fig. 8 shows a hand with one retroreflector in view of the video camera Fig. 9 shows a hand with two retroreflectors in view of the video camera Fig. 10 shows a finger 2 with a first coloured retroreflector 9 on the front of the tip of the finger and a second retroreflector of a different colour 1 on the back of the finger Fig. 11 shows a hand 15 with finger 2 curled down to expose retroreflector 1 Fig. 12 shows a hand 15 with finger 2 held up to expose retroreflector 9 Fig. 13 shows a person in the video image combined with one of three active zones highlighted by one retroreflector Fig. 14 shows a person in the video image combined with one of three active zones highlighted by one retroreflector and selected by a second retroreflector of a different colour Fig. 15 shows a hand held device with a retroreflector on one side Fig. 16 shows a hand held device with a different coloured retroreflector on the other side Fig. 17 shows a hand held device with a single retroreflector on display Fig. 18 shows a hand held device, which had been pressed to display a second coloured retroreflector Fig. 19 shows the invention within a rig that displays the video image of the user in a three dimensional setting Referring to the drawing Fig. 1 a user 3 stands in a position in the field of view of a video camera 4. A light source 6 is positioned close to the video camera 4 so that they are both close to the same centre line. The light from the light source 6 shines on the retroreflector which reflects light directly back toward the light source 6. Since the light source 6 is close to the video camera 4, the light will also pass through the taking lens of the video camera 4. The movement of the retroreflector 1 will be displayed on the monitor 7 so that the user 3 can transpose his/her movement to the movement of a cursor or other functions on the computer program.

In Fig. 2 a mirror 17 is placed close to the taking lens of the video camera 4. The mirror 17 is placed at 45 degrees to the centre line of the receiving lens of the video 11 camera 4 so that the centre line of the projection of light from the light source 6 facing the mirror 17 is aligned at 90 degrees to the centre line of the receiving lens of the video camera and the centre line of the mirror reflection of the projection of light is closely aligned with the centre line of the receiving lens of the video camera so that the retroreflected light enters the receiving lens of the video camera 4.

In Fig. 3 a user 3 stands in a position in the field of view of a video camera 4, which is behind a two-way mirror 5. A light source 6 projects light onto a two-way mirror 5, which is angled so that the light reflects to cover the area of the user 3. The light from the light source 6 shines on the retroreflector which reflects light directly back toward the light source 6. Since the light source 6 and the video camera 4 are aligned on the same axis, the light will also pass through the two-way mirror directly to the video camera 4.

In Fig. 4 a retroreflector 1 is held in position on the front of afinger 2 by an adhesive backing on the retroreflector.

In Fig. 5 a thimble shaped device 13 is placed on a finger 2. The device 13 has a retroreflector 1 on one side.

Fig. 6 shows a band or ring 14, which holds a retrorefiector 1 on one side of a finger 2. lle band or ring 14 could be adjustable.

Fig. 7 shows a clasp hand 15 with fingers 2 and 8 blocking the view of the retroreflectors.

Fig. 8 shows a hand 15 with one finger 2 extended to expose a clear view of the retroreflector 1.

In Fig. 9 a second finger 8 is extended so that a different coloured retroreflector 9 is exposed in clear view.

12 Fig. 10 shows a finger 2 with a first coloured retroreflector 9 on the front tip of the finger and a second retroreflector of a different colour 1 on the back tip of the finger. The method of holding the two retroreflectors can be any of the devices illustrated in Figs. 7, 8 or 9.

Fig. 11 shows a hand 15 that is clasp so that the retroreflector 1 on finger 2 is exposed _to the said light source and the said imaging device.

Fig. 12 shows a hand 15 with the finger 2 held up so that the retroreflector 9 is exposed to the said light source and the said imaging device.

Fig. 13 shows one embodiment of the invention where a monitor 7 displays the computer program where the user 3 is shown as a video image in combination with a set of interactive buttons 10, 11 and 12. In this illustration the user 3 has exposed a clear view of a retroreflector 1 in the position of interactive button 10, which has created a bright reflection on the video image in this position to trigger the computer program to highlight this interactive button.

Fig. 14 shows the user exposing a second retroreflector 9 in the position of the highlighted interactive button 10 which will cause a bright light of a different colour to be viewed by the video camera so that the computer software program will read this as a command to trigger the function designated to the interactive buttonIO.

In Fig. 15 a hand 15 is holding a device 20 that has a retroreflector 1 attached to one side. The device 20 can be a card, stick or other handheld object that is suitable for holding the retroreflector 1.

Fig. 16 is the same device 20 rotated 180 degrees on its vertical axis so that a different coloured retroreflector 9 is displayed.

In Fig. 17 a handheld device 21 has a retroreflector 1 displayed. A different coloured retroreflector 9 is not in view because it is behind a cover 22 for the device 21.

13 Fig. 18 is the same device 21 with the cover 22 moved so that the different coloured retroreflector 9 is displayed.

In Fig. 19 the present invention is shown in a rig using a large two-way mirror 30. The user 3 stands in the field of view of the video camera 4 and displays a retroreflector 1 that reflects light from the light source 6. The video camera 4 picks up the image of the user 3 against a chromakey backdrop 34. The colour of the chromakey is dropped out in the video image so that the image of the user 3 is displayed against a black background. A video projector 33 projects onto a mirror 32 that reflects the projection to a screen 3 1. A two-way mirror 30 is positioned so that the reflected image of the screen 31 is viewed is position 35. By making the video image of the user at life size the user will see an image of himself/herself that appears to be like a mirror image. The user will move so that the video image of retroreflector 1 moves into position to trigger interactive buttons with the retroreflectors 1 and 9 for operating the computer program.

14

Claims

1. A control arrangement comprising:

an imaging device that forms an image of an object space; a light source that illuminates said object space a retroreflector capable of being moved within said object space and reflecting said light source such that the retroreflected image is detected by said imaging device; and a means to effect a control operation in dependence on the position and/or colour of said retroreflected image.

2. A control arrangement according to claim 1 in which said imaging device comprises a video camera.

3. A control arrangement according to claims I and/or 2 in which said image formed by said imaging device is displayed on a monitor or screen visible to a user widiin said object space.

4. A control arrangement according to any of the claims 1 to 3 in which said retroreflector is illuminated by a configuration of said light source and said imaging device such that the centre line of the projection of light and the centre line of the receiving lens of said imaging device are closely aligned by using a mirror positioned close to the receiving lens of said imaging device so that the said light will reflect light onto the said object space.

5. A control arrangement according to any of the claims 1 to 3 in which said retroreflector is illuminated by a configuration of said light source and said imaging device such that the centre line of the projection of light and the centre line of the receiving lens of said imaging device are closely aligned by using a two-way mirror.

6. A control arrangement of any of the claims 1 to 5 in which said control operation is mediated through a computer processor which is programmed to recognise and respond to the position and/or colour of said retroreflective images.

7. A control arrangement according to claim 6 in which a user can see a visualisation generated by said computer processor and displayed on a monitor or screen such that the user can transpose his/her movement of the retroreflector in the object space to the relative position of the image of the retroreflector displayed on the said monitor or screen to control functions within the programmed domains of said computer processor

8. A control arrangement according to claims 1 to 7 in which a plurality of retroreflectors are employed in the said object space such that the said imaging device captures the retroreflected images and the said computer processor recognises and responds to all of the retroreflected images respectively in the programmed domain to perforin more than one discrete function simultaneously and/or to perform functions based on the quantity and movement of the said plurality of retroreflected images.

9. A control arrangement according to claims 1 to 8 in which said computer processor can recognise the colour of a retroreflected image and differentiate between two or more colours to perforin different functions corresponding to colour of the retroreflected image.

10. A control arrangement according to claims 1 to 9 in which the cursor for controlling the computer program of the said computer processor is operated by the movement of one specific colour of retroreflector in the said object space and the mouse click function is operated by the display of a second 16 specific colour of retroreflector either simultaneously with the colour for the cursor as an alternation with the colour for the cursor.

11. A control arrangement according to any of the claims 1 to 10 in which said retroreflector is attached to a finger of a user with an adhesive substance or device.

12. A control arrangement according to any of the claims 1 to 10 in which said retroreflector is attached to a device that fits onto a finger of the user.

13. A control arrangement according to any of the claims 1 to 10 in which said retroreflector is strapped onto a finger of said user.

14. A control arrangement according to any of the claims 1 to 13 in which a retroreflector of one colour is attached to the front of the tip of one finger and a second retroreflector of a different colour is attached to the back of the tip of the same finger so that the user can expose the first coloured retroreflector with the finger curled down and expose the second coloured retroreflector with the finger held up.

15. A control arrangement according to any of the claims 1 to 10 in which said retroreflector is a rotatable device that exposes different coloured retroreflectors attached to opposite faces of said rotatable device to imaging device.

16. A control arrangement according to any of the claims 1 to 10 in which said retroreflector is a device that exposes different coloured retroreflectors according to movement of a slidable cover.

17. A control arrangement according to any of the claims 1 to 16 in which the user can see an image of himself/herself in combination with the image of the said retroreflector on the said monitor or screen so that he/she can more 17 intuitively transpose their movement to the movement of the image of the retroreflector displayed on the monitor or screen.

18. A control arrangement according to any of the claims 1 to 17 in which a large two-way mirror is positioned between the user and the said imaging device and said light source so that the two-way mirror is angled to reflect the said monitor or screen displaying the image of the user and the said retroreflectors in a way'that the user will look forward to see the reflected image of himself/herself directly in front as a video generated mirror image of himself/herself so that he/she can intuitively interact with the program domain of the computer processor as a life-size image of himself/herself.