WO1998014798A1

WO1998014798A1 - System for 3d tracking of a remote point

Info

Publication number: WO1998014798A1
Application number: PCT/CA1997/000724
Authority: WO
Inventors: Will N. Bauer; Rafael Lozano-Hemmer
Original assignee: Bauer Will N; Lozano Hemmer Rafael
Priority date: 1996-10-02
Filing date: 1997-09-30
Publication date: 1998-04-09
Also published as: CA2186994A1; AU4448197A

Abstract

The invention concerns a system for and a method of three dimensional tracking to establish the three dimensional position and/or orientation of an object to be tracked. The method includes the steps of transmitting pulses between a pulse emitter/receiver (24) and a trackable devie (26) associated with the object to be tracked, utilizing lock on and follow means (24, 13, 14) and ranging means (24, 20). The pan/tilt angles between and the distance between the emitter/receiver (24) and the device (26) are obtained for each pulse and the three dimensional position/orientation is calculated for this data.

Description

SYSTEM FOR 3D TRACKING OF A REMOTE POINT

Technical Field

This invention relates to a system for and to a method of 3D tracking which can be used to establish three dimensional position, velocity and/or orientation of tracked objects.

Background Art 3D position and orientation information may be used to operate and control a wide variety of "variably operable systems" for example as described and claimed in U.S.

Patent No. 5,107,746 issued April 28, 1992 to Bauer; U.S.

Patent No. 5,214,615 issued. May 25, 1993 to Bauer; and U.S. Patent No. 5,412,619 issued May 2, 1995 to Bauer.

Disclosure of the Invention

According to the invention a tracking method to establish the three dimensional position and/or orientation of an object to be tracked is characterized by the steps of: transmitting a continuous modulated signal or signal pulses between a trackable device associated with an object to be tracked and a signal emitter/receiver mounted on a platform in line of sight of the trackable device; panning and tilting the platform or a reflecting surface to orientate the relative positions of the signal emitter/receiver and the trackable device whereby the emitter/receiver points continuously at the trackable device; locking on and following the trackable device to find, for each pulse or signal modulation the angular position, i.e. azimuth (pan) and its elevation (tilt) relative to the initial orientation; measuring the time taken by each pulse or signal modulation between the trackable device and the signal emitter/receiver.

Also according to the invention there is provided a tracking system to establish the three dimensional position, velocity and/or orientation of an object to be tracked by transmitting a continuous modulated signal or signal pulses between a trackable device associated with the object to be tracked and a signal emitter/receiver mounted on a platform; characterized in that it comprises pan/tilt control means to turn the platform or reflecting surface so that it follows changes in the position of the trackable device so that the signal emitter/receiver continuously points at the trackable device; lock on and follow means to find and follow the angular position of the trackable device, i.e. azimuth (pan) and elevation (tilt) relative to the initial orientation; and ranging means for measurement of distance between the platform and the trackable device.

The "lock on and follow" means locate a tracking sensor "point" and provide signals allowing said pan/tilt motors to position a platform in such a way as to follow the motion of a point or associated set of points as they move through 3D space. The pan/tilt tracking means may use high precision motors to measure extremely small angular resolutions. The ranging means allow for the measurement of the distance from said moving platform to the point (s) being tracked. There are a variety of technologies that can be used to embody each of these three sub-systems. By combination of these three technologies it may be possible to achieve wireless tracking of the 3D orientation and position of a large number of points at high sampling rates over large areas (radii of up to 20m or more around each tracker assembly) .

In practice both a "lock-on and follow" sensor and a ranging sensor may be mounted on a small platform whose movement along two orthogonal axes is controlled by high resolution pan/tilt motors such as those found in intelligent lighting units. These motors are digitally controlled, such control being exercised by a conventional micro-controller. Alternatively, said sensors may be mounted in a fixed position while a reflecting surface such as a mirror is turned by the motors. The surface reflects signals to/from the sensors and directs and receives them from angular coordinates depending on the surfaces pan/tilt orientation.

The lock-on sensor finds the initial location and follows, by controlling the pan/tilt movements of the platform, the motion of one or more tracking point sensors which are carried by an object as it moves about within the tracking area. Due to the pan/tilt calculations needed for this "following" behaviour, the pan and tilt angles of the platform are known at any given moment. The distance from the platform to the point or associated set of points being tracked may be determined using the ranging sensor. Given these three pieces of information, i.e. the range from the platform to a tracking point; the orthogonal pan and tilt angles of the platform when it is oriented directly toward the tracking points, the 3D position relative to the platform can be calculated.

If one tracks the 3D positions of a set of at least three associated points mounted on a rigid planar structure such as a sheet of plastic, it is possible to calculate the 3D angular orientation (pitch, yaw, and roll angles) of said planar structure. The technique described above forms the basis of the inventive tracking of 3D position and orientation of moving objects. A discussion of the three "base" technologies and their relation to possible embodiments of the invention follows:

Pan/Tilt Platform Movement Normally, so-called "intelligent" lights use their pan/tilt motors to position a moving mirror which reflects the light coming from a non-moving bulb, allowing the reflected beam of light to point in different directions over a wide range of angles. These lights are commonly used in the entertainment industry and are increasingly being used in other areas such as marketing and promotional fields. The utility of this technology in the present application is that the pan/tilt control allows extremely precise two axis angular control of a platform comprising a mounting bracket for mounting the range and lock/follow sensors. Such a platform may replace the mirror of the intelligent light system. Alternatively, the mirror may be used to reflect signals to/from sensors mounted in a fixed position. Using this technology, angular resolution of at least 360 degrees in 65535 steps (0.00549 degrees per step) are possible. This allows very precise determination of where the platform is pointing along each of its orthogonal pan and tilt axes and thus allows great precision in angular measurements even when the point (s) being tracked are relatively far from the tracker assembly.

Lock-On and Follow (L.O.A.F.) Sensors There are a variety of sensor technologies that can be used to generate the pan/tilt signals necessary to find and follow a point or associated set of points. These technologies may or may not also provide the ranging function as part of their functionality. Some examples of L.O.A.F. sensors are as follows: a) Infrared emitter/reflector/receiver. In this situation, an infrared light emitter and receiver are mounted on the tracking platform and a reflector is mounted on the point to be tracked. Modulated pulses or a continuous modulated beam of light are/is sent from the emitter and, when the pan/tilt motors are positioned properly, the reflected modulated light is detected by the receiver. The receiver may be a rectangular array of detectors such as a charged coupled device (CCD device) and, depending where on the array the reflected light falls, a micro-controller may generate signals for the pan/tilt motors to position the platform so that the reflected light is in the centre of the rectangular array. Alternatively, the receiver can be a single "narrow beam" receiver element which may only sense light coming along a narrow beam collinear with the receiving element. In this case, the pan/tilt motors will be positioned properly by the micro-controller when the received light is at maximum intensity in the receiver i.e. when the receiver and the point are collinearly aligned.

b) Infrared receiver/emitter. A variation on the same basic technology is possible where an infrared emitter is attached to each point being tracked. This emits modulated infrared light in a manner such as described in a) above. A receiver element, again such as those mentioned in a), receives the infrared light and a micro- controller generates pan/tilt signals to position the platform so that it is centred on the point(s) being tracked.

c) Infrared emitter/receiver/transducer. It is also possible to use a system whereby light, modulated as in a) and b) above, is emitted from a sensor on the platform, received by a receiver located on the tracking point, converted into some other transmission medium (radio waves or ultrasonic sound for example) or a different coding of the same medium, and telemetered back to the digital signal processor in charge of the L.O.A.F. positioning and 3D calculations .

d) It is also possible to lock on and follow using either laser or longer wavelength electromagnetic wave emitters/receivers in a manner analogous to those discussed in a), b), and c) above but transposing light with longer electromagnetic waves such as radio. This transposition may also be done using acoustic waves in place of light.

In order to precisely align the platform to point directly at the point(s) being tracked, it is possible to use a source with a narrow beam pattern where maximum signal strength is obtained when the platform is pointed directly at the receiver and where mis-alignment of the platform can be continuously monitored by comparing the current received amplitude with the most recent local maximum. Alternatively, it is possible to use three or more sources with overlapping beam patterns and align the platform so that all three sources generate the same amplitude (indicating that the receiver is situated equidistant from the three sources in the centre of the triangle formed by the sources).

It is also possible to spatially and temporally modulate a signal of wider beam width so that the signal's modulation contains positional information. This could be done by, for example, placing a rotating filter (cut in a specific geometric shape) in front of the beam of light. Still further, it is possible to have a spatially distributed array of receivers which can generate information about the position of a tracked point relative to a centre origin.

Ranging Sensors

There are also a variety of technologies which may be used to determine the distance between the pan/tilt platform and the point(s) being tracked. These technologies can be similar in form to those mentioned above with regard to L.O.A.F. activity. Some examples are as follows: a) Infrared emitter/reflector/receiver: In this situation, an infrared light emitter and receiver are mounted on the tracking platform and a reflector is mounted on the point to be tracked. Modulated pulses or a continuous modulated beam of light are/is sent from the emitter and, when the pan/tilt motors are positioned properly, the reflected modulated light is detected by the receiver. Either time-of-flight or phase measurements can be used to determine the distance between the platform and the point(s) being tracked. The receiver can be a rectangular array of detectors capable of detecting light coming from a variety of angles. Alternatively, the receiver can be a single "narrow beam" receiver element which can only sense light coming along a narrow beam collinear with the receiving element. b) Infrared emitter/receiver. A variation on the same basic technology is possible where an infrared emitter is attached to each point being tracked. This emits modulated infrared light in a manner such as described in a) above. A receiver element, again such as those mentioned in a), receives the infrared light and a microcontroller generates pan/tilt signals to position the platform so that it is centred on the point(s) being tracked.

c) Infrared emitter/receiver/transducer. It is also possible to use a system whereby light, modulated as in a) above, is emitted from a sensor on the platform, received by a receiver located on the tracking point, converted into some other transmission medium (radio waves or ultrasonic sound for example) or into a different coding scheme in the same medium, and telemetered back to the micro-controller in charge of the platform orientation and 3D calculation.

d) Use of other spectra. It is also possible to range using either sound or laser or longer wavelength electromagnetic wave emitters/receivers in a manner analogous to those discussed in a), b) and c) above but transposing light with sound or longer electromagnetic waves such as radio.

Brief Description of the Drawings

A specific embodiment will now be described by way of example with reference to the drawings in which: Figure 1 shows one system according to the invention;

Figure 2 shows another system according to the invention;

Figure 3 shows yet another system according to the invention; Figure 4 shows still yet another system according to the invention; and

Figure 5 shows a flow chart showing digital signal processor activities. Modes of Carrying Out the Invention

The drawings show a system based on ultrasonic and radio technology. In Figure 1 a tracker assembly 12 including a pulse emitter/receiver 24 having a tiltable platform 13 may be mounted relatively high up (perhaps 3m or more for a 6m or more tracking radius) above the centre of the area in which 3D tracking is desired. The reason for mounting the system above the object(s) being tracked is to ensure that a clean line of sight is maintained between the tracker assembly and the tracked points as much as possible. In cases where such mounting is difficult, it may be possible to use several tracker assemblies to ensure that at least one has a clear line of sight to the tracked points at all times.

The tracker assembly 12 may be comprised of an "intelligent" light fixture such as a Martin PAL1200 with its mirror replaced by a platform containing a narrow-beam ultrasonic speaker. This fixture provides a built-in pan/tilt micro-controller 14 which can be controlled via a digital signal processor 34. The range of motion of the platform is reasonably wide, about 150° for pan and 70° for tilt.

The digital signal processor 34 generates ultrasonic pulses of about 22KHz and sends them through a D/A converter 20, an amplifier 22, and a speaker 24. The pulses then travel through the air to a microphone 26 connected to a radio transmitter 27 which radios them to a radio receiver 28 which is connected to a filter/amplifier 33 and an A/D converter 29 and finally back to the digital signal processor 34.

Since the pulses are ultrasonic, it should be noted that normal speech can also be taken in from audible sound emitter 31 through a trackable device 26 which, in Figure 1, is a wireless microphone 26, simultaneously with the ultrasound (since audible sounds are at different frequencies from the ultrasonic pulses). This speech is radioed to the receiver 28 where it is sent to a public address amplifier/speaker system 32 to be heard by others in the room. In order to remove the possibility of audible "sideband" noise from the ultrasonic pulses being broadcast through the public address 32, it may sometimes be necessary to include a low pass filter 30 to remove some of the energy of the ultrasonic pulses from the public address signal. Thus the signal received by the radio receiver is split with one branch of the signal going through a low pass filter 30 to the public address system 32 and another branch going to the input of the digital signal processing sub-system 34.

The incoming ultrasound pulses from radio receiver 28 are digitized by the A/D converter 29 and then analyzed by the digital signal processor 20 to calculate the range between the ultrasonic speaker 24 and the microphone 26. This can be done by measuring the time-of-flight of the pulse between when it was sent from the speaker 24 and when it was detected by the digital signal processor 20. The speed of sound is roughly 30 cm per millisecond so timing of these durations to within a few microseconds yields range measurements that are precise to within at least a few centimetres. It may also be possible to use phase measurements of the incoming pulses to improve this accuracy even further.

In order to find the location of the microphone 26, the digital signal processor 34 must first direct the micro-controller 14 to scan the available area using pan/tilt indicated by double headed arrows X and Y motors until the pulses are picked up by the microphone 26.

Once the pulses are being detected, the pan/tilt settings can be further refined by examining the incoming pulse amplitude and how it changes with small variations of pan and tilt angle. The local maximum of this restricted search will yield the "true" pan and tilt settings which align the speaker 24 to be pointed directly at the microphone 26. As the microphone 26 is moved, its received pulse amplitude will lessen as it moves away from the centre of the speakers 24 beam pattern. By constantly making slight changes in the pan/tilt angle of the speaker 24, the digital signal processor 34 can ascertain which direction of pan/tilt movement restores the received pulse amplitude (thus realigning the speaker 24 with the microphone 26).

A typical pulse frequency might be around 22KHz and might have a duration of about 1ms . Up to a thousand of these pulses per second may be fired and analyzed so the maximum position update frequency could also be about one thousand per second. Alternatively, the energy of a number of pulses may be summed together in order to reduce timing errors or be able to use pulses with a lower amplitude or to overcome the directional pick-up pattern of the wireless microphone. This might result in an overall positional update rate of thirty to sixty hertz.

The system of Figure 2 differs from that of Figure 1 in that the emitter/receiver 24 comprises an infrared transmitter 24A and an infrared receiver or receiver array 24B and the trackable device 26 is an infrared reflector 26A. Reflected pulses go to amplifier/filter 33 and are processed similarly to those of the embodiment of Figure 1.

The system of Figure 3 differs from that of Figure 1 in that the emitter/receiver 24 comprises an infrared receiver (or receiver array) 24C and the trackable device 26 comprises an infrared emitter 26B. Pulses received by receiver 24C go to amplifier/filter 33 and are processed as previously described.

The system of Figure 4 differs from that of Figure 3 in that, before delivery to amplifier/filter 33 pulses from IR receiver 26C are passed to transducer 37 and resulting telemetering is passed to telemetry transmitter 27. Remote telemetry receiver 28 receives the pulses which then go to amplifier/filter 33 for processing as previously described.

Figure 4 is a flow chart showing the operation of the digital signal processor for all described embodiments.

The 3D position/orientation information may be calculated by the digital signal processor 34.

Calculation of the 3D Position of the Tracked Point

Beginning with a light whose initial rotation, pan, and tilt orientation angles are θ, γ and φ, a 3 space coordinate system x" , y" , z" may be defined such that these axes are turned with the light and are positioned at orientation angles θ, γ, and φ. Conceptually, it is as if the starting point was with the light having θ = γ = φ = 0 (where, by definition x"=x, y"=y, and z"=z) a rotational transformation having been performed, the light having been taken through three rotations in a particular sequence. The transformation equations between (x,y,z) coordinates and (x",y",z") coordinates as rotations are applied in this order :

a) Rotation about z axis: x' = xCos(-θ) + ySin(-θ) y' = -xSin(-θ) + yCos (-Θ) b) Rotation about x' axis: y" = y' Cosγ + zSin γ z' = -y'Sinγ + zCos γ c) Rotation about y' axis: x" = x'Cosφ + z'Sinφ z" = -x'Sinφ + z'Cosφ

Now it is necessary to back calculate the (x,y,z) coordinates of the tracked point given initial angular orientations θ_{i r} γ_{i r} φ₄ plus pan angle β, tilt angle ψ and range R.

The first step is to calculate the position of the tracked point in (x",y",z") coordinates using the following equations:

TanB = x"

Tanψ

R² = x" ² + y" ² + z " ²

With some allowances for choosing the right trigonometric quadrants to get the signs right, these three equations can easily be solved to yield values for x", y" , and z" .

Back-rotating the (x",y",z") coordinates yields the equivalent (x,y,z) coordinates. This must be done in the reverse order to the conversion from (x,y,z) to (x",y",z") using the following transformations in this order:

a) Rotation about the y" axis: x' = x"Cosφ - z"Sinφ z' = x"Sinφ + z"Cosφ b) Rotation about the x' axis: y' = y"Cosγ - z'Sinγ z = z'Coεγ + y"Sinγ c) Rotation about the z axis: y = x'Sin(-φ) + y'Cos(-φ) x = x'Cos(-φ) - y'Sin(-φ)

This yields the 3D (x,y,z) coordinates of the tracked point.

Once known, the 3D position/orientation information could be used to control intelligent lighting fixtures via the DMX or other industry standard protocols in order to make them respond to the movements of the holder of microphone 26. Such controlled apparatus 36 is generally referred to as Electronic Media and Show Control Systems. These responses could include making lights "follow" the microphone holder as they moved about so that they would always be illuminated, controlling various parameters of the light such as colour or gobo setting/rotation speed in real-time, or triggering preset cues to send lights to established positions. The position/orientation information could also be mapped to control other media such as security systems, electronic music, etc., via the M.I.D.I. protocol, computer generated video, laser disk or CD ROM playback, interfaces to mechanical equipments, virtual reality systems, or any other electronic media over which real-time control was desired in a manner similar to that described in U.S. Patent No. 5,107,746, U.S. Patent No. 5,214,615 and U.S. Patent No. 5,412,619, previously referred to.

Claims

CLAIMS :

1. A method of tracking to establish the three dimensional position, velocity and/or orientation of an object to be tracked characterized by the steps of: transmitting a continuous modulated signal or signal pulses between a trackable device (26) associated with an object to be tracked and a signal emitter/receiver (24) mounted on a platform (13) in line of sight of the trackable device (26) ; panning and tilting the platform or a reflecting surface (13) to orientate the relative positions of the signal emitter/receiver (24) and the trackable device (26) whereby the emitter/receiver (24) points continuously at the trackable device (26) locking on and following the trackable device to find, for each pulse or signal modulation the angular position, i.e. azimuth (pan) and its elevation (tilt) relative to the initial orientation; and measuring the time taken by each pulse or signal modulation between the trackable device and the pulse emitter/receiver .

2. A method as claimed in claim 1 characterized by the step of emitting pulses from the emitter/receiver (24) and detecting received pulses at the trackable device (26) .

3. A method as claimed in claim 1 characterized by the step of emitting pulses from the trackable device (26A) and detecting received pulses at the emitter/receiver (24C) .

. A method as claimed in claim 1 characterized by the step of emitting pulses from the emitter/receiver (24A, 24B) , reflecting pulses from the trackable device (26B) and detecting received pulses at the emitter/receiver.

5. A method as claimed in claim 2 characterized by the step of transmitting detected pulses at the trackable device (26) by radio to a radio receiver (28) at a location spaced from the trackable device.

6. A method as claimed in claim 1 in which the pulses are pulses of infrared radiation.

7. A method as claimed in claim 1 in which the pulses are ultrasound pulses.

8. A system for tracking to establish the three dimensional, velocity position and/or orientation of an object to be tracked by transmitting a continuous modulated signal or signal pulses between a trackable device (26) associated with an object to be tracked and tracking means therefor, characterized in that the tracking means comprises:

a signal emitter/receiver mounted on a platform; pan/tilt control means (14) to turn the platform or reflecting surface (13) so that the pulse emitter/receiver follows changes in the position of the trackable device (26) to continuously point at the trackable device; lock on and follow means (24, 13, 14) to find and follow the angular position of the trackable device, i.e. azimuth (pan) and elevation (tilt) relative to the initial orientation; and ranging means (24, 20) for measurement of distance between the platform (13) and the trackable device (26) .

9. A system as claimed in claim 8 characterized in that a radio transmitter (27) is provided for pulses from the trackable device (26) and a radio receiver (28) is provided at a location remote from the trackable device to receive radio signals therefrom.

10. A system for tracking to follow a moving object to be tracked with an aimable device comprising a trackable device associated with the object to be tracked and tracking means therefor, characterized in that the tracking means comprises :

a signal emitter/receiver mounted on a platform; pan/tilt control means (14) to turn the platform or reflecting surface (13) so that the pulse emitter/receiver follows changes in the position of the trackable device (26) to continuously point at the trackable device; lock on and follow means (24, 13, 14) to find and follow the angular position of the trackable device, i.e. azimuth (pan) and elevation (tilt) relative to the initial orientation; and an aimable device programmed from said angular position and elevation to aim at the object to be tracked.