Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N5/00—Details of television systems
  • H04N5/74—Projection arrangements for image reproduction, e.g. using eidophor
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/70—Determining position or orientation of objects or cameras
  • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/10—Image acquisition
  • G06V10/12—Details of acquisition arrangements; Constructional details thereof
  • G06V10/14—Optical characteristics of the device performing the acquisition or on the illumination arrangements
  • G06V10/145—Illumination specially adapted for pattern recognition, e.g. using gratings
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/10—Image acquisition
  • G06V10/12—Details of acquisition arrangements; Constructional details thereof
  • G06V10/14—Optical characteristics of the device performing the acquisition or on the illumination arrangements
  • G06V10/147—Details of sensors, e.g. sensor lenses
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N9/00—Details of colour television systems
  • H04N9/12—Picture reproducers
  • H04N9/31—Projection devices for colour picture display, e.g. using electronic spatial light modulators [ESLM]
  • H04N9/3179—Video signal processing therefor
  • H04N9/3185—Geometric adjustment, e.g. keystone or convergence
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N9/00—Details of colour television systems
  • H04N9/12—Picture reproducers
  • H04N9/31—Projection devices for colour picture display, e.g. using electronic spatial light modulators [ESLM]
  • H04N9/3191—Testing thereof
  • H04N9/3194—Testing thereof including sensor feedback

Definitions

• the invention relates generally to computer vision, and more particularly to determining the position and orientation of an object using a projected image.
• Position sensors are useful for many applications in which it is necessary to dete ⁇ nine the position and orientation of an object.
• the term 'position sensor' means both position and orientation sensor.
• the position is the location of the object in a three-dimensional coordinate frame, and the orientation of the object is its rotation relative to that coordinate frame.
• a position sensor can be attached to a moving camera that is being used for 3D scanning. The position and orientation of the camera are then known, to aid creation of a 3D reconstruction of a scene from the h ⁇ ages acquired by the moving camera.
• a position sensor can be attached to a projector to aid correct projection of an image onto a surface or an object.
• a position sensor can also be attached to a tool so that the tool can be properly positioned with respect to a part or a working surface. If a user is manipulating the tool, the position sensor enables haptic or other feedback, for example in tele-operation, or for computer-aided surgery.
• a position sensor can also be used to track the location of a moving object such as a vehicle or person.
• position sensors have been used in prior art applications is to place visual markers or active emitters, such as LEDs, in known positions in the environment.
• the position sensor can be a camera that observes the markers or emitters in order to infer its own position and orientation.
• Other applications use ultrasound instead of light.
• visual markers or active emitters can be attached to the object of interest so that the object's position and orientation can be determined directly using a camera observing the object.
• a system which uses LED emitters in the environment is described in "The HiBall Tracker: High-Performance Wide- Area Tracking for Virtual and Augmented Environments," by Welch et al, Proc. of the ACM Symposium on Virtual Reality Software and Technology, 1999.
• a system which uses ultrasound emitters in the environment is the IS-600 motion tracker by Intersense Corporation, Burlington, Massachusetts.
• a position and orientation sensor for an object, which works without having to modify the environment, and which does not require a complicated calibration.
• a position and orientation sensor that is compact, i.e., the sensor does not require markers or emitters that are distributed across the environment, or across the surface of an object of interest.
• a position and orientation of a projector are dete ⁇ nined from projected images. First, a homography between a camera and a planar surface, with the camera and planar surface in a fixed relationship, is determined. Next, a known pattern is projected on the planar surface using a projector having known intrinsic parameters and an unknown position and orientation.
• An image of the known pattern is acquired with the camera, and a position and orientation of the projector is determined from the image, using the homography and the intrinsic parameters.
• Figure 1 is a block diagram of a position and orientation sensing system according to the invention.
• Figure 2 is a flow diagram of a calibration process according to the invention.
• Figure 3 is a flow diagram of a position and orientation sensing method according to the invention.
• Figure 1 shows a position and orientation sensing system 100 according to the invention.
• the system includes a fixed camera 110 and a movable projector 120. Both the camera 110 and the projector 120 view a fixed, planar (2D) surface 130.
• the camera and the planar surface have a fixed relative position with respect to each other.
• the camera and projector can be on the same side of the surface, or alternatively, if the surface is translucent, the camera and the projector can be on opposite sides of the planar surface.
• the camera 110 and the projector 120 are digital devices. Analog devices, with A/D converters, are also possible.
• a processor (CPU) 140 is coupled to the camera 110.
• the processor 140 is equipped with a memory and I/O ports as are known in the art.
• the processor 140 is capable of executing a position and orientation sensing method 300 according to the invention, as described below in greater detail.
• a calibration procedure 200 of the system 100 has two stages 210-220.
• the second is optional.
• the fixed camera 110 is calibrated relative to the fixed planar surface 130 to determine a homography H 211 between a coordinate frame of an image plane of the camera 110, and a Euclidean (metric) coordinate frame of the fixed surface 130. This can be achieved, using well known geometric relationships, given four points 131 of known position on the fixed surface 130, e.g., the four corners of a square on the fixed planar surface 130.
• the homography 211 relates the coordinate frame of the camera 110 to the coordinate frame of the fixed surface 130 in the following way.
• an overall scale of the coordinate frame of the surface 130 is unknown, e.g., the four points 131 on the surface 130 are known to be the corners of a square but the size of the square is unknown, then the position and orientation of the projector 120 is determined up to an unknown scale in the subsequent processing. This is sufficient for many applications e.g. for 3D scanning of a scene in which the goal is to create a 3D reconstruction but the overall scale may be immaterial.
• the calibration procedure 200 determines the intrinsic parameters IP 221 of the projector 120. Determining the intrinsic parameters of a pinhole device, such as the projector 120, is a well- known process. Alternatively, the intrinsic parameters of the projector 120 can be determined while the system 100 is operating, as described in further detail below.
• the position sensing system 100 can be used for sensing the position and orientation of an object using the projector 120 and the method 300.
• the homography 211 between the camera 110 and the planar surface 130 is dete ⁇ nined 310, as described above.
• a known pattern 230 is projected 320 onto the planar surface 210 by the projector 120 having known intrinsic parameters 221, but an unknown position and orientation.
• the pattern 230 includes at least three distinct non-collinear points.
• the camera 110 acquires 330 an image 331 of the known pattern 230, and the features of the pattern 230 are measured in image coordinates of the projector 120.
• the homography 211 is used to transfonn features of the pattern from the coordinate frame of the image 331 to the coordinate frame of the planar surface 130.
• the position and orientation of the projector 120 are determined.
• the approach is an adaptation of a method described by DeMenthon et al., "Exact and Approximate Solutions of the Perspective-Three-Point Problem," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 14, no. 11, pp. 1100-1105, November 1992.
• the DeMenthon method is for computing the position and orientation of a camera with known intrinsic parameters observing a physical pattern with known coordinates.
• that method is adapted for the case of a projector which is projecting a pattern onto a plane, with a coordinate frame imposed on the projected pattern by means of a fixed camera observing the plane.
• the projected pattern can be adapted as necessary during operation, to provide the best possible view to the fixed camera(s).
• Any system that can be modeled as a pinhole projector is suitable for making the projection, e.g., three different-colored laser pointers with distinct beam directions and with the beams concurrent at a point, or with the beams non-concurrent but with calibration data to take account of the non-concurrency.
• the intrinsic parameters 221 of the projector 210 can be determined during operation of the system 100.
• the pattern 230 is projected for two different positions of the projector 220.
• the minimum requirements on the pattern in this case are that there are four distinct points in general positions, or any equivalent pattern.
• the projector's intrinsic parameters can be detenriined as follows. First, the homography H 211 is used to dete ⁇ nine coordinates of the projected pattern features on the planar surface 130. Then, a first homography Gl is dete ⁇ nined between the image plane of the projector 120, and the planar surface for a first position PI of the projector. Similarly a second homography G2 is computed for a second position P2 of the projector. Given Gl and G2, it is possible to dete ⁇ nine the intrinsic parameters 221 of the projector 120 as described by Zhang in "A flexible new technique for camera calibration," IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11): 1330-1334, 2000.
• That technique was developed for intrinsic parameters of a camera. However, we apply that technique to determine the intrinsic parameters of a projector.
• the operation 200 of the system 100 described above uses one fixed camera 110. If the projector 120 is moving across an extended area, so that the projected pattern is likely to move out of the field of view of any one camera, then multiple fixed cameras can be used, each observing a different part of the fixed planar surface 130. In this case, the calibration of the fixed cameras requires the dete ⁇ nination of multiple homographies HI, H2, etc., between the image plane of each camera, and a common coordinate frame of the fixed surface 130.
• the projector 120 can be moving inside a room, and the ceiling, floor, or walls provide the fixed planar surfaces.
• a wide-angle or omni-directional projector is projecting simultaneously on multiple surfaces for a wide field of view, all of the projection information can be used simultaneously for determining the position and orientation of the projector 120 to a greater accuracy.
• the projector 120 can be in a fixed relationship to an object 160 of interest. Then, the position and orientation of the projector can be used to infer a position and orientation of the object 160. If the object has its own coordinate frame, then the projector and object can be calibrated in another calibration step.
• the object 160 can be a second moving camera.
• the projector 120 can track the position and orientation of the camera 160. This is useful when the second camera 160 is used for 3D scanning.
• the inferred positions and orientations of second camera can be used to determine a 3D reconstruction from the scanned images.
• the object can be another projector.
• the second projector projects images into the environment according to the inferred position and orientation of the second projector. This is useful for augmenting virtual reality scenes, or active display areas.
• the object can also be a tool so that feedback dependent on the inferred position and orientation of the tool can be provided to a user.
• the feedback can be haptic.
• the projector can be attached to an object, which is moving in 2D, e.g., a car or robot, in which case only the position is relevant, and the orientation is irrelevant.

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Signal Processing (AREA)
• Theoretical Computer Science (AREA)
• Computer Vision & Pattern Recognition (AREA)
• General Health & Medical Sciences (AREA)
• Vascular Medicine (AREA)
• Artificial Intelligence (AREA)
• Health & Medical Sciences (AREA)
• Geometry (AREA)
• Length Measuring Devices By Optical Means (AREA)
• Image Processing (AREA)
• Studio Devices (AREA)
• Image Analysis (AREA)
• Projection Apparatus (AREA)

Priority Applications (3)

Applications Claiming Priority (2)

Publications (1)

Family

ID=32712151

Family Applications (1)

Country Status (6)

Cited By (1)

Families Citing this family (40)

Family Cites Families (11)

• 2003
  • 2003-01-17 US US10/346,442 patent/US7242818B2/en not_active Expired - Fee Related
  • 2003-12-25 EP EP03768261A patent/EP1483740B1/en not_active Expired - Fee Related
  • 2003-12-25 CN CNB2003801003098A patent/CN100343876C/zh not_active Expired - Fee Related
  • 2003-12-25 DE DE60323945T patent/DE60323945D1/de not_active Expired - Lifetime
  • 2003-12-25 JP JP2004567159A patent/JP4437748B2/ja not_active Expired - Fee Related
  • 2003-12-25 WO PCT/JP2003/016777 patent/WO2004066213A1/en active Application Filing

Non-Patent Citations (4)

Also Published As

Similar Documents

Legal Events