US20060116850A1 - Relative range camera calibration - Google Patents
Relative range camera calibration Download PDFInfo
- Publication number
- US20060116850A1 US20060116850A1 US11/263,810 US26381005A US2006116850A1 US 20060116850 A1 US20060116850 A1 US 20060116850A1 US 26381005 A US26381005 A US 26381005A US 2006116850 A1 US2006116850 A1 US 2006116850A1
- Authority
- US
- United States
- Prior art keywords
- data
- cameras
- time offset
- camera
- range
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Images
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
- G01C3/00—Measuring distances in line of sight; Optical rangefinders
- G01C3/02—Details
- G01C3/06—Use of electric means to obtain final indication
- G01C3/08—Use of electric radiation detectors
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
- G01C11/00—Photogrammetry or videogrammetry, e.g. stereogrammetry; Photographic surveying
- G01C11/04—Interpretation of pictures
- G01C11/06—Interpretation of pictures by comparison of two or more pictures of the same area
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
- G01C25/00—Manufacturing, calibrating, cleaning, or repairing instruments or devices referred to in the other groups of this subclass
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/02—Systems using the reflection of electromagnetic waves other than radio waves
- G01S17/06—Systems determining position data of a target
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/87—Combinations of systems using electromagnetic waves other than radio waves
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/88—Lidar systems specially adapted for specific applications
- G01S17/89—Lidar systems specially adapted for specific applications for mapping or imaging
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
- G01S7/48—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00
- G01S7/497—Means for monitoring or calibrating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/10—Image acquisition
- G06V10/12—Details of acquisition arrangements; Constructional details thereof
- G06V10/14—Optical characteristics of the device performing the acquisition or on the illumination arrangements
- G06V10/147—Details of sensors, e.g. sensor lenses
Definitions
- the present invention relates in general to range imaging systems and more particularly to a method and a system for measuring a relative position and orientation of range cameras using a movement of an object within a scene.
- Range imaging systems are used in a variety of applications to determine the three-dimensional (3-D) characteristics of a scene (a scene is an environment of interest).
- these applications include 3-D scene reconstruction, 3-D object recognition, robot navigation, terrain mapping and object tracking.
- An important component of a range imaging system is a range camera.
- a range camera is a device that is used to measure a 3-D structure of a scene by providing range (or depth) information as measured from a plane on the camera.
- range camera provides a range (or distance to the 3-D scene) of each pixel.
- Range cameras use a variety of techniques to measure range including lasers, projected light patterns and stereo vision.
- the range imaging system may include more than one range camera because a single range camera may not have a sufficiently large field of view to monitor the entire scene.
- the cameras In order for multiple range cameras to work together, however, the cameras must be calibrated to determine a position and an orientation of each camera relative to one of the cameras (known as a relative pose). This calibration of multiple cameras enables the ranging system to convert 3-D measurements obtained from each camera into a common coordinate frame. For example, a path of a person in a scene may be measured by each camera in its local coordinate frame and converted to a common coordinate frame (such as a room-based coordinate system).
- One type of calibration technique uses a three-dimensional calibration chart to determine the relative position of each camera. This technique, however, is difficult to use and time-consuming because it requires that the calibration chart be positioned correctly within a scene.
- Another type of calibration technique requires a user to monitor a scene and determine a plurality of reference points in the scene until the relative position of each camera can be determined. For example, a user references a number of common points in a scene (within each camera's field of view) and, if enough of these common points are found, the relative pose of the cameras may be determined.
- One disadvantage of this technique is that it is difficult to implement in a consumer-based product because it is unlikely the consumer would want to perform such a complicated and time-consuming calibration process.
- both types of calibration techniques if the consumer performed the calibration process improperly any results obtained from the range imaging system would be erroneous.
- the present invention includes a method and system for determining a relative position and orientation of a plurality of range cameras using spatial movement.
- a path of an object is measured by each range camera in the camera's local coordinate frame.
- the path of the object is observed by each camera but, because each camera has a different view of the object's path, the object path is reported by each camera in different local coordinate frames.
- the present invention determines the relative location of each range camera by converting the object path as measured in each of the local coordinate frames to a common coordinate frame.
- the common coordinate frame may be, for example, with respect to one of the cameras or with respect to the scene (such as a room-based coordinate system).
- the novel method of the present invention includes measuring a path of an object in a scene as observed by each camera, performing matching of points of the path and obtaining transformation parameters (such as an offset distance ( )x, )y) and a rotation angle (2)), preferably by solving a system of transformation equations. These transformation parameters are used to determine the relative position of each camera.
- the present invention includes other novel features such a data synchronization feature that uses a time shift between cameras to obtain the transformation parameters.
- the present invention includes a unique process that improves the robustness and accuracy of solving the system of transformation equations by using a process that is less sensitive to outlying points.
- the present invention includes using a least median of squares technique to reduce the sensitivity of the solution to points extremely removed from the correct solution.
- the present invention also includes an interpolation process that interpolates between sampled points if there is no data at a particular instant in time.
- the present invention includes a system for determining a relative position and orientation of range cameras using spatial movement that incorporates the method of the present invention.
- FIG. 1 is a block diagram illustrating an apparatus for carrying out the present invention.
- FIG. 2 is an overall block diagram of a range imaging system incorporating the present invention.
- FIG. 3 is a general block diagram of the object tracker of the present invention.
- FIG. 4 is a block diagram illustrating the calibration module of the object tracker shown in FIG. 3 .
- FIG. 5 is a block diagram illustrating the transformation processor the calibration module shown in FIG. 4 .
- FIG. 6 is a general flow diagram of the operation of the calibration module of the present invention.
- FIG. 7A-7C are general block diagrams illustrating exemplary operations of the calibration module shown in FIG. 4 .
- FIG. 8 is a detailed flow diagram illustrating a preferred embodiment of the present invention.
- FIGS. 9A-9D illustrate an example of how the present invention can perform data matching.
- FIG. 10 illustrates an example of how the present invention can determine an accurate time offset value.
- FIG. 1 and the following discussion are intended to provide a brief, general description of a suitable computing environment in which the invention may be implemented.
- the invention will be described in the general context of computer-executable instructions, such as program modules, being executed by a computer.
- program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types.
- program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types.
- program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types.
- the invention may be practiced with a variety of computer system configurations, including personal computers, server computers, hand-held devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, and the like.
- the invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network.
- program modules may be located on both
- an exemplary system for implementing the invention includes a general-purpose computing device in the form of a conventional personal computer 100 , including a processing unit 102 , a system memory 104 , and a system bus 106 that couples various system components including the system memory 104 to the processing unit 102 .
- the system bus 106 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures.
- the system memory includes read only memory (ROM) 110 and random access memory (RAM) 112 .
- a basic input/output system (BIOS) 114 containing the basic routines that help to transfer information between elements within the personal computer 100 , such as during start-up, is stored in ROM 110 .
- the personal computer 100 further includes a hard disk drive 116 for reading from and writing to a hard disk, not shown, a magnetic disk drive 118 for reading from or writing to a removable magnetic disk 120 , and an optical disk drive 122 for reading from or writing to a removable optical disk 124 such as a CD-ROM or other optical media.
- the hard disk drive 116 , magnetic disk drive 128 and optical disk drive 122 are connected to the system bus 106 by a hard disk drive interface 126 , a magnetic disk drive interface 128 and an optical disk drive interface 130 , respectively.
- the drives and their associated computer-readable media provide nonvolatile storage of computer readable instructions, data structures, program modules and other data for the personal computer 100 .
- exemplary environment described herein employs a hard disk, a removable magnetic disk 120 and a removable optical disk 124 , it should be appreciated by those skilled in the art that other types of computer readable media that can store data that is accessible by a computer, such as magnetic cassettes, flash memory cards, digital video disks, Bernoulli cartridges, random access memories (RAMs), read-only memories (ROMs), and the like, may also be used in the exemplary operating environment.
- RAMs random access memories
- ROMs read-only memories
- a number of program modules may be stored on the hard disk, magnetic disk 120 , optical disk 124 , ROM 110 or RAM 112 , including an operating system 132 , one or more application programs 134 , other program modules 136 and program data 138 .
- a user may enter commands and information into the personal computer 100 through input devices such as a keyboard 140 and a pointing device 142 .
- a camera 143 (or other types of imaging devices) may be connected to the personal computer 100 as well as other input devices (not shown) including, for example, a microphone, joystick, game pad, satellite dish, scanner, or the like.
- serial port interface 144 that is coupled to the system bus 106 , but may be connected by other interfaces, such as a parallel port, a game port or a universal serial bus (USB).
- a monitor 146 or other type of display device is also connected to the system bus 106 via an interface, such as a video adapter 148 .
- personal computers typically include other peripheral output devices (not shown), such as speakers and printers.
- the personal computer 100 may operate in a networked environment using logical connections to one or more remote computers, such as a remote computer 150 .
- the remote computer 150 may be another personal computer, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to the personal computer 100 , although only a memory storage device 152 has been illustrated in FIG. 1 .
- the logical connections depicted in FIG. 1 include a local area network (LAN) 154 and a wide area network (WAN) 156 .
- LAN local area network
- WAN wide area network
- the personal computer 100 When used in a LAN networking environment, the personal computer 100 is connected to the local network 154 through a network interface or adapter 158 . When used in a WAN networking environment, the personal computer 100 typically includes a modem 160 or other means for establishing communications over the wide area network 156 , such as the Internet.
- the modem 160 which may be internal or external, is connected to the system bus 106 via the serial port interface 144 .
- program modules depicted relative to the personal computer 100 may be stored in the remote memory storage device 152 . It will be appreciated that the network connections shown are exemplary and other means of establishing a communications link between the computers may be used.
- the method and system of the present invention include measuring the relative position and orientation of at least two range cameras.
- Range cameras which are used to measure the 3-D structure of a scene, give the range (or depth) of each pixel.
- the system such as a range imaging system
- the system using the range cameras must be able to determine a relative position and orientation of each camera.
- This process of determining a relative pose of each camera (also known as calibration) enables the system to convert 3-D measurements from each camera into a common coordinate frame. Data from each camera is in the camera's local coordinate frame, and calibration of each camera makes the 3-D measurements from different cameras (in different local coordinate frames) consistent with each other.
- the present invention measures a relative pose between a plurality of range cameras by measuring a relative pose between two cameras at a time.
- One camera is designated as a base camera and relative poses of the remainder of the cameras can be measured relative to the base camera.
- the present invention calibrates range cameras based on a path of an object around a scene. The object path is determined in a ground plane (such as a floor of a room) as a function of time as measured by a range camera.
- the present invention determines the transformation parameters that take a point on the object path measured by a non-base camera and convert it to a point as it would be seen from the base camera.
- the present invention includes synchronizing data obtained from each camera, interpolating between sampled data points and using a robust error minimization technique to determine the transformation parameters.
- FIG. 2 is an overall block diagram of a range imaging system incorporating the present invention.
- the range imaging system illustrated is only one example of several systems that could incorporate the relative range camera calibration method and system of the present invention.
- the range imaging system 200 includes a first camera 208 and a second camera 216 .
- Each of the cameras 208 , 216 may use any of the various techniques available to measure range, such as, for example, lasers, projected light patterns and stereo vision. Both of the cameras 208 , 216 are directed toward a scene 224 and are capable of measuring a 3-D structure of the scene 224 .
- the range imaging system also includes a first data module 232 that samples raw position data from the first camera 208 and a second data module 236 that samples raw position data from the second camera 216 .
- These data modules 232 , 236 may be, for example, computers or microprocessors.
- the first camera 208 supplies position data about the scene 224 in a first local coordinate frame and the second camera 216 supplies position data about the scene 224 in a second local coordinate frame. These two local coordinate frames generally are not the same, and calibration of the two cameras 208 , 216 is necessary to express the position data from each camera in a common coordinate frame.
- the sampled data from each camera is sent to an object tracker 240 , which inputs the sampled data, calibrates the cameras 208 , 216 and performs a coordinate transformation of the data.
- an output module 248 is included in the range imaging system 200 that outputs scene data in a common coordinate system (such as a room-based coordinate system).
- the scene 224 includes a room 256 containing a first sofa 264 on one side of the room 256 and a second sofa 272 opposite the first sofa 264 .
- a chair 280 is situated between to sofas 264 , 272 .
- calibration of the range cameras 208 , 216 generally is performed by having a person 288 (denoted by an “X”) move in a path 296 around the room 256 .
- This path 296 is observed by the cameras 208 , 216 in their respective local coordinate frames and the raw position data (such as (x,y) coordinates) of the path 296 is sampled by the data modules 232 , 236 .
- the data modules 232 , 236 sample raw position data from each camera that includes the object path 296 described in a first local coordinate frame (as observed by the first camera 208 ) and the object path 296 described in a second local coordinate frame (as observed by the second camera 216 ).
- the object tracker 240 receives the sampled data from the data modules 232 , 236 and, using the present invention, calibrates cameras 208 , 216 by determining the relative position and orientation of each camera. Once the cameras 208 , 216 are calibrated any data from the cameras 208 , 216 is converted into a common coordinate frame. This means, for example, a path of an object around the room 256 is expressed by the object tracker 240 in a common coordinate frame. The object tracker 240 sends data in a common coordinate frame to the output module 248 , for output from the range imaging system 200 .
- the range imaging system 200 may transmit the data to a post-processing module 298 that may include, for example, a three-dimensional (3-D) scene reconstruction system, a 3-D object recognition system or a 3-D tracking system (which may be part of a vision-based computer interface system).
- a post-processing module 298 may include, for example, a three-dimensional (3-D) scene reconstruction system, a 3-D object recognition system or a 3-D tracking system (which may be part of a vision-based computer interface system).
- FIG. 3 is a general block diagram of the object tracker 300 (the object tracker 240 in FIG. 2 is one example of the object tracker 300 ) of the present invention.
- position data from cameras (box 310 ) in local coordinate frames is received by the object tracker 300 , processed and data is sent as output in a common coordinate frame (box 320 ).
- the object tracker 300 includes a calibration module 330 , which determines transformation parameters that will transform position data in local coordinate frames into a common coordinate frame, and a coordinate processor 340 , which uses the transformation parameters computed by the calibration module 330 to transform data observed by the cameras into a desired common coordinate frame.
- FIG. 4 is a block diagram illustrating the calibration module 330 of the object tracker 300 shown in FIG. 3 .
- the calibration module determines transformation parameters that are used to convert data in a local coordinate frame of each camera into a common coordinate frame.
- data from each camera is received as input (box 410 ) and a data synchronizer 420 is used to synchronize the data received from multiple cameras.
- a coordinate selector 430 determines the desired coordinate frame of the transformation. For example, a first camera may be selected as the base camera and data from the other cameras are expressed in the coordinate frame of the base camera.
- a transformation processor 440 computes transformation parameters that convert data from a local coordinate frame to be expressed in the base coordinate frame. These transformation parameters are sent as output (box 450 ) of the calibration module 330 .
- FIG. 5 is a block diagram illustrating the transformation processor 440 the calibration module shown in FIG. 4 .
- the transformation processor 440 includes an interpolation module 510 , for interpolating between data points, a data matching processor 520 , for matching up data points from different cameras at a certain time, and an error minimization processor 530 , for determining the data points that yield the most accurate transformation parameters.
- the transformation processor 440 inputs synchronized data from the data synchronizer 420 . A time is then selected by the interpolation module 510 along with position data corresponding to that time. If there was no data point sampled by the data modules at the selected time then the interpolation module 510 interpolates a data point, as described further below.
- the data points at the selected time are received by the data matching processor 520 .
- the data matching processor 520 receives a desired coordinate frame as determined by the coordinate selector 430 .
- the desired coordinate frame may be, for example, chosen by the user or selected at random. Any data from the cameras is expressed in the selected coordinate frame (also called the base coordinate frame).
- the data matching processor 520 matches data points at the selected time and computes transformation parameters using the data points.
- the error minimization processor 530 determines which data points give the most accurate transformation parameters.
- FIG. 6 is a general flow diagram of the operation of the calibration module 330 of the present invention.
- data observed by cameras is received as input (box 610 ).
- each camera positioned to observe data within a scene.
- the calibration method of the present invention only needs to measure the relative position and orientation between two cameras at a time. This is because one camera is designated as the base camera and the position and orientation of the remainder of the cameras are measured from that base camera.
- the raw data from each camera is sampled and sent to the calibration module 330 and the sampled data from each camera in its respective local coordinate frame.
- Transformation parameters are computed (box 630 ) from the sampled data received by the calibration module 330 . These transformation parameters are then used to express data received from each camera in the base coordinate frame. Once this calibration process is performed, any data observed by a non-base camera can be expressed in the base coordinate frame as if the data had been observed by the base camera.
- FIGS. 7A-7C are general block diagrams illustrating the operations of the transformation parameters computed by the calibration module 330 shown in FIG. 4 .
- a first camera 700 and a second camera 705 observe an object path in a scene.
- a first object path 710 is observed by the first camera 700 in a first local coordinate frame and a second object path 715 is observed by the second camera in a second local coordinate frame.
- two cameras 700 , 705 observe the same path but in different local coordinate frames.
- FIG. 7B illustrates a set of transformation parameters computed by the calibration module 330 applied to the data of the second camera 705 .
- local coordinate frame of the first camera 700 has been selected as the base coordinate frame and, in accordance with the present invention, one purpose of the calibration module 330 is to compute transformation parameters that cause the second object path 715 to overlap with the first object path 710 as closely as possible.
- the transformation parameters include a change in the “x” coordinate ( ⁇ x) 730 , a change in the “y” coordinate ( ⁇ y) 735 and an angle of rotation ( ⁇ ) 740 . As shown in FIG.
- FIG. 7C illustrates another set of transformation parameters computed by the calibration module 330 applied to the data of the second camera 705 .
- the transformation parameters ⁇ x* 750 , Ay* 755 and ⁇ * 760 are used to achieve an exact overlap of the first object path 710 and the second object path 715 into a single object path 770 .
- the exact overlap represents minimum error in the transformation parameters ( ⁇ x*, ⁇ y*, ⁇ *) and means that these transformation parameters can be used to express data from the second camera 705 in the base coordinate frame.
- FIG. 8 is a detailed flow diagram illustrating a preferred embodiment of the present invention.
- the present invention designates one of a plurality of cameras as a base camera and measures the relative pose of the remainder of the cameras with respect to the base camera. Initially, one camera is selected as a base camera and that camera's local coordinate frame becomes the base coordinate frame (box 800 ). Moreover, data from each camera is received as input (box 805 ).
- the first problem occurs if the clocks on the computers used to sample the data are unsynchronized by a constant time offset so that equivalent time readings on the computers do not correspond to the same actual time.
- the present invention corrects this problem by adding a time offset to the data.
- a time offset value is chosen (box 810 ) and applied to the camera data (box 815 ) in order to synchronize the data.
- the second problem occurs if the data from the cameras is not sampled at the same time leaving, for example, a data point at time t from a first camera without a corresponding data point from a second camera.
- the present invention corrects this problem by performing a linear interpolation (box 820 ) between two data points sampled before and after time t. This linear interpolation approximates where a data point would have been seen at time t.
- data matching is performed to provide enough data points to compute the corresponding transformation parameters. Data matching matches data from different cameras at certain absolute times and uses these data points to compute transformation parameters.
- FIGS. 9A-9D illustrate an example of how the present invention can perform data matching.
- data from a first camera (camera 1 ) in a first local coordinate frame (x 1 , y 1 ) and data from a second camera (camera 2 ) in a second coordinate frame (x 2 , y 2 ) are graphed as a function of time.
- FIG. 9A illustrates a graph of x, versus time
- FIG. 9B illustrates a graph of y 1 versus time
- FIG. 9C illustrates x 2 versus time
- FIG. 9D illustrates y 2 versus time.
- a time T* is selected such that there is data available at T* from, for example, camera 1 .
- the x 1 and y 1 coordinates from camera 1 will be perfectly synchronized, but there may be no data (i.e. (x 2 , y 2 )) available from camera 2 .
- data from camera 1 at time T* is matched such that a first point 910 on the x 1 versus time graph (or a second point 920 on the y 1 versus time graph) are matched with a third point 930 on the x 2 versus time graph and a fourth point 940 on the y 2 versus time graph. Note that in FIGS. 9C and 9D there are no sampled data points from camera 2 at time T*.
- the present invention performs an interpolation and chooses sampled data points 950 , 955 prior to T* and sampled data points 960 , 965 after T*. These sampled points 950 , 955 , 960 , 965 are used to interpolate values of x 2 and y 2 at time T* to obtain the third and fourth data points 930 , 940 .
- the invention determines whether more data points are needed (box 830 ). If so, then a different time is chosen and data matching is performed (box 825 ) at that time. Otherwise, if there are enough data points, an error minimization technique is used to find the data points that give the transformation parameters with the least error (box 835 ).
- the present invention may determine minimum error by using a least squares technique that is discussed by S. Ma and Z. Zhang in “Computer Vision” (Chinese Academy of Science, 1998), the entire contents of which are hereby incorporated by reference.
- the present invention uses a least median of squares technique to determine minimum error.
- the least median of squares technique is more robust and less affected by data points that lie well away from the majority of data points.
- the least median of squares technique is discussed in detail by P. J. Rousseeuw and A. M. Leroy in “Robust Regression and Outlier Detection” (New York: John Wiley and Sons, 1987), the entire contents of which are hereby incorporated by reference.
- the transformation parameters with the least amount of error are stored along with the time offset value used to synchronize the data (box 840 ).
- FIG. 10 illustrates an example of how the present invention can determine an accurate time offset. Specifically, the error for each time offset values is determined and plotted as shown in FIG. 10 . A point 1000 at which a minimum error occurs a corresponding time offset value is noted. When the time offset value at minimum error is determined, both the time offset value and the corresponding transformation parameters are sent as output (box 860 ).
- the transformation parameters are changes in the x and y coordinates and the rotation angle (such as ⁇ x, ⁇ y, ⁇ ).
- other transformation parameters may be used depending on the type of coordinate systems used (such as, for example, polar coordinate systems).
- the following working example uses a range imaging system to track the movement of a person around a room and is provided for illustrative purposes only.
- the method and system of the present invention are used to calibrate two range cameras in prior to using the range imaging system.
- a variety of techniques such as lasers and projected light patterns
- stereo cameras were used. Stereo cameras were chosen because of their fast frame rate and because they are inexpensive and safe.
- the application was tracking people as they move around a room.
- two range cameras (camera 1 and camera 2 ) were used and calibrated based on a person's path when the person walked around the room.
- the calibration process began by determining an (x,y) location of the person on a ground plane (in this working example, the floor of the room) as a function of time as measured by each range camera. This was accomplished using a technique described in co-pending U.S. patent application Ser. No. 09/455,822 entitled “A System and Process for Locating and Tracking a Person or Object in a Scene Using a Series of Range Images” by Barry Brumitt, filed on Dec. 6, 1999, the entire contents of which are hereby incorporated by reference.
- the present invention then chose a first camera as the base camera and designated the location measured by the base camera as (x 1 ,y 1 ) and a corresponding point from a second (non-base) camera (camera 2 ) as (x 2 ,y 2 ).
- the present invention was used to calibrate the two cameras by computing the transformation parameters of an angle ⁇ and an offset ( ⁇ x, ⁇ y) that made the following equation true:
- ( x 1 y 1 1 ) [ cos ⁇ ( ⁇ ) - sin ⁇ ( ⁇ ) ⁇ ⁇ ⁇ x sin ⁇ ( ⁇ ) cos ⁇ ( ⁇ ) ⁇ ⁇ ⁇ y 0 0 1 ] ⁇ ( x 2 y 2 1 )
- any point seen by camera 2 (x 2 ,y 2 ) could be transformed into the coordinates of camera 1 (x 1 ,y 1 ). This means that a point as seen by camera 2 could be expressed in the local coordinate frame of camera 1 as if the point was actually seen by camera 1 .
- a time offset ⁇ t was used to correct for the fact that a clock on the computer associated with the first camera (clock 1 ) was not synchronized with a clock on the computer associated with the second camera (clock 2 ).
- the points from the first camera and the second camera became (x 1i ,y 1i ,t 1i ) and (x 2j ,y 2j ,t 2j + ⁇ t), respectively.
- An initial guess of the time offset ⁇ t was chosen and a point from each camera was sampled. Because the sampled points from each camera did not exactly match up with each other, the data obtained from the second camera was interpolated as follows.
- point matching was performed and the transformation parameters corresponding to the least squared error was selected.
- the least median of squares technique was used because it is a robust method. This method was implemented by picking random pairs of corresponding points from the data set ((x 1k *,y 1k *, x 2k *, y 2k *), 1 ⁇ k ⁇ n). A pair of points was the minimum number needed to compute the candidate transformation parameters (i.e., ⁇ and ( ⁇ x, ⁇ y)).
- This ⁇ and ( ⁇ x, ⁇ y) served as a trial solution for the calibration problem based on the two randomly chosen pair of points.
- the quality of the solution was the median value of this list of squared errors.
- our implementation 100 random pairs of corresponding points were chosen and the transformation parameters ⁇ and ( ⁇ x, ⁇ y) that corresponded to the least median of squares were used.
- the least median of square technique was used as above to compute the best ⁇ and ( ⁇ x, ⁇ y) for a whole series of values of a time offset value ( ⁇ t). Whichever ⁇ t gave the minimum least median of squares was chosen as the best one, and the corresponding ⁇ and ( ⁇ x, ⁇ y) were used for the final solution.
- the least squares solution to the calibration problem computes the transformation parameters ⁇ and ( ⁇ x, ⁇ y) that minimize the sum of the Euclidean distances between corresponding points in (x 1k *,y 1k *,x 2k *,y 2k ), 1 ⁇ k ⁇ n.
- ⁇ k 1 n ⁇ ( ( x 1 ⁇ k * - x _ 1 * ) ⁇ ( x 2 ⁇ k * - x _ 2 * ) - ( y 1 ⁇ k * - y _ 1 * ) ⁇ ( y 2 ⁇ k * - y _ 2 * ) )
- the transformation parameters ⁇ , ( ⁇ x, ⁇ y) and e 2 are computed.
- the average squared distance between corresponding points, e 2 will be a minimum for some value of ⁇ t. We take the corresponding values of ⁇ and ( ⁇ x, ⁇ y) at the
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Radar, Positioning & Navigation (AREA)
- Remote Sensing (AREA)
- Electromagnetism (AREA)
- Computer Networks & Wireless Communication (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- Vascular Medicine (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Manufacturing & Machinery (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Length Measuring Devices By Optical Means (AREA)
Abstract
Method and system for measuring a relative position and orientation of range cameras using a movement of an object within a scene. In general, the method and system determine the relative pose between two cameras by measuring a path the movement of the object makes within a scene and calculating transformation parameters based on these measurements. These transformation parameters are used to determine the relative position of each camera with respect to a base camera. The system and method include other novel features, such as a data synchronization feature that uses a time offset between cameras to obtain the transformation parameters, and a technique that improves the robustness and accuracy of solving for the transformation parameters, and an interpolation process that interpolates between sampled points if there is no data at a particular instant in time.
Description
- This application is a continuation application of U.S. patent application Ser. No. 10/927,373, entitled “Relative Range Camera Calibration,” filed Aug. 25, 2004, which is now pending, the entire contents of which are hereby incorporated by reference.
- 1. Field of the Invention
- The present invention relates in general to range imaging systems and more particularly to a method and a system for measuring a relative position and orientation of range cameras using a movement of an object within a scene.
- 2. Related Art
- Range imaging systems are used in a variety of applications to determine the three-dimensional (3-D) characteristics of a scene (a scene is an environment of interest). By way of example, these applications include 3-D scene reconstruction, 3-D object recognition, robot navigation, terrain mapping and object tracking. An important component of a range imaging system is a range camera. A range camera is a device that is used to measure a 3-D structure of a scene by providing range (or depth) information as measured from a plane on the camera. Thus, while a black and white camera provides a grayscale intensity of each pixel and a color camera provides a color of each pixel, a range camera provides a range (or distance to the 3-D scene) of each pixel. Range cameras use a variety of techniques to measure range including lasers, projected light patterns and stereo vision.
- For some applications (such as tracking persons within a scene) the range imaging system may include more than one range camera because a single range camera may not have a sufficiently large field of view to monitor the entire scene. In order for multiple range cameras to work together, however, the cameras must be calibrated to determine a position and an orientation of each camera relative to one of the cameras (known as a relative pose). This calibration of multiple cameras enables the ranging system to convert 3-D measurements obtained from each camera into a common coordinate frame. For example, a path of a person in a scene may be measured by each camera in its local coordinate frame and converted to a common coordinate frame (such as a room-based coordinate system).
- Several types of manual calibration techniques are used to calibrate the range cameras. One type of calibration technique uses a three-dimensional calibration chart to determine the relative position of each camera. This technique, however, is difficult to use and time-consuming because it requires that the calibration chart be positioned correctly within a scene.
- Another type of calibration technique requires a user to monitor a scene and determine a plurality of reference points in the scene until the relative position of each camera can be determined. For example, a user references a number of common points in a scene (within each camera's field of view) and, if enough of these common points are found, the relative pose of the cameras may be determined. One disadvantage of this technique, however, is that it is difficult to implement in a consumer-based product because it is unlikely the consumer would want to perform such a complicated and time-consuming calibration process. Moreover, with both types of calibration techniques, if the consumer performed the calibration process improperly any results obtained from the range imaging system would be erroneous.
- Accordingly, there exists a need for a range camera calibration method and system that is accurate and simple to use. Whatever the merits of the above-mentioned systems and methods, they do not achieve the benefits of the present invention.
- To overcome the limitations in the prior art as described above and other limitations that will become apparent upon reading and understanding the present specification, the present invention includes a method and system for determining a relative position and orientation of a plurality of range cameras using spatial movement. In particular, a path of an object is measured by each range camera in the camera's local coordinate frame. Thus, the path of the object is observed by each camera but, because each camera has a different view of the object's path, the object path is reported by each camera in different local coordinate frames.
- The present invention determines the relative location of each range camera by converting the object path as measured in each of the local coordinate frames to a common coordinate frame. The common coordinate frame may be, for example, with respect to one of the cameras or with respect to the scene (such as a room-based coordinate system).
- In general, the novel method of the present invention includes measuring a path of an object in a scene as observed by each camera, performing matching of points of the path and obtaining transformation parameters (such as an offset distance ( )x, )y) and a rotation angle (2)), preferably by solving a system of transformation equations. These transformation parameters are used to determine the relative position of each camera. Moreover, the present invention includes other novel features such a data synchronization feature that uses a time shift between cameras to obtain the transformation parameters. In addition, the present invention includes a unique process that improves the robustness and accuracy of solving the system of transformation equations by using a process that is less sensitive to outlying points. For example, in a preferred implementation the present invention includes using a least median of squares technique to reduce the sensitivity of the solution to points extremely removed from the correct solution. The present invention also includes an interpolation process that interpolates between sampled points if there is no data at a particular instant in time. Further, the present invention includes a system for determining a relative position and orientation of range cameras using spatial movement that incorporates the method of the present invention.
- Other aspects and advantages of the present invention as well as a more complete understanding thereof will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrating by way of example the principles of the invention. Moreover, it is intended that the scope of the invention be limited by the claims and not by the preceding summary or the following detailed description.
- The present invention can be further understood by reference to the following description and attached drawings that illustrate the preferred embodiments. Other features and advantages will be apparent from the following detailed description of the invention, taken in conjunction with the accompanying drawings, which illustrate, by way of example, the principles of the present invention.
- Referring now to the drawings in which like reference numbers represent corresponding parts throughout:
-
FIG. 1 is a block diagram illustrating an apparatus for carrying out the present invention. -
FIG. 2 is an overall block diagram of a range imaging system incorporating the present invention. -
FIG. 3 is a general block diagram of the object tracker of the present invention. -
FIG. 4 is a block diagram illustrating the calibration module of the object tracker shown inFIG. 3 . -
FIG. 5 is a block diagram illustrating the transformation processor the calibration module shown inFIG. 4 . -
FIG. 6 is a general flow diagram of the operation of the calibration module of the present invention. -
FIG. 7A-7C are general block diagrams illustrating exemplary operations of the calibration module shown inFIG. 4 . -
FIG. 8 is a detailed flow diagram illustrating a preferred embodiment of the present invention. -
FIGS. 9A-9D illustrate an example of how the present invention can perform data matching. -
FIG. 10 illustrates an example of how the present invention can determine an accurate time offset value. - In the following description of the invention, reference is made to the accompanying drawings, which form a part thereof, and in which is shown by way of illustration a specific example whereby the invention may be practiced. It is to be understood that other embodiments may be utilized and structural changes may be made without departing from the scope of the present invention.
- I. Exemplary Operating Environment
-
FIG. 1 and the following discussion are intended to provide a brief, general description of a suitable computing environment in which the invention may be implemented. Although not required, the invention will be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Moreover, those skilled in the art will appreciate that the invention may be practiced with a variety of computer system configurations, including personal computers, server computers, hand-held devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, and the like. The invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located on both local and remote computer storage media including memory storage devices. - With reference to
FIG. 1 , an exemplary system for implementing the invention includes a general-purpose computing device in the form of a conventionalpersonal computer 100, including aprocessing unit 102, asystem memory 104, and asystem bus 106 that couples various system components including thesystem memory 104 to theprocessing unit 102. Thesystem bus 106 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. The system memory includes read only memory (ROM) 110 and random access memory (RAM) 112. A basic input/output system (BIOS) 114, containing the basic routines that help to transfer information between elements within thepersonal computer 100, such as during start-up, is stored inROM 110. Thepersonal computer 100 further includes ahard disk drive 116 for reading from and writing to a hard disk, not shown, amagnetic disk drive 118 for reading from or writing to a removablemagnetic disk 120, and anoptical disk drive 122 for reading from or writing to a removableoptical disk 124 such as a CD-ROM or other optical media. Thehard disk drive 116,magnetic disk drive 128 andoptical disk drive 122 are connected to thesystem bus 106 by a harddisk drive interface 126, a magneticdisk drive interface 128 and an opticaldisk drive interface 130, respectively. The drives and their associated computer-readable media provide nonvolatile storage of computer readable instructions, data structures, program modules and other data for thepersonal computer 100. - Although the exemplary environment described herein employs a hard disk, a removable
magnetic disk 120 and a removableoptical disk 124, it should be appreciated by those skilled in the art that other types of computer readable media that can store data that is accessible by a computer, such as magnetic cassettes, flash memory cards, digital video disks, Bernoulli cartridges, random access memories (RAMs), read-only memories (ROMs), and the like, may also be used in the exemplary operating environment. - A number of program modules may be stored on the hard disk,
magnetic disk 120,optical disk 124,ROM 110 orRAM 112, including anoperating system 132, one ormore application programs 134,other program modules 136 andprogram data 138. A user (not shown) may enter commands and information into thepersonal computer 100 through input devices such as akeyboard 140 and apointing device 142. In addition, a camera 143 (or other types of imaging devices) may be connected to thepersonal computer 100 as well as other input devices (not shown) including, for example, a microphone, joystick, game pad, satellite dish, scanner, or the like. These other input devices are often connected to theprocessing unit 102 through aserial port interface 144 that is coupled to thesystem bus 106, but may be connected by other interfaces, such as a parallel port, a game port or a universal serial bus (USB). Amonitor 146 or other type of display device is also connected to thesystem bus 106 via an interface, such as avideo adapter 148. In addition to themonitor 146, personal computers typically include other peripheral output devices (not shown), such as speakers and printers. - The
personal computer 100 may operate in a networked environment using logical connections to one or more remote computers, such as aremote computer 150. Theremote computer 150 may be another personal computer, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to thepersonal computer 100, although only amemory storage device 152 has been illustrated inFIG. 1 . The logical connections depicted inFIG. 1 include a local area network (LAN) 154 and a wide area network (WAN) 156. Such networking environments are commonplace in offices, enterprise-wide computer networks, intranets and the Internet. - When used in a LAN networking environment, the
personal computer 100 is connected to thelocal network 154 through a network interface oradapter 158. When used in a WAN networking environment, thepersonal computer 100 typically includes amodem 160 or other means for establishing communications over thewide area network 156, such as the Internet. Themodem 160, which may be internal or external, is connected to thesystem bus 106 via theserial port interface 144. In a networked environment, program modules depicted relative to thepersonal computer 100, or portions thereof, may be stored in the remotememory storage device 152. It will be appreciated that the network connections shown are exemplary and other means of establishing a communications link between the computers may be used. - II. Introduction
- The method and system of the present invention include measuring the relative position and orientation of at least two range cameras. Range cameras, which are used to measure the 3-D structure of a scene, give the range (or depth) of each pixel. In order for two or more range cameras to work properly together, the system (such as a range imaging system) using the range cameras must be able to determine a relative position and orientation of each camera. This process of determining a relative pose of each camera (also known as calibration) enables the system to convert 3-D measurements from each camera into a common coordinate frame. Data from each camera is in the camera's local coordinate frame, and calibration of each camera makes the 3-D measurements from different cameras (in different local coordinate frames) consistent with each other.
- The present invention measures a relative pose between a plurality of range cameras by measuring a relative pose between two cameras at a time. One camera is designated as a base camera and relative poses of the remainder of the cameras can be measured relative to the base camera. In general, the present invention calibrates range cameras based on a path of an object around a scene. The object path is determined in a ground plane (such as a floor of a room) as a function of time as measured by a range camera. The present invention determines the transformation parameters that take a point on the object path measured by a non-base camera and convert it to a point as it would be seen from the base camera. In addition, the present invention includes synchronizing data obtained from each camera, interpolating between sampled data points and using a robust error minimization technique to determine the transformation parameters.
- III. General Overview
- As shown in
FIGS. 2-10 for the purposes of illustration, the invention is embodied in a method and a system for measuring a relative position and orientation of range cameras using a movement of an object within a scene.FIG. 2 is an overall block diagram of a range imaging system incorporating the present invention. The range imaging system illustrated is only one example of several systems that could incorporate the relative range camera calibration method and system of the present invention. In general, therange imaging system 200 includes afirst camera 208 and asecond camera 216. Each of thecameras cameras scene 224 and are capable of measuring a 3-D structure of thescene 224. - The range imaging system also includes a
first data module 232 that samples raw position data from thefirst camera 208 and asecond data module 236 that samples raw position data from thesecond camera 216. Thesedata modules first camera 208 supplies position data about thescene 224 in a first local coordinate frame and thesecond camera 216 supplies position data about thescene 224 in a second local coordinate frame. These two local coordinate frames generally are not the same, and calibration of the twocameras - The sampled data from each camera is sent to an
object tracker 240, which inputs the sampled data, calibrates thecameras output module 248 is included in therange imaging system 200 that outputs scene data in a common coordinate system (such as a room-based coordinate system). In this example, thescene 224 includes aroom 256 containing afirst sofa 264 on one side of theroom 256 and asecond sofa 272 opposite thefirst sofa 264. In addition, achair 280 is situated between tosofas - In this range imaging system, calibration of the
range cameras path 296 around theroom 256. Thispath 296 is observed by thecameras path 296 is sampled by thedata modules data modules object path 296 described in a first local coordinate frame (as observed by the first camera 208) and theobject path 296 described in a second local coordinate frame (as observed by the second camera 216). - The
object tracker 240 receives the sampled data from thedata modules cameras cameras cameras room 256 is expressed by theobject tracker 240 in a common coordinate frame. Theobject tracker 240 sends data in a common coordinate frame to theoutput module 248, for output from therange imaging system 200. Further, therange imaging system 200 may transmit the data to apost-processing module 298 that may include, for example, a three-dimensional (3-D) scene reconstruction system, a 3-D object recognition system or a 3-D tracking system (which may be part of a vision-based computer interface system). - IV. Component Overview
-
FIG. 3 is a general block diagram of the object tracker 300 (theobject tracker 240 inFIG. 2 is one example of the object tracker 300) of the present invention. In general, position data from cameras (box 310) in local coordinate frames is received by theobject tracker 300, processed and data is sent as output in a common coordinate frame (box 320). Theobject tracker 300 includes acalibration module 330, which determines transformation parameters that will transform position data in local coordinate frames into a common coordinate frame, and a coordinateprocessor 340, which uses the transformation parameters computed by thecalibration module 330 to transform data observed by the cameras into a desired common coordinate frame. -
FIG. 4 is a block diagram illustrating thecalibration module 330 of theobject tracker 300 shown inFIG. 3 . The calibration module determines transformation parameters that are used to convert data in a local coordinate frame of each camera into a common coordinate frame. In general, data from each camera is received as input (box 410) and adata synchronizer 420 is used to synchronize the data received from multiple cameras. A coordinateselector 430 determines the desired coordinate frame of the transformation. For example, a first camera may be selected as the base camera and data from the other cameras are expressed in the coordinate frame of the base camera. Atransformation processor 440 computes transformation parameters that convert data from a local coordinate frame to be expressed in the base coordinate frame. These transformation parameters are sent as output (box 450) of thecalibration module 330. -
FIG. 5 is a block diagram illustrating thetransformation processor 440 the calibration module shown inFIG. 4 . Thetransformation processor 440 includes aninterpolation module 510, for interpolating between data points, adata matching processor 520, for matching up data points from different cameras at a certain time, and anerror minimization processor 530, for determining the data points that yield the most accurate transformation parameters. Thetransformation processor 440 inputs synchronized data from thedata synchronizer 420. A time is then selected by theinterpolation module 510 along with position data corresponding to that time. If there was no data point sampled by the data modules at the selected time then theinterpolation module 510 interpolates a data point, as described further below. - The data points at the selected time are received by the
data matching processor 520. In addition, thedata matching processor 520 receives a desired coordinate frame as determined by the coordinateselector 430. The desired coordinate frame may be, for example, chosen by the user or selected at random. Any data from the cameras is expressed in the selected coordinate frame (also called the base coordinate frame). Thedata matching processor 520 matches data points at the selected time and computes transformation parameters using the data points. Theerror minimization processor 530 determines which data points give the most accurate transformation parameters. - V. Details of the Components and Operation
-
FIG. 6 is a general flow diagram of the operation of thecalibration module 330 of the present invention. Generally, data observed by cameras is received as input (box 610). Depending on the application there may be two or more cameras, with each camera positioned to observe data within a scene. Even if there are more than two cameras, however, the calibration method of the present invention only needs to measure the relative position and orientation between two cameras at a time. This is because one camera is designated as the base camera and the position and orientation of the remainder of the cameras are measured from that base camera. The raw data from each camera is sampled and sent to thecalibration module 330 and the sampled data from each camera in its respective local coordinate frame. - One of the cameras is selected as the base camera and the coordinate frame of chosen camera becomes the base coordinate frame (box 620). Transformation parameters are computed (box 630) from the sampled data received by the
calibration module 330. These transformation parameters are then used to express data received from each camera in the base coordinate frame. Once this calibration process is performed, any data observed by a non-base camera can be expressed in the base coordinate frame as if the data had been observed by the base camera. -
FIGS. 7A-7C are general block diagrams illustrating the operations of the transformation parameters computed by thecalibration module 330 shown inFIG. 4 . InFIG. 7A , afirst camera 700 and asecond camera 705 observe an object path in a scene. Afirst object path 710 is observed by thefirst camera 700 in a first local coordinate frame and asecond object path 715 is observed by the second camera in a second local coordinate frame. It should be noted that twocameras -
FIG. 7B illustrates a set of transformation parameters computed by thecalibration module 330 applied to the data of thesecond camera 705. In particular, local coordinate frame of thefirst camera 700 has been selected as the base coordinate frame and, in accordance with the present invention, one purpose of thecalibration module 330 is to compute transformation parameters that cause thesecond object path 715 to overlap with thefirst object path 710 as closely as possible. The transformation parameters include a change in the “x” coordinate (Δx) 730, a change in the “y” coordinate (Δy) 735 and an angle of rotation (θ) 740. As shown inFIG. 7B , when the transformation parameters (Δx, Δy, θ) are applied to thesecond object path 715, thefirst object path 710 andsecond object path 715 nearly overlap. The lack of exact overlap is due to a slight amount of error in the calculation of the transformation parameters. -
FIG. 7C illustrates another set of transformation parameters computed by thecalibration module 330 applied to the data of thesecond camera 705. InFIG. 7C the transformation parameters Δx* 750, Ay* 755 and θ* 760 are used to achieve an exact overlap of thefirst object path 710 and thesecond object path 715 into asingle object path 770. The exact overlap represents minimum error in the transformation parameters (Δx*, Δy*, θ*) and means that these transformation parameters can be used to express data from thesecond camera 705 in the base coordinate frame. -
FIG. 8 is a detailed flow diagram illustrating a preferred embodiment of the present invention. In this preferred embodiment, the present invention designates one of a plurality of cameras as a base camera and measures the relative pose of the remainder of the cameras with respect to the base camera. Initially, one camera is selected as a base camera and that camera's local coordinate frame becomes the base coordinate frame (box 800). Moreover, data from each camera is received as input (box 805). - Before this data can be used to compute transformation parameters, however, at least two problems must be overcome. The first problem occurs if the clocks on the computers used to sample the data are unsynchronized by a constant time offset so that equivalent time readings on the computers do not correspond to the same actual time. The present invention corrects this problem by adding a time offset to the data. In particular, a time offset value is chosen (box 810) and applied to the camera data (box 815) in order to synchronize the data. The second problem occurs if the data from the cameras is not sampled at the same time leaving, for example, a data point at time t from a first camera without a corresponding data point from a second camera.
- +
- The present invention corrects this problem by performing a linear interpolation (box 820) between two data points sampled before and after time t. This linear interpolation approximates where a data point would have been seen at time t. Next, data matching is performed to provide enough data points to compute the corresponding transformation parameters. Data matching matches data from different cameras at certain absolute times and uses these data points to compute transformation parameters.
-
FIGS. 9A-9D illustrate an example of how the present invention can perform data matching. In particular, data from a first camera (camera 1) in a first local coordinate frame (x1, y1) and data from a second camera (camera 2) in a second coordinate frame (x2, y2) are graphed as a function of time.FIG. 9A illustrates a graph of x, versus time,FIG. 9B illustrates a graph of y1 versus time,FIG. 9C illustrates x2 versus time andFIG. 9D illustrates y2 versus time. A time T* is selected such that there is data available at T* from, for example, camera 1. At time T*, the x1 and y1 coordinates from camera 1 (assuming that camera 1 was selected) will be perfectly synchronized, but there may be no data (i.e. (x2, y2)) available from camera 2. In this situation, data from camera 1 at time T* is matched such that afirst point 910 on the x1 versus time graph (or asecond point 920 on the y1 versus time graph) are matched with athird point 930 on the x2 versus time graph and afourth point 940 on the y2 versus time graph. Note that inFIGS. 9C and 9D there are no sampled data points from camera 2 at time T*. The present invention performs an interpolation and chooses sampleddata points data points points - For example, the present invention may determine minimum error by using a least squares technique that is discussed by S. Ma and Z. Zhang in “Computer Vision” (Chinese Academy of Science, 1998), the entire contents of which are hereby incorporated by reference. In a preferred embodiment, however, the present invention uses a least median of squares technique to determine minimum error. The least median of squares technique is more robust and less affected by data points that lie well away from the majority of data points. The least median of squares technique is discussed in detail by P. J. Rousseeuw and A. M. Leroy in “Robust Regression and Outlier Detection” (New York: John Wiley and Sons, 1987), the entire contents of which are hereby incorporated by reference.
- When the transformation parameters with the least amount of error have been determined, they are stored along with the time offset value used to synchronize the data (box 840). Next, a determination is made whether more time offset points are needed (box 845). If more are needed, then another time offset value is selected (box 850) and the process begins again at
box 815. Otherwise, an error minimization technique is used to find the time offset value with the least amount of error (box 855). As before, the least median of squares technique is a preferred technique to determine the minimum error. -
FIG. 10 illustrates an example of how the present invention can determine an accurate time offset. Specifically, the error for each time offset values is determined and plotted as shown inFIG. 10 . Apoint 1000 at which a minimum error occurs a corresponding time offset value is noted. When the time offset value at minimum error is determined, both the time offset value and the corresponding transformation parameters are sent as output (box 860). - It should be noted that in a preferred embodiment the transformation parameters are changes in the x and y coordinates and the rotation angle (such as Δx, Δy, θ). In addition, other transformation parameters may be used depending on the type of coordinate systems used (such as, for example, polar coordinate systems).
- VI. Working Example
- The following working example uses a range imaging system to track the movement of a person around a room and is provided for illustrative purposes only. In this working example, the method and system of the present invention are used to calibrate two range cameras in prior to using the range imaging system. As mentioned above, a variety of techniques (such as lasers and projected light patterns) are available for measuring range. Although in general the present invention is capable of using any ranging technique, in this working example stereo cameras were used. Stereo cameras were chosen because of their fast frame rate and because they are inexpensive and safe. In this working example, the application was tracking people as they move around a room. Further, two range cameras (camera 1 and camera 2) were used and calibrated based on a person's path when the person walked around the room.
- The calibration process began by determining an (x,y) location of the person on a ground plane (in this working example, the floor of the room) as a function of time as measured by each range camera. This was accomplished using a technique described in co-pending U.S. patent application Ser. No. 09/455,822 entitled “A System and Process for Locating and Tracking a Person or Object in a Scene Using a Series of Range Images” by Barry Brumitt, filed on Dec. 6, 1999, the entire contents of which are hereby incorporated by reference. The present invention then chose a first camera as the base camera and designated the location measured by the base camera as (x1,y1) and a corresponding point from a second (non-base) camera (camera 2) as (x2,y2). The present invention was used to calibrate the two cameras by computing the transformation parameters of an angle θ and an offset (Δx,Δy) that made the following equation true:
Once the values of θ and (Δx,Δy) were determined, using this equation, any point seen by camera 2, (x2,y2) could be transformed into the coordinates of camera 1 (x1,y1). This means that a point as seen by camera 2 could be expressed in the local coordinate frame of camera 1 as if the point was actually seen by camera 1. - In order to synchronize the data coming from the first and second cameras, a time offset Δt was used to correct for the fact that a clock on the computer associated with the first camera (clock 1) was not synchronized with a clock on the computer associated with the second camera (clock 2). Thus, the points from the first camera and the second camera became (x1i,y1i,t1i) and (x2j,y2j,t2j+Δt), respectively. An initial guess of the time offset Δt was chosen and a point from each camera was sampled. Because the sampled points from each camera did not exactly match up with each other, the data obtained from the second camera was interpolated as follows.
- First, for every point in from the first camera taken at time t1i, two points from the second camera were found that were taken as close as possible on either side of that time (i.e., points j− and j+ were found such that t2j
− +Δt<t1i<t2j+ +Δt). Next, a linear interpolation was performed on the two points from the second camera, (x2j− , y2j− ) and (x2j+ , y2j+ ), to approximate where the point would have been had it been seen at time t1i. If, for any point in the first data set, surrounding points in the second data set could not be found, that point in time was ignored. After ignoring such points and after interpolation, there was a set of corresponding (x,y) points that were designated as (x1k*, y1k*, x2k*, y2k*), 1≦k≦n. The time data in this data set was ignored because it made no difference in the subsequent computations. - Next, point matching was performed and the transformation parameters corresponding to the least squared error was selected. Specifically, in this working example the least median of squares technique was used because it is a robust method. This method was implemented by picking random pairs of corresponding points from the data set ((x1k*,y1k*, x2k*, y2k*), 1≦k≦n). A pair of points was the minimum number needed to compute the candidate transformation parameters (i.e., θ and (Δx,Δy)). The two pairs of randomly chosen points were (x1a*,y1a*), (x1b*,y1b*), (x2a*, y2a*), (x2b*,y2b*), and the angle θ was computed as:
and the translation (Δx,Δy) was:
Δx=x 1a *−x 2a*cos(θ)+y 2a*sin(θ)
Δy=y 1a *−y 2a*sin (θ)+y 2a*cos(θ) - This θ and (Δx,Δy) served as a trial solution for the calibration problem based on the two randomly chosen pair of points. The solution was evaluated by computing a list of the squared errors between corresponding points:
e k=(x 1k *−x 2k cos(θ)+y 2k*sin(θ)−Δx)2+(y 1k *−x 2k*sin(θ)−y 2k*cos(θ)−Δy)2
The quality of the solution was the median value of this list of squared errors. In this working example, our implementation, 100 random pairs of corresponding points were chosen and the transformation parameters θ and (Δx,Δy) that corresponded to the least median of squares were used. - The least median of square technique was used as above to compute the best θ and (Δx,Δy) for a whole series of values of a time offset value (Δt). Whichever Δt gave the minimum least median of squares was chosen as the best one, and the corresponding θ and (Δx,Δy) were used for the final solution.
- As an alternative to the least median of square technique described above, a least square solution could have been used to determine a minimum error. The least squares solution to the calibration problem computes the transformation parameters θ and (Δx,Δy) that minimize the sum of the Euclidean distances between corresponding points in (x1k*,y1k*,x2k*,y2k), 1≦k≦n. The angle, θ, is given by
- The above equation depends on the following equation, which computes the centroids of the points from each camera
- The translation (Δx,Δy) is then given by
The θ and (Δx,Δy) computed are the solution to the calibration problem. - The quality (or amount of error) of the solution is given by the average squared distance between corresponding points:
For a series of values of the time offset, Δt, the transformation parameters θ, (Δx,Δy) and e2 are computed. The average squared distance between corresponding points, e2, will be a minimum for some value of Δt. We take the corresponding values of θ and (Δx,Δy) at the minimum value of Δt as the solution to the calibration problem. - This least squares solution works well in spite of small errors in tracking the position of the person in the room. However, there can be outlier points due to gross errors in the process that tracks the person. These outlier points are (x, y) locations that deviate greatly from the actual location of the person. In this case, the least squares solution will be drawn away from the right answer, and the a technique that is robust to such errors should be used, such as the least median of square technique described above.
- The foregoing description of the preferred embodiments of the invention has been presented for the purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications and variations are possible in light of the above teaching. It is intended that the scope of the invention be limited not by this detailed description of the invention, but rather by the claims appended hereto.
Claims (20)
1. A method of measuring a relative pose between two cameras, comprising:
a first selection step for selecting a time offset value corresponding to a time shift between the two cameras;
a first calculation step for calculating a transformation parameter using the time offset value capable of transforming data in a coordinate frame of one of the two cameras into a coordinate frame of the other of the two cameras so as to obtain the relative pose; and
an application step for applying the time offset value to data from at least one of the two cameras;
wherein the data are measurements by the two cameras of a path of an object.
2. The method of claim 1 , wherein the object is moving in the path.
3. The method of claim 2 , wherein the object path is a person moving around a scene.
4. The method of claim 1 , further comprising:
a second selection step for selecting a plurality of time offset values; and
a second calculation step for calculating a corresponding transformation parameter for each of the plurality of time offset values.
5. The method of claim 4 , further comprising a third selection step for choosing one of the plurality of time offset values as a most correct time offset based on an error function.
6. The method of claim 5 , wherein the error function includes a least squares solution.
7. The method of claim 5 , wherein the error function includes a least median of square technique.
8. A computer-readable medium having computer-executable instructions for measuring a relative pose between two cameras, comprising:
selecting a time offset value corresponding to a time shift between the two cameras;
calculating a transformation parameter using the time offset value capable of transforming data in a coordinate frame of one of the two cameras into a coordinate frame of the other of the two cameras so as to obtain the relative pose; and
applying the time offset value to data from at least one of the two cameras;
wherein the data are measurements by the two cameras of a path of an object.
9. The computer-readable medium of claim 8 , wherein the object path is a person moving around a scene.
10. The computer-readable medium of claim 8 , wherein a plurality of time offset values are selected and a corresponding transformation parameter is calculated for each of the plurality of time offset values.
11. The computer-readable medium of claim 10 , wherein one of the plurality of time offset values is chosen as a most correct time offset based on an error function.
12. The computer-readable medium of claim 11 , wherein the error function includes a least squares solution.
13. The computer-readable medium of claim 11 , wherein the error function includes a least median of square technique.
14. A range imaging system for measuring a relative pose between two range cameras, comprising:
time offset value corresponding to a time shift between the two range cameras;
a transformation parameter that uses the time offset value to transforms data in a coordinate frame of one of the two range cameras into a coordinate frame of the other of the two range cameras so as to obtain the relative pose; and
a calibration module that applies the time offset value to data from at least one of the two range cameras;
wherein the data are measurements by the two range cameras of a path of an object.
15. The range imaging system of claim 14 , wherein the data is obtained from the two range cameras measuring a path of a moving object in a scene.
16. The range imaging system of claim 14 , further comprising:
a data synchronizer that can synchronize the data from each of the range cameras to produce synchronized data; and
a data matching processor that can match the synchronized data from each of the two range cameras.
17. The range imaging system of claim 16 , further comprising an error minimization processor that can use the synchronized data to compute a transformation parameter having a minimum error.
18. The range imaging system of claim 17 , wherein the data synchronizer uses the time offset value to synchronize the data and the time offset value is used by the error minimization processor to compute the transformation parameter.
19. The range imaging system of claim 17 , further comprising a coordinate selector that can select a base coordinate system for use in the calculation of the transformation parameter.
20. The range imaging system of claim 19 , further comprising an interpolation module that can interpolate data for use in the data matching processor.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US11/263,810 US20060116850A1 (en) | 2004-08-25 | 2005-10-31 | Relative range camera calibration |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US10/927,373 US7003427B2 (en) | 2000-04-05 | 2004-08-25 | Relative range camera calibration |
US11/263,810 US20060116850A1 (en) | 2004-08-25 | 2005-10-31 | Relative range camera calibration |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US10/927,373 Continuation US7003427B2 (en) | 2000-04-05 | 2004-08-25 | Relative range camera calibration |
Publications (1)
Publication Number | Publication Date |
---|---|
US20060116850A1 true US20060116850A1 (en) | 2006-06-01 |
Family
ID=35944488
Family Applications (6)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US11/237,406 Expired - Fee Related US7050928B2 (en) | 2000-04-05 | 2005-09-27 | Relative range camera calibration |
US11/237,405 Expired - Fee Related US7113885B2 (en) | 2000-04-05 | 2005-09-27 | Relative range camera calibration |
US11/263,810 Abandoned US20060116850A1 (en) | 2004-08-25 | 2005-10-31 | Relative range camera calibration |
US11/263,720 Expired - Fee Related US7392153B2 (en) | 2004-08-25 | 2005-10-31 | Relative range camera calibration |
US11/379,352 Expired - Fee Related US7526401B2 (en) | 2000-04-05 | 2006-04-19 | Relative range camera calibration |
US11/382,022 Expired - Fee Related US7321838B2 (en) | 2000-04-05 | 2006-05-06 | Relative range camera calibration |
Family Applications Before (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US11/237,406 Expired - Fee Related US7050928B2 (en) | 2000-04-05 | 2005-09-27 | Relative range camera calibration |
US11/237,405 Expired - Fee Related US7113885B2 (en) | 2000-04-05 | 2005-09-27 | Relative range camera calibration |
Family Applications After (3)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US11/263,720 Expired - Fee Related US7392153B2 (en) | 2004-08-25 | 2005-10-31 | Relative range camera calibration |
US11/379,352 Expired - Fee Related US7526401B2 (en) | 2000-04-05 | 2006-04-19 | Relative range camera calibration |
US11/382,022 Expired - Fee Related US7321838B2 (en) | 2000-04-05 | 2006-05-06 | Relative range camera calibration |
Country Status (1)
Country | Link |
---|---|
US (6) | US7050928B2 (en) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20120140063A1 (en) * | 2009-08-13 | 2012-06-07 | Pasco Corporation | System and program for generating integrated database of imaged map |
CN102608614A (en) * | 2010-11-15 | 2012-07-25 | 塞德斯股份公司 | Apparatus for monitoring a door with a 3d sensor |
Families Citing this family (21)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7892178B1 (en) | 2009-09-28 | 2011-02-22 | Impact Sports Technologies, Inc. | Monitoring device for an interactive game |
US7864159B2 (en) | 2005-01-12 | 2011-01-04 | Thinkoptics, Inc. | Handheld vision based absolute pointing system |
US8913003B2 (en) * | 2006-07-17 | 2014-12-16 | Thinkoptics, Inc. | Free-space multi-dimensional absolute pointer using a projection marker system |
JP4969279B2 (en) * | 2007-03-22 | 2012-07-04 | 本田技研工業株式会社 | Position detection method and position detection apparatus |
US9176598B2 (en) | 2007-05-08 | 2015-11-03 | Thinkoptics, Inc. | Free-space multi-dimensional absolute pointer with improved performance |
US7995055B1 (en) | 2007-05-25 | 2011-08-09 | Google Inc. | Classifying objects in a scene |
US20110035174A1 (en) * | 2008-04-07 | 2011-02-10 | Nxp B.V. | Time synchronization in an image processing circuit |
WO2010048146A1 (en) * | 2008-10-20 | 2010-04-29 | Carnegie Mellon University | System, method and device for predicting navigational decision-making behavior |
TWI402479B (en) * | 2009-12-15 | 2013-07-21 | Ind Tech Res Inst | Depth detection method and system using thereof |
WO2011123509A1 (en) * | 2010-03-31 | 2011-10-06 | Design & Test Technology, Inc. | 3d video processing unit |
MY170579A (en) * | 2010-06-07 | 2019-08-19 | Mimos Berhad | A method of determining an optimum transformation point |
US8760513B2 (en) * | 2011-09-30 | 2014-06-24 | Siemens Industry, Inc. | Methods and system for stabilizing live video in the presence of long-term image drift |
US10285141B1 (en) | 2012-09-19 | 2019-05-07 | Safeco Insurance Company Of America | Data synchronization across multiple sensors |
US10664994B2 (en) * | 2013-02-25 | 2020-05-26 | Cognex Corporation | System and method for calibration of machine vision cameras along at least three discrete planes |
CN103512558B (en) * | 2013-10-08 | 2016-08-17 | 北京理工大学 | A kind of conical target binocular video pose measuring method and target pattern |
WO2015082383A1 (en) * | 2013-12-06 | 2015-06-11 | Siemens Aktiengesellschaft | Method for determining a position of at least two sensors, and sensor network |
WO2018076154A1 (en) * | 2016-10-25 | 2018-05-03 | 成都通甲优博科技有限责任公司 | Spatial positioning calibration of fisheye camera-based panoramic video generating method |
KR102552815B1 (en) * | 2017-12-13 | 2023-07-07 | 콘메드 코포레이션 | Multi-density all suture anchor |
CN109218619A (en) * | 2018-10-12 | 2019-01-15 | 北京旷视科技有限公司 | Image acquiring method, device and system |
CN111121743B (en) * | 2018-10-30 | 2023-11-24 | 阿里巴巴集团控股有限公司 | Position calibration method and device and electronic equipment |
US11619708B2 (en) | 2020-07-28 | 2023-04-04 | Trackman A/S | System and method for inter-sensor calibration |
Family Cites Families (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US4639878A (en) * | 1985-06-04 | 1987-01-27 | Gmf Robotics Corporation | Method and system for automatically determining the position and attitude of an object |
EP1494476A1 (en) * | 1994-09-09 | 2005-01-05 | Canon Kabushiki Kaisha | Image processing method and apparatus |
US5905568A (en) * | 1997-12-15 | 1999-05-18 | The United States Of America As Represented By The Administrator Of The National Aeronautics And Space Administration | Stereo imaging velocimetry |
US6789039B1 (en) * | 2000-04-05 | 2004-09-07 | Microsoft Corporation | Relative range camera calibration |
EP1491038A2 (en) * | 2001-12-26 | 2004-12-29 | YEDA RESEARCH AND DEVELOPMENT CO., Ltd. | A system and method for increasing space or time resolution in video |
-
2005
- 2005-09-27 US US11/237,406 patent/US7050928B2/en not_active Expired - Fee Related
- 2005-09-27 US US11/237,405 patent/US7113885B2/en not_active Expired - Fee Related
- 2005-10-31 US US11/263,810 patent/US20060116850A1/en not_active Abandoned
- 2005-10-31 US US11/263,720 patent/US7392153B2/en not_active Expired - Fee Related
-
2006
- 2006-04-19 US US11/379,352 patent/US7526401B2/en not_active Expired - Fee Related
- 2006-05-06 US US11/382,022 patent/US7321838B2/en not_active Expired - Fee Related
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20120140063A1 (en) * | 2009-08-13 | 2012-06-07 | Pasco Corporation | System and program for generating integrated database of imaged map |
US9001203B2 (en) * | 2009-08-13 | 2015-04-07 | Pasco Corporation | System and program for generating integrated database of imaged map |
CN102608614A (en) * | 2010-11-15 | 2012-07-25 | 塞德斯股份公司 | Apparatus for monitoring a door with a 3d sensor |
Also Published As
Publication number | Publication date |
---|---|
US20060052975A1 (en) | 2006-03-09 |
US20060047471A1 (en) | 2006-03-02 |
US7050928B2 (en) | 2006-05-23 |
US7321838B2 (en) | 2008-01-22 |
US20060195292A1 (en) | 2006-08-31 |
US20060047472A1 (en) | 2006-03-02 |
US7526401B2 (en) | 2009-04-28 |
US20060195291A1 (en) | 2006-08-31 |
US7392153B2 (en) | 2008-06-24 |
US7113885B2 (en) | 2006-09-26 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US7050928B2 (en) | Relative range camera calibration | |
US6789039B1 (en) | Relative range camera calibration | |
CN102472609B (en) | Position and orientation calibration method and apparatus | |
US6438507B1 (en) | Data processing method and processing device | |
Dang et al. | Continuous stereo self-calibration by camera parameter tracking | |
US8953847B2 (en) | Method and apparatus for solving position and orientation from correlated point features in images | |
US10373339B2 (en) | Hyperspectral scene analysis via structure from motion | |
EP1882895A1 (en) | 3-dimensional shape measuring method and device thereof | |
US20060256200A1 (en) | Method and system for improving video metadata through the use of frame-to-frame correspondences | |
CN106526593B (en) | Sub-pixel-level corner reflector automatic positioning method based on the tight imaging model of SAR | |
Lourakis et al. | Camera self-calibration using the singular value decomposition of the fundamental matrix: From point correspondences to 3D measurements | |
JP7306192B2 (en) | Synthesis processing device, synthesis processing system, and synthesis processing method | |
US20230350067A1 (en) | Robust laser scanning for generating a 3d model | |
CN114529615B (en) | Radar calibration method, device and storage medium | |
O'Byrne et al. | A comparison of image based 3D recovery methods for underwater inspections | |
Kunz et al. | Stereo self-calibration for seafloor mapping using AUVs | |
JP4379626B2 (en) | Three-dimensional shape measuring method and apparatus | |
CN112419427A (en) | Method for improving time-of-flight camera accuracy | |
Habib et al. | DEM generation from high resolution satellite imagery using parallel projection model | |
Sekkati et al. | Direct and indirect 3-D reconstruction from opti-acoustic stereo imaging | |
Voges et al. | Timestamp offset determination between an actuated laser scanner and its corresponding motor | |
US20240114119A1 (en) | Image processing device, image processing method, and program | |
Negaharipour | On 3-D motion estimation from 2-D sonar image flow | |
EP3574472B1 (en) | Apparatus and method for registering recorded images | |
US20240045041A1 (en) | Multi-sensor navigation |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |
|
AS | Assignment |
Owner name: MICROSOFT TECHNOLOGY LICENSING, LLC, WASHINGTON Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:MICROSOFT CORPORATION;REEL/FRAME:034766/0001 Effective date: 20141014 |