Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
  • H04N23/60—Control of cameras or camera modules
  • H04N23/695—Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects

Definitions

• This invention relates to the field of video systems, and in particular to the automation of cameras having pan, tilt, and zoom adjustments.
• mapping of object, or "world”, coordinates to image coordinates can be used in a variety of applications to either control the field of view of the camera to include particular world coordinates, or to determine world coordinates based on image coordinates.
• U.S. patent application "An Optimized User Interface for Selecting and Adjusting a Camera in a Multiple Camera System", Attorney docket 23,353, US serial number 09/002,105, filed 12/31/97 for S. Sengupta, D. Lyons, and T.
• Murphy discloses a user interface wherein the user selects a location on a displayed image, such as a floorplan, and in response to this selection, a controller adjusts one or more cameras so that their fields of view include the world coordinate corresponding to the selected location.
• a controller adjusts one or more cameras so that their fields of view include the world coordinate corresponding to the selected location.
• Reese discloses a technique for automatically tracking a person or object as the person or object traverses from one camera's field of view to another by automatically controlling a camera based on world coordinates that are derived from images in another camera's field of view.
• sm Q(RP + T), (1) where m denotes the image coordinates of a world point P, s denotes the projective scale factor, Q is an internal camera calibration matrix, and R and T are the rotation and translation that align the camera and world coordinate systems.
• m denotes the image coordinates of a world point P
• s denotes the projective scale factor
• Q is an internal camera calibration matrix
• R and T are the rotation and translation that align the camera and world coordinate systems.
• (xO, yO) denotes the optical center, or "principal point" on the camera's imaging plane, and f denotes the focal length.
• f denotes the focal length.
• a common x and y focal length is shown; alternatively, the "f in the first row of the matrix Q can be expressed as "fx”, the x focal length, and the "f" in the second row can be expressed as "fy”, the y focal length.
• the optical center of a camera is the intersection of the camera's optical axis with the camera's imaging plane.
• the parameters of the matrix Q are typically expressed in pixels.
• a camera is typically calibrated by collecting data using varying camera settings, and then processing the data to determine, or estimate, the camera's optical center and focal length.
• a calibration target is built by identifying the 3-D coordinates of reference points. The corresponding coordinates of these points in the camera image are determined for a variety of camera settings. Curve fitting techniques are employed to determine the parameters of equations (1) and (2), above, based on the mappings of the camera image coordinates and the actual 3-D coordinates of each reference point.
• U.S. patent 5,930,740, issued 27 July 1999 discloses a camera calibration method comprising establishing reference points within a 3-D space and triangulating the camera position and orientation with respect to the reference points.
• a database is created that maps the camera settings corresponding to each of a variety of camera positions and/or orientations.
• a zoom camera To calibrate a zoom camera, it is necessary to determine the relationship between the control that is used to change the focal length and the effect produced by this change. For example, if a stepping motor is used to change the separation between lenses in a camera to increase or decrease the effective focal length, there is typically a non-linear relationship between a single step of the motor and the amount of change in the focal length. Calibration provides an algorithmic mapping to determine the camera's focal length at each step of the stepping motor, or other device that effects a change to the camera's focal length.
• the term "zoom setting" is used herein as a generic term for a control of the camera that can be used to provide a repeatable zoom adjustment. That is, a "zoom setting of 7" will repeatedly provide substantially the same focal length, although this focal length may be unknown (before calibration). Conventionally, a camera's zoom setting is inversely proportional to the focal length.
• a zoom setting of "8" for example, conventionally provides a focal length that is shorter (lower magnification) than a zoom factor of 7, a zoom setting of "6" provides a longer focal length, and so on.
• an algorithm is often determined to mathematically determine the focal length corresponding to each possible zoom setting. To determine the algorithmic mapping between a camera's zoom setting and corresponding focal length, a set of measurements z(i) and f(z(i)) are collected.
• the corresponding focal length value f(z(i)) is determined, using for example the calibration techniques presented above. Because the focal length is inversely proportional to zoom settings (a high zoom setting corresponds to a short focal length), the conventional model used to determine a focal length corresponding to a zoom setting is an equation in the form:
• the order of the polynomial in the denominator of equation (3) is, in general, unknown. Experience indicates that an order of 2 is optimal for most camera zoom configurations.
• the coefficients aO, al, a2 that best match the set of measurements z(i), f(z(i)) for the camera are determined by minimizing the objective function:
• the larger values of f(z(i)) at low zoom settings z(i) are substantially less accurate than the lower values of f(z(i)) at high zoom settings z(i).
• the larger and more inaccurate measures of f(z(i)) will have a greater influence on the objective function than the smaller and more accurate measures of f(z(i)), and hence the determination of the coefficients aO, al, and a2 will be substantially affected by these inaccurate measures at low zoom settings.
• a first image is captured, at the camera's minimum zoom setting (zoomed-in as much as possible), that contains a scene containing distinguishing details. After capturing this image, the camera is maintained at the same orientation, set to its maximum zoom setting (zoomed- out as much as possible), and a second image is captured. The first image is scaled down to form a template image, using the known ratio of a camera's minimum and maximum focal lengths.
• the location of this template image within the second image is determined by searching the area in the vicinity of the center of the second image for a block that corresponds to the template image. Because the orientation of the camera is fixed between the first image and the second image, the location of this block defines the optical center, or principal point, of the camera's image plane. Having determined the optical center of the camera's image plane, the focal length of the camera at various zoom settings can be determined by measuring the displacement of the image at the optical center corresponding to a slight adjustment of the camera through a known angle at each of the various zoom settings.
• Conventional curve fitting techniques are used for determining an algorithmic mapping between zoom settings and corresponding field of view. In a preferred embodiment, the curve fitting is based on a determination of lens power rather than focal length. The use of a lens power measure facilitates a linear determination of the coefficients of the mapping equation, and provides an objective function that is less sensitive to inaccurate focal length determinations at low zoom settings.
• FIG. 1 illustrates an example determination of the optical center of a camera system by using a template image in accordance with this invention.
• FIGs. 2 A and 2B illustrate a technique for determining the focal length of a camera lens system based on a displacement of objects in an image in accordance with this invention.
• FIG. 3 illustrates the correspondence between the focal length of a camera and an apparent displacement of an object in an image as the camera is rotated.
• FIGs. 4A and 4B illustrate an example flow diagram for a camera calibration process in accordance with this invention.
• FIG. 5 illustrates an example camera system in accordance with this invention.
• the same reference numerals indicate similar or corresponding features or functions.
• FIG. 1 illustrates an example sequence for the determination of the optical center (xO, yO) 101 of a camera system (FIG. 5) by using a template image 130 in accordance with this invention.
• Images 110 and 120 correspond to images captured by the camera system at two different zoom settings but at identical camera orientations (constant pan and tilt settings).
• Image 120 illustrates an image corresponding to a low zoom setting (long focal length) of the camera system.
• this image 120 contains one or more clearly distinguishable objects, having a high degree of image-information, such as distinct edges and vertices.
• Image 110 illustrates an image corresponding to a higher zoom setting (shorter focal length) of the camera system, and corresponds to a "zoomed-out" image corresponding to the image 120.
• the image 110 is captured at the minimum focal length, and image 120 is captured at the maximum focal length, so that the magnification ratio of the two images 110, 120 is known. That is, for example, a zoom lens is typically specified in terms of the ratio of focal lengths between maximum and minimum zoom. If the camera system has a "16x" zoom capability, and the images 110 and 120 are set to minimum and maximum focal lengths, respectively, the scale of image 120 will be 16 times the scale of image 110. Other techniques common in the art can alternatively be used to determine the scale of image 120 compared to image 110.
• the template image 130 is created by scaling the magnified image 120 by the scale factor to produce the template image 130 at the same scale as the less magnified image 110.
• This template image 130 is then compared to the image 110 to determine the location of an area 140 within image 110 that matches the template image 130.
• the search for a matching area 140 is preferably commenced at the image center of the image 110, and progresses outward from this image center until the matching area is found. In this manner, the likelihood of matching the template image 130 to a similar area at another location in the image 110 is minimal.
• the image 120 is preferably selected as an image having a high image-information content, the determination of a location of the template 130 within the image 110 can be very accurate, using pattern matching techniques that are common in the art.
• the location of the template 130 within the image 110 corresponds to the optical center of the camera system.
• a very accurate determination of the optical center of the camera system can be obtained by zooming into an arbitrary area within the camera's field of view that contains a distinguishable object image. Thereafter, the determination of the optical center can be automated, as discussed further below.
• panning for a distinguishable object in the camera's field of view can also be automated, thereby providing a completed automated camera calibration process.
• a template-finding approach to pattern matching is also used to determine the focal length of the camera system in a preferred embodiment of the invention.
• FIG. 3 illustrate a method of determining a camera's focal length in accordance with the principles of this invention.
• FIG. 2A illustrates a camera image 200. Within the camera image 200, a variety of object images 211, 212, etc. are displayed.
• FIG. 2B illustrates a second camera image 200' that is produced when the camera is tilted down slightly.
• the object images 211', 212', etc. in FIG. 2B correspond to the object images 211, 212 of FIG. 2A, and are illustrated as being displaced slightly, relative to their locations in FIG. 2A.
• FIG. 3 illustrates a lens 250, an image plane 260, and a focal point 270 that is located at one focal length f 255 from the lens 250.
• the optical axis of the lens 250 is illustrated by the line 251. Also illustrated in FIG. 3 is the lens 250 tilted down slightly, identified as lens 250', with a corresponding change in the optical axis, illustrated by the line 251', and corresponding change in focal point 270'. Assuming that the image plane 260 is fixed relative to the lens 250, and thus also rotates, the new location of the image plan 260 is identified at 260'.
• the distance D 265 corresponds to vertical displacement of the focal point.
• the displacement D 265 corresponds to an apparent displacement d 220 of the image on the image plane 260'.
• the image 200' on the image plane 260' corresponds to an view that is slightly lower than the original view, and an object at the optical center of the image plane will appear to have been shifted by an amount d 220 compared to the original image 200.
• the distance D 265 and the focal length F 255 are in units of actual distances; for the purposes of image processing, these actual distances must be converted to image units.
• the displacement d 220 represents a conversion of the distance D into image units
• the image focal length f 225 represents a conversion of the physical focal length F 255 into image units.
• the tangent of the . angle A is equal to the ratio of the distance D to the focal length E.
• the image focal length f 225 can be determined as:
• the displacement of the image can be determined by measuring the distance that each of the object images 211, 212, etc. are displaced between FIGs. 2A and 2B.
• the distances dl 221 and d2 223 are shown for clarification purposes.
• the displacement of the image can be determined as a composite of the individual object movement distances dl, d2, etc., such as an average, a least square error estimate, and so on.
• the above focal length determination requires an automatic identification and location determination of corresponding points (211-211', 212-212', etc.) from one image to another, and is adversely affected by variances among determined distances dl, d2, etc.
• This variance can be significant if the camera image 200 contains object images that do not contain sharp edges, or contains similar object images in close proximity to each other.
• a template region 230 of an image 200 is used in lieu of the entire image, to reduce computation complexity, and to minimize the variances among determined distances between corresponding points of successive images.
• a template 230 is created by extracting a small image area from the first image 200, preferably centered about the above determined optical center 101 of the camera image 200.
• the amount of displacement that is introduced by the tilting through the angle A is based solely on the location of the template 230 in the new image 200'.
• the difference between the location of the template 230 in the new image 200' and the location of the template 230 in the original first image 200 corresponds to the displacement d 220 of FIG. 3, and equation (5), above.
• the selection of the template 230 as a small area at the optical center 101 facilitates an easy and accurate determination of the displacement d 220.
• the template 230 can be easily found in the new image 200' by searching along the displacement direction from the optical center 101' in the new image 200', and little, if any, ambiguity will be present if the selected template area 230 is substantially smaller than the entire image 200, 200'.
• the tilt-determined focal length f corresponds to a focal length in the vertical direction. If potentially different horizontal and vertical focal lengths, fx and fy, need to be determined, the template-matching process is repeated to determine fx via a slight panning movement of the camera system, in lieu of a tilt movement.
• the conventional camera calibration process provides an algorithmic mapping between a zoom setting of the camera system and the corresponding focal length achieved at this zoom setting.
• the above tilt-displacement template- matching technique is applied for a variety of zoom settings to collect sets of data for subsequent curve fitting to determine coefficients of a polynomial that models the relationship between zoom settings and focal lengths.
• a "lens power" term p is used to determine the algorithmic mapping between the zoom settings and the focal length, where p is the inverse off:
• the lens power is the inverse of the focal length, and the focal length is inversely proportional to the zoom setting
• the objective function is expressed as:
• z(i) m i n is preferably the zoom setting at the minimum camera zoom setting
• z(i) max is the zoom setting at the maximum zoom setting
• s is the magnification factor that characterizes the ratio of the minimum and maximum zoom capabilities of the camera system.
• FIG. 4A and 4B illustrate example flow diagrams for a camera calibration process in accordance with this invention.
• FIG. 4 A illustrates a determination of the optical center, or optical center, of the camera
• FIG. 4B illustrates a determination of lens power as a function of the zoom settings of the camera.
• the camera is oriented so that it contains a high amount of image-information at its maximum zoom setting, such as object images with clearly defined edges, and the image is captured as a reference image.
• This process is typically performed manually, although an automated panning and tilting process can be employed, such that the camera searches until it locates and captures a sufficiently discriminating image.
• a template is created by scaling the reference image down by the maximum zoom-ratio of the camera. That is, for example, if the camera is a "16X" zoom camera, the template is created by scaling the reference image down by a factor of 16.
• the orientation used to capture the reference image is fixed while an image is captured 430 at the minimum zoom setting. Note that by maintaining a fixed orientation, the optical center the captured image will be common with the optical center of the reference image.
• image pattern matching techniques are used to determine the location of the template within the captured image at the minimum zoom setting, and this location defines the optical center of the camera.
• the loop 450-464 of FIG. 4B illustrates an example determination, or calibration, of the lens power corresponding to each of a plurality of zoom settings.
• the number of zoom settings selected for calibration to lens power is dependent upon the degree of precision required. Typically, between five and ten settings across the range of zoom settings are calibrated, to provide a set of five to ten points for a polynomial curve fitting of zoom settings to lens power, at 470. If the order of the polynomial is N, at least N+l calibration points must be provided. In a preferred embodiment, a polynomial of order 2 is used, and thus at least 3 calibration points must be provided.
• the camera is oriented to an information laden center image, and the image is captured as a reference image at the current zoom setting. For convenience, the same orientation that was used to capture the reference image used to determine the optical center of the camera, at 410, can be used.
• a template is created by selecting a center portion of the reference image. Because the optical center of the camera is known, and the determination of the focal length, or its inverse, is based solely on the displacement of the optical center, as discussed above, the scaled template can be as small as a single pixel.
• the size of the template is inversely proportional to the zoom setting, and, for effective pattern matching, is at least 5x5 pixels in size.
• the camera is offset by a known angle A, preferably in a vertical direction, to minimize the variables that may be introduced by a multi-dimensional displacement, and the displaced image is captured.
• the location of the template is determined within the displaced image. The distance between the optical center of the image and the displaced location of the template defines the image displacement D corresponding to the known angular displacement A. If the displacement of the camera is constrained to a vertical angle, and the optical center of the image is defined as an origin (0, 0), the displacement D is the magnitude of the vertical coordinate of the location of the template.
• the lens power, p, corresponding to the particular zoom setting is thereafter determined, at 462, using the inverse of equation (5) above:
• FIG. 5 illustrates an example camera system 500 that is configured in accordance with this invention.
• the camera system 500 includes a camera 510, an image processing system 550, a controller 560, and a set of calibration parameters 570.
• the orientation of the camera is adjusted via a tilt motor 520 and pan motor 530, and the focal length of the camera is adjusted by the zoom motor and lenses 540.
• the controller 560 controls the motors 520, 530, 540 to execute the functions and sequences exemplified in FIGs. 4A-4B to derive the calibration parameters 570.
• the image processing system 550 performs a variety of functions, including: the capture of the images; the scaling of the reference image to create the scaled template that is used to determine the optical center; the determination of the location of each template in each image of the pair of images associated with each zoom setting; the corresponding determination of the lens power and/or focal length corresponding to each zoom setting; and the determination of the algorithmic mapping of the zoom setting to the lens power and/or focal length.
• the controller 560 receives requests for desired fields of view 580 and applies the calibration parameters and other factors to determine the appropriate commands to communicate to the motors 520, 530, 540, such as the appropriate zoom setting for the zoom motor 540, to achieve the desired field of view 580.

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Length Measuring Devices By Optical Means (AREA)
• Studio Devices (AREA)
• Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)

Applications Claiming Priority (3)

Publications (1)

Family

ID=24166232

Family Applications (1)

Country Status (4)

Families Citing this family (7)

Family Cites Families (1)

• 2001
  • 2001-03-28 WO PCT/EP2001/003556 patent/WO2001076261A2/en not_active Application Discontinuation
  • 2001-03-28 EP EP01933772A patent/EP1275240A2/de not_active Withdrawn
  • 2001-03-28 JP JP2001573805A patent/JP2003530039A/ja active Pending
  • 2001-04-04 TW TW090108053A patent/TW505820B/zh not_active IP Right Cessation

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events