WO2006050456A2

WO2006050456A2 - Automatic mirror stabilization

Info

Publication number: WO2006050456A2
Application number: PCT/US2005/039813
Authority: WO
Inventors: Yuri Ivanov
Original assignee: Honda Motor Co Ltd
Current assignee: Honda Motor Co Ltd
Priority date: 2004-11-02
Filing date: 2005-11-02
Publication date: 2006-05-11
Anticipated expiration: 2007-05-02
Also published as: US7373249B2; WO2006050456A3; US20060092534A1

Abstract

A system and a method (10) are disclosed for automatically controlling the position of a mirror (102) to stabilize the view in the mirror (102) in spite of changes in viewing position. Initially, a mirror (102) position is set by a user so that the user can comfortably view the reflection of a desired area. A tracking system estimates the user's viewing position. As a user's viewing position changes, the system reorients the mirror (102) such that the reflection in the mirror (102) is stabilized.

Description

AUTOMATIC MIRROR STABILIZATION

INVENTOR: YURI IVANOV, BOSTON, MA

CROSS REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority under 35 USC § 119(e) to U.S. Provisional

Patent Application No. 60/624,768 titled "Automatic Rear-View Mirror Stabilization" filed on November 2, 2004, which is incorporated by reference herein in its entirety.

BACKGROUND OF THE INVENTION

1. FIELD OF THE INVENTION

[0002] The present invention generally relates to the field of control systems of mirrors of a car, and more specifically, to automatic stabilization of mirrors with respect to head position.

2. DESCRIPTION OF THE RELATED ART

[0003] Conventional mirror positioning systems allow car mirrors to be moved manually or by using electronic controls located on the dashboard or elsewhere in the interior of the car within easy reach of the user. Some conventional mirror positioning systems also offer the feature of saving one or more settings that allow customization to the driver. For example, when one driver positions the mirrors at a comfortable viewing angle, those settings can be saved by the system. Then, if that driver or another driver alters the positions of the mirrors, the mirrors can be returned to the positions corresponding to the saved settings at the touch of a button. [0004] Despite these features, conventional mirror positioning systems remain limited in several ways. For example, the settings that can be saved in conventional mirror positioning systems are static. If the driver changes position in the seat, for example by slouching or shifting to one side or the other, the mirrors may no longer be at a desired angle to view the area behind the vehicle. In other words, as the driver shifts viewing position, what is seen in the reflection shifts as well. Accordingly, the driver's ability to perceive the space behind the vehicle that he set up the mirrors to view is impaired. To maintain the optimal view of the scene reflected in the mirror using conventional systems, the user must either retain precisely the same head position throughout an entire drive or repeatedly readjust mirrors. Maintaining the same head position throughout an entire drive, particularly a lengthy one, is uncomfortable and, in practice, not possible. It unduly restricts a driver's natural inclination/to change posture periodically. On the other hand, repeatedly readjusting mirrors manually or by manipulating electronic controls while driving is dangerous because it takes the driver's attention away from the road.

[0005] Accordingly, there is a need for a system and process of positioning mirrors that is capable of automatically responding to changes in a driver's viewing position so that the view reflected in the mirror is stabilized over time.

SUMMARY OF THE INVENTION

[0006] The present invention includes a system and a method for automatically controlling the position of mirrors to stabilize the view in spite of changes in viewing position. Initially, a mirror position is set by a user so that the user can comfortably view a desired area, for example a portion of the area outside of a vehicle. A tracking system estimates the user's viewing position. As a user's viewing position changes, the system reorients the mirrors such that the apparent reflection in the mirror is stabilized.

[0007] According to one embodiment of the present invention, a method is used to automatically stabilize the view in a mirror as a user's viewing position changes. The user sets the mirror to a desired orientation to see a desired reflected area. The mirror stabilization is engaged. By tracking the user's head position, the system registers a change in the user's viewing angle. The system adjusts the position of the mirror so that the view of the desired reflected area is maintained. While engaged, the system continues to compensate for changes in the user's head position.

[0008] In order to operate effectively, the method above relies on the system being accurately calibrated such that any given change in view position will result in the appropriate corresponding change in the mirror position to keep the user's view in the mirror stabilized. According to one embodiment of the present invention, the tracking system provides three-dimensional coordinates of the user's head, for example by means of a camera. A calibration is performed so that the coordinate transformation between the camera and the mirror is known. An auxiliary mirror, rigidly attached to the camera, is calibrated. Then the camera-mirror system can be calibrated through the calibrated auxiliary mirror. Then the resulting transformation is augmented by reflection of the coordinate system of the mirror through the auxiliary plane. The control surface for the movement of the mirror is found. The solution to the control surface determines how the mirror should be adjusted to maintain a constant reflection in the mirror as the user's viewing position changes.

[0009] The features and advantages described in the specification are not all inclusive and, in particular, many additional features and advantages will be apparent to one of ordinary skill in the art in view of the drawings, specification, and claims. Moreover, it should be noted that the language used in the specification has been principally selected for readability and instructional purposes, and may not have been selected to delineate or circumscribe the inventive subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The invention has other advantages and features which will be more readily apparent from the following detailed description of the invention and the appended claims, when taken in conjunction with the accompanying drawings, in which:

[0011] Figure ("FIG.") 1 illustrates a system used to automatically stabilize the view in a mirror in accordance with one embodiment of the present invention.

[0012] FIG. 2 is a flow chart of a method used to automatically stabilize the view in the mirror in accordance with one embodiment of the present invention.

[0013] FIG. 3 illustrates a set-up used to calibrate the mirror control system according to one embodiment of the present invention.

[0014] FIG. 4 is a flow chart of a method used to calibrate the mirror control system, according to one embodiment of the present invention.

[0015] FIG. 5 illustrates the coordinate systems according to the calibration procedure used in one embodiment of the present invention.

[0016] FIG. 6 illustrates a portion of the control surface of the mirror drive mechanism used in the system according to one embodiment of the present invention.

[0017] FIG. 7 is a graph of the control surface of the mirror drive mechanism. [0018] FIG. 8 illustrates a portion of the control surface of the mirror drive mechanism showing an initial position and an altered position of the mirror in response to an altered viewing position in order to stabilize the view in the mirror, indicated by parallel reflected rays.

[0019] FIG. 9 illustrates a set of solutions, each solution generated from a distinct head position.

[0020] FIG. 10 is a flowchart of the method of stabilizing the view reflected in the mirror in accordance with one embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] The Figures and the following description relate to preferred embodiments of the present invention by way of illustration only. It should be noted that from the following discussion, alternative embodiments of the structures and methods disclosed herein will be readily recognized as viable alternatives that may be employed without departing from the principles of the claimed invention.

[0022] Reference will now be made in detail to several embodiments of the present invention(s), examples of which are illustrated in the accompanying figures. It is noted that wherever practicable similar or like reference numbers may be used in the figures and may indicate similar or like functionality. The figures depict embodiments of the present invention for purposes of illustration only. One skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles of the invention described herein. [0023] FIG. 1 shows a system 100 for automatically controlling the position of mirrors

102 to stabilize the reflected view in spite of changes in viewing position 104. In this embodiment, system 100 includes a camera 106 for tracking head position. Many systems are available that use cameras to provide three-dimensional coordinates of a user's head, such as pfinder (http://vismod.media.mit.edu/vismod/demos/pfinder/), and IBM 3D head tracking system (http://www.research.ibm.com/ecvg/lisabjproj/headtrack.html). A conventional single camera method can be employed based on learning the differential relations between the apparent head dimensions and the depth in a projective geometric system of the calibrated camera lens. In alternative embodiments, camera 106 can be any device for receiving an electromagnetic reflection, e.g., a radar system, or any optical system, that can determine head position. In this implementation, camera 106 is coupled to control 108. In one embodiment, control 108 comprises a computing device for controlling a mirror drive mechanism (not shown), for example, for controlling electronic mirror positioners. In one embodiment, the coupling between control 108 and a mirror drive mechanism in mirrors 102 comprises a wire connection. Alternatively, the coupling is a wireless connection. Preferably, control 108 interprets signals from camera 106, determines the change in viewing position, and alters the position of mirrors 102 to compensate for the change in viewing position. For the sake of clarity, the rest of the description will focus on a single-mirror system. It will be readily recognized that the systems and methods described herein may be employed in a system with more than one mirror without departing from the principles of the present invention.

[0024] FIG. 2 is a flow chart of a method 200 used to automatically stabilize the view in the mirror in accordance with one embodiment of the present invention. The method 200 begins by engaging 202 the automatic mirror stabilization system. In some embodiments, the system is engaged when the car is started. Alternatively, the system can be engaged by the user turning on the system separately from the car. In some embodiments, the user sets the mirrors at the desired position prior to engaging the system. In other embodiments, the user sets the mirrors at the desired position after the system has been engaged. In step 204, the system tracks the user's head position. In step 206, the system determines if there has been a change in the user's head position. A change in the user's head position corresponds to a change in the user's viewpoint 104. If there has not been a change in the user's head position, the system continues to track the user's head position. If there has been a change in the user's head position, the system adjusts 208 the mirror to stabilize the reflected view. Control 108 sends signals to the positioning devices of the mirrors to alter the position of the mirrors. Then the system returns to tracking 204 the user's head position. Preferably, the system is capable of responding to changes in the user's viewpoint rapidly enough to keep up with the user's shifting^" position, for example, on the order of less than a second. Preferably, the system continues to iteratively track 204 the user's head position and adjust 208 the mirror to compensate for the change until the automatic mirror stabilization system is disengaged.

CALIBRATION

[0025] Referring now to FIG. 3, there is shown a system 300 used to calibrate a mirror control system, according to one embodiment of the present invention. The system includes a camera 106, a mirror 102, and an auxiliary mirror 302. A camera 106 mounted in a car in a position suitable for observing the user's face typically cannot see the rear-view mirror, for example. Hence, an auxiliary mirror 302 is used to enable the camera 106 to view the mirror 102 as a reflection in the auxiliary mirror 302. In this setup, the camera-mirror system can effectively be calibrated through the calibrated auxiliary mirror 302, after which the resulting transformation is augmented by reflection of the coordinate system of the mirror through the auxiliary plane. The task of the calibration of the camera-mirror setup can be accomplished according to the method described in FIG. 4.

[0026] FIG. 4 shows a flow chart of a method 400 used to calibrate the mirror control system, according to one embodiment of the present invention. In describing the calibration method 400 illustrated in FIG. 4, reference will be made to the coordinate systems of FIG. 5.

[0027] The task of the calibration of the camera-mirror setup can be formulated as

follows: Given "77 and %° _> find ^ » where indices c, a, I, andm stand for the camera

coordinate system (c), the auxiliary mirror coordinate system (a), the coordinate system of the Image of the mirror in the auxiliary mirror (I), and the mirror coordinate systems (m), respectively. Notationally, the transformation from coordinate system a to coordinates

system b is designated herein by T_a ^b .

[0028] At the outset, T° is unknown. Thus, the first step in calibration is to determine

402 the transformation from auxiliary mirror coordinates 504 to camera frame coordinates 502. For this purpose, in one embodiment, a calibration pattern is placed directly on the auxiliary mirror surface. Using any traditional camera calibration technique, the coordinate transformation matrix, which represents the coordinate transformation from the auxiliary mirror coordinate system 504 to the camera frame coordinates system 502 is found:

τ_a ^c = [ κ \ t_a ^c \ (l) [0029] R_a ^c is the matrix of rotation and t_a ^c is the translation between origins of the

coordinate systems. For mathematical clarity, we will proceed in homogeneous 3D coordinates, where the transformation matrix is a 4 by 4 matrix formed as follows:

[0030] At the outset, T₁ ⁰ is also unknown. Thus, the next step in the method 400 is to

determine 404 the transformation from the coordinates of the image of the mirror in the auxiliary mirror to the camera frame coordinates. Similarly to the calibration of the auxiliary mirror, using the image of the mirror 102 through the auxiliary mirror 302 results in finding the parameters of the transformation from the coordinates of the image of the mirror in the auxiliary mirror to the camera coordinates, T₁' .

[0031] The next step in method 400 is to determine 406 the transformation from the coordinates of the image of the mirror in the auxiliary mirror 506 to auxiliary mirror coordinates 504. T₁" can be determined from TJ^C and T° :

[0032] Then, the transformation from the mirror coordinates 508 to the auxiliary mirror coordinates 504 is determined 408. A reflection matrix, R₂510, is applied to T₁" to move

from the apparent mirror position coordinates 506 to the true mirror position coordinates 508. The reflection of T₁" through the XY plane of the auxiliary frame is found by reversing the

Z direction with the reflection matrix R, 510:

[0033] Thus, the transformation from the mirror coordinates 508 to the auxiliary mirror coordinates 504 is found using the following equation:

T: = K₁T₁" (5)

[0034] Lastly, the transformation from the mirror coordinates to the camera coordinates is determined 410. T^ is found using the following equation.

[0035] Thus, the calibration of the camera-mirror setup is complete. As described in the task formulation, given T° and T₁" , T^ has been found, using equation 6.

ESTIMATION OF CONTROL SURFACE PARAMETERS

[0036] Mirrors 102 are mounted on positioning units (not shown). For example, the positioning unit may be a pan-and-tilt unit. Such a unit is capable of sweeping some control surface that can range in shape from spherical to a more complicated curved surface due in part to misalignment of the axis and lateral displacement of the servo motors.

[0037] FIG. 6 illustrates a portion of the control surface 601 of an example mirror drive mechanism used in the system according to one embodiment of the present invention. Control surface 601 comprises points for which the center point of mirror 102 can positioned. A three dimensional graph of an example control surface 702 derived from the inventors'

10 experimental data is shown in FIG. 7. The dots 702 on the control surface 701 represent data points collected during the calibration process.

[0038] The control surface 601 can be approximated with a general quadratic surface that is described by a polynomial of a second degree:

z = a_xx² + a₂y² + a₃xy + O₄JC + a₅y + a₆ (7)

[0039] A least squares approximation to this surface is found by fixing a point on the mirror during the calibration process and measuring its position while changing the mirror orientation. The corresponding coordinate transformation allows the computation of the position of this point in the camera coordinate system.

[0040] Let/?_m denote a position of such point in the mirror coordinate frame and p_m

denote its homogeneous coordinates. Then the point p_c , in the camera coordinates is

computed with the corresponding transformation:

[0041] Denoting the x, y and z components of the z-th point by x,, y,, Z₁ respectively, the parameters of the surface given by equation 7 can be found as a quadratic regression:

Z = a^TD (9)

[0042] where a is a 6-vector of coefficients and Z and D are formed as follows:

Z = [ Z₁ Z₂ - z_N ] (10)

D = ^{• • • χ} _Ny

(H)

X₁ X₂ ^{• • ■} χ_N

1 1 1

[0043] The solution for the coefficients, a, is found in the least squares sense:

[0044] Alternatively, the surface can also be computed in the mirror coordinates by

first selecting one of the mirror frames as a basis, T^, , and then transforming all collected

points to that basis:

= V m' ) ^Jm,,Pn (13)

[0045] This approach may provide a better approximation in some geometrically degenerate cases, such as the case where the real surface is not a valid function of the coordinates.

[0046] Additionally, estimations can be made of the set of functions that allow the actual control signals for pan and tilt of the mirror drive mechanism to be derived:

[0047] The inverses of these functions are used to find the initial conditions:

\x=X{pan,tih)

[0048] In experiments conducted by the inventors, the last three functions have been found to be well approximated by the quadratic expansion similar to that of equation 11. However, the pan signal is sometimes more problematic. Thus, a 3^rd degree full polynomial expansion can be used:

3 pan= ^{^}Y a_p,q,rx^py^qz^r (16) p,q,r=o

[0049] This equation represents a general polynomial where a_Pιq,_r are the unknown coefficients of the terms x?y^q/. These terms can be determined by taking a pseudo-inverse, similar to the solution of equation 12 above.

ITERATIVE SOLUTION FOR MIRROR POSITION

[0050] Referring now to FIG. 6, there is shown a head position F and the reference

reflection direction r . For any new position F , the solution for the appropriate mirror

position is the position of the mirror center P , on the control surface such that the resulting

viewing vector V from F in the direction of P , is reflected around the normal, n , to be

parallel with r . As an example, FIG. 8 shows the solution P₂ for the position of the mirror

center on the control surface given the new head position F₂ so that the reflection around the

normal, n₂ , results in a ray, r₂ , that is parallel with r . This indicates that the reflected view

has been maintained despite a change in viewing position.

[0051] Given the normal, n , and the position of the mirror center, P , the reference vector can be determined as follows:

r =2(V n)n -V (17)

[0052] The mirror position necessary to maintain the pre-set reflection direction, r ,

can be found by specifying the head position F , and solving the system of equations for the intersection of a ray through the point F with the control surface such that the reflected ray is parallel with r .

[0053] The normal to the surface z = f(x,y) is given by:

[0054] The point of intersection P of the viewing ray with the control surface can be

found from a joint solution to the parametric line equation, P = F + Vt , where V is a vector

in the direction from F to P and the surface equation is given by equation 7. This is a

system of 7 equations with 10 unknowns (P , V , t, and n ). Additionally, vector V is

constrained to be of unit length and r is constrained to be a reflection of V through the surface normal n . The resulting system of equations is:

+ a₆

[0055] This is a system of 10 equations (lines 2 and 3 provide three constraints each) with 10 unknowns. The direct solution is difficult due to the presence of the quadratic terms in lines 1, 3, and 4. However, this system can be iteratively solved using the Newton- Raphson method, for example.

[0056] The Newton-Raphson solution to the system of equations given in 19 involves first transforming the system of equations to the homogeneous form,

f(x) = 0 (20)

[0057] Then starting with the initial guess, xo, the solution is iteratively obtained as:

[0058] Where the matrix J is the Jacobian of equation 20.

[0059] Equation 19 can be solved iteratively by means of recurrence of equation 21 setting xo to the known values initially set by the user:

[0060] A set of solutions resulting from the initial set up and then changing the head position is shown in FIG. 9. In this figure, the viewing position changes along the circular pattern shown by a series of points 901 on the right. Each point induces a new set of the system parameters that are illustrated here. The surface normal originating from the point on the surface that provides the constant direction of the reflection rays 904 is designated by

lines 902. Lines 903 show the viewing direction. These lines 903 correspond to V in FIG. 4 and 6. Note that reflection rays 904 are all parallel to each other. Thus, the direction of the reflection is constant regardless of the viewing position.

[0061] Having found the required position P of the mirror center on the control surface, the values for pan and tilt controls of the mirror drive are determined via the approximation of equation 14. Thus, the mirror can be positioned appropriately to maintain the constant reflection direction.

[0062] Referring now to FIG. 10, there is shown a flow chart describing a method used to stabilize the view in a mirror, in accordance with one embodiment of the present invention. In describing the method 1000 shown in FIG. 10, reference will be made to the equations described above.

[0063] The method 1000 describes the steps of the system after the user has adjusted the mirror to a comfortable viewing angle. The method 1000 begins with initialization 1002 of the system. The starting viewing position is determined 1004. This point corresponds to

where the user's viewpoint is, and is referred to in earlier figures as F . Preferably, this reference point is obtained from a properly calibrated camera positioned to view the user's face. Then the initial pan and tilt values from the mirror drive are determined 1006. These values identify the starting position of the mirror after it has been comfortably adjusted for the user. In the next step, the values of x and y correspond to the starting position of the mirror on the control surface, are determined 1008, preferably by plugging the pan and tilt values taken from the mirror drive into the system of equations given in 15. Then z, the last coordinate corresponding to the starting position of the mirror on the control surface is determined 1010, preferably by plugging in the determined values of* andjy into equation 7 which approximates the control surface. After the initial position of the mirror (x, y, and z) is

determined, the viewing vector V , the normal to the control surface n , and t are determined 1012, preferably using lines 2 and 3 of the system of equations given in 19. In the next step, the reflection vector r is determined 1014, preferably using equation 17. Note that in

equation 17, all the unknowns (V and n ) have already been determined. At this point, all of the components of the initial conditions xo from equation 23 have been determined and thus xo can be determined 1016. Thus concludes the initialization 1002 of the system.

[0064] Upon the completion of the initialization 1002, each of the steps of the method

1000 shown inside of box 1018 are completed for each new video frame. The new viewing position is determined 1020. Preferably, the new viewing position is determined in a similar manner as was used to determine the reference viewing position described above with reference to step 1004. If the new viewing position is the same as the previous viewing position, then no alterations are necessary to the mirror position and the method is complete for that video frame. However, if the new viewing position for a video frame is a change in position from the previous viewing position, then the method proceeds with the next steps in box 1018. The iterative solution to the system of equations 19 is obtained 1022, preferably by iterating equation 21, beginning withxo, until convergence is achieved. Then, the pan and tilt values for the mirror drive to reposition the mirror is determined 1024. These pan and tilt values are preferably determined using the system of equations given in equation 14. These pan and tilt values position the mirror according to the iterative solution obtained. This compensates for the change to the new viewing position determined in step 1020 such that the reflected view is stabilized. [0065] Upon reading this disclosure, those of skill in the art will appreciate still additional alternative structural and functional designs for a system and a process for automatically controlling the position of mirrors to stabilize the view in spite of changes in viewing position through the disclosed principles of the present invention. Thus, while particular embodiments and applications of the present invention have been illustrated and described, it is to be understood that the invention is not limited to the precise construction and components disclosed herein and that various modifications, changes and variations which will be apparent to those skilled in the art may be made in the arrangement, operation and details of the method and apparatus of the present invention disclosed herein without departing from the spirit and scope of the invention as defined in the appended claims.

Claims

WHAT IS CLAIMED IS:

1 1. A method for stabilizing a first view in a mirror, the method comprising the steps of:

2 identifying the first view in the mirror with respect to a first viewpoint of a user;

3 automatically tracking a change in viewpoint of said user to a second viewpoint; and

4 automatically adjusting a position of the mirror in response to said change in

5 viewpoint so that substantially the first view is presented to said user when

6 said user is at said second viewpoint.

/ 2. The method of claim 1, wherein the mirror comprises a rear-view mirror.

/ 3. The method of claim 1, wherein said user comprises a driver of a motor vehicle.

1 4. The method of claim 1, wherein automatically tracking a change in viewpoint of said user comprises monitoring a head position of said user with a camera.

1 5. The method of claim 1, wherein adjusting the position of the mirror comprises controlling electronic mirror positioners.

6. A method for maintaining a substantially constant reflection direction in a mirror despite a change from a first viewpoint to a second viewpoint, the method comprising the steps of: PATENT identifying a first reflection direction in the mirror having a first position, the first reflection direction determined from the first viewpoint and the first position of the mirror; automatically tracking a change in viewpoint from the first viewpoint to the second viewpoint; and responsive to said change in viewpoint, automatically reorienting the mirror from said first position to a second position, wherein the second viewpoint and said second position determine a second reflection direction, wherein said second reflection direction is substantially parallel to said first reflection direction.

7. The method of claim 6, wherein the mirror comprises a rear-view mirror.

8. The method of claim 6, wherein automatically tracking a change in viewpoint from the first viewpoint to the second viewpoint comprises monitoring with a camera a head position of a user.

9. The method of claim 6, wherein automatically reorienting the mirror from said first position to said second position comprises sending a signal to a positioning device of the mirror.

10. A system for stabilizing a first view in a mirror, the system comprising: a first device positioned to observe changes in a viewing position of a user; a control coupled to said camera; and a mirror coupled to said control, wherein said mirror is positioned responsive to observed changes in said viewing position of said user to maintain a substantially constant first view. PATENT

11. A method of calibrating an automatic mirror stabilization system, the system comprising a first mirror, a second mirror, and a camera, the method comprising the steps of: determining a transformation from a coordinate system associated with the first mirror to a coordinate system associated with the camera; determining a transformation from a coordinate system associated with an image of the second mirror in the first mirror to said coordinate system associated with the camera; determining a transformation from said coordinate system associated with said image of the first mirror in the second mirror to a coordinate system associated with the second mirror; determining a transformation from said coordinate system associated with the second mirror to said coordinate system associated with the first mirror; and determining a transformation from said coordinate system associated with the second mirror to said coordinate system associated with the camera.

12. The method of claim 11, wherein the second mirror comprises an auxiliary mirror temporarily in a fixed position relative to the camera.