Classifications

• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
  • B60K35/00—Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
  • B60K35/00—Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
  • B60K35/20—Output arrangements, i.e. from vehicle to user, associated with vehicle functions or specially adapted therefor
  • B60K35/21—Output arrangements, i.e. from vehicle to user, associated with vehicle functions or specially adapted therefor using visual output, e.g. blinking lights or matrix displays
  • B60K35/23—Head-up displays [HUD]
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
  • B60K35/00—Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
  • B60K35/90—Calibration of instruments, e.g. setting initial or reference parameters; Testing of instruments, e.g. detecting malfunction
• • G—PHYSICS
  • G02—OPTICS
  • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
  • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
  • G02B27/01—Head-up displays
  • G02B27/017—Head mounted
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
  • B60K2360/00—Indexing scheme associated with groups B60K35/00 or B60K37/00 relating to details of instruments or dashboards
  • B60K2360/40—Hardware adaptations for dashboards or instruments
  • B60K2360/48—Sensors
• • G—PHYSICS
  • G02—OPTICS
  • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
  • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
  • G02B27/01—Head-up displays
  • G02B27/0179—Display position adjusting means not related to the information to be displayed
  • G02B2027/0181—Adaptation to the pilot/driver
• • G—PHYSICS
  • G02—OPTICS
  • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
  • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
  • G02B27/01—Head-up displays
  • G02B27/0179—Display position adjusting means not related to the information to be displayed
  • G02B2027/0183—Adaptation to parameters characterising the motion of the vehicle
• • G—PHYSICS
  • G02—OPTICS
  • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
  • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
  • G02B27/01—Head-up displays
  • G02B27/0179—Display position adjusting means not related to the information to be displayed
  • G02B2027/0187—Display position adjusting means not related to the information to be displayed slaved to motion of at least a part of the body of the user, e.g. head, eye

Definitions

• the invention is concerned with a method and a processing unit for determining calibration parameters for a calibration function of a head- mounted device, HMD, for use in a vehicle.
• the invention also provides such a HMD with the calibration function that uses the calibration parameters.
• HMD head- mounted display
• VR and AR headsets It is fundamentally important for the use of VR and AR headsets to precisely localize the HMDs during use (6-DOF i.e. 6 degrees of freedom: translation in the X, Y and Z directions as well as rotation around the aforementioned axes, i.e. roll around x-axis, pitch around y-axis, and yaw around z-axis) in order to be able to present the virtual graphical content to the user in a visually correct manner.
• the localization of the HMDs is called tracking.
• inside-out from the HMD to the outside
• outside-in tracking from the outside to the glasses.
• outside-in tracking for stationary use cases with external tracking sensors (e.g. via infrared) has been the state of the art.
• HMD development will also focus on mobile HMDs - without external trackers - with inside-out tracking.
• These mobile HMDs without external tracking sensors use, for example, cameras built into the HMD for computer vision based tracking (see e.g. US 2018 / 0200614 A1).
• Today's use cases are limited to static environments.
• the technology for inside-out tracking of mobile HMDs in a dynamic environment e.g.
• a 3-DOF mobile FIMD can be used.
• a 3-DOF FIMD has only one Inertial Measurement Unit (IMU) installed, which determines the rotations of the FIMD through multiple inertial sensors such as accelerometers and angular rate sensors.
• IMU Inertial Measurement Unit
• the translation in X, Y, and Z directions is assumed to be 0/0/0 and only the rotations are considered from this point.
• Inertial sensors such as accelerometers and angular rate sensors (angular velocity sensors) are generally subject to error. Without the compensating correction factor of e.g. tracking by camera or infrared sensors, purely IMU- based systems tend to drift. Drift in this case is the shift of the origin of the system to which the rotations are calculated, i.e. angular values caused by the drift are added. Particularly with larger accelerations and longer use without calibration the drift can become very important. Especially for the use case of a mobile FIMD with 3DOF tracking in a vehicle, the drift over time is crucial. If this is too large, the visual representation of the virtual environment shifts too much to the actual direction of travel for the user. This can result in motion sickness and an unpleasant experience.
• Mobile FIMDs are to be used for the series implementation, in particular for the use in vehicles. In all likelihood, these will only have 3DOF tracking. Depending on the manufacturer and generation of the FIMDs, different IMUs with different inertial sensors will be installed. It follows that the FIMDs will have different drift errors.
• Document DE 10 2017 221 871 A1 describes a method for operating a head- mounted device, FIMD, for measuring angle values of the roll angle, pitch angle and yaw angle by means of an inertial measurement unit, IMU.
• IMU inertial measurement unit
• an additional video camera is used that provides camera data showing the absolute position of the FIMD in the room.
• Document RU 2527 132 C1 describes a method for compensating the drift of an IMU by means of camera data that show the user wearing the HMD.
• Optical markers are attached to the HMD such that the absolute position of the HMD in the room can be derived from the camera image data. However, this requires a camera that is detached from the HMD and that is mounted in a known fixed position in the room.
• the invention provides a method for determining calibration parameters for a calibration function of a head-mounted device, HMD, for use in a vehicle, wherein an inertial measurement unit, IMU, of the HMD generates at least one HMD sensor signal and from the at least one HMD sensor signal a respective HMD angle value for roll and/or pitch and/or yaw of the HMD is determined and updated over time.
• the calibration function may be active while the HMD is used by a user, i.e. a continuous compensation of the drift error may be available.
• the IMU of the HMD may measure or provide angle values for the roll angle and/or pitch angle and/or yaw angle of the HMD.
• angle values may be subject to a drift (drift error) of the IMU, they may be continuously or periodically corrected by the calibration function.
• Said HMD sensor signal may indicate a respective angle value directly (for roll and/or pitch and/or yaw) and/or it may signal an angular acceleration and/or angular velocity over time for at least one of the angles.
• the HMD angle values are updated over time such that they constitute a signal indicating the current value of at least one of the said angles.
• the IMU may comprise at least one sensor for acceleration and/or angular velocity (angular rate) for generating the at least one HMU sensor signal.
• the IMU may be taken from the prior art and may therefore be subject to said drift.
• the amount of correction or adaption that is applied by the calibration function is set or determined by the calibration parameters that need to be adapted to the specific IMU that is used in a HMD.
• the following first calibration phase is performed that comprises: while in the vehicle the HMD is held in a mounting position that is fixed with regard to the vehicle and while the vehicle performs at least one test drive:
• a respective vehicle sensor signal of at least one vehicle sensor of a localization module of the vehicle and/or a respective vehicle angle value for roll angle and/or pitch angle and/or yaw angle of the vehicle derived from the at least one vehicle sensor signal is recorded, wherein the localization module is also in a fixed arrangement with regard to the vehicle, and
• the at least one HMD sensor signal and/or the at least one HMD angle value is recorded.
• any signal and/or angle value determined by the IMU of the HMD has the prefix “HMD” (e.g. HMD angle value), while the corresponding signal and/or angle value that is determined by the localization module of the vehicle has the prefix “vehicle” (e.g. vehicle angle value).
• the signals taken or received from the localization module of the vehicle and the signals received from the IMU should indicate the same changes in values regarding roll angle and/or pitch angle and/or yaw angle (if the axes X, Y, Z are aligned or if a corresponding transformation corresponding to such an aligning is applied to the values). Any difference in the resulting angle values derived from the vehicle sensor signal on one side and the HMD sensor signal on the other hand describes the drift of the IMU of the HMD with respect to the vehicle.
• a processing unit determines the drift or drift error of the respective HMD angle value (i.e. for roll angle and/or pitch angel and/or yaw angle) by comparing the recorded signals and/or the recorded angle values (of the IMU and the vehicle) and determines the first calibration parameters as a function of the determined drift or drift error.
• the calibrations function generates a corrected respective HMD angle value (for roll angle and/or pitch angle and/or yaw angle of the HMD) over time based on at least the first calibration parameters.
• the comparison of the signals yields a difference in angular acceleration and/or angular velocity and/or angular values for roll angle, pitch angle and/or yaw angle, depending on which angle values are monitored or analyzed.
• the first calibration parameters describe the observed difference or drift.
• the drift can be described as the difference in corresponding signals and/or angle values.
• the invention provides the advantage that the drift and the resulting first calibration parameters can be derived or determined without the need of a specific additional sensor, for example, a camera, as the localization module of the vehicle may be used that may be available in a vehicle anyway for other vehicle-specific functions, like vehicle body control and/or an autonomous or automated driving function and/or driver assistance systems, for example, for stabilizing the driving dynamics of the vehicle.
• the calibration function can be configured on the basis of these calibration parameters and this configured calibration function can be used in the HMD or even in several other HMDs that use the same type of IMU. While the respective IMU is operated, the calibration function does not need further adaptation and can run or can be operated in a self-sufficient or self-sustained mode. No further input from the localization module of the vehicle is needed.
• the localization module of a vehicle may comprise at least one acceleration sensor and/or at least one sensor for an angular speed and/or a gyroscope, just to name examples.
• the localization module may be designed according to the known prior art.
• the axes of the IMU are aligned with the corresponding axes of the vehicle (x-axis corresponding to the longitudinal direction of the vehicle, y-axis corresponding to the cross axis and z-axis corresponding to the vertical axis).
• a movement around the x-axis results in a roll angle
• a movement around the cross axis results in a pitch angle
• a movement around the vertical axis results in a yaw angle.
• the at least one signal of the localization module and/or the vehicle angle value derived from this at least one vehicle sensor signal can be adapted or transformed by using, for example, a mathematical matrix operation for mapping the signal values and/or vehicle angle values into the coordinate system of the HMD.
• the invention also comprises embodiments that provide features which afford additional technical advantages.
• An embodiment may comprise that the processing unit determines the drift as a function over time starting from a starting point that indicates the beginning of a user session of the HMD and the first set of calibration parameters describes a correction value for the respective HMD angle value for roll and/or pitch and/or yaw that is to be applied per time unit starting from the starting point (e.g. degrees per minute). Whenever the user starts a user session for using the HMD, this indicates the starting point from where the calibration function may set off or begin. While the user uses or operates the HMD, the calibration function may apply the correction value for the respective HMD angle value for roll and/or pitch and/or yaw. In this manner, the calibration function can be designed as a static function that continuously or repeatedly applies the correction value without the need for further measurements. This results in a very simple design which is very robust.
• An embodiment may comprise that the processing unit determines the drift as a function of the magnitude value of the acceleration and/or the magnitude value of the angular velocity of the HMD and/or of the vehicle and the first calibration parameters describe a respective dynamic correction value for the respective HMD angle value, wherein the respective dynamic correction value is a function of the corresponding angular velocity and/or acceleration value that is currently observed by the IMU of the HMD and/or the localization module of the vehicle.
• the processing unit determines the drift as a function of the magnitude value of the acceleration and/or the magnitude value of the angular velocity of the HMD and/or of the vehicle and the first calibration parameters describe a respective dynamic correction value for the respective HMD angle value, wherein the respective dynamic correction value is a function of the corresponding angular velocity and/or acceleration value that is currently observed by the IMU of the HMD and/or the localization module of the vehicle.
• the corresponding dynamic correction value can be calculated or derived or chosen for example from a table or a characteristic function that has been derived from the observed drift in the calibration phase. Especially, the higher the magnitude value, the larger the magnitude of the correction value. This results in a dynamic calibration function that takes into account the current use situation especially the current angular velocity and/or acceleration value of the HMD.
• An embodiment may comprise that the processing unit determines the first calibration parameters by (e.g. iteratively) minimizing a mean square error of the at least one corrected HMD angle value with regard to a respective corresponding vehicle angle value that is determined from the at least one recorded vehicle sensor signal.
• the first calibration parameter can describe the correction value that needs to be applied for the HMD angle value at specific points in time (e.g. X degrees per minute for the roll angle, Y degrees per minute for the pitch angle and/or Z degrees per minute for the yaw angle).
• the mean square error provides an optimized way for compensating the drift or minimizing the effect of the drift error.
• the procedure for minimizing the mean square error may start with an initial set of correction values or at least one correction value and the calibration function may be applied to the at least one HMD angle value based on these initial first calibration parameters.
• This yields an updated value for the drift that can be, for example, the sum of the square values of the differences of the angle values of the HMD and the vehicle at corresponding sampling times. For example, if 500 measurements (500 samples) have been used for deriving the HMD angle value and the vehicle angle value, respectively, the difference between the HMD angle value and the vehicle angle value for the same point in time or time interval yields one difference value that is then squared and summed with the remaining square difference values.
• the values for the first calibration parameters can be applied and a new mean square error can be calculated such that a minimum for the mean square error is achieved by iteratively changing the calibration parameters such that a reduced mean square error is obtained.
• This method can be interrupted or stopped once a change in mean square error from one iteration to the next is smaller than a predefined threshold value. This will end the optimization procedure.
• An embodiment may comprise that a dynamic drift model for the IMU of the HMD is determined based on a machine learning algorithm, wherein for training the drift model, the at least one recorded HMD sensor signal and/or the at least one HMD angle value is provided as a training input and the at least one recorded vehicle sensor signal and/or the at least one recorded vehicle angle value is provided as a training label.
• a machine learning algorithm is the training of an artificial neural network. The machine learning algorithm derives or calculates first calibration parameters that yield a minimum difference between the training input and the corresponding training label.
• the described dynamic calibration function can be obtained that applies dynamic correction values depending on the magnitude of the observed acceleration and/or observed angular velocity value.
• An embodiment may comprise that a second calibration phase is performed.
• the HMD is mounted on the head of a vehicle passenger or an artificial head that may move in the vehicle. While the vehicle performs the at least one test drive and/or at least one additional test drive, the HMD is positioned on the head that moves inside the vehicle.
• the calibration function is applied to the at least one HMD sensor signal and/or the at least one un corrected HMD angle value resulting in at least one “corrected HMD angle value” based on the first calibration parameters. If the user (vehicle passenger) uses the HMD inside the vehicle, additional angular velocity and/or angular acceleration is existent and additional drift may be caused by the usage of the HMD in the vehicle. This is compensated by deriving second calibration parameters that describe the drift caused by using the HMD in the vehicle. The “base drift” as caused by the vehicle movements is compensated at that point, as the first calibration parameters are already applied.
• the second recording (in the second calibration phase) is performed resulting in additional recorded signals and/or additional recorded angle values (from HMD and vehicle) and based on these additional recorded signals and/or additional recorded angle values second calibration parameters are determined, and the second calibration parameters are integrated into the calibration function.
• the same routines may be used (minimization of mean square error and/or machine learning algorithm) with the difference that the HMD sensor signal and/or the HMD angle value is corrected using the calibration function with a configuration based on the first calibration parameters.
• the resulting second calibration parameters describe the amount of drift that is caused by the movement of the HMD inside the vehicle.
• the precise measurements from the localization module of the vehicle may be used to derive or determine at least a part of the drift error using the at least one vehicle sensor as a reference.
• a reference may also be necessary in order to calculate the difference values between the true head position and the head position as indicated by the IMU.
• the user vehicle passenger
• the user may be instructed to position the head in a specific posture, for example, by resting the head against the head rest and looking straight forward. Instructions of this type can be presented to the user using the HMD itself. Then, the current HMD angle value for roll and/or pitch and/or yaw can be measured while the head is arranged in this known posture. Any observed difference is the drift that thus becomes measurable or determinable.
• An embodiment may comprise that during the second calibration phase, a graphical content is displayed to the user via the HMD instructing the user to move the head at different or varying values for angular acceleration and/or angular speed according to a predefined motion profile.
• a graphical content is displayed to the user via the HMD instructing the user to move the head at different or varying values for angular acceleration and/or angular speed according to a predefined motion profile.
• An embodiment may comprise that for integrating the second calibration parameters, the calibration function operates a first correction stage that uses the first calibration parameters and a subsequent second correction stage that uses the second calibration parameters.
• This is a two step or two stage calibration function providing the benefit that, for example, by means of weighing values or weighing factors, the influence of the first calibration parameters and the second calibration parameters can be tuned or adapted individually.
• An embodiment may comprise that for integrating the second calibration parameters, from the first and second calibration parameters, combined calibration parameters are derived for a single correction stage.
• This single stage calibration function provides the advantage that less computational load is caused.
• a mathematical matrix operation can be used or the corresponding correction values may be added.
• a further aspect of the invention is given by a processing unit for determining calibration parameters for a calibration function of a head-mounted device, HMD, for use in a vehicle, wherein the processing unit is designed to perform an embodiment the inventive method.
• the processing unit may be provided for deriving the calibration parameters (first and/or second calibration parameters or combined calibration parameters) in a prototype vehicle using a prototype HMD.
• the resulting calibration parameters or the resulting calibration function configured by means of the calibration parameters may then be provided in several different vehicles that use a corresponding type of HMD or HMDs with corresponding IMUs that exhibit the same type or amount of drift, as they have the same design and/or the same technology.
• the processing unit may comprise at least one microprocessor and/or at least one ASIC (application specific integrated circuit).
• ASIC application specific integrated circuit
• a software or program code may be stored in a data storage of the processing unit.
• a further aspect of the invention is given by a head-mounted device, HMD, with a calibration function for generating a respective corrected HMD angle value for roll and/or pitch and/or yaw of the HMD based on at least one HMD sensor signal of an inertial measurement unit, IMU, of the HMD.
• a control circuit of the HMD is designed to execute the calibration function while a user wears the HMD on the head, wherein the calibration function is designed to apply calibration parameters that describe a static drift of the respective un corrected HMD angle value over time.
• the calibration function is designed to apply calibration parameters that describe a dynamic drift of the respective un-corrected HMD angle value as a function of a magnitude of angular acceleration and/or angular speed of the HMD.
• the calibration function may be configured based on first calibration parameters, first and second calibration parameters or combined calibration parameters as has been described above.
• the control circuit may be based on a microprocessor or several microprocessors and/or an ASIC of the HMD.
• a corresponding program code or software code may be provided in the control circuit.
• the calibration function may perform the static drift compensation (constant correction value) or the described dynamic correction value that is adapted to the magnitude of the observed angular acceleration and/or angular speed of the HMD.
• Said vehicle is preferably designed as a motor vehicle, in particular as a passenger vehicle or a truck, or as a bus or a motorcycle.
• the invention also comprises the combinations of the features of the different embodiments.
• an exemplary implementation of the invention is described.
• the figures show:
• Fig. 1 a schematic illustration of an embodiment of the inventive processing unit
• Fig. 2 a schematic illustration of diagrams of recordings.
• the embodiment explained in the following is a preferred embodiment of the invention.
• the described components of the embodiment each represent individual features of the invention which are to be considered independently of each other and which each develop the invention also independently of each other and thereby are also to be regarded as a component of the invention in individual manner or in another than the shown combination.
• the described embodiment can also be supplemented by further features of the invention already described.
• Fig. 1 shows a vehicle 10 that may be used in a test drive for calibrating a head-mounted device, FIMD, that is to be used by a user 11 in later drives for consuming or regarding augmented reality AR and/or virtual reality VR contents.
• the FIMD may be a prototype of a specific FIMD model for which a calibration is necessary.
• the vehicle 10 may comprise a localization module 12 that may generate at least one vehicle sensor signal 13 and/or a continuously or periodically updated vehicle angle value 14 indicating the roll angle 15 and/or pitch angle 16 and/or yaw angle 17 of the vehicle.
• the roll angle 15 may be measured as the rotation around the rotation axis X
• the pitch angle 16 may be measured with regard to the rotation axis Y
• the yaw angle 17 may measured for the rotation axis Z.
• at least one vehicle sensor 18 may be provided in the localization module 12.
• Examples for a vehicle sensor are an inertial measurement unit for the vehicle, a gyroscope, a camera, a laser gyroscope, in general at least one vehicle sensor as is available in the prior art.
• the at least one vehicle sensor signal 13 and/or the at least one vehicle angle value 14 may be recorded using a processing unit 19.
• the HMD may also comprise an inertial measurement unit, IMU, for generating and providing at least one HMD sensor signal 20 and/or at least one HMD angle value 21 indicating the respective current roll angle 22 and/or pitch angle 23 and/or yaw angle 24 of the HMD with respect its local axes X, Y and Z.
• IMU inertial measurement unit
• the at least one HMD sensor signal 20 and/or the at least one HMD angle value 21 may also be provided to the processing unit 19 for recording.
• a second calibration phase 27 may comprise that the user 11 is wearing the HMD on a head 28 and performs head movements 29 during the test drive.
• additional HMD sensor signals 30 and/or additional HMD angle values 31 are then generated in the second calibration phase and these may also be recorded together with corresponding or simultaneously generated additional vehicle sensor signals 13 and/or additional vehicle angle values 14.
• the user 11 may be instructed to position the head 28 in a known predefined fixed position 26’ from time to time between the head movements 29 or at certain points in time such that a comparison between the HMD sensor signal 30 and/or HMD angle value 31 with corresponding or simultaneously determined vehicle sensor signals 13 and/or vehicle angle values 14 can be obtained.
• the processing unit 19 may derive a respective drift error or drift 32 by comparing the recorded vehicle sensor signals 13 and/or recorded vehicle angle values 14 with corresponding recorded HMD sensor signals 20, 30 and/or recorded HMD angle values 21 , 31. From the drift error, from the first calibration phase 25, first calibration parameters 33 may be derived and from the second calibration phase 27, second calibration parameters 34 may be derived. The calibration parameters 33, 34 may be combined to obtain combined calibration parameters 35. The first and the second calibration parameters 33, 34 and/or the combined calibration parameters 35 may be provided to a control circuit 36 of the HMD that may operate a calibration function 37 for compensating the drift 32 of the IMU of the HMD.
• Fig. 2 illustrates how as calibration parameters either static correction values
• each HMD angle value 21 may be generated or alternatively dynamic correction values 39 may be generated that depend on a respective current acceleration value 40 and/or angular speed value of the HMD.
• the graphs show the angle value for the vehicle as measured by the localization module 12 and for the HMD as measured by the IMU for one possible angle, for example, roll, pitch or yaw.
• the drift error or drift 32 may be determined and from the drift error, the correction value 38 may be derived that may be applied by the correction function 37 such that a corrected HMD angle value is obtained.
• the correction value 38 may indicate the amount of degrees per minute or per second to be added or subtracted from the measured HMD angle value 31 , 32.
• the angular velocity and/or angular acceleration value 40 may be measured or derived from the angle value and for different values of the angular velocity and/or angular acceleration value 40, a respective correction value 39 may be derived or calculated.
• different correction values 39 for different values of the angular velocity and/or angular acceleration value 40 result.
• the corrected HMD angle value 41 results over time.
• the derived correction parameters 33, 34 or 35 may be stored in the control circuit 36 of the HMD and/or in the respective control circuit of at least one other HMD.
• the vehicle 10 and the HMD may be used by a manufacturer of HMDs and/or vehicles.
• the idea is based on the approach of estimating and correcting the drift error using the vehicle as a reference. This results in a calibration function that is minimizing drift error in 3-DOF HMDs in a dynamic environment. This may improve the overall HMD experience in production vehicles. Additional HMD manufacturers may be enabled whose 3-DOF tracking quality would have been too low for the vehicle use case so far.
• the premise for using a vehicle as a reference is a state of the art localization module for the ego position of the vehicle.
• the vehicle fuses data from one or more vehicle IMUs as well as GNSS data (GNSS - global navigation satellite system), wheel speed, steering angle and/or, if necessary, other sensor data and/or estimates of the ego position using a so-called Kalman filter or a filter technique based on it.
• the vehicle localization module may thus provide the ego position with a higher precision than the IMU of the HMD, as it and corrects the estimate of the ego-position over time on a wider data basis and/or more precise measurement hardware.
• a reference localization system can be installed in the vehicle, e.g. localization by using a mobile communication modem and/or a receiver for a position signal of a GNSS (e.g. GPS - global positioning system). The localization of the vehicle and the accuracy of this can therefore be taken as given.
• a GNSS e.g. GPS - global positioning system
• First calibration phase Measurement and correction of the HMD IMU drift error without head movement:
• the output of the HMD IMU and the output of the vehicle's localization module is recorded simultaneously during one or several dynamic journeys or test drives.
• the HMD is preferably aligned forward in the vehicle interior along the vehicle X-axis and is fixed in place.
• the results of the measurement runs may be subsequently evaluated.
• the decisive question is how the HMD IMU behaves in comparison to the vehicle, as any difference indicates the drift error.
• the square distance of the individual measurement points can be compared, for example. With the assumption that the vehicle localization is correct, the squared distance is the error of the HMD IMU.
• One solution is to correct the HMD IMU with the mean squared error over time.
• Another possibility is a machine learning algorithm or machine learning model. Using the measured inertial sensor data as input, a generalizing model can be created. Instead of the mean error, the HMD IMU is corrected over time with the dynamic output of the model.
• Second calibration phase Measurement and correction of the HMD error with head movements:
• the HMD is carried by a user during one or several dynamic journeys or test drives and moved around the three axes of rotation by a head motion. The movement during this is supposed to be random and different for each measurement ride. The movement around the three rotation axes during the ride represents the normal use of the HMD during a dynamic ride by a later user.
• the HMD IMU may be corrected during the measurement runs with the result from the first calibration phase. According to the chosen approach, either the mean square error is included over time, or the input of the sensor data is given to the machine learning model and then the output is used as correction.
• the output of the HMD IMU may be recorded analogously to the first calibration phase together with that of the localization module in the vehicle.
• the evaluation may also be done analogous to the first evaluation (quadratic error or machine learning model). This procedure only considers the error caused by the user's head movement.
• the correction of the mean square error is a static correction.
• the mean square error may be determined based on the measurement journeys or test drives and then assumed to be constant, regardless of how dynamic the user's actual journey is and also regardless of whether the user moves the HMD a lot and moves it quickly or keeps it still. Therefore, the mean square error correction is assumed to be the easier solution to implement.
• the advantage of a machine learning model is that it generalizes.
• the error is not static, but is determined depending on the input.
• the Machine Learning model 'knows' the error for different types of runs as output labels (e.g. as categories or intervals like very dynamic with head movement, medium dynamic, without head movement) and can thus adjust the model parameters of the respective algorithm so that the output fits each input.
• the output is accurate. It is possible to combine the mean square error or machine learning models from calibration phases 1 and 2 into one correction stage, as well as to correct them separately in two successive correction stages. Which gives more accurate results for a specific IMU can be empirically determined by the skilled person.
• the machine learning model to be used may also be determined empirically by the skilled person depending on the IMU. Likewise, another metric can be used instead of the squared error for the static correction.
• the IMU data of the vehicle and HMD should preferably be determined by several measurement runs or test drives in a two-phase calibration procedure. Subsequently, based on the data evaluation - either via the mean square error or a generalizing machine learning model - the drift error for the specific HMD IMU may be estimated. Option 1 would be to subtract the error (a correction value) determined for the HMD per measurement point for further live drives or to implement the machine learning model on the HMD (for dynamic correction values). The use of additional measurement units such as a camera is not necessary.
• the IMU data of the vehicle may be used as a reference to determine a mean error or to create a machine learning model.
• the example shows how an automated estimation and correction of a drift of sensor signals in a HMD can be provided in a dynamic environment.

Landscapes

• Engineering & Computer Science (AREA)
• Chemical & Material Sciences (AREA)
• Combustion & Propulsion (AREA)
• Transportation (AREA)
• Mechanical Engineering (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Optics & Photonics (AREA)
• Navigation (AREA)

Applications Claiming Priority (1)

Publications (1)

Family

ID=74884932

Family Applications (1)

Country Status (2)

Family Cites Families (6)

• 2021
  • 2021-03-11 EP EP21712444.5A patent/EP4291940A1/de active Pending
  • 2021-03-11 WO PCT/EP2021/056128 patent/WO2022188975A1/en active Application Filing

Also Published As

Similar Documents

Legal Events