US20240068839A1 - Artificial intelligence-enabled imu calibration and sensor fusion - Google Patents
Artificial intelligence-enabled imu calibration and sensor fusion Download PDFInfo
- Publication number
- US20240068839A1 US20240068839A1 US18/238,880 US202318238880A US2024068839A1 US 20240068839 A1 US20240068839 A1 US 20240068839A1 US 202318238880 A US202318238880 A US 202318238880A US 2024068839 A1 US2024068839 A1 US 2024068839A1
- Authority
- US
- United States
- Prior art keywords
- data
- calibration
- calibrated
- raw
- sensor fusion
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000004927 fusion Effects 0.000 title claims description 41
- 238000005259 measurement Methods 0.000 claims abstract description 21
- 238000000034 method Methods 0.000 claims abstract description 19
- 230000001133 acceleration Effects 0.000 claims abstract description 16
- 238000004422 calculation algorithm Methods 0.000 claims description 20
- 230000005484 gravity Effects 0.000 claims description 15
- 230000035945 sensitivity Effects 0.000 claims description 14
- 238000012937 correction Methods 0.000 claims description 5
- 230000008569 process Effects 0.000 claims description 3
- 238000013473 artificial intelligence Methods 0.000 description 9
- 230000002093 peripheral effect Effects 0.000 description 9
- 238000012546 transfer Methods 0.000 description 7
- 238000012545 processing Methods 0.000 description 6
- 230000003068 static effect Effects 0.000 description 6
- 230000008901 benefit Effects 0.000 description 5
- 230000008859 change Effects 0.000 description 5
- 238000010801 machine learning Methods 0.000 description 5
- 238000004891 communication Methods 0.000 description 4
- 230000000694 effects Effects 0.000 description 4
- 238000011084 recovery Methods 0.000 description 4
- 230000001186 cumulative effect Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 230000035939 shock Effects 0.000 description 3
- 238000012549 training Methods 0.000 description 3
- 238000011156 evaluation Methods 0.000 description 2
- 230000026676 system process Effects 0.000 description 2
- 230000002411 adverse Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 230000036541 health Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 239000002245 particle Substances 0.000 description 1
- 238000013186 photoplethysmography Methods 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 231100000430 skin reaction Toxicity 0.000 description 1
- 239000011800 void material Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
- G01C25/00—Manufacturing, calibrating, cleaning, or repairing instruments or devices referred to in the other groups of this subclass
- G01C25/005—Manufacturing, calibrating, cleaning, or repairing instruments or devices referred to in the other groups of this subclass initial alignment, calibration or starting-up of inertial devices
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01P—MEASURING LINEAR OR ANGULAR SPEED, ACCELERATION, DECELERATION, OR SHOCK; INDICATING PRESENCE, ABSENCE, OR DIRECTION, OF MOVEMENT
- G01P21/00—Testing or calibrating of apparatus or devices covered by the preceding groups
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04W—WIRELESS COMMUNICATION NETWORKS
- H04W4/00—Services specially adapted for wireless communication networks; Facilities therefor
- H04W4/02—Services making use of location information
- H04W4/025—Services making use of location information using location based information parameters
- H04W4/026—Services making use of location information using location based information parameters using orientation information, e.g. compass
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04W—WIRELESS COMMUNICATION NETWORKS
- H04W4/00—Services specially adapted for wireless communication networks; Facilities therefor
- H04W4/02—Services making use of location information
- H04W4/025—Services making use of location information using location based information parameters
- H04W4/027—Services making use of location information using location based information parameters using movement velocity, acceleration information
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04W—WIRELESS COMMUNICATION NETWORKS
- H04W4/00—Services specially adapted for wireless communication networks; Facilities therefor
- H04W4/80—Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication
Definitions
- the current state of the art is to perform calibration online (i.e. in real time, with some latency) because typically these measurements are used downstream for real-time applications. Given the time-critical nature of some real-time applications, it becomes pointless to calibrate if it is not done so in a timely manner. So, existing calibration algorithms have to be optimized for highly constrained hardware and software limitations. Therefore, existing algorithms trade-off quality for reduced latency. There is interest in receiving highly accurate measurements before it is used offline for training an AI/ML model to make a critical (e.g., a health-related) inference or decision. So, technically, the calibration algorithms have no theoretical limitations on the compute power or latency.
- One novel aspect of the present invention is to use all of the data needed to effectively back-calibrate the erroneous measurements offline resulting in lower overall cumulative error.
- the current state of the art for calibrating the different IMU sensors does not account for dynamically changing bias and sensitivity parameters due to time-varying factors such as temperature and shock.
- a factory calibration is performed at the time the device is manufactured and/or a recalibration routine is performed every so often to proactively adjust for the drift or other things that causes the calibration to become void.
- one of the novel ideas is to systematically react to inconsistency in the calibration to enable faster recovery from erroneous calibration parameter estimates using some of the techniques discussed in this paragraph.
- Another novel idea is to use a suboptimal online calibration to provide time-sensitive situational awareness but do an offline correction to provide a more accurate inference at an acceptable later time with relaxed time and compute-power constraints.
- the disclosed embodiments provide a system for sensor calibration.
- the system receives raw accelerometer data, raw gyroscope data, and raw magnetometer data and performs spatial calibration at an inertial measurement unit (IMU).
- IMU inertial measurement unit
- a plurality of calibration parameters are then output from the inertial measurement unit.
- Calibrated data from the accelerometer, gyroscope, and magnetometer are transmitted to a sensor fusion module, where the sensor fusion module computes an orientation.
- the orientation and calibrated accelerometer data are then transmitted to a gravity removal module, where the gravity removal module calculates corrected linear acceleration.
- the calibration is dynamically performed.
- system further transmits calibrated accelerometer data, calibrated magnetometer data, and calibrated gyroscope data from the sensor fusion module to an AI algorithm module.
- the AI algorithm module is trained to perform an offline correction using the calibrated accelerometer data, calibrated magnetometer data, and calibrated gyroscope data for subsequent iterations
- FIG. 1 is a diagram of an exemplary embodiment of the hardware of the system of the present invention
- FIG. 2 is a flowchart outlining how the system processes raw IMU data to produce an orientation estimate from an exemplary sensor fusion algorithm that uses online calibration;
- FIG. 3 is a chart showing an evaluation of online calibration for IMU accelerometer bias errors
- FIG. 4 is a chart showing the advantages of dynamic IMU calibration over static IMU calibration
- FIG. 5 is a flowchart outlining how the system processes raw IMU data to produce an orientation estimate from an exemplary sensor fusion algorithm that uses offline calibration;
- FIG. 6 is a chart showing an evaluation of offline calibration for IMU accelerometer bias errors
- FIG. 7 is a chart showing cumulative accelerometer errors over time for raw IMU signals vs. calibrated IMU signals.
- FIG. 8 is a diagram showing how the system of the present invention operates with regard to a wearable device.
- a medical-grade wearable device which is comprised of physiological sensors including PPG (“Photoplethysmography”), GSR (“Galvanic Skin Response”), and temperature sensors.
- the device is utilized in conjunction with custom data pipeline and backend to collect physiological data that will be used for training AI/ML models to perform medically relevant diagnostic inferences as well as for patient monitoring.
- Rajant desires the most accurate and precise physiological sensor data that can be provided by the device.
- the device is further equipped with a 9-axis IMU (“Inertial Measurement Unit”) comprising one or more accelerometers, gyroscopes, and magnetometers.
- the IMU may also include one or more temperature sensors.
- the IMU is used to reduce motion artifacts in the PPG sensor data introduced due to motion by the patient.
- the accelerometer data will be utilized to correct the PPG signals by removing motion-induced artifacts.
- the gravitational component of the acceleration must then be removed from the signal in order to compute the true linear acceleration of the device. That is achieved using a process referred to as sensor fusion, which combines the outputs of the accelerometer, gyroscope and the magnetometer to determine the orientation (and heading) of the device and subtracting the contribution of gravity along each axis.
- the accelerometer, gyroscope, and magnetometer produce noisy measurements which need to be corrected. Furthermore, the measurements are adversely impacted if the sensors are not calibrated properly. Calibration is an additional correctional step that is performed to offset the error in the raw measurements to account for the bias and sensitivity parameters of the sensors.
- One novel idea is to come up with better and more informative categories/labels for each data point enabled by AI to assist the calibration and sensor fusion. For example, performing activity recognition (i.e. walking, running, etc.) could inform the calibration/fusion procedures about potential trajectories that the device might take, thereby enabling faster and more effective calibration/fusion (both online and offline).
- activity recognition i.e. walking, running, etc.
- FIG. 1 is an exemplary embodiment of the system of the present invention.
- one or more peripheral devices 110 are connected to one or more computers 120 through a network 130 .
- peripheral devices/locations 110 include smartphones, tablets, wearables devices, and any other electronic devices that collect and transmit data over a network that are known in the art.
- the network 130 may be a wide-area network, like the Internet, or a local area network, like an intranet. Because of the network 130 , the physical location of the peripheral devices 110 and the computers 120 has no effect on the functionality of the hardware and software of the invention. Both implementations are described herein, and unless specified, it is contemplated that the peripheral devices 110 and the computers 120 may be in the same or in different physical locations.
- Communication between the hardware of the system may be accomplished in numerous known ways, for example using network connectivity components such as a modem or Ethernet adapter.
- the peripheral devices/locations 110 and the computers 120 will both include or be attached to communication equipment. Communications are contemplated as occurring through industry-standard protocols such as HTTP or HTTPS.
- Each computer 120 is comprised of a central processing unit 122 , a storage medium 124 , a user-input device 126 , and a display 128 .
- Examples of computers that may be used are: commercially available personal computers, open source computing devices (e.g. Raspberry Pi), commercially available servers, and commercially available portable devices (e.g. smartphones, smartwatches, tablets).
- each of the peripheral devices 110 and each of the computers 120 of the system may have software related to the system installed on it.
- system data may be stored locally on the networked computers 120 or alternately, on one or more remote servers 140 that are accessible to any of the peripheral devices 110 or the networked computers 120 through a network 130 .
- the software runs as an application on the peripheral devices 110 , and includes web-based software and iOS-based and Android-based mobile applications.
- FIG. 2 characterizes the processing that raw IMU data follows to produce an accurate orientation estimate from the sensor fusion algorithm.
- Bias and sensitivity parameters are estimated for each of the 3 sensors (accelerometer 202 , gyroscope 204 & magnetometer 206 ).
- IMU calibration is then performed 208 and a total of 18 calibration parameters 210 are preferably estimated (bias and sensitivity for the x, y & z axis of each sensor).
- Parameters are updated when particular angular velocities and static orientations are measured for the accelerometer 212 , the gyroscopes 214 and the magnetometer 216 .
- the IMU rotates by an arc length of 120° and comes to rest at a specific angle relative to gravity.
- Sensor fusion 218 utilizes calibrated IMU data to estimate the orientation of the device in world coordinates. Dynamic calibration results in more accurate sensor fusion estimation for time and temperature varying IMU bias and sensitivity parameters than static calibration. Faster recovery is achieved by the sensor fusion with dynamic calibration. Sensor fusion detects inaccuracies in IMU parameter estimates.
- the quaternion 222 and calibrated accelerometer 220 describing IMU orientation from the sensor fusion algorithm is used to determine acceleration due to gravity that is projected onto the x, y & z axes of the IMU.
- the gravitational projection is removed 224 from the accelerometer readings to provide linear acceleration estimates 226 in world coordinates.
- FIG. 3 demonstrates the ability of the calibration procedure to estimate IMU parameters.
- the accelerometer has static bias offset errors of 0.6, ⁇ 0.2 & 0.1 milligees on the x, y, & z axes respectively.
- the calibration goes into effect at approximately 24 seconds.
- the calibrated accelerometer signals have errors that are less than 10 milligees on each axis, whereas the uncalibrated accelerometer measurements contain errors that are reflective of the biases.
- FIG. 4 demonstrates the necessity of continuous calibration to ensure accurate IMU calibration.
- total accelerometer errors are initially 100 milligees.
- the IMU parameters change. This can be due to temperature change or mechanical shock (or other phenomena). If calibration is only executed once and IMU parameters change at some instance in the future, the change in parameters is likely to produce orientation estimates that are out of tolerance. Conversely, continuous calibration is able to recover from IMU parameter dynamics.
- Real time IMU calibration makes use of historical data only to update bias and sensitivity parameters. However, given a set of recorded raw IMU data, offline calibration can be executed to possibly achieve more accurate IMU parameters. At any point in time, past and future IMU samples can be used to determine the optimal set of bias and sensitivity estimates. Just as in the online application, offline calibration is capable of tracking dynamic IMU parameters that are time and temperature varying.
- FIG. 5 characterizes the processing that raw IMU data follows to produce an accurate orientation estimate from the sensor fusion algorithm.
- Data can be stored locally to an IMU device, smartphone, edge processing nodes, etc. until ready for processing. Offline processing can occur on a backend machine or at remote nodes.
- bias and sensitivity parameters are estimated for each of the 3 sensors (accelerometer 502 , gyroscope 504 & magnetometer 506 ) and sent to offline storage 508 .
- the readings from the sensor 502 , 504 , 506 are used to perform IMU calibration 516 and a total of 18 calibration parameters 518 are preferably estimated (bias and sensitivity for the x, y & z axis of each sensor), using raw acceleration data 510 , raw gyroscope data 512 , and raw magnetometer data 514 .
- Parameters are updated when particular angular velocities and static orientations are measured for the calibrated accelerometer 520 , the calibrated gyroscope 522 , and the calibrated magnetometer 524 .
- the IMU rotates by an arc length of 120° and comes to rest at a specific angle relative to gravity.
- Sensor fusion 526 utilizes calibrated IMU data to estimate the orientation of the device in real world coordinates. Dynamic calibration results in a more accurate sensor fusion estimation for time and temperature varying IMU bias and sensitivity parameters than static calibration. Faster recovery is achieved by the sensor fusion with dynamic calibration. Sensor fusion detects inaccuracies in IMU parameter estimates.
- Sensor fusion 526 data is transmitted to AI algorithm modules 528 that are used for training an AI/ML model systematically to react to inconsistency in the calibration to enable faster recovery from erroneous calibration parameter estimates using some of the techniques described above.
- a suboptimal online calibration may be used to provide time-sensitive situational awareness but does an offline correction to provide a more accurate inference at an acceptable later time with relaxed time and compute-power constraints.
- the offline calibration demonstrates that by incorporating the entire history of IMU samples, all accelerometer errors are close to zero. That is in contrast to FIG. 3 , where the online calibration errors are minimized only after a complete set of calibration measurements have been captured.
- the quaternion 532 and calibrated acceleration 530 describing IMU orientation from the sensor fusion algorithm is used to determine acceleration due to gravity that is projected onto the x, y & z axes of the IMU.
- the gravitational projection is removed 534 from the accelerometer readings to provide linear acceleration estimates 536 in real world coordinates.
- the benefit of offline and online calibration is realized in comparison to the scenario of no calibration by evaluating cumulative accelerometer error.
- errors accumulate over time at a significant rate until the calibration goes into effect at 24 seconds. At that point, the error contribution is significantly reduced.
- the bias and sensitivity errors are minimized over the entire data set. The substantial error contributed by the raw IMU signals will continue to accumulate at a rate that is not tolerable for maintaining accurate IMU orientation.
- FIG. 8 is a diagram showing how the system of the present invention operates with regard to a wearable device in an online-offline hybrid implementation, in accordance with an exemplary embodiment of the invention.
- the system is comprised of a wearable 802 and a backend 804 .
- the wearable 802 is the peripheral device 110 shown in FIG. 1
- the backend 140 may be one or computers 120 , as shown in FIG. 1 .
- the backend 140 may also be a smartphone or tablet.
- the wearable 110 is comprised of one or more IMU Sensors 806 , a Data Transfer Service 808 , an Online Calibration and Sensor Fusion (“CNSF”) 810 , an External Calibration Updater 812 , and a Bluetooth Module 814 .
- Each of the Data Transfer Service 808 and the Online CNSF 810 have their own Local Store 816 , 818 .
- the IMU Sensors 806 collect various sensor data and transmit that raw IMU data to the Online CNSF 810 and the Data Transfer Service 808 .
- the Online CNSF 810 is primarily responsible for performing calculations and coverts the raw IMU data to calibrated IMU data with annotations, while also calculating calibration values with metrics and quality. Recalibration of the raw IMU data is necessitated by such factors as a change in temperature, mechanical shock and/or time. Other reasons for recalibration will be readily apparent to one of ordinary skill in the art.
- the Online CNSF 810 stores the calibrated IMU data with annotations and calibration values at a Local Store.
- the Online CNSF 810 then transmits the calibrated IMU data with annotations and calibration values to the Data Transfer Service 808 .
- the Data Transfer Service 808 stores the calibrated IMU data with annotations, the calibration values, and raw IMU data at a Local Store 818 .
- the Online CNSF 810 is also connected to the External Calibration Updater 812 , which provides real-time external calibration data to the Online CNSF 810 . That external calibration data can be used by the Online CNSF 810 to calculate the calibrated IMU data and the calibration values with metrics and quality.
- the Data Transfer Service 808 and the External Calibration Updater 812 are each connected to the Bluetooth Module 814 .
- the Data Transfer Service 808 and External Calibration Updater 812 transmit one or more of raw IMU data, calibration data, calibrated IMU data with annotations, and calibration values with metrics and quality to and from the Bluetooth Module 814 .
- the Bluetooth Module 814 is capable of communicating with the Edge App 820 , an application that connects the wearable 110 to the backend 140 .
- the wearable 802 and the backend 804 are in continuous two-way communication through the Edge App 820 .
- the backend 804 is comprised of a Recalibration Service 824 , a DB Manager 826 , an Offline CNSF 828 , and a Remote Store 830 .
- the Offline CNSF 828 receives the calibration data from the Edge App and compares it against calibration data that is received from the DB Manager 826 , which queries the Remote Store 830 for historical, previously calculated data received from the wearable 802 . If the calibration data received from the wearable 802 is determined to be less accurate than the historical data, then the Offline CNSF 828 signals the Recalibration Service 824 to request that the External Calibration Updater 812 perform an updated calibration and analysis of the IMU data at the wearable 802 . That request for recalibration is transmitted through the Edge App 820 and the Bluetooth Module 814 , to the External Calibration Updater 812 .
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Remote Sensing (AREA)
- Radar, Positioning & Navigation (AREA)
- Manufacturing & Machinery (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Theoretical Computer Science (AREA)
- General Health & Medical Sciences (AREA)
- Biophysics (AREA)
- Evolutionary Computation (AREA)
- Computational Linguistics (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Biomedical Technology (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Health & Medical Sciences (AREA)
- Navigation (AREA)
- Measuring Magnetic Variables (AREA)
Abstract
Disclosed herein are systems and methods for computer-assisted sensor calibration where the computer receives raw accelerometer data, raw gyroscope data, and raw magnetometer data and performs spatial calibration. The calibration data is used to calculate orientation data, and then then calibrated data and calibration data are used to provide corrected linear acceleration data that may be used by sensors to provide more accurate readings, for example, with regard to physiological data. In certain embodiments, in certain embodiments, the sensor calibration further employs a 9-axis IMU (“Inertial Measurement Unit”) comprising an accelerometer, gyroscope and magnetometer to provide calibration data.
Description
- This application claims the benefit of U.S. Prov. App. No. 63/401,448, filed Aug. 26, 2022, the entire contents of which are incorporated herein by reference.
- The current state of the art is to perform calibration online (i.e. in real time, with some latency) because typically these measurements are used downstream for real-time applications. Given the time-critical nature of some real-time applications, it becomes pointless to calibrate if it is not done so in a timely manner. So, existing calibration algorithms have to be optimized for highly constrained hardware and software limitations. Therefore, existing algorithms trade-off quality for reduced latency. There is interest in receiving highly accurate measurements before it is used offline for training an AI/ML model to make a critical (e.g., a health-related) inference or decision. So, technically, the calibration algorithms have no theoretical limitations on the compute power or latency. One novel aspect of the present invention is to use all of the data needed to effectively back-calibrate the erroneous measurements offline resulting in lower overall cumulative error.
- The current state of the art for calibrating the different IMU sensors does not account for dynamically changing bias and sensitivity parameters due to time-varying factors such as temperature and shock. Typically, a factory calibration is performed at the time the device is manufactured and/or a recalibration routine is performed every so often to proactively adjust for the drift or other things that causes the calibration to become void. Instead, one of the novel ideas is to systematically react to inconsistency in the calibration to enable faster recovery from erroneous calibration parameter estimates using some of the techniques discussed in this paragraph. Another novel idea is to use a suboptimal online calibration to provide time-sensitive situational awareness but do an offline correction to provide a more accurate inference at an acceptable later time with relaxed time and compute-power constraints.
- The ability to perform back-calibration of sensors impacted by time-varying calibration parameters is also novel and does not currently exist and that deficiency is remedied by the present invention.
- The disclosed embodiments provide a system for sensor calibration. The system receives raw accelerometer data, raw gyroscope data, and raw magnetometer data and performs spatial calibration at an inertial measurement unit (IMU). A plurality of calibration parameters are then output from the inertial measurement unit. Calibrated data from the accelerometer, gyroscope, and magnetometer are transmitted to a sensor fusion module, where the sensor fusion module computes an orientation. The orientation and calibrated accelerometer data are then transmitted to a gravity removal module, where the gravity removal module calculates corrected linear acceleration.
- In certain embodiments, there may be eighteen calibration parameters and in others, up to thirty-six calibration parameters.
- In other embodiments, the calibration is dynamically performed.
- In yet other embodiments, the system further transmits calibrated accelerometer data, calibrated magnetometer data, and calibrated gyroscope data from the sensor fusion module to an AI algorithm module.
- In other embodiments, the AI algorithm module is trained to perform an offline correction using the calibrated accelerometer data, calibrated magnetometer data, and calibrated gyroscope data for subsequent iterations
- A more complete appreciation of the invention and many of the attendant advantages thereof will be readily obtained as the same becomes better understood by reference to the following detailed description when considered in connection with the accompanying drawings, wherein:
-
FIG. 1 is a diagram of an exemplary embodiment of the hardware of the system of the present invention; -
FIG. 2 is a flowchart outlining how the system processes raw IMU data to produce an orientation estimate from an exemplary sensor fusion algorithm that uses online calibration; -
FIG. 3 is a chart showing an evaluation of online calibration for IMU accelerometer bias errors; -
FIG. 4 is a chart showing the advantages of dynamic IMU calibration over static IMU calibration; -
FIG. 5 is a flowchart outlining how the system processes raw IMU data to produce an orientation estimate from an exemplary sensor fusion algorithm that uses offline calibration; -
FIG. 6 is a chart showing an evaluation of offline calibration for IMU accelerometer bias errors; -
FIG. 7 is a chart showing cumulative accelerometer errors over time for raw IMU signals vs. calibrated IMU signals; and -
FIG. 8 is a diagram showing how the system of the present invention operates with regard to a wearable device. - In describing a preferred embodiment of the invention illustrated in the drawings, specific terminology will be resorted to for the sake of clarity. However, the invention is not intended to be limited to the specific terms so selected, and it is to be understood that each specific term includes all technical equivalents that operate in a similar manner to accomplish a similar purpose. Several preferred embodiments of the invention are described for illustrative purposes, it being understood that the invention may be embodied in other forms not specifically shown in the drawings.
- Disclosed herein is a medical-grade wearable device which is comprised of physiological sensors including PPG (“Photoplethysmography”), GSR (“Galvanic Skin Response”), and temperature sensors. In certain embodiments, the device is utilized in conjunction with custom data pipeline and backend to collect physiological data that will be used for training AI/ML models to perform medically relevant diagnostic inferences as well as for patient monitoring. Rajant desires the most accurate and precise physiological sensor data that can be provided by the device. To that end, the device is further equipped with a 9-axis IMU (“Inertial Measurement Unit”) comprising one or more accelerometers, gyroscopes, and magnetometers. In certain embodiments, the IMU may also include one or more temperature sensors.
- Among other uses, the IMU is used to reduce motion artifacts in the PPG sensor data introduced due to motion by the patient. In particular, the accelerometer data will be utilized to correct the PPG signals by removing motion-induced artifacts. The gravitational component of the acceleration must then be removed from the signal in order to compute the true linear acceleration of the device. That is achieved using a process referred to as sensor fusion, which combines the outputs of the accelerometer, gyroscope and the magnetometer to determine the orientation (and heading) of the device and subtracting the contribution of gravity along each axis. The accelerometer, gyroscope, and magnetometer produce noisy measurements which need to be corrected. Furthermore, the measurements are adversely impacted if the sensors are not calibrated properly. Calibration is an additional correctional step that is performed to offset the error in the raw measurements to account for the bias and sensitivity parameters of the sensors.
- In order to perform a calibration online, one common way is to categorize each new data point obtained into groups (i.e. labels) indicating usability of the data point toward calibrating a certain sensor. So, when sufficient data points have been collected, the sensors can then be calibrated. Subsequently, the sensor measurements are then fused to provide a better orientation estimate for the device using one of several estimation methods such as Kalman or Particle filter (or its variants). Better estimates can be provided at a faster rate if the search for potential solutions is focused on solution spaces that are more probable given the estimated trajectory of the device (such as a car that is driving on a road). However, there is interest in applications where the trajectory of the device in question may seem ad hoc (such as human motion). One novel idea is to come up with better and more informative categories/labels for each data point enabled by AI to assist the calibration and sensor fusion. For example, performing activity recognition (i.e. walking, running, etc.) could inform the calibration/fusion procedures about potential trajectories that the device might take, thereby enabling faster and more effective calibration/fusion (both online and offline).
-
FIG. 1 is an exemplary embodiment of the system of the present invention. In theexemplary system 100, one or moreperipheral devices 110 are connected to one ormore computers 120 through anetwork 130. Examples of peripheral devices/locations 110 include smartphones, tablets, wearables devices, and any other electronic devices that collect and transmit data over a network that are known in the art. Thenetwork 130 may be a wide-area network, like the Internet, or a local area network, like an intranet. Because of thenetwork 130, the physical location of theperipheral devices 110 and thecomputers 120 has no effect on the functionality of the hardware and software of the invention. Both implementations are described herein, and unless specified, it is contemplated that theperipheral devices 110 and thecomputers 120 may be in the same or in different physical locations. Communication between the hardware of the system may be accomplished in numerous known ways, for example using network connectivity components such as a modem or Ethernet adapter. The peripheral devices/locations 110 and thecomputers 120 will both include or be attached to communication equipment. Communications are contemplated as occurring through industry-standard protocols such as HTTP or HTTPS. - Each
computer 120 is comprised of acentral processing unit 122, astorage medium 124, a user-input device 126, and adisplay 128. Examples of computers that may be used are: commercially available personal computers, open source computing devices (e.g. Raspberry Pi), commercially available servers, and commercially available portable devices (e.g. smartphones, smartwatches, tablets). In one embodiment, each of theperipheral devices 110 and each of thecomputers 120 of the system may have software related to the system installed on it. In such an embodiment, system data may be stored locally on thenetworked computers 120 or alternately, on one or moreremote servers 140 that are accessible to any of theperipheral devices 110 or thenetworked computers 120 through anetwork 130. In alternate embodiments, the software runs as an application on theperipheral devices 110, and includes web-based software and iOS-based and Android-based mobile applications. - Online IMU Calibration and Sensor Fusion
- Online calibration and sensor fusion utilizes real-time IMU measurements to update the bias and sensitivity parameters of the accelerometer, gyroscope and magnetometer. When an appropriate solution to the parameters has been found, errors introduced by the biases and offsets are significantly reduced.
FIG. 2 characterizes the processing that raw IMU data follows to produce an accurate orientation estimate from the sensor fusion algorithm. Bias and sensitivity parameters are estimated for each of the 3 sensors (accelerometer 202,gyroscope 204 & magnetometer 206). IMU calibration is then performed 208 and a total of 18calibration parameters 210 are preferably estimated (bias and sensitivity for the x, y & z axis of each sensor). Parameters are updated when particular angular velocities and static orientations are measured for theaccelerometer 212, thegyroscopes 214 and themagnetometer 216. For example, the IMU rotates by an arc length of 120° and comes to rest at a specific angle relative to gravity. - Sensor Fusion
-
Sensor fusion 218 utilizes calibrated IMU data to estimate the orientation of the device in world coordinates. Dynamic calibration results in more accurate sensor fusion estimation for time and temperature varying IMU bias and sensitivity parameters than static calibration. Faster recovery is achieved by the sensor fusion with dynamic calibration. Sensor fusion detects inaccuracies in IMU parameter estimates. - Gravity Removal
- The
quaternion 222 and calibratedaccelerometer 220 describing IMU orientation from the sensor fusion algorithm is used to determine acceleration due to gravity that is projected onto the x, y & z axes of the IMU. The gravitational projection is removed 224 from the accelerometer readings to provide linear acceleration estimates 226 in world coordinates. -
FIG. 3 demonstrates the ability of the calibration procedure to estimate IMU parameters. In this simulated example, the accelerometer has static bias offset errors of 0.6, −0.2 & 0.1 milligees on the x, y, & z axes respectively. The calibration goes into effect at approximately 24 seconds. After that point in time, the calibrated accelerometer signals have errors that are less than 10 milligees on each axis, whereas the uncalibrated accelerometer measurements contain errors that are reflective of the biases. -
FIG. 4 demonstrates the necessity of continuous calibration to ensure accurate IMU calibration. In this simulation, total accelerometer errors are initially 100 milligees. At 24 seconds, it is determined that a set valid calibration parameter has been computed and the accelerometer errors drop to 10 milligees. At the 40 second time position, the IMU parameters change. This can be due to temperature change or mechanical shock (or other phenomena). If calibration is only executed once and IMU parameters change at some instance in the future, the change in parameters is likely to produce orientation estimates that are out of tolerance. Conversely, continuous calibration is able to recover from IMU parameter dynamics. - Offline IMU Calibration and Sensor Fusion
- Real time IMU calibration makes use of historical data only to update bias and sensitivity parameters. However, given a set of recorded raw IMU data, offline calibration can be executed to possibly achieve more accurate IMU parameters. At any point in time, past and future IMU samples can be used to determine the optimal set of bias and sensitivity estimates. Just as in the online application, offline calibration is capable of tracking dynamic IMU parameters that are time and temperature varying.
- Offline calibration takes advantage of large data sets by leveraging machine learning (ML) and artificial intelligence (AI) algorithms. The AI algorithms predict when calibration parameters are out of tolerance and also quantify the quality of calibration measurements.
FIG. 5 characterizes the processing that raw IMU data follows to produce an accurate orientation estimate from the sensor fusion algorithm. - Offline Storage
- Data can be stored locally to an IMU device, smartphone, edge processing nodes, etc. until ready for processing. Offline processing can occur on a backend machine or at remote nodes.
- As shown in
FIG. 5 , bias and sensitivity parameters are estimated for each of the 3 sensors (accelerometer 502,gyroscope 504 & magnetometer 506) and sent tooffline storage 508. From theoffline storage 508, the readings from thesensor IMU calibration 516 and a total of 18calibration parameters 518 are preferably estimated (bias and sensitivity for the x, y & z axis of each sensor), usingraw acceleration data 510,raw gyroscope data 512, andraw magnetometer data 514. In certain embodiments, there may be up to 36 calibration parameters. Parameters are updated when particular angular velocities and static orientations are measured for the calibratedaccelerometer 520, the calibratedgyroscope 522, and the calibratedmagnetometer 524. For example, the IMU rotates by an arc length of 120° and comes to rest at a specific angle relative to gravity. - Sensor Fusion
- Sensor fusion 526 utilizes calibrated IMU data to estimate the orientation of the device in real world coordinates. Dynamic calibration results in a more accurate sensor fusion estimation for time and temperature varying IMU bias and sensitivity parameters than static calibration. Faster recovery is achieved by the sensor fusion with dynamic calibration. Sensor fusion detects inaccuracies in IMU parameter estimates.
- Sensor fusion 526 data is transmitted to
AI algorithm modules 528 that are used for training an AI/ML model systematically to react to inconsistency in the calibration to enable faster recovery from erroneous calibration parameter estimates using some of the techniques described above. A suboptimal online calibration may be used to provide time-sensitive situational awareness but does an offline correction to provide a more accurate inference at an acceptable later time with relaxed time and compute-power constraints. As shown inFIGS. 5 and 6 , the offline calibration demonstrates that by incorporating the entire history of IMU samples, all accelerometer errors are close to zero. That is in contrast toFIG. 3 , where the online calibration errors are minimized only after a complete set of calibration measurements have been captured. - Gravity Removal
- The
quaternion 532 and calibratedacceleration 530 describing IMU orientation from the sensor fusion algorithm is used to determine acceleration due to gravity that is projected onto the x, y & z axes of the IMU. The gravitational projection is removed 534 from the accelerometer readings to provide linear acceleration estimates 536 in real world coordinates. - As shown in
FIG. 7 , the benefit of offline and online calibration is realized in comparison to the scenario of no calibration by evaluating cumulative accelerometer error. For online calibration, errors accumulate over time at a significant rate until the calibration goes into effect at 24 seconds. At that point, the error contribution is significantly reduced. In the offline case, since both historical and future data knowledge is available, the bias and sensitivity errors are minimized over the entire data set. The substantial error contributed by the raw IMU signals will continue to accumulate at a rate that is not tolerable for maintaining accurate IMU orientation. -
FIG. 8 is a diagram showing how the system of the present invention operates with regard to a wearable device in an online-offline hybrid implementation, in accordance with an exemplary embodiment of the invention. As shown inFIG. 8 , the system is comprised of a wearable 802 and abackend 804. In certain embodiments, the wearable 802 is theperipheral device 110 shown inFIG. 1 , while thebackend 140 may be one orcomputers 120, as shown inFIG. 1 . In certain embodiments, thebackend 140 may also be a smartphone or tablet. The wearable 110 is comprised of one ormore IMU Sensors 806, aData Transfer Service 808, an Online Calibration and Sensor Fusion (“CNSF”) 810, anExternal Calibration Updater 812, and aBluetooth Module 814. Each of theData Transfer Service 808 and theOnline CNSF 810 have theirown Local Store - In operation, the
IMU Sensors 806 collect various sensor data and transmit that raw IMU data to theOnline CNSF 810 and theData Transfer Service 808. TheOnline CNSF 810 is primarily responsible for performing calculations and coverts the raw IMU data to calibrated IMU data with annotations, while also calculating calibration values with metrics and quality. Recalibration of the raw IMU data is necessitated by such factors as a change in temperature, mechanical shock and/or time. Other reasons for recalibration will be readily apparent to one of ordinary skill in the art. TheOnline CNSF 810 stores the calibrated IMU data with annotations and calibration values at a Local Store. TheOnline CNSF 810 then transmits the calibrated IMU data with annotations and calibration values to theData Transfer Service 808. TheData Transfer Service 808 stores the calibrated IMU data with annotations, the calibration values, and raw IMU data at aLocal Store 818. TheOnline CNSF 810 is also connected to theExternal Calibration Updater 812, which provides real-time external calibration data to theOnline CNSF 810. That external calibration data can be used by theOnline CNSF 810 to calculate the calibrated IMU data and the calibration values with metrics and quality. - The
Data Transfer Service 808 and theExternal Calibration Updater 812 are each connected to theBluetooth Module 814. TheData Transfer Service 808 andExternal Calibration Updater 812 transmit one or more of raw IMU data, calibration data, calibrated IMU data with annotations, and calibration values with metrics and quality to and from theBluetooth Module 814. TheBluetooth Module 814 is capable of communicating with theEdge App 820, an application that connects the wearable 110 to thebackend 140. The wearable 802 and thebackend 804 are in continuous two-way communication through theEdge App 820. - The
backend 804 is comprised of aRecalibration Service 824, aDB Manager 826, anOffline CNSF 828, and aRemote Store 830. TheOffline CNSF 828 receives the calibration data from the Edge App and compares it against calibration data that is received from theDB Manager 826, which queries theRemote Store 830 for historical, previously calculated data received from the wearable 802. If the calibration data received from the wearable 802 is determined to be less accurate than the historical data, then theOffline CNSF 828 signals theRecalibration Service 824 to request that theExternal Calibration Updater 812 perform an updated calibration and analysis of the IMU data at the wearable 802. That request for recalibration is transmitted through theEdge App 820 and theBluetooth Module 814, to theExternal Calibration Updater 812. - The foregoing description and drawings should be considered as illustrative only of the principles of the invention. The invention is not intended to be limited by the preferred embodiment and may be implemented in a variety of ways that will be clear to one of ordinary skill in the art. Numerous applications of the invention will readily occur to those skilled in the art. Therefore, it is not desired to limit the invention to the specific examples disclosed or the exact construction and operation shown and described. Rather, all suitable modifications and equivalents may be resorted to, falling within the scope of the invention. All references cited herein are incorporated by reference.
Claims (20)
1. A computer-implemented method for sensor calibration comprising:
receiving raw accelerometer data, raw gyroscope data, and raw magnetometer data;
performing sensor calibration at an inertial measurement unit using the raw accelerometer data, the raw gyroscope data, and the raw magnetometer data;
outputting a plurality of calibration parameters from the inertial measurement unit;
transmitting the plurality of calibrated parameters to a sensor fusion module, wherein the sensor fusion module computes calibrated accelerometer data an orientation; and
transmitting calibrated accelerometer data and orientation to a gravity removal module, wherein the gravity removal module calculates corrected linear acceleration data.
2. The method of claim 1 , wherein there are up to thirty-six calibration parameters.
3. The method of claim 1 , wherein the calibration is dynamically performed.
4. The method of claim 1 , further comprising transmitting calibrated accelerometer data, calibrated magnetometer data, and calibrated gyroscope data from the sensor fusion module to an AI algorithm module.
5. The method of claim 4 , wherein the AI algorithm module is trained to perform an offline correction using the calibrated accelerometer data, calibrated magnetometer data, and calibrated gyroscope data for subsequent iterations.
6. The method of claim 1 , further comprising:
comparing a plurality of previously calculated calibration parameters from the inertial measurement unit to the plurality of calibration parameters; and
updating the plurality of calibration parameters at an external calibration updater located at a wearable device prior their transmittal to the sensor fusion module.
7. The method of claim 1 , wherein the sensor fusion module is used to determine acceleration due to gravity that is projected onto x, y, and z axes.
8. The method of claim 4 , wherein data from the sensor fusion module is used to train the AI algorithm module.
9. The method of claim 1 , wherein historical calibration data is used to update bias and sensitivity parameters.
10. The method of claim 1 , further comprising transmitting the corrected linear acceleration data to a Bluetooth Module for delivery to a second computer.
11. A non-transitory computer storage medium that stores a program thereon that causes a computer to execute a process comprising:
receiving raw accelerometer data, raw gyroscope data, and raw magnetometer data;
performing sensor calibration at an inertial measurement unit using the raw accelerometer data, the raw gyroscope data, and the raw magnetometer data;
outputting a plurality of calibration parameters from the inertial measurement unit;
transmitting the plurality of calibrated parameters to a sensor fusion module, wherein the sensor fusion module computes calibrated accelerometer data and an orientation; and
transmitting calibrated accelerometer and orientation to a gravity removal module, wherein the gravity removal module calculates corrected linear acceleration data.
12. The non-transitory computer medium of claim 11 , wherein there are up to thirty-six calibration parameters.
13. The non-transitory computer medium of claim 11 , wherein the calibration is dynamically performed.
14. The non-transitory computer medium of claim 11 , further comprising transmitting calibrated accelerometer data, calibrated magnetometer data, and calibrated gyroscope data from the sensor fusion module to an AI algorithm module.
15. The non-transitory computer medium of claim 14 , wherein the AI algorithm module is trained to perform an offline correction using the calibrated accelerometer data, calibrated magnetometer data, and calibrated gyroscope data for subsequent iterations.
16. The non-transitory computer medium of claim 11 , wherein the program further causes the computer to execute a process comprising:
comparing a plurality of previously calculated calibration parameters from the inertial measurement unit to the plurality of calibration parameters; and
updating the plurality of calibration parameters at an external calibration updater located at a wearable device prior their transmittal to the sensor fusion module.
17. The non-transitory computer medium of claim 11 , wherein the sensor fusion module is used to determine acceleration due to gravity that is projected onto x, y, and z axes.
18. The non-transitory computer medium of claim 11 , wherein data from the sensor fusion module is used to train the AI algorithm module.
19. The non-transitory computer medium of claim 11 , wherein historical calibration data is used to update bias and sensitivity parameters.
20. The non-transitory computer medium of claim 11 , wherein the corrected linear acceleration data is transmitted to a Bluetooth Module for delivery to a second computer.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US18/238,880 US20240068839A1 (en) | 2022-08-26 | 2023-08-28 | Artificial intelligence-enabled imu calibration and sensor fusion |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US202263401448P | 2022-08-26 | 2022-08-26 | |
US18/238,880 US20240068839A1 (en) | 2022-08-26 | 2023-08-28 | Artificial intelligence-enabled imu calibration and sensor fusion |
Publications (1)
Publication Number | Publication Date |
---|---|
US20240068839A1 true US20240068839A1 (en) | 2024-02-29 |
Family
ID=89998420
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US18/238,880 Pending US20240068839A1 (en) | 2022-08-26 | 2023-08-28 | Artificial intelligence-enabled imu calibration and sensor fusion |
Country Status (2)
Country | Link |
---|---|
US (1) | US20240068839A1 (en) |
WO (1) | WO2024123399A2 (en) |
Family Cites Families (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN107850901B (en) * | 2015-05-23 | 2021-04-16 | 深圳市大疆创新科技有限公司 | Sensor fusion using inertial and image sensors |
US9891245B2 (en) * | 2015-06-29 | 2018-02-13 | CloudNav Inc. | Real-time accelerometer calibration |
US11209456B2 (en) * | 2019-02-22 | 2021-12-28 | Woven Planet North America, Inc. | Integrated movement measurement unit |
-
2023
- 2023-08-25 WO PCT/US2023/031187 patent/WO2024123399A2/en unknown
- 2023-08-28 US US18/238,880 patent/US20240068839A1/en active Pending
Also Published As
Publication number | Publication date |
---|---|
WO2024123399A9 (en) | 2024-08-02 |
WO2024123399A3 (en) | 2024-07-04 |
WO2024123399A2 (en) | 2024-06-13 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10182746B1 (en) | Decoupling body movement features from sensor location | |
Shen et al. | Closing the gaps in inertial motion tracking | |
US10416755B1 (en) | Motion predictions of overlapping kinematic chains of a skeleton model used to control a computer system | |
US9316513B2 (en) | System and method for calibrating sensors for different operating environments | |
JP6668239B2 (en) | Motion tracking using a reduced number of wearing sensor sets | |
CN105850113B (en) | The calibration of virtual reality system | |
EP1959831B1 (en) | Motion tracking system | |
US20130311133A1 (en) | Method for step detection and gait direction estimation | |
US9599634B2 (en) | System and method for calibrating inertial measurement units | |
WO2015146047A1 (en) | Reference-value generation method, motion analysis method, reference-value generation device, and program | |
US11774246B2 (en) | Correction of heading errors of inertial measurement units of a motion tracking system | |
WO2018076205A1 (en) | Stride calibrating method, and relevant device and system | |
US20210108923A1 (en) | Information processing apparatus, information processing method, and program | |
US20230213549A1 (en) | Virtual Reality System with Modeling Poses of Tracked Objects by Predicting Sensor Data | |
CN115999139A (en) | Real-time sports action training aid | |
CN111895997B (en) | Human body action acquisition method based on inertial sensor without standard posture correction | |
CN111624547A (en) | Tracking objects in an electromagnetic field | |
Wang | Multisensor data fusion of motion monitoring system based on BP neural network | |
US20240068839A1 (en) | Artificial intelligence-enabled imu calibration and sensor fusion | |
CN113229806A (en) | Wearable human body gait detection and navigation system and operation method thereof | |
US11369316B2 (en) | Heart rate detection architecture | |
US20240032820A1 (en) | System and method for self-learning and reference tuning activity monitor | |
Wu et al. | Indoor positioning method for pedestrian dead reckoning based on multi-source sensors | |
US20230214027A1 (en) | Reduction of Time Lag Between Positions and Orientations Being Measured and Display Corresponding to the Measurements | |
US20230011082A1 (en) | Combine Orientation Tracking Techniques of Different Data Rates to Generate Inputs to a Computing System |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: RAJANT HEALTH INC., PENNSYLVANIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:CHANDRASEKARAN, MUTHUKUMARAN;MOUNTNEY, JACK;SCHENA, ROBERT JOSEPH;REEL/FRAME:064725/0544 Effective date: 20221129 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |