JP6275887B2 - Sensor calibration method and sensor calibration apparatus - Google Patents

Description

  Modern unmanned aerial vehicles (UAVs), due to their small size and flexibility, are suitable for various mass-use applications such as surveillance and tracking, remote sensing, search and rescue operations, scientific research, entertainment and hobbies, etc. Widely used. A plurality of UAVs are generally wirelessly controlled by one of a remote control device and a control program installed via a plurality of communication links, and the performance of such a plurality of communication links is a plurality of May have a direct impact on the safety and effectiveness of UAV missions. If multiple sensors deviate from the initial configuration, multiple problems may occur during the UAV mission.

  When an unmanned aerial vehicle (UAV) is equipped with multiple sensors, one useful use of UAV is to collect large amounts of data. Some non-limiting examples include multiple terrain photography, audio data recording, air pollution level measurement, and air quality monitoring. Before the UAV takes off, calibration is typically performed to compensate for the lack of data collection. When the UAV is in operation (including but not limited to when the UAV is in flight), the calibration may be performed remotely via multiple wireless communications. Nevertheless, remote calibration is impractical because in many cases the UAV may move out of range of multiple wireless communications. The UAV may have a sensor device having an internal configuration for multiple sensors or multiple sets of parameters. If the instrument settings deviate from the initial or optimal configuration, the collected data will be unreliable or inaccurate. To overcome this difficulty, the subject matter described herein discloses multiple onboard systems and methods for automatically calibrating multiple UAVs and multiple sensors. Automatic calibration may be performed during flight of the UAV (eg, inline calibration) and may streamline the data collection process. In particular, avoiding collection of unnecessary data or improving data quality can significantly reduce costs.

  In one aspect, a sensor calibration method is disclosed herein. The method includes obtaining sensor data during flight of a UAV from a plurality of different types of sensors connected to an unmanned aerial vehicle (UAV), the plurality of sensors having an initial spatial configuration relative to each other. And detecting a change in the spatial configuration of the plurality of sensors from the initial spatial configuration to the next spatial configuration based on the sensor data via the processor, determining the next spatial configuration, Adjusting data from at least one of the sensors during UAV flight based on the following spatial configuration:

  In another aspect, an inline sensor calibration apparatus is disclosed herein, wherein the apparatus is (a) (i) connected to an unmanned aerial vehicle (UAV) and (ii) sensor data during flight of the UAV. A plurality of different types of sensors configured to provide a plurality of sensors having an initial spatial configuration relative to each other, (b) individually or collectively, (i) based on sensor data Detecting a change in the spatial configuration from the initial spatial configuration to the next spatial configuration of each other, (ii) determining the next spatial configuration, and (iii) moving to the next spatial configuration during the flight of the UAV And one or more processors configured to condition data from at least one of the plurality of sensors.

  In another aspect, a sensor calibration method is disclosed herein, the method comprising: obtaining sensor data from a plurality of different types of sensors connected to an unmanned aerial vehicle (UAV); and a reference coordinate system. Selecting the sensor data from the plurality of sensors in a reference coordinate system based on the predicted spatial relationship between the plurality of sensors via the processor, and between the plurality of sensors via the processor Detecting a mismatch in sensor data, wherein the mismatch indicates an error in a predicted spatial relationship between a plurality of sensors; (e) determining an actual spatial configuration; and (f) actual Adjusting data from at least one of the plurality of sensors based on the spatial configuration of the plurality of sensors.

  In other aspects, disclosed herein is a sensor calibration device, wherein the device is (a) (i) connected to an unmanned aerial vehicle (UAV) and (ii) configured to provide sensor data. (B) individually or collectively, (i) select a reference coordinate system, and (ii) from a plurality of sensors based on a predicted spatial relationship between the plurality of sensors. (Iii) is detected in the reference coordinate system, and (iii) a mismatch in sensor data between a plurality of sensors, which indicates an error in a predicted spatial relationship between the plurality of sensors, is detected. One or more processors configured to determine a spatial configuration and (v) adjust data from at least one of the plurality of sensors based on the actual spatial configuration.

  In another aspect, disclosed herein is a sensor calibration method, the method comprising: obtaining sensor data from a plurality of different types of sensors connected to an unmanned aerial vehicle (UAV); and via a processor. Grouping a plurality of sensors into a plurality of subsets, each subset comprising (i) at least two sensors and (ii) different combinations of the plurality of sensors, and via a processor, Calculating a predicted spatial relationship between at least two sensors in each subset based on sensor data and determining an actual spatial relationship between at least two sensors in each subset based on the predicted spatial relationship; And each subset based on the actual spatial relationship between at least two sensors via the processor Comprising definitive of the plurality of sensors, and calculating a spatial configuration relative to one another.

  In other aspects, disclosed herein is a sensor calibration device, wherein the device is (a) (i) connected to an unmanned aerial vehicle (UAV) and (ii) configured to provide sensor data. A plurality of different types of sensors; (b) individually or collectively, (i) a plurality of sensors, a plurality of subsets, each subset comprising: (i) at least two sensors; and (ii) Grouping into a plurality of subsets having different combinations of sensors, (ii) calculating a predicted spatial relationship between at least two sensors in each subset based on sensor data, and (iii) a predicted space Based on the relationship, determine an actual spatial relationship between at least two sensors in each subset; and (iv) an actual spatial relationship between the at least two sensors. And Zui, a plurality of sensors in each subset, configured to calculate the spatial configuration relative to one another, and a one or more processors.

  In another aspect, a sensor calibration method is disclosed herein, the method comprising obtaining sensor data from a plurality of different types of sensors and, via a processor, the plurality of sensors into a plurality of subsets. Grouping, each subset comprising (i) at least two sensors comprising a reference sensor and one or more measurement sensors, and (ii) different combinations of the plurality of sensors, At least two of which have a plurality of different reference sensors and an actual spatial relationship between at least two sensors in each subset using a processor and at least one Kalman filter per subset based on sensor data And the actual space between at least two sensors via the processor Based on engagement, a plurality of sensors in each subset, and a step of calculating a spatial configuration relative to one another.

  In another aspect, disclosed herein is a sensor calibration device, the device comprising: (a) a plurality of different types of sensors configured to provide sensor data; and (b) individually or collectively. (1) a plurality of sensors in a plurality of subsets, each subset comprising (i) at least two sensors comprising a reference sensor and one or more measurement sensors; and (ii) a plurality of sensors At least two of the plurality of subsets having different combinations, grouped into a plurality of subsets having a plurality of different reference sensors, and (2) using at least one Kalman filter per subset based on sensor data, Determining an actual spatial relationship between at least two sensors in the subset; (3) an actual spatial relationship between at least two sensors; Based on the provided plurality of sensors in each subset, configured to calculate the spatial configuration relative to one another, and one or more processors.

  Some or all of the plurality of systems and methods include a plurality of different types of sensors. The plurality of different types of sensors includes at least two of a vision sensor, a GPS sensor, an inertial sensor, an infrared sensor, an ultrasonic sensor, or a lidar sensor. At least one of the plurality of sensors is an inertial sensor, and sensor data from the plurality of sensors is provided to a coordinate system associated with the inertial sensor.

  In some embodiments, the initial spatial configuration of the plurality of systems and methods comprises an initial position and an initial direction relative to each other for each of the plurality of sensors. The change in the spatial configuration comprises a change in an initial position and / or an initial direction with respect to at least one of the plurality of sensors and a plurality of other sensors of the plurality of sensors.

  In some embodiments, the initial spatial configuration of systems and methods is provided prior to UAV takeoff. Alternatively, the initial spatial configuration is given during UAV flight.

  In some embodiments, changes in the spatial configuration of multiple systems and methods are caused by UAV movement. In additional embodiments, the change in spatial configuration is caused by UAV oscillation.

  In some examples, the one or more processors use a plurality of Kalman filters to determine a next spatial configuration, the plurality of Kalman filters comprising one or more extended Kalman filters and / or unscented Kalman filters.

  In some embodiments, one or more processors of multiple systems and methods are mounted on a UAV. Alternatively, the one or more processors are in an external device of the UAV and communicate with the UAV during the UAV flight.

  In some embodiments, the plurality of different types of sensors of the plurality of systems and methods comprise at least two of vision sensors, GPS sensors, inertial sensors, infrared sensors, ultrasonic sensors, or lidar sensors. At least one of the plurality of sensors is an inertial sensor. In particular examples, the reference coordinate system is a coordinate system associated with the inertial sensor.

  In some embodiments, the predicted spatial relationship between the plurality of sensors is in a configuration from an initial spatial configuration of the plurality of sensors comprising an initial position and an initial direction relative to each other for each of the plurality of sensors. Based on predicted spatial changes. In additional embodiments, the initial spatial configuration is provided prior to UAV take-off or during UAV flight.

  In some embodiments, the plurality of sensors comprises a spatial position and a relative spatial direction relative to each other.

  In some applications, the actual spatial configuration of multiple sensors is determined during UAV operation. The actual spatial configuration of the sensors is determined during flight of the UAV. In some examples, data from at least one of the plurality of sensors is adjusted during operation of the UAV.

  In some specific embodiments, the data adjusted based on the actual spatial configuration is image data recorded by a vision sensor.

  In various examples, each subset has a reference sensor and one or more measurement sensors. The reference sensor for each subset is the same or different.

  In some embodiments, the plurality of systems and methods further comprise adjusting data from at least one of the plurality of sensors based on the spatial configuration. Furthermore, the data adjusted based on the actual space configuration is image data recorded by the vision sensor.

  In various embodiments, the predicted spatial relationship between the plurality of sensors in each subset is derived from an initial spatial configuration of the plurality of sensors with an initial position and an initial direction relative to each other for each of the plurality of sensors. Based on the expected spatial variation in the configuration. In some further embodiments, the initial spatial configuration is provided prior to UAV take-off or during UAV flight.

  Some or all of the systems and / or methods comprise a predicted spatial relationship between the plurality of sensors, including the relative spatial position and relative spatial direction of the plurality of sensors relative to each other.

  In some embodiments, multiple systems and / or multiple methods group multiple sensors into multiple subsets. Further, the plurality of systems and / or the plurality of methods calculate a predicted spatial relationship between at least two sensors in each subset based on the sensor data, and the Kalman filter is based on the predicted spatial relationship. Determine the spatial relationship.

  It should be understood that multiple different aspects of the present invention may be recognized individually, collectively or in combination with each other. The various aspects of the invention described herein may be used for any of the specific applications described below or data communication between any type of movable and / or fixed objects. Applicable.

  Other objects and features of the invention will become apparent by reference to the specification, claims and appended drawings.

[Built-in reference]
All documents, patents and patent applications mentioned in this specification are all to the same extent as if each individual document, patent or patent application was specifically and individually indicated to be incorporated by reference. Which is incorporated herein by reference.

  The novel features of the invention are set forth with particularity in the appended claims. A better understanding of the features and advantages of the present invention will be obtained by reference to the following detailed description that sets forth illustrative embodiments, in which the principles of the invention are utilized, and the accompanying drawings of which: It is done.

It is an example of UAV which has the some mounted sensor which concerns on embodiment of this invention.

It is an example of the reference | standard frame conversion which concerns on embodiment of this invention.

It is an example of the image from the stereo vision camera which has the 1st and 2nd camera which concerns on embodiment of this invention.

It is a flowchart which shows the whole automated error detection and in-line calibration process.

It is a flowchart of the sensor error detection method which concerns on embodiment of this invention.

It is a flowchart which shows the in-line calibration method which concerns on embodiment of this invention.

2 is an example of a system architecture with a uniform reference sensor according to an embodiment of the present invention.

2 is an example of a system architecture having a plurality of reference sensors according to an embodiment of the present invention.

1 shows an unmanned aerial vehicle according to an embodiment of the present invention.

1 shows a movable object including a support mechanism and a load according to an embodiment of the present invention.

It is a schematic diagram by the block diagram of the system for controlling the movable object which concerns on embodiment of this invention.

  The systems, devices and methods of the present invention provide a plurality of sensors mounted on an unmanned aerial vehicle (UAV) and a method for calibrating sensors mounted on a UAV. The UAV description is applicable to any kind of unmanned transporter or any kind of movable object. The description of the transporter is applicable to any sensor that is mounted on the UAV. Examples of multiple sensors mounted on a UAV include multiple vision sensors, GPS or IMU.

  The systems, devices and methods of the present invention provide a plurality of sensors mounted on an unmanned aerial vehicle (UAV) and a plurality of methods for calibrating a plurality of sensors mounted on a UAV. The various aspects of the described disclosure are applicable to any of the applications identified herein, in addition to any of the plurality of transporters comprising a plurality of sensors. It should be understood that different aspects of the invention may be recognized individually, collectively or in combination with each other.

  FIG. 1 shows an example of an unmanned aerial vehicle (UAV). Any description of UAV 101 herein is applicable to any type of movable object. The UAV description is applicable to any type of unmanned movable object (eg, capable of traversing air, land, water or space). The UAV may be able to respond to multiple commands from the remote controller. The remote controller may not be connected to the UAV, and the remote controller may perform wireless communication with the UAV from a remote location. In some examples, the UAV may be able to operate autonomously or semi-autonomously. A UAV may be capable of following a set of pre-programmed instructions. In some examples, the UAV may operate semi-autonomously by responding to one or more commands from a remote controller, or may operate autonomously. For example, one or more commands from a remote controller may initiate a series of autonomous or semi-autonomous operations by the UAV according to one or more parameters.

  The UAV 101 may be an aircraft. The UAV 101 may have one or more propulsion units 102 that may allow the UAV 101 to move about in the air. One or more propulsion units 102 have one or more UAVs 101, two or more, three or more, four or more, five or more, six or more It may be possible to move around with more degrees of freedom. In some examples, the UAV 101 may be rotatable about 1, 2, 3, or more axes of rotation. The plurality of rotation axes may be orthogonal to each other. The plurality of rotation axes may remain orthogonal to each other over the entire flight course of the UAV. The plurality of rotation axes may include a pitch axis, a roll axis, and / or a yaw axis. The UAV 101 may be movable along one or more dimensions. For example, the UAV 101 may be able to move upward by lift generated by one or more rotor blades. In some examples, the UAV is movable along the Z-axis (which may be upward relative to the direction of the UAV), the X-axis and / or the Y-axis (which may be lateral). May be. The UAV may be movable along one, two or three axes that may be orthogonal to each other.

  The UAV 101 may be a rotary wing aircraft. In some examples, UAV 101 may be a multi-rotor aircraft that may include multiple rotor blades. The plurality of rotor blades may be rotatable to generate lift for the UAV. The plurality of rotor blades may be a plurality of propulsion units that may allow the UAV to freely move about in the air. The plurality of rotor blades may rotate at the same speed and / or generate the same amount of lift or thrust. The plurality of rotors may optionally rotate at different speeds, which may generate different amounts of lift or thrust and / or allow UAV rotation. In some examples, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 or more rotors may be provided to the UAV. The plurality of rotor blades may be configured such that the plurality of rotation axes are parallel to each other. In some examples, a plurality of rotor blades may have a plurality of axes of rotation that are at an arbitrary angle with respect to each other, which can affect UAV movement.

  The UAV 101 may have a plurality of on-board sensors that collect data during UAV operation. The plurality of sensors may have an initial configuration. The initial configuration of the plurality of sensors may deviate from the initial and / or optimal settings. Thus, automatic in-line calibration of multiple sensors when the UAV is in flight is beneficial and in some cases may be necessary to ensure the quality and reliability of the collected data. The subject matter described herein discloses an onboard system and method for automatically calibrating UAVs and sensors. Multiple automated in-line calibration systems and methods can facilitate the data collection process.

  The UAV 101 may include a plurality of vision sensors. For example, the image sensor may be a monocular camera, a stereo vision camera, a radar, a sonar, or an infrared camera. The UAV 101 may further include a plurality of position sensors, for example, GPS, IMU, or LIDAR. The plurality of sensors mounted on the UAV may collect information such as the position of the UAV, the positions of other objects, the direction of the UAV 101, or environment information. A single sensor may be able to fully determine any one of the above-described parameters, or a group of sensors determines one of the listed parameters You may cooperate with. Multiple sensors may be used for location mapping, navigation between multiple locations, multiple obstacle detection or target detection.

  The plurality of sensors may be mounted on the UAV or not mounted. A plurality of mounted sensors may be arranged in the main body of the UAV 101. The plurality of sensors 103 may be attached outside the main body of the UAV 101 and / or the sensor 104 may be attached inside the main body of the UAV 101. The plurality of sensors may be arranged in the center of a single region of the body. Alternatively, the plurality of sensors may be disposed at a plurality of different positions on the body. The plurality of sensors may be attached to the UAV 101 permanently or detachably. The UAV 101 may have a support mechanism 105 that transports the load. The plurality of sensors may be attached to the support mechanism 106.

  The plurality of sensors may be characterized by a plurality of parameters of the one or more sensors. The plurality of sensor parameters may be a plurality of internal or external parameters. The internal parameter may relate to the internal configuration of the sensor. Examples of multiple internal parameters may include focal length, magnification, multiple radial distortion coefficients, and multiple tangential distortion coefficients. The internal parameters may be any number of parameters that depend on multiple hardware configurations, and in some cases, the multiple internal parameters may be set by factory settings for the sensor. The plurality of external parameters may relate to a spatial relationship between any two or more sensors. Each sensor may have a relative coordinate system that does not depend on a plurality of other sensors mounted on the UAV. Multiple external characteristics may be important for sensor fusion that combines data from multiple sensors at different locations in the UAV. Sensor fusion may involve processing to transform a plurality of relative coordinates of a given sensor to match the reference frame of another sensor.

  FIG. 2 illustrates the conversion process so that the relative coordinates of a given sensor coincide with the reference frame of another sensor. FIG. 2 shows a cube 201 viewed from the sensor 1 in the coordinate system 202 of the sensor 1 and a cube 203 viewed from the sensor 2 in the coordinate system 204 of the sensor 2. The transformation may be performed such that the coordinate system 204 of the sensor 2 is rotated so as to coincide with the coordinate system 202 of the sensor 1. The transformation may be described mathematically by a transformation matrix.

  The plurality of internal characteristics may be unique to the sensor or may differ infrequently. Recalibration of multiple internal characteristics may be performed periodically while the UAV is not operating. Recalibrating multiple internal characteristics may not be essential during UAV 101 operation. This is because these multiple characteristics are maintained in a relatively consistent state during operation of the UAV 101 compared to multiple external characteristics. The plurality of internal characteristics may be calibrated by interpreting images of known calibration standards or targets. Calibration standards or elimination of multiple lines or points on the target may be used to calibrate multiple internal characteristics such as focal length and distortion.

  The plurality of external characteristics may change more frequently than the plurality of internal characteristics. Shifts, vibrations and thermal drift during UAV landing and takeoff can cause multiple changes in multiple external characteristics of multiple sensors. For example, the camera position may shift due to UAV vibration during flight. Several external characteristics may deviate from their initial configuration during UAV operation. Therefore, it may be preferable to perform recalibration of multiple external characteristics during UAV operation. Recalibration of a plurality of external characteristics during operation of the UAV 101 may require a plurality of computing resources, and the plurality of computing resources may be mounted on or not mounted on the UAV 101. Recalibration of multiple external characteristics may be performed at a set time frequency, for example, multiple external characteristics may be 1 minute, 5 minutes, 10 minutes, 20 minutes, 30 minutes, 40 minutes, 50 minutes, It may be recalibrated every 1 hour, 2 hours, 3 hours, 4 hours, 5 hours, 10 hours, every 12 hours, or once daily. Alternatively, recalibration of a plurality of external characteristics may be performed at a set distance frequency, for example, recalibration may be performed by another 0.5 mile, 1 mile, 2 mile, 3 mile from the UAV 101's initial starting position. It may be performed every time a mile, 4 miles, 5 miles, 10 miles, 15 miles, 20 miles, 25 miles, 30 miles, 35 miles, 40 miles, 45 miles, 50 miles or 100 miles are moved. The frequency of calibration of on-board external characteristics may be determined based on available computing resources, fuel or power requirements, or flight conditions. Some flight conditions may increase or decrease the expected deviation in the calibration of multiple external sensors. For example, when the UAV is flying at a low speed, the vibration of the UAV main body is reduced, so that the deviation in calibration of a plurality of external sensors can be reduced.

  The plurality of external parameters may have an initial calibration. Initial calibration of multiple external parameters accounts for relative differences between multiple sensors, for example, relative multiple positions, multiple rotations and / or multiple displacements of two or more sensors. May be. The plurality of parameters may include a plurality of changes over time to the plurality of sensors, for example, a plurality of displacements of the plurality of sensors between a specific time and a next time. The plurality of displacements may include translational displacements and / or rotational displacements. Translational displacement may be performed along one or more of the three axes. Similarly, rotational displacement may be performed on one or more of the three axes. In general, calibration is achieved by a filtering process. Non-limiting examples include various types of multiple Kalman filters.

  Calibration of multiple external parameters may be adjusted during operation of the UAV 101 (eg, during flight). A method of calibrating a plurality of external parameters may comprise detecting, via a processor, a change in spatial configuration from an initial spatial configuration to a next spatial configuration with respect to each other of two or more sensors. . In further embodiments, the method uses a plurality of filters, such as a plurality of Kalman filters, to determine the next spatial configuration. Finally, the method may include adjusting data from at least one of the plurality of sensors based on the following spatial configuration during the flight of the UAV.

  A UAV may have one or more onboard processors. The processors, individually or collectively, detect (i) a change in the spatial configuration of one or more sensors from one initial configuration to the next based on the sensor data, ii) determine a next spatial configuration using a plurality of Kalman filters, and (iii) adjust data from at least one of the plurality of sensors based on the next spatial configuration during flight of the UAV It may be configured. Alternatively, the processor or processors may be unmounted on the UAV. The UAV may send information about the spatial configuration of a given sensor to a non-mounted processor, which performs the steps (i)-(iii) described above and reverses the information. May be configured to transmit to the UAV.

[Calibrate multiple cameras]
In some embodiments, multiple systems, devices and methods may include multiple sensors. Calibration of the plurality of sensors may comprise integrating sensor data. Referring to FIG. 3, one camera may capture the image shown at 301, and another camera may capture the second image 302 with different displacements and different directions. Thus, the two cameras need to be calibrated, and the calibration may use both an image 301 taken by the first camera and a second image 302 taken by the second camera. In the following, a plurality of calibration mathematical formulas are disclosed.

In an embodiment, two or more cameras are assembled to form a stereo camera system. Calibration of two or more cameras is as follows. First, each camera takes an image. Next, the specific system is
Select individual features. In mathematical formulas,
as well as
Represents two cameras. The plurality of features identified in the plurality of images are a plurality of vectors.
as well as
Where, where
It is. Multiple features
as well as
Multiple coordinate systems for multiple cameras
as well as
Respectively. For faithful mapping, multiple features have the same reference coordinate system
Needs to be analyzed. Multiple features
as well as
And reference coordinate system
The relationship between and the projection
as well as
Where may be described by:
as well as
Is a standardized set of coordinates, i.e.
as well as
A plurality of features described in.
as well as
Each with multiple cameras
as well as
Multiple projections, which are multiple internal parameters
And multiple external parameters (eg rotation
And translation
as well as
May be determined by: Multiple projections once
as well as
Multiple internal parameters are calculated
as well as
When is known, multiple external parameters
And T may be calculated.

  If multiple parameters R and T are obtained, calibration is complete.

In general, multiple internal parameters
as well as
Does not change. Even if these change, the amount of change is small. As such, multiple internal parameters may be calibrated offline. That is, in some applications, multiple internal parameters may be determined prior to UAV takeoff. In several examples, multiple internal parameters
as well as
Is kept stationary during flight, so calibration is
as well as
Is to calculate a plurality of optimal solutions. An example uses multiple projection error minimizations to find multiple solutions.

This problem is a nonlinear optimization problem. Various multiple solutions may be included in multiple embodiments. In some applications, multiple solutions are realized by the bundle adjustment method. In bundle adjustment method, multiple projections
as well as
Are a plurality of given initial values. Required matrix
Use epipolar constraints to obtain
Perform a decomposition (eg, singular value decomposition) to obtain
Is the distortion target matrix
It is.

  This solution finds a corresponding mapping between these features and features in other images taken by other cameras.

  In some embodiments, the spatial configuration of the two cameras α and β forming the stereo camera system configures one camera on the left and the other on the right.

[Camera and IMU calibration]
In some embodiments, the multiple systems and multiple methods include camera and IMU calibration. The principle of this calibration is based on obtaining multiple images over time by the camera and predicting multiple position changes of the self by the camera. In some examples, the self-calibration method includes two cameras
as well as
Multiple times depending on
as well as
Similar to calibrating two impending cameras by examining two images of cameras taken in Similarly, a self-calibration scheme may be applied to the IMU.
as well as
Each represents a plurality of changes in the camera and IMU self-coordinates. Multiple subscripts
Time by placing
Multiple coordinate system mapping in
as well as
To express. As a result, the multiple mappings at time 2 related to time 1 are
as well as
It becomes.
By letting you represent the mapping between camera and IMU, the visual and hand calibration equations are
Where
Is a standardized mapping in the following form:
,
,

further,
as well as
It is. According to these equations and considering the properties of the rotation matrices,
as well as
There are several solutions for. In order to guarantee a unique solution, the requirement n ≧ 3 must be met.

  In some embodiments, calibration is performed between the stereo camera system and the IMU. Calibration is achieved by calibrating one camera in the system with the IMU.

  The UAV may include a device that interprets the sensor data to determine if the sensor requires recalibration. The device may interpret the sensor data in real time. The device may interpret sensor data from any or all of a plurality of sensors mounted on the UAV. The device may comprise one or more sensors that can be connected to the UAV. The plurality of sensors may provide the UAV with information about the environment, nearby objects, UAV location, and / or nearby objects or obstacles. The plurality of sensors may have a known initial configuration that may include a plurality of internal and external characteristics. The initial configuration may comprise an initial position and orientation of the sensor relative to at least one other sensor. The initial configuration of the plurality of sensors may be communicated to an on-board or non-on-board processor prior to UAV take-off or during UAV flight. The processor may monitor a plurality of changes in the plurality of sensor configurations that may occur as a result of UAV movement or vibration to perform in-line sensor calibration when the plurality of sensor configurations change.

  A processor installed or not in the UAV may acquire sensor data continuously or at separate time intervals. FIG. 4 shows a flowchart illustrating the steps performed by the processor in performing calibration of the inline sensor. First, the processor may obtain sensor data (401). The plurality of sensors may have an initial spatial configuration relative to each other. The initial spatial configuration of the plurality of sensors may be empirically known. The initial spatial configuration may be recorded and stored in a memory location installed in the UAV prior to or after the UAV takes off. The initial configuration may be an initial position and an initial direction of the sensor with respect to other sensors mounted on the UAV. Based on the sensor data received by the processor, the processor may detect changes in the spatial configuration of the one or more sensors (402). The change in the spatial configuration for one or more of the plurality of sensors may be a change relative to a known initial configuration. The processor may then determine a new spatial configuration deviation from the initial spatial configuration (403). The processor may use one or more Kalman filters to determine the deviation of the new spatial configuration relative to the initial spatial configuration. The plurality of Kalman filters may be a plurality of extended Kalman filters or a plurality of unscented Kalman filters. The plurality of Kalman filters may all be of the same type, or they may be different so that some of the plurality of filters can be expanded and the remaining part can be a plurality of unscented Kalman filters. After the deviation is determined, the processor may correct or adjust the data from the plurality of sensors to reflect the newly determined position of the plurality of sensors relative to each other (404). The data adjustment may be performed during operation of the UAV (eg, during flight of the UAV). The steps described in FIG. 4 may be performed by an apparatus that includes a plurality of sensors and a plurality of processors. The device may be connected to the UAV.

  The processor or the plurality of processor systems is mounted on the UAV or is not mounted on the UAV and communicates with the UAV wirelessly or by wire, but the deviation of the spatial configuration of one or more sensors from the initial spatial configuration It may be configured to detect. Detection of spatial configuration deviations may involve the application of one or more statistical methods to the sensor data. One example of a statistical method that can be used for spatial configuration deviation detection is the Mahalanobis distance method.

  The processor may detect discrepancies in the spatial configuration of one or more sensors using the method shown in FIG. The processor may receive data from one or more sensors mounted on the UAV (501). The plurality of sensors may have a plurality of different spatial directions relative to each other so that data from each sensor may have a different relative coordinate system. For data interpretation, it may be preferable to convert the data into a consistent unified reference coordinate system. The reference coordinate system may be any coordinate system or it may be the relative coordinate system of one of the plurality of sensors, for example, the coordinate system of an inertial measurement unit (IMU). The processor may select a reference coordinate system to transform the data prior to data interpretation (502). The reference coordinate system may be selected before the UAV takes off, or it may be selected during the flight. The reference coordinate system may be consistent throughout the operation of the UAV, or it may differ over time. In some cases, the relative coordinate system of the IMU may be selected as the reference coordinate system. The processor may collect sensor data and represent all data in a unified reference coordinate system (503). After the data is converted to a unified reference coordinate system, the data may be analyzed to detect a plurality of inconsistencies in the spatial configuration of the one or more sensors (504). A discrepancy may indicate an error in an expected or predicted spatial relationship between one or more sensors. The detected mismatches may indicate that the sensor or sensors require recalibration.

  Detection of multiple discrepancies in the spatial configuration of one or more sensors may be performed using one or more statistical methods. In one example, the Mahalanobis distance method may be used to detect a plurality of mismatches in the spatial configuration of one or more sensors. The Mahalanobis distance method may compare a plurality of measured values from a plurality of different sensors after the plurality of measured values are converted into a unified reference coordinate system. The processor may detect a plurality of inconsistencies by generating a covariance matrix between a plurality of measurements from a plurality of different sensors in the unified reference coordinate system using the Mahalanobis distance method. As the processor receives measurement data from multiple sensors, the covariance matrix may be updated in real time. The processor may calculate a Mahalanobis distance between a plurality of measured values from a plurality of different sensors in the unified reference coordinate system. If the Mahalanobis distance exceeds a preset threshold, the Mahalanobis distance may indicate a sensor abnormality while indicating that the plurality of sensors have deviated from their initial calibration. The threshold preset for the Mahalanobis distance to indicate an error may be at least 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 15 or 20. The threshold value may be a uniform value for all of the plurality of sensors, or the value may be different for each sensor. The threshold may be fixed or it may be different for independent variables such as UAV travel time or distance.

  If it is detected that the mismatch indicated by the Mahalanobis distance calculation is greater than a predetermined threshold, recalibration of the sensor with the detected mismatch may be initiated. The mismatch detection may be a result of sensor movement away from the sensor's initial direction and toward a new spatial direction. Multiple sensors can be used during UAV takeoff and landing, in flight as a result of UAV vibrations, or with other objects (eg, animals or insects, terrain (eg, trees, cliffs). , Waterfalls) or other UAVs) as a result of movement.

  Sensor recalibration may require determining a new spatial orientation of the sensor. The new spatial direction of the sensor may be determined using one or more Kalman filters. The plurality of Kalman filters may be a plurality of extended Kalman filters or a plurality of unscented Kalman filters. The new spatial direction determined by the multiple Kalman filters is used to update a plurality of external characteristics of one or more sensors that may correspond to a detected inconsistent sensor or a spatial direction of the plurality of sensors. Also good. The updated plurality of external characteristics may be used to adjust the data from the plurality of sensors or the plurality of sensors.

  Multiple Kalman filters may continuously update sensor calibration using an iterative method. The plurality of Kalman filters may use measurement data from a plurality of sensors and constraints between the measurement data as measurement formulas.

  All of the multiple sensors may use a single Kalman filter to perform calibration. Alternatively, calibration may be achieved by connecting multiple Kalman filters in parallel with each other. Each of the plurality of Kalman filters in a parallel architecture may be independent of the rest of the plurality of filters. Each Kalman filter may perform calibration for one or more sensors. If the Kalman filter is responsible for calibrating more than one sensor, multiple sensors calibrated by the Kalman filter may be related. Multiple parallel Kalman filters can improve the scalability of the system. Further, the plurality of parallel Kalman filters can improve the scalability of the system and further improve the response speed while reducing the calculation resources necessary for the calibration of the plurality of sensors.

  An example of a sensor calibration method that can be performed by the system described herein is shown in FIG. The first stage of the method may be obtaining data from one or more sensors mounted on the UAV (601). Data may be collected by the processor, which may be mounted on the UAV or in a non-mounted location on the UAV, where the non-mounted location is wireless or wired It may be communicable with the UAV via a connection. The processor may group the plurality of sensors into one or more subsets such that each subset has at least two sensors (602). Each subset may include a plurality of groups of measurement sensors. The subset may further have a reference sensor, the reference sensor may be the same for each subset, or each subset may have a plurality of unique reference sensors. A subset of multiple sensors may be associated with multiple sensors, for example, they may be multiple sensors with associated functionality. Each subset may comprise a different subset of multiple measurement sensors such that each subset is distinct and there is no overlap between multiple measurement subsets. For example, if there are five sensors A, B, C, D and E, two subsets may be generated using these multiple sensors. Subset 1 may comprise a plurality of sensors A and B, and subset 2 may comprise a plurality of sensors C, D and E. Grouping with subset 1 comprising A, B and C and subset 2 comprising A, D and E is not allowed because sensor A is included in both subsets. After the multiple sensors are grouped into multiple subsets, the next step may be to calculate the expected spatial relationship between the multiple sensors based on data from the multiple sensors (603). ). Next, the actual spatial relationship between the plurality of sensors may be determined using at least one Kalman filter (604). Each subset may have at least one Kalman filter to perform multiple calculations using sensor data. In the last step, the spatial configuration of the one or more sensors may be calculated using a processor (605). The calculation of the spatial configuration of multiple sensors may be based on the actual spatial relationship between multiple sensors in the subset, as determined by one or more Kalman filters.

  A UAV may comprise different types of sensors. The plurality of sensors may be further configured into a plurality of sensors or a plurality of subset groups. Each subset may include at least two sensors comprising a reference sensor and one or more measurement sensors. Each subset may comprise a different subset of the plurality of measurement sensors such that each subset is distinct and there is no overlap between the plurality of subsets. All of the plurality of subsets may have a single uniform reference sensor. For example, all of the plurality of subsets may be analyzed against an IMU that can be selected as a reference sensor. Alternatively, each subset may have a different reference sensor. At least one Kalman filter per each subset may determine an actual spatial relationship between at least two sensors in each subset based on the sensor data. Multiple Kalman filters may function in parallel to determine the actual spatial relationship between multiple sensors in multiple subsets. Further calculations may be performed in each subset to determine the spatial configuration based on the actual spatial relationship between the at least two sensors relative to each other of the plurality of sensors.

  FIG. 7 visually illustrates an example of an applicable system architecture. At the highest level, the system may comprise a system 701 of multiple sensors or subsets of multiple sensors. One of the plurality of sensors may be designated as the reference sensor 701a. The remaining plurality of measurement sensors 701b may collect data in a plurality of relative coordinate systems different from the relative coordinate system of the reference sensor 701a. The plurality of measurement values 702 from the plurality of measurement sensors 701b may be converted into the coordinate system of the reference sensor 701a at level 703. After conversion, the data from each subset may be replaced with a Kalman filter 704. Each subset of the plurality of filters may replace the data with a different Kalman filter 704. The multiple Kalman filters may determine the exact spatial relationship of the sensors in subset 705 and then the accurate relationship of the sensors in system 706. The plurality of Kalman filters may be connected in parallel.

  The sensor may be calibrated by a Kalman filter by first establishing a dynamic equation for describing the movement of the system and a conversion equation between the selected sensor and the reference sensor. The dynamic equations for system motion may vary primarily with data from the IMU, including system acceleration and angular acceleration. Next, the initial state of the Kalman filter may be determined from a plurality of factory settings, for example. The initial state may be described by a state variable vector and a covariance matrix. For multiple internal receptive sensors (eg, IMU), the measured data may be read directly, and for multiple external receptive sensors (eg, cameras), the measured data May be obtained by selecting an appropriate observation spot based on a plurality of external parameters of the sensor at the previous stage. In the last stage, the status of the Kalman filter may be updated by replacing the measured data with an update formula in the Kalman filter. The update formula may vary depending on the type of Kalman filter (eg, extended or uncentred).

  An example of a possible system architecture is visually shown in FIG. The system may have multiple sensors that can be grouped into multiple subsets. Each of the plurality of subsets may have a reference sensor. A reference sensor may be used for one or more subsets. In the example shown in FIG. 8, the left camera 801 and the IMU 802 may be a plurality of reference sensors. The remaining plurality of sensors (for example, the right camera, GPS, and ultrasound) may be the measurement sensor 803. Similar to the process described in FIG. 7, each of the sensor multiple measurements may be converted to an associated reference coordinate system 804. The transformed multiple measurements may be calibrated at step 805 using a Kalman filter. The plurality of Kalman filters may be unscented or a plurality of extended Kalman filters. The multiple Kalman filters may determine the exact spatial relationship of the sensors in subset 806 and then the exact relationship of the sensors in system 807.

  The multiple systems, multiple devices, and multiple methods described herein may be applied to a wide variety of movable objects. As previously mentioned, any description of an aircraft such as a UAV herein is applicable to and can be used with any movable object. Any description herein for aircraft is specifically applicable to multiple UAVs. The movable object of the present invention can be in any suitable environment, for example, in the air (eg, fixed wing aircraft, rotary wing aircraft, or aircraft having neither fixed wings nor rotary wings), underwater (eg, ships or submarines). , Ground (eg, automobiles such as passenger cars, trucks, buses, vans, motorcycles, bicycles, or trains), underground (eg, subways), space (eg, space planes, satellites or probes), or multiple environments thereof It may be configured to move through any combination of The movable object may be a transporter, for example, a transporter described elsewhere herein. In some embodiments, the movable object may be carried by a live object, such as a human or animal, or may take off from the live object. Suitable animals may include birds, dogs, cats, horses, cows, sheep, pigs, dolphins, rodents or insects.

  The movable object may be freely movable in the environment with 6 degrees of freedom (eg, 3 degrees of freedom in translation and 3 degrees of freedom in rotation). Alternatively, the movement of the movable object may be constrained for one or more degrees of freedom, for example by a predetermined path, trajectory or direction. The movement may be driven by any suitable drive mechanism, such as an engine or motor. The drive mechanism of the movable object is powered by any suitable energy source, for example, electrical energy, magnetic energy, solar energy, wind energy, gravity energy, chemical energy, nuclear energy or any suitable combination thereof. May be. The movable object may self-propell through a propulsion system as described elsewhere herein. The propulsion system may optionally operate from an energy source, such as electrical energy, magnetic energy, solar energy, wind energy, gravity energy, chemical energy, nuclear energy, or any suitable combination thereof. Alternatively, the movable object may be carried by a living thing.

  In some examples, the movable object may be an aircraft. For example, a plurality of aircraft may be fixed wing aircraft (eg, airplanes, gliders), rotary wing aircraft (eg, helicopters, rotary wing aircraft), aircraft having both fixed wings and rotary wings, or aircraft that do not have either ( For example, an airship or a hot air balloon) may be used. An aircraft may self-propel, for example, as it self-propels in the air. A self-propelled aircraft may employ a propulsion system, such as a propulsion system that includes one or more engines, motors, wheels, axles, magnets, rotors, propellers, blades, nozzles, or any suitable combination thereof. . In some examples, a movable object may take off from a plane, land on a plane, maintain its current position and / or direction (eg, hover), change direction, and / or change position A propulsion system may be used to enable.

  The movable object may be remotely controlled by a user or may be controlled on-site by a passenger in or on the movable object. The movable object may be remotely controlled via another vehicle occupant. In some embodiments, a movable object, an unattended movable object such as a UAV. In an unmanned movable object such as UAV, a passenger may not board the movable object. The movable object may be controlled by a human or autonomous control system (eg, a computer control system) or any suitable combination thereof. The movable object may be an autonomous or semi-autonomous robot, for example, a robot configured to have artificial intelligence.

  The movable object may have any suitable size and / or dimensions. In some embodiments, the movable object may be of a size and / or dimension that allows the crew to board in or on the transport. Alternatively, the movable object may be smaller than the size and / or dimensions that allow the crew to board in or on the transport. The movable object may be of a size and / or dimension suitable for lifting or carrying by a human. Alternatively, the movable object may be larger than the size and / or dimensions suitable for lifting or carrying by a human. In some examples, the movable object may have a maximum dimension (eg, length, width, height, diameter, diagonal) of about 2 cm, 5 cm, 10 cm, 50 cm, 1 m, 2 m, 5 m or 10 m or less. Good. The maximum dimension may be about 2 cm, 5 cm, 10 cm, 50 cm, 1 m, 2 m, 5 m, or 10 m or more. For example, the distance between the shafts of the opposed rotor blades of the movable object may be about 2 cm, 5 cm, 10 cm, 50 cm, 1 m, 2 m, 5 m, or 10 m or less. Alternatively, the distance between the shafts of the opposing rotor blades may be about 2 cm, 5 cm, 10 cm, 50 cm, 1 m, 2 m, 5 m, or 10 m or more.

In some embodiments, the movable object may have a volume of less than 100 cm × 100 cm × 100 cm, less than 50 cm × 50 cm × 30 cm, or less than 5 cm × 5 cm × 3 cm. The total volume of the movable object is about 1 cm ³ , 2 cm ³ , 5 cm ³ , 10 cm ³ , 20 cm ³ , 30 cm ³ , 40 cm ³ , 50 cm ³ , 60 cm ³ , 70 cm ³ , 80 cm ³ , 90 cm ³ , 100 cm ³ , 150 cm ³ , ^{200cm 3, 300cm 3, 500cm 3} , 750cm 3, 1000cm 3, 5000cm 3, 10,000cm 3, may be 100,000 ^3, 1 m ³ or 10 m ³ or less. Conversely, the total volume of the movable object is about 1 cm ³ , 2 cm ³ , 5 cm ³ , 10 cm ³ , 20 cm ³ , 30 cm ³ , 40 cm ³ , 50 cm ³ , 60 cm ³ , 70 cm ³ , 80 cm ³ , 90 cm ³ , 100 cm ³ , ^{150cm 3, 200cm 3, 300cm 3} , 500cm 3, 750cm 3, 1000cm 3, 5000cm 3, 10,000cm 3, may be 100,000 ^3, 1 m ³ or 10 m ³ or more.

In some embodiments, the footprint of the movable object (which may refer to the side cross-sectional area covered by the movable object) is about 32,000 cm ² , 20,000 cm ² , 10,000 cm ² , 1,000 cm ^{2. , 500cm 2, 100cm 2, 50cm} 2, 10cm 2 , or may be 5 cm ² or less. Conversely, footprint, of about ^{32,000cm 2, 20,000cm 2, 10,000cm 2} , 1,000cm 2, 500cm 2, 100cm 2, 50cm 2, 10cm may be ^two or 5 cm ² or more.

  In some examples, the weight of the movable object may be 1000 kg or less. The weight of the movable object is about 1000 kg, 750 kg, 500 kg, 200 kg, 150 kg, 100 kg, 80 kg, 70 kg, 60 kg, 50 kg, 45 kg, 40 kg, 35 kg, 30 kg, 25 kg, 20 kg, 15 kg, 12 kg, 10 kg, 9 kg, 8 kg, 7 kg. 6 kg, 5 kg, 4 kg, 3 kg, 2 kg, 1 kg, 0.5 kg, 0.1 kg, 0.05 kg, or 0.01 kg or less. Conversely, the weight is about 1000 kg, 750 kg, 500 kg, 200 kg, 150 kg, 100 kg, 80 kg, 70 kg, 60 kg, 50 kg, 45 kg, 40 kg, 35 kg, 30 kg, 25 kg, 20 kg, 15 kg, 12 kg, 10 kg, 9 kg, 8 kg, 7 kg. 6 kg, 5 kg, 4 kg, 3 kg, 2 kg, 1 kg, 0.5 kg, 0.1 kg, 0.05 kg, or 0.01 kg or more.

  In some embodiments, the movable object may be small relative to the load carried by the movable object. The load may include a load and / or support mechanism, as described in further detail elsewhere herein. In some examples, the ratio of the weight of the movable object to the applied load may be greater than about 1: 1, less than or equal to this. In some examples, the ratio of the weight of the movable object to the applied load may be greater than about 1: 1, less than or equal to this. Optionally, the ratio of support mechanism weight to applied load may be greater than about 1: 1, less than or equal to this. If desired, the ratio of the weight of the movable object to the load may be 1: 2, 1: 3, 1: 4, 1: 5, 1:10 or less, or even smaller. Conversely, the ratio of the weight of the movable object to the load may also be 2: 1, 3: 1, 4: 1, 5: 1, 10: 1 or more, or even greater.

  In some embodiments, the movable object may have low energy consumption. For example, the use of movable objects may be about 5 W / h, 4 W / h, 3 W / h, 2 W / h, less than 1 W / h or less. In some examples, the support mechanism of the movable object may be low energy consumption. For example, the use of a support mechanism may be about 5 W / h, 4 W / h, 3 W / h, 2 W / h, less than 1 W / h or less. Optionally, the load of movable objects may have a low energy consumption of, for example, about 5 W / h, 4 W / h, 3 W / h, 2 W / h, less than 1 W / h or less.

  FIG. 9 shows an unmanned aerial vehicle (UAV) 900 according to embodiments of the present invention. A UAV may be an example of a movable object, as described herein. The UAV 900 may include a propulsion system having four rotor blades 902, 904, 906 and 908. Any number of rotor blades (eg, 1, 2, 3, 4, 5, 6 or more) may be provided. The unmanned aerial vehicle's multiple wings, multiple wing components, or other propulsion systems allow the unmanned aircraft to maintain hovering / position, change direction, and / or change position. The distance between the shafts of the plurality of opposed rotor blades may be any suitable length 910. For example, the length 910 may be 2 m or less, or 5 m or less. In some embodiments, the length 910 may be in the range of 40 cm to 1 m, 10 cm to 2 m, or 5 cm to 5 m. Any description herein for UAV is applicable to movable objects, eg, different types of movable objects, and vice versa. The UAV may use an auxiliary takeoff system or method as described herein.

  In some embodiments, the movable object may be configured to carry a load. The load may include one or more passengers, cargo, equipment, equipment, and the like. A load may be provided in the housing. The housing may be separate from the movable object housing or may be part of the movable object housing. Alternatively, the movable object may not have a housing and the load may be provided with the housing. Alternatively, part of the load or the entire load may be provided without a housing. The load may be firmly fixed to the movable object. Optionally, the load may be movable relative to the movable object (eg, translatable or rotatable relative to the movable object). The load may include a load and / or a support mechanism, as described elsewhere herein.

  In some embodiments, the movement of the fixed reference frame (eg, ambient environment) and / or movable objects, support mechanisms, and load relative to each other may be controlled by the terminal. The terminal may be a remote control device located remotely from the movable object, support mechanism and / or load. The terminal may be arranged or fixed on the support base. Alternatively, the terminal may be a handheld or wearable device. For example, the terminal may include a smartphone, tablet, laptop, computer, glasses, gloves, helmet, microphone, or any suitable combination thereof. The terminal may include a user interface such as a keyboard, mouse, joystick, touch screen or display. Any suitable user input, e.g., manually entered commands (e.g., via terminal movement, position or tilt), voice control, gesture control or position control are used for interaction with the terminal May be.

  The terminal may be used to control any suitable state of the movable object, the support mechanism and / or the load. For example, the terminal may be used to control the position and / or orientation of movable objects, support mechanisms and / or loads relative to a fixed reference and / or each other. In some embodiments, the terminal is used to control a movable object, a support mechanism and / or a plurality of individual elements of the load, such as a drive part of the support mechanism, a load sensor or a load emitter. May be. The terminal may include a wireless communication device configured to communicate with one or more of the movable object, the support mechanism, or the load.

  The terminal may include a display unit suitable for viewing information on movable objects, support mechanisms and / or loads. For example, the terminal may be configured to display movable object, support mechanism, and / or load information regarding position, translation velocity, translation acceleration, direction, angular velocity, angular acceleration, or any suitable combination thereof. . In some embodiments, the terminal displays information provided by the load, eg, data provided by a functional load (eg, multiple images recorded by a camera or other image recording device). May be.

  Optionally, the same terminal receives and / or displays information from the movable object, support mechanism and / or load, and the movable object, support mechanism and / or load, or movable object, support mechanism and / or load. You may control both the state of an object. For example, the terminal may control the positioning of the load with respect to the environment while displaying the image data recorded by the load or information on the position of the load. Alternatively, different terminals may be used for different functions. For example, the first terminal may control the movement or state of the movable object, support mechanism and / or load, and the second terminal may receive information from the movable object, support mechanism and / or load, It may be received and / or displayed. For example, the first terminal may be used to control the positioning of the load with respect to the environment, and the second terminal displays the image data recorded by the load. Various multiples between a movable object and an integrated terminal that performs both control of the movable object and data reception, or between multiple terminals that perform both control of the movable object and data reception. May be used. For example, at least two different communication modes may be formed between a movable object and a terminal that performs both control of the movable object and data reception from the movable object.

  FIG. 10 shows a movable object 1000 including a support mechanism 1002 and a load 1004 according to a plurality of embodiments. Although the movable object 1000 is shown as an aircraft, this display is not intended to be limiting and any suitable type of movable object may be used, as previously described herein. One skilled in the art will appreciate that any of the embodiments described herein with respect to multiple aircraft systems can be applied to any suitable movable object (eg, UAV). In some examples, the load 1004 may be provided to the movable object 1000 without the need for a support mechanism 1002. The movable object 1000 may include a plurality of propulsion mechanisms 1006, a detection system 1008, and a communication system 1010.

  The plurality of propulsion mechanisms 1006 may include one or more of a plurality of rotor blades, propellers, blades, engines, motors, wheels, axles, magnets or nozzles, as described above. A movable object may have one or more, two or more, three or more, or four or more propulsion mechanisms. The plurality of propulsion mechanisms may all be the same type. Alternatively, the one or more propulsion mechanisms may be a plurality of different types of propulsion mechanisms. The plurality of propulsion mechanisms 1006 may be mounted on the movable object 1000 using any suitable means, such as a support element (eg, a drive shaft), as described elsewhere herein. . The plurality of propulsion mechanisms 1006 may be mounted on any suitable portion of the movable object 1000, such as a top surface, a bottom surface, a front surface, a back surface, a side surface, or any suitable combination thereof.

  In some embodiments, the plurality of propulsion mechanisms 1006 allows the movable object 1000 to take off vertically from the surface without requiring any horizontal movement of the movable object 1000 (eg, without going down the runway), Or it may be possible to land perpendicular to the surface. Optionally, the plurality of propulsion mechanisms 1006 may be operable to allow the movable object 1000 to hover in the air at a particular position and / or orientation. One or more of the plurality of propulsion mechanisms 1000 may be controlled independently of other propulsion mechanisms. Alternatively, the plurality of propulsion mechanisms 1000 may be configured to be controlled simultaneously. For example, the movable object 1000 may have a plurality of horizontally oriented rotor blades that can provide lift and / or thrust to the movable object. A plurality of horizontally oriented rotors may be driven to provide the movable object 1000 with vertical takeoff, vertical landing and hover performance. In some embodiments, one or more of the plurality of horizontally oriented rotors may rotate in a clockwise direction and one or more of the plurality of horizontal rotors is , It may rotate counterclockwise. For example, the number of clockwise rotor blades may be equal to the number of counterclockwise rotor blades. The rotational speed of each of the plurality of rotor blades oriented in the horizontal direction may be independently varied to control the lift and / or thrust generated by each rotor so that the movable object 1000 Adjust spatial placement, speed and / or acceleration (eg, for up to 3 degrees of translation and up to 3 degrees of rotation).

  The sensing system 1008 may include one or more sensors that can sense the spatial arrangement, velocity, and / or acceleration of the movable object 1000 (eg, for a maximum of 3 degrees of translation and a maximum of 3 degrees of rotation). The one or more sensors may include multiple global positioning system (GPS) sensors, multiple motion sensors, multiple inertial sensors, multiple proximity sensors, or multiple image sensors. Sensing data provided by sensing system 1008 is used to control the spatial placement, speed and / or direction of movable object 1000 (eg, using an appropriate processing unit and / or control module, as described below). May be used. Alternatively, the sensing system 1008 may provide data about the surrounding environment of the movable object, eg, multiple weather conditions, proximity to potential obstacles, multiple geographic feature locations, multiple artificial structure locations. May be used to provide etc.

  The communication system 1010 enables communication with a terminal 1012 having a communication system 1014 via a plurality of radio signals 1016. The communication systems 1010, 1014 may include any number of transmitters, receivers, and / or transceivers suitable for wireless communication. The communication may be one-way communication so that data can be transmitted in only one direction. For example, one-way communication may involve only the movable object 1000 that transmits data to the terminal 1012 and vice versa. Data may be transmitted from one or more transmitters of communication system 1010 to one or more receivers of communication system 1012 and vice versa. Alternatively, the communication may be two-way communication so that data can be transmitted in both directions between the movable object 1000 and the terminal 1012. Two-way communication may involve data transmission from one or more transmitters of the communication system 1010 to one or more receivers of the communication system 1014, and vice versa.

  In some embodiments, the terminal 1012 provides control data to one or more of the movable object 1000, the support mechanism 1002 and the load 1004, and from one or more of the movable object 1000, the support mechanism 1002 and the load 1004. Information (eg, data detected by the load, such as movable object, support mechanism or load position and / or motion information, image data recorded by the load camera) may be received. In some examples, control data from the terminal may include multiple commands for relative positions, multiple movements, multiple drives or multiple controls of the movable object, support mechanism and / or load. . For example, the control data may be a change in the position and / or orientation of the movable object (eg, via control of a plurality of propulsion mechanisms 1006) or movement of the load relative to the movable object (eg, via control of the support mechanism 1002). May bring about. Control data from the terminal can be used to control the load, for example, control of the operation of a camera or other image recording device (e.g., taking still images or videos, zooming in or out, power on or off, switching between multiple imaging modes) Change of image resolution, change of focus, change of depth of field, change of exposure time, change of viewing angle or field of view). In some examples, multiple communications from movable objects, support mechanisms, and / or loads may include information from one or more sensors (eg, sensing system 1008 or load 1004). The plurality of communications may include information detected from one or more different types of sensors (eg, multiple GPS sensors, multiple motion sensors, inertial sensors, multiple proximity sensors, or multiple image sensors). Good. Such information may relate to the position (eg, position, direction), movement or acceleration of the movable object, support mechanism and / or load. Such information from the load may include data recorded by the load or a sensed state of the load. The control data transmitted and provided by the terminal 1012 may be configured to control one or more states of the movable object 1000, the support mechanism 1002, or the load 1004. Alternatively, or in combination, the support mechanism 1002 and the load 1004 may each communicate with and control each of the movable object 1000, the support mechanism 1002, and the load 1004 independently. A communication module configured to communicate with the terminal 1012 may be further included.

  In some embodiments, the movable object 1000 may be configured to communicate with other remote devices in addition to or in place of the terminal 1012. The terminal 1012 may be configured to communicate not only with the movable object 1000 but also with other remote devices. For example, the movable object 1000 and / or the terminal 1012 may communicate with other movable objects or other movable object support mechanisms or loads. If desired, the remote device may be a second terminal or other computing device (eg, a computer, laptop, tablet, smartphone or other mobile device). The remote device may be configured to send data to the movable object 1000, receive data from the movable object 1000, send data to the terminal 1012, and / or receive data from the terminal 1012. Optionally, the remote device may be connected to the Internet or other communication network so that data received from movable object 1000 and / or terminal 1012 can be uploaded to a website or server.

  FIG. 11 is a schematic block diagram of a system 1100 according to embodiments for controlling a movable object. System 1100 may be used in combination with any suitable embodiment of systems, devices, and methods disclosed herein. The system 1100 may include a detection module 1102, a processing unit 1104, a non-transitory computer readable medium 1106, a control module 1108 and a communication module 1110.

  The detection module 1102 may use a plurality of different types of sensors that collect information about a plurality of movable objects in a plurality of different ways. The plurality of different types of sensors may detect a plurality of different types of signals or a plurality of signals from different sources. For example, the plurality of sensors may include a plurality of inertial sensors, a plurality of GPS sensors, a plurality of proximity sensors (eg, a lidar), or a plurality of vision / image sensors (eg, a camera). The detection module 1102 may be operatively connected to a processing unit 1104 having a plurality of processors. In some embodiments, the detection module may be operatively connected to a transmission module 1112 (eg, a Wi-Fi image transmission module) configured to transmit detection data directly to a suitable external device or system. Good. For example, the transmission module 1112 may be used to transmit a plurality of images recorded by the camera of the detection module 1102 to a remote terminal.

  The processing unit 1104 may include one or more processors, such as a programmable processor (eg, a central processing unit (CPU)). Processing unit 1104 may be operatively connected to non-transitory computer readable media 1106. Non-transitory computer readable media 1106 may store logic, code for executing one or more steps, and / or a plurality of program instructions executable by processing unit 1104. Non-transitory computer readable media may include one or more memory units (eg, removable media such as SD card or random access memory (RAM) or external storage). In some embodiments, data from the sensing module 1102 may be transmitted directly to and stored within a plurality of memory units of the non-transitory computer readable medium 1106. Multiple memory units of non-transitory computer readable medium 1106 may be executed by logic, code and / or processing unit 1104 to perform any suitable embodiment of the methods described herein. A plurality of program instructions may be stored. For example, the processing unit 1104 may be configured to execute a plurality of instructions that cause one or more processors of the processing unit 1104 to analyze detection data generated by the detection module. The plurality of memory units may store detection data from the detection module to be processed by the processing unit 1104. In some embodiments, multiple memory units of non-transitory computer readable media 1106 may be used to store multiple processing results generated by processing unit 1104.

  In some embodiments, the processing unit 1104 may be operatively connected to a control module 1108 that is configured to control the state of the movable object. For example, the control module 1108 may be configured to control a plurality of propulsion mechanisms of the movable object to adjust the spatial arrangement, velocity, and / or acceleration of the movable object for six degrees of freedom. Alternatively or in combination, the control module 1108 may control one or more of the states of the support mechanism, load or detection module.

  The processing unit 1104 is operatively connected to a communication module 1110 that is configured to send and / or receive data from one or more external devices (eg, a terminal, display device or other remote controller). Also good. Any suitable communication means such as wired communication or wireless communication may be used. For example, the communication module 1110 uses one or more of local area network (LAN), wide area network (WAN), infrared, wireless, WiFi, point-to-point (P2P) network, communication network, cloud communication, and the like. Also good. Optionally, multiple relay stations such as towers, satellites or mobile stations may be used. Multiple wireless communications may depend on proximity or may not depend on proximity. In some embodiments, visibility may be required or not required for multiple communications. The communication module 1110 receives one or more of detection data from the detection module 1102, a plurality of processing results generated by the processing unit 1104, predetermined control data, and a plurality of user commands from a terminal or a remote controller. May be transmitted and / or received.

  The multiple components of system 1100 may be configured in any suitable configuration. For example, one or more of the components of the system 1100 may be located on a movable object, a support mechanism, a load, a terminal, a sensing system, or an additional external device that communicates with one or more of the above. Good. Further, although FIG. 11 shows a single processing unit 1104 and a single non-transitory computer readable medium 1106, those skilled in the art are not intended to limit this, and the system 1100 may include multiple processing units. It will be appreciated that and / or may include non-transitory computer readable media. In some embodiments, any suitable aspect of the processing performed by the system 1100 and / or the plurality of memory functions may be performed at a plurality of different locations, eg, movable objects, support mechanisms, loads, terminals, sensing modules, One or more processing units and / or non-transitory computer-readable media as performed by one or more of the additional external devices communicating with one or more of the above, or any suitable combination thereof May be located at a plurality of positions as described above.

While several preferred embodiments of the present invention have been illustrated and described herein, it will be apparent to those skilled in the art that such embodiments are provided for illustrative purposes only. Here, many changes, multiple changes, and multiple substitutions will occur to those skilled in the art without departing from the present invention. It should be understood that various alternatives to the embodiments of the invention described herein can be used to practice the invention. The following claims are intended to define the scope of the invention, and thereby cover multiple methods and structures within the scope of these claims and their equivalents. Intended.
[Item 1]
A calibration method for an inline sensor,
Obtaining sensor data during flight of the UAV from a plurality of different types of sensors connected to an unmanned aerial vehicle (UAV) having an initial spatial configuration relative to each other;
Detecting a change in the spatial configuration from the initial spatial configuration to the next spatial configuration with respect to each other of the plurality of sensors based on the sensor data via a processor;
Determining the next spatial configuration;
Adjusting data from at least one of the plurality of sensors during flight of the UAV based on the next spatial configuration.
[Item 2]
The method of item 1, wherein the plurality of different types of sensors includes at least two of a vision sensor, a GPS sensor, an inertial sensor, an infrared sensor, an ultrasonic sensor, or a lidar sensor.
[Item 3]
Item 3. The method according to Item 2, wherein at least one of the plurality of sensors is an inertial sensor.
[Item 4]
4. The method of item 3, wherein the sensor data from the plurality of sensors is provided in a coordinate system associated with the inertial sensor.
[Item 5]
The method of item 1, wherein the initial spatial configuration comprises an initial position and an initial direction relative to each other for each of the plurality of sensors.
[Item 6]
The change in the spatial configuration includes an change in at least one of the initial position and the initial direction with respect to at least one of the plurality of sensors and other sensors of the plurality of sensors. 5. The method according to 5.
[Item 7]
The method of item 1, wherein the initial spatial configuration is provided prior to take-off of the UAV.
[Item 8]
Item 2. The method of item 1, wherein the initial spatial configuration is provided during flight of the UAV.
[Item 9]
The method according to item 1, wherein the change in the spatial configuration is caused by the movement of the UAV.
[Item 10]
Item 10. The method according to Item 9, wherein the change in the spatial configuration is caused by the vibration of the UAV.
[Item 11]
In item 1, further comprising determining the next spatial configuration using a plurality of Kalman filters via the processor, wherein the plurality of Kalman filters includes one or more extended Kalman filters and / or unscented Kalman filters. The method described.
[Item 12]
The method according to item 1, wherein the processor is mounted on the UAV.
[Item 13]
Item 12. The method of item 11, wherein the processor is in an external device of the UAV and communicates with the UAV during the flight of the UAV.
[Item 14]
An in-line sensor calibration device,
(I) a plurality of different types of sensors connected to an unmanned aerial vehicle (UAV) and (ii) configured to provide sensor data during flight of the UAV and having an initial spatial configuration relative to each other;
(B) individually or collectively, (i) detecting a change in the spatial configuration of the plurality of sensors from the initial spatial configuration relative to each other to the next spatial configuration based on the sensor data; (ii) (Iii) configured to adjust data from at least one of the plurality of sensors based on the next spatial configuration during flight of the UAV during the UAV flight; An apparatus comprising one or more processors.
[Item 15]
15. The apparatus of item 14, wherein the plurality of different types of sensors comprises at least two of a vision sensor, a GPS sensor, an inertial sensor, an infrared sensor, an ultrasonic sensor, or a lidar sensor.
[Item 16]
Item 16. The apparatus according to Item 15, wherein at least one of the plurality of sensors is an inertial sensor.
[Item 17]
Item 17. The apparatus of item 16, wherein the sensor data from the plurality of sensors is provided in a coordinate system associated with the inertial sensor.
[Item 18]
Item 15. The apparatus of item 14, wherein the initial spatial configuration comprises an initial position and an initial direction relative to each other for each of the plurality of sensors.
[Item 19]
The change in the spatial configuration comprises a change in at least one of the initial position and the initial direction with respect to at least one of the plurality of sensors and other sensors of the plurality of sensors. The device according to claim 18.
[Item 20]
Item 15. The apparatus of item 14, wherein the initial spatial configuration is provided prior to take-off of the UAV.
[Item 21]
Item 15. The apparatus of item 14, wherein the initial spatial configuration is provided during flight of the UAV.
[Item 22]
Item 15. The apparatus of item 14, wherein the change in the spatial configuration is caused by the movement of the UAV.
[Item 23]
Item 23. The apparatus according to Item 22, wherein the change in the spatial configuration is caused by vibration of the UAV.
[Item 24]
15. The item 14 wherein the one or more processors determine the next spatial configuration using a plurality of Kalman filters, the plurality of Kalman filters comprising one or more extended Kalman filters and / or unscented Kalman filters. Equipment.
[Item 25]
Item 15. The apparatus of item 14, wherein the one or more processors are mounted on the UAV.
[Item 26]
Item 15. The apparatus of item 14, wherein the one or more processors are in an external device of the UAV and communicate with the UAV during the flight of the UAV.
[Item 27]
A sensor calibration method comprising:
Obtaining sensor data from a plurality of different types of sensors coupled with an unmanned aerial vehicle (UAV);
Selecting a reference coordinate system;
Representing the sensor data from the plurality of sensors in the reference coordinate system based on a predicted spatial relationship between the plurality of sensors via a processor;
Detecting the sensor data mismatch between the plurality of sensors via the processor, wherein the mismatch indicates an error in the predicted spatial relationship between the plurality of sensors; and
Determining the actual spatial configuration;
Adjusting data from at least one of the plurality of sensors based on the actual spatial configuration.
[Item 28]
28. The method of item 27, wherein the plurality of different types of sensors comprises at least two of vision sensors, GPS sensors, inertial sensors, infrared sensors, ultrasonic sensors, or lidar sensors.
[Item 29]
29. A method according to item 28, wherein at least one of the plurality of sensors is an inertial sensor.
[Item 30]
30. A method according to item 29, wherein the reference coordinate system is a coordinate system associated with the inertial sensor.
[Item 31]
The predicted spatial relationship between the plurality of sensors is predicted in the configuration from an initial spatial configuration of the plurality of sensors having an initial position and an initial direction relative to each other for each of the plurality of sensors. 28. A method according to item 27, based on a spatial change.
[Item 32]
32. The method of item 31, wherein the initial spatial configuration is provided prior to the UAV taking off.
[Item 33]
32. The method of item 31, wherein the initial spatial configuration is provided during flight of the UAV.
[Item 34]
28. The method of item 27, wherein the predicted spatial relationship between the plurality of sensors comprises a relative spatial position and a relative spatial direction of the plurality of sensors relative to each other.
[Item 35]
Item 27. further comprising determining the next spatial configuration using a plurality of Kalman filters via the processor, the plurality of Kalman filters comprising one or more extended Kalman filters and / or unscented Kalman filters. The method described.
[Item 36]
28. The method according to item 27, wherein the processor is installed in the UAV.
[Item 37]
28. The method of item 27, wherein the plurality of processors are in an external device of the UAV and communicate with the UAV during the flight of the UAV.
[Item 38]
28. The method of item 27, wherein the actual spatial configuration of the plurality of sensors is determined during operation of the UAV.
[Item 39]
40. The method of item 38, wherein the actual spatial configuration of the plurality of sensors is determined during flight of the UAV.
[Item 40]
28. The method of item 27, wherein the data from the plurality of sensors is adjusted during operation of the UAV.
[Item 41]
28. The method according to item 27, wherein the data adjusted based on the actual space configuration is image data recorded by a vision sensor.
[Item 42]
A sensor calibration device,
(A) (i) a plurality of sensors of different types coupled to an unmanned aerial vehicle (UAV) and (ii) configured to provide sensor data;
(B) individually or collectively,
(I) Select a reference coordinate system,
(Ii) representing the sensor data from the plurality of sensors in the reference coordinate system based on a predicted spatial relationship between the plurality of sensors;
(Iii) detecting a mismatch of the sensor data between the plurality of sensors indicating an error in the predicted spatial relationship between the plurality of sensors;
(Iv) determine the actual spatial configuration;
(V) an apparatus comprising one or more processors configured to adjust data from at least one of the plurality of sensors based on the actual spatial configuration.
[Item 43]
45. The apparatus of item 42, wherein the plurality of different types of sensors comprises at least two of a vision sensor, a GPS sensor, an inertial sensor, an infrared sensor, an ultrasonic sensor, or a lidar sensor.
[Item 44]
44. The apparatus according to item 43, wherein at least one of the plurality of sensors is an inertial sensor.
[Item 45]
45. An apparatus according to item 44, wherein the reference coordinate system is a coordinate system associated with the inertial sensor.
[Item 46]
The predicted spatial relationship between the plurality of sensors is predicted in the configuration from an initial spatial configuration of the plurality of sensors having an initial position and an initial direction relative to each other for each of the plurality of sensors. 45. Apparatus according to item 42, based on a spatial change.
[Item 47]
47. Apparatus according to item 46, wherein the initial spatial configuration is provided prior to take-off of the UAV.
[Item 48]
47. Apparatus according to item 46, wherein the initial spatial configuration is provided during flight of the UAV.
[Item 49]
43. The apparatus of item 42, wherein the predicted spatial relationship between the plurality of sensors comprises a relative spatial position and a relative spatial direction of the plurality of sensors relative to each other.
[Item 50]
45. The item 42, wherein the one or more processors determine the next spatial configuration using a plurality of Kalman filters, the plurality of Kalman filters comprising one or more extended Kalman filters and / or unscented Kalman filters. Equipment.
[Item 51]
Item 43. The apparatus according to Item 42, wherein the one or more processors are mounted on the UAV.
[Item 52]
43. The apparatus of item 42, wherein the one or more processors are in an external device of the UAV and communicate with the UAV during the flight of the UAV.
[Item 53]
43. The apparatus of item 42, wherein the actual spatial configuration of the plurality of sensors is determined during operation of the UAV.
[Item 54]
54. The apparatus of item 53, wherein the actual spatial configuration of the plurality of sensors is determined during flight of the UAV.
[Item 55]
45. The apparatus of item 42, wherein the data from the at least one of the plurality of sensors is adjusted during operation of the UAV.
[Item 56]
Item 43. The apparatus according to Item 42, wherein the data adjusted based on the actual space configuration is image data recorded by a vision sensor.
[Item 57]
A sensor calibration method comprising:
Obtaining sensor data from a plurality of different types of sensors coupled with an unmanned aerial vehicle (UAV);
Grouping the plurality of sensors into a plurality of subsets via a processor, each subset having (i) at least two sensors and (ii) different combinations of the plurality of sensors;
Calculating a predicted spatial relationship between the at least two sensors in each subset based on the sensor data via the processor;
Determining an actual spatial relationship between the at least two sensors in each subset based on the predicted spatial relationship;
Calculating the spatial configuration of each of the plurality of sensors in each subset with respect to each other based on the actual spatial relationship between the at least two sensors via the processor.
[Item 58]
58. The method of item 57, wherein each subset comprises a reference sensor and one or more measurement sensors.
[Item 59]
59. The method of item 58, wherein the reference sensor for each subset is the same.
[Item 60]
59. The method of item 58, wherein the reference sensor for each subset is different.
[Item 61]
58. The method of item 57, wherein the plurality of different types of sensors comprises at least two of a vision sensor, a GPS sensor, an inertial sensor, an infrared sensor, an ultrasonic sensor, or a lidar sensor.
[Item 62]
62. A method according to item 61, wherein at least one of the plurality of sensors is an inertial sensor.
[Item 63]
The predicted spatial relationship between the plurality of sensors in each subset is determined from the initial spatial configuration of the plurality of sensors with an initial position and initial direction relative to each other for each of the plurality of sensors. 59. The method of item 58, based on a predicted spatial change in.
[Item 64]
64. The method of item 63, wherein the initial spatial configuration is provided prior to take-off of the UAV.
[Item 65]
64. The method of item 63, wherein the initial spatial configuration is provided during flight of the UAV.
[Item 66]
59. The method of item 58, wherein the predicted spatial relationship between the plurality of sensors comprises a relative spatial position and a relative spatial direction of the plurality of sensors relative to each other.
[Item 67]
The processor determines an actual spatial relationship between the at least two sensors in each subset using a plurality of Kalman filters, the plurality of Kalman filters comprising one or more extended Kalman filters and / or unscented Kalman filters. 59. A method according to item 58.
[Item 68]
59. A method according to item 58, wherein the processor is installed in the UAV.
[Item 69]
59. The method of item 58, wherein the processor is in an external device of the UAV and communicates with the UAV during the flight of the UAV.
[Item 70]
59. The method of item 58, wherein the actual spatial configuration of the plurality of sensors is determined during operation of the UAV.
[Item 71]
59. The method of item 58, wherein the actual spatial configuration of the plurality of sensors is determined during flight of the UAV.
[Item 72]
59. The method of item 58, wherein the data from the plurality of sensors is adjusted during operation of the UAV.
[Item 73]
Adjusting the data from at least one of the plurality of sensors based on the spatial configuration, and via the processor, based on the actual spatial relationship between the at least two sensors, in each subset; 58. The method of item 57, further comprising calculating a spatial configuration of the plurality of sensors relative to each other.
[Item 74]
74. A method according to item 73, wherein the data adjusted based on the actual spatial configuration is image data recorded by a vision sensor.
[Item 75]
A sensor calibration device,
(A) (i) a plurality of sensors of different types coupled to an unmanned aerial vehicle (UAV) and (ii) configured to provide sensor data;
(B) individually or collectively,
(I) grouping the plurality of sensors into a plurality of subsets, each subset having (i) at least two sensors and (ii) different combinations of the plurality of sensors;
(Ii) calculating a predicted spatial relationship between the at least two sensors in each subset based on the sensor data;
(Iii) determining an actual spatial relationship between the at least two sensors in each subset based on the predicted spatial relationship;
(Iv) one or more processors configured to calculate a spatial configuration of the plurality of sensors in each subset relative to each other based on the actual spatial relationship between the at least two sensors. apparatus.
[Item 76]
76. Apparatus according to item 75, wherein each subset comprises a reference sensor and one or more measurement sensors.
[Item 77]
77. Apparatus according to item 76, wherein the reference sensor for each subset is the same.
[Item 78]
79. Apparatus according to item 76, wherein the reference sensor for each subset is different.
[Item 79]
The apparatus of item 75, wherein the plurality of different types of sensors comprises at least two of a vision sensor, a GPS sensor, an inertial sensor, an infrared sensor, an ultrasonic sensor, or a lidar sensor.
[Item 80]
80. Apparatus according to item 79, wherein at least one of the plurality of sensors is an inertial sensor.
[Item 81]
The predicted spatial relationship between the plurality of sensors in each subset is determined from the initial spatial configuration of the plurality of sensors with an initial position and initial direction relative to each other for each of the plurality of sensors. 76. Apparatus according to item 75, based on a predicted spatial change in
[Item 82]
82. The apparatus of item 81, wherein the initial spatial configuration is provided prior to the UAV taking off.
[Item 83]
82. Apparatus according to item 81, wherein the initial spatial configuration is provided during flight of the UAV.
[Item 84]
The apparatus of item 75, wherein the predicted spatial relationship between the plurality of sensors comprises a relative spatial position and a relative spatial direction of the plurality of sensors relative to each other.
[Item 85]
The one or more processors determine an actual spatial relationship between the at least two sensors in each subset using a plurality of Kalman filters, the plurality of Kalman filters comprising one or more extended Kalman filters and / or 76. Apparatus according to item 75, comprising an unscented Kalman filter.
[Item 86]
76. The apparatus of item 75, wherein the one or more processors are mounted on the UAV.
[Item 87]
76. The apparatus of item 75, wherein the one or more processors are in an external device of the UAV and communicate with the UAV during the flight of the UAV.
[Item 88]
The apparatus of item 75, wherein the actual spatial configuration of the plurality of sensors is determined during operation of the UAV.
[Item 89]
The apparatus of item 75, wherein the actual spatial configuration of the plurality of sensors is determined during flight of the UAV.
[Item 90]
76. The apparatus of item 75, wherein the data from the at least one of the plurality of sensors is adjusted during operation of the UAV.
[Item 91]
A sensor calibration method comprising:
Acquiring sensor data from a plurality of different types of sensors;
Grouping the plurality of sensors into a plurality of subsets via a processor, each subset comprising: (i) at least two sensors comprising a reference sensor and one or more measurement sensors; and (ii) ) Having different combinations of sensors, wherein at least two of the plurality of subsets have a plurality of different reference sensors;
Determining, via the processor, an actual spatial relationship between the at least two sensors in each subset using at least one Kalman filter per subset based on the sensor data;
Calculating the spatial configuration of each of the plurality of sensors in each subset with respect to each other based on the actual spatial relationship between the at least two sensors via the processor.
[Item 92]
92. The method of item 91, wherein the plurality of sensors are coupled to an unmanned aerial vehicle (UAV).
[Item 93]
93. A method according to item 92, wherein the processor is installed in the UAV.
[Item 94]
93. The method of item 92, wherein the processor is in an external device of the UAV and communicates with the UAV during the flight of the UAV.
[Item 95]
93. The method of item 92, wherein the actual spatial configuration of the plurality of sensors is determined during operation of the UAV.
[Item 96]
93. The method of item 92, wherein the actual spatial configuration of the plurality of sensors is determined during flight of the UAV.
[Item 97]
93. The method of item 92, wherein the data from the plurality of sensors is adjusted during operation of the UAV.
[Item 98]
92. The method of item 91, wherein the plurality of different types of sensors comprises at least two of a vision sensor, a GPS sensor, an inertial sensor, an infrared sensor, an ultrasonic sensor, or a lidar sensor.
[Item 99]
99. A method according to item 98, wherein at least one of the plurality of sensors is an inertial sensor.
[Item 100]
99. A method according to item 98, wherein at least one of the plurality of sensors is a vision sensor.
[Item 101]
92. The method of item 91, wherein the plurality of Kalman filters comprises one or more extended Kalman filters and / or unscented Kalman filters.
[Item 102]
After grouping the plurality of sensors into a plurality of subsets, the method further includes calculating a predicted spatial relationship between the at least two sensors in each subset based on the sensor data, wherein the Kalman filter includes the prediction 92. A method according to item 91, wherein an actual spatial relationship is determined based on the spatial relationship to be performed.
[Item 103]
92. The method of item 91, wherein the predicted spatial relationship between the plurality of sensors comprises a relative spatial position and a relative spatial direction of the plurality of sensors relative to each other.
[Item 104]
A sensor calibration device,
(A) a plurality of different types of sensors configured to provide sensor data;
(B) individually or collectively,
(1) the plurality of sensors as a plurality of subsets, each subset comprising (i) at least two sensors comprising a reference sensor and one or more measurement sensors; and (ii) different of the plurality of sensors. Grouping into a plurality of subsets having a combination, wherein at least two of the plurality of subsets have a plurality of different reference sensors;
(2) determining an actual spatial relationship between the at least two sensors in each subset using at least one Kalman filter per subset based on the sensor data;
(3) one or more processors configured to calculate a spatial configuration relative to each other of the plurality of sensors in each subset based on the actual spatial relationship between the at least two sensors. ,apparatus.
[Item 105]
105. The apparatus of item 104, wherein the plurality of sensors are coupled to an unmanned aerial vehicle (UAV).
[Item 106]
106. The apparatus according to item 105, wherein the one or more processors are mounted on the UAV.
[Item 107]
106. The apparatus of item 105, wherein the one or more processors are in an external device of the UAV and communicate with the UAV during the flight of the UAV.
[Item 108]
106. The apparatus according to item 105, wherein the actual spatial configuration of the plurality of sensors is determined during operation of the UAV.
[Item 109]
106. The apparatus of item 105, wherein the actual spatial configuration of the plurality of sensors is determined during flight of the UAV.
[Item 110]
106. The apparatus of item 105, wherein the data from the at least one of the plurality of sensors is adjusted during operation of the UAV.
[Item 111]
105. The apparatus of item 104, wherein the plurality of different types of sensors comprises at least two of a vision sensor, a GPS sensor, an inertial sensor, an infrared sensor, an ultrasonic sensor, or a lidar sensor.
[Item 112]
119. The device according to Item 111, wherein at least one of the plurality of sensors is an inertial sensor.
[Item 113]
120. An apparatus according to item 111, wherein at least one of the plurality of sensors is a vision sensor.
[Item 114]
105. The apparatus of item 104, wherein the plurality of Kalman filters comprises one or more extended Kalman filters and / or unscented Kalman filters.

Claims (26)

A sensor calibration method comprising:
Obtaining sensor data from a plurality of different types of sensors connected to an unmanned aerial vehicle (UAV);
Grouping the plurality of sensors into a plurality of subsets via a processor, each subset comprising (i) at least two sensors and (ii) different combinations of the plurality of sensors;
Calculating a predicted spatial relationship between the at least two sensors in each subset based on the sensor data via the processor;
Determining an actual spatial relationship between the at least two sensors in each subset based on the predicted spatial relationship;
Calculating a spatial configuration of the plurality of sensors in each subset relative to each other based on the actual spatial relationship between the at least two sensors.   The method of claim 1, wherein each subset comprises a reference sensor and one or more measurement sensors.   The method according to claim 1 or 2, wherein the plurality of different types of sensors comprises at least two of a vision sensor, a GPS sensor, an inertial sensor, an infrared sensor, an ultrasonic sensor, or a lidar sensor.   The method of claim 3, wherein at least one of the plurality of sensors is an inertial sensor.   The predicted spatial relationship is based on a predicted spatial change in the configuration from an initial spatial configuration of the plurality of sensors with an initial position and initial direction relative to each other for each of the plurality of sensors. 5. A method according to any one of claims 1 to 4.   The method of claim 5, wherein the initial spatial configuration is set during take-off or flight of the UAV.   The method of any one of claims 1 to 6, wherein the predicted spatial relationship comprises a relative spatial position and a relative spatial orientation of the plurality of sensors relative to each other.   The processor determines an actual spatial relationship between the at least two sensors in each subset using a plurality of Kalman filters, the plurality of Kalman filters comprising one or more extended Kalman filters and / or unscented Kalman filters. 8. A method according to any one of claims 1 to 7.   The method according to claim 1, wherein the processor is installed in the UAV.   The method according to any one of claims 1 to 9, wherein the processor is in an external device of the UAV and communicates with the UAV during the flight of the UAV.   11. A method according to any one of the preceding claims, wherein the actual spatial configuration of the plurality of sensors is determined during operation of the UAV.   12. A method according to any one of the preceding claims, wherein the data from the plurality of sensors is adjusted during operation of the UAV.   Adjusting data from at least one of the plurality of sensors based on the spatial configuration, and via the processor, based on the actual spatial relationship between the at least two sensors, in each subset The method of any one of claims 1 to 12, further comprising calculating a spatial configuration of the plurality of sensors relative to each other.   The method of claim 13, wherein the data adjusted based on the actual spatial configuration is image data recorded by a vision sensor. A sensor calibration device,
(A) (i) a plurality of sensors of different types coupled to an unmanned aerial vehicle (UAV) and (ii) configured to provide sensor data;
(B)
(I) grouping the plurality of sensors into a plurality of subsets, each subset having (i) at least two sensors and (ii) different combinations of the plurality of sensors;
(Ii) calculating a predicted spatial relationship between the at least two sensors in each subset based on the sensor data;
(Iii) determining an actual spatial relationship between the at least two sensors in each subset based on the predicted spatial relationship;
(Iv) one or more processors configured to calculate a spatial configuration of the plurality of sensors in each subset relative to each other based on the actual spatial relationship between the at least two sensors. apparatus.   The apparatus of claim 15, wherein each subset comprises a reference sensor and one or more measurement sensors.   17. The apparatus according to claim 15 or 16, wherein the plurality of different types of sensors comprises at least two of a vision sensor, a GPS sensor, an inertial sensor, an infrared sensor, an ultrasonic sensor, or a lidar sensor.   The apparatus of claim 17, wherein at least one of the plurality of sensors is an inertial sensor.   The predicted spatial relationship between the plurality of sensors in each subset is determined from the initial spatial configuration of the plurality of sensors comprising an initial position and an initial direction relative to each other for each of the plurality of sensors. 19. An apparatus according to any one of claims 15 to 18 based on a predicted spatial change in.   The apparatus of claim 19, wherein the initial spatial configuration is set during take-off or flight of the UAV.   21. The apparatus according to any one of claims 15 to 20, wherein the predicted spatial relationship between the plurality of sensors comprises a relative spatial position and a relative spatial direction of the plurality of sensors relative to each other.   The one or more processors determine an actual spatial relationship between the at least two sensors in each subset using a plurality of Kalman filters, the plurality of Kalman filters comprising one or more extended Kalman filters and / or 22. Apparatus according to any one of claims 15 to 21 comprising an unscented Kalman filter.   23. The apparatus according to any one of claims 15 to 22, wherein the one or more processors are mounted on the UAV.   24. The apparatus according to any one of claims 15 to 23, wherein the one or more processors are in an external device of the UAV and communicate with the UAV during flight of the UAV.   25. The apparatus according to any one of claims 15 to 24, wherein the actual spatial configuration of the plurality of sensors is determined during operation of the UAV.   26. The apparatus according to any one of claims 15 to 25, wherein the data from the at least one of the plurality of sensors is adjusted during operation of the UAV.