JP6800918B2 - Methods, systems, and programs for performing error recovery - Google Patents

Description

Unmanned aerial vehicles (UAVs) and other unmanned transporters can be used to perform surveillance, reconnaissance, and exploration tasks for a wide variety of civilian, commercial, and military applications. The UAV can be manually controlled by a remote user, or can operate in a semi-autonomous or fully autonomous manner. Such a UAV may include multiple sensors configured to collect data that may be used during the operation of the UAV, such as state information.

Existing approaches to data collection and processing for UAV operations may not be optimal in some cases. For example, some techniques for estimating state information about a UAV may not be accurate enough, which can be detrimental to the functioning of the UAV.

The present disclosure provides multiple systems, methods, and devices for the control and operation of moving objects such as unmanned aerial vehicles (UAVs). In some embodiments, the plurality of systems, methods, and devices described herein are sensor fusions that allow enhanced accuracy and flexibility to determine information related to the operation of the UAV. To use. For example, fusion of one or more inertial sensors and one or more image sensors can be used to improve initialization, error recovery, parameter calibration, and / or state estimation. The information obtained from these procedures can be used to facilitate UAV operations such as autonomous or semi-autonomous navigation, obstacle avoidance, mapping, and more.

In one aspect, a method for determining initialization information about a moving object using a plurality of sensors is provided. The method involves receiving inertial data in one or more processors from at least one inertial sensor mounted by the movable object, with the step of detecting that the moving object has begun to move with the assistance of one or more processors. A step, a step of receiving image data in one or more processors from at least two image sensors mounted by the moving object, and initialization information about the moving object based on inertial data and image data of the moving object. The initialization information includes one or more of the position of the movable object, the speed of the movable object, or the orientation of the movable object with respect to the direction of gravity when the movable object starts to move, with a step of determining with assistance. Including.

In some embodiments, the movable object is an unmanned aerial vehicle (UAV). The step of detecting that the UAV has started operation may have to detect that the UAV has started flying. The UAV may start flying from the free fall state and / or by being thrown by the user. The step of detecting that the UAV has started operation may include a step of detecting that the UAV has taken off from a surface, and the surface can be an inclined surface or a non-inclined surface.

In some embodiments, the inertial data includes one or more measurements that indicate the 3D acceleration and 3D angular velocity of the moving object. Inertia data may include one or more measurements acquired by at least one inertial sensor over a time interval that begins when the moving object begins to move.

In some embodiments, the image data includes one or more images of the environment surrounding the moving object. The image data may include one or more images acquired by each of at least two image sensors over a time interval starting from the time the movable object starts moving. In some embodiments, the determining step comprises processing one or more images using a feature point detection algorithm. The determining step may include processing one or more images using an optical flow algorithm. The determining step may include processing one or more images using a feature matching algorithm. The step of determining is to compare one or more images acquired by the first image sensor of at least two image sensors with one or more images acquired by the second image sensor of at least two image sensors. May have steps.

In some embodiments, at least two image sensors are configured to synchronously acquire a plurality of images. Alternatively, at least two image sensors may be configured to acquire multiple images asynchronously. At least two image sensors include a plurality of image sensor subsets switchably coupled to one or more processors via a switching mechanism, the switching mechanism including a single image sensor subset at one time to one or more processors. It is configured to bind to. The steps of receiving image data include a step of coupling the first image sensor subset to one or more processors via a switching mechanism, a step of receiving image data from the first image sensor subset, and a step of receiving image data via a switching mechanism. It may have a step of combining the second image sensor subset into one or more processors and a step of receiving image data from the second image sensor subset.

In some embodiments, the initialization information further includes the position and velocity of the moving object when it begins to move. The steps to be determined are a step of generating a first estimated value of initialization information using inertial data, a step of generating a second estimated value of initialization information using image data, and a first estimated value and a second estimated value. It may have a step of combining estimates to obtain initialization information about the moving object. Initialization information can be determined using a linear optimization algorithm. Initialization information can be determined based solely on inertial data and image data.

In some embodiments, the method further comprises a step of correcting subsequent inertial data obtained from at least one inertial sensor using the orientation of the moving object with respect to the direction of gravity.

In another aspect, a system is provided for determining initialization information about a moving object using multiple sensors. The system may include at least one inertial sensor mounted by a moving object, at least two image sensors mounted by a moving object, and one or more processors, each of which may be individual or. Collectively, it detects that a moving object has started to move, receives inertial data from at least one inertial sensor, receives image data from at least two image sensors, and is based on the inertial data and the image data. It is configured to determine the initialization information about, which is one or more of the position of the moving object, the speed of the moving object, or the orientation of the moving object with respect to the direction of gravity when the moving object starts to move. Including.

In other embodiments, one or more non-temporary computer-readable storage media are provided. One or more non-temporary computer-readable storage media may store multiple executable instructions, which are executed by one or more processors of the computer system for determining initialization information about the moving object. At least two image sensors mounted by the movable object, causing the computer system to detect that the movable object has started to move, receive inertial data from at least one inertial sensor mounted by the movable object. The image data is received from, and the initialization information about the moving object is determined based on the inertial data and the image data. Or includes one or more of the orientations of the moving object with respect to the direction of gravity.

In other embodiments, methods are provided for performing error recovery on moving objects using multiple sensors. The method involves the step of detecting an error during the operation of a moving object with the assistance of one or more processors, and one or more inertial data from at least one inertial sensor mounted by the moving object. One step of receiving in the processor, one step of receiving image data in one or more processors from at least two image sensors mounted by the movable object, and one reinitialization information about the movable object based on inertial data and image data. Or it may be equipped with a step of determining with the assistance of multiple processors, and the reinitialization information is the position of the moving object, the speed of the moving object, or the orientation of the moving object with respect to the direction of gravity when an error occurs. Includes one or more.

In some embodiments, the movable object is an unmanned aerial vehicle (UAV).

In some embodiments, the error includes a malfunction in one or more of at least one inertial sensor or at least two image sensors. The error may include a malfunction in the state estimation module used by the moving object. The state estimation module can use an iterative state estimation algorithm, and malfunctions can include the iterative state estimation algorithm failing to converge to a solution. The method may further include a step of reinitializing the iterative state estimation algorithm with the reinitialization information.

In some embodiments, the inertial data includes one or more measurements that indicate the 3D acceleration and 3D angular velocity of the moving object. Inertia data may include one or more measurements taken by at least one inertial sensor over a time interval starting from the time the error occurs.

In some embodiments, the image data includes one or more images of the environment surrounding the moving object. The image data may include one or more images acquired by each of at least two image sensors over a time interval starting from the time the error occurs. In some embodiments, the determining step comprises processing one or more images using a feature point detection algorithm. The determining step may include processing one or more images using an optical flow algorithm. The determining step may include processing one or more images using a feature matching algorithm. The step of determining is to compare one or more images acquired by the first image sensor of at least two image sensors with one or more images acquired by the second image sensor of at least two image sensors. May have steps.

In some embodiments, at least two image sensors are configured to synchronously acquire a plurality of images. Alternatively, at least two image sensors may be configured to acquire multiple images asynchronously. At least two image sensors include a plurality of image sensor subsets switchably coupled to one or more processors via a switching mechanism, the switching mechanism including a single image sensor subset at one time to one or more processors. It is configured to bind to. The steps of receiving image data include a step of coupling the first image sensor subset to one or more processors via a switching mechanism, a step of receiving image data from the first image sensor subset, and a step of receiving image data via a switching mechanism. It may have a step of combining the second image sensor subset into one or more processors and a step of receiving image data from the second image sensor subset.

In some embodiments, the reinitialization information further includes the position and velocity of the moving object when the error occurs. The steps to determine are the step of generating the first estimated value of the reinitialization information using the inertial data, the step of generating the second estimated value of the reinitialized information using the image data, the first estimated value, and the step. It may have a step of combining the second estimates to obtain reinitialization information about the moving object. The reinitialization information can be determined using a linear optimization algorithm. The reinitialization information can be determined based solely on inertial data and image data.

In some embodiments, the method further comprises a step of correcting subsequent inertial data obtained from at least one inertial sensor using the orientation of the moving object with respect to the direction of gravity.

In another embodiment, a system is provided for performing error recovery on moving objects using multiple sensors. The system may include at least one inertial sensor mounted by a moving object, at least two image sensors mounted by a moving object, and one or more processors, each of which may be individual or. Collectively, it detects that an error has occurred during the movement of a moving object, receives inertial data from at least one inertial sensor, receives image data from at least two image sensors, and converts it into inertial data and image data. It is configured to determine reinitialization information about a moving object based on, which is one of the position of the moving object at the time of the error, the speed of the moving object, or the orientation of the moving object with respect to the direction of gravity. Including multiple.

In other embodiments, one or more non-temporary computer-readable storage media are provided. One or more non-temporary computer-readable storage media may store multiple executable instructions, which are executed by one or more processors of the computer system to perform error recovery on moving objects. At least when the computer system is made to detect that an error has occurred during the movement of the moving object, to receive inertial data from at least one inertial sensor mounted by the moving object, and at least mounted by the moving object. Image data is received from two image sensors, and reinitialization information about the movable object is determined based on the inertial data and the image data. The reinitialization information is the position of the movable object and the movable object when an error occurs. Includes one or more of the speeds of, or the orientation of the moving object with respect to the direction of gravity.

In another aspect, a method is provided for calibrating one or more external parameters of a moving object using multiple sensors during the movement of the moving object. The method is that one or more external parameters receive multiple initial values for one or more external parameters including multiple spatial relationships between at least two image sensors mounted by the moving object in one or more processors. A step of receiving inertial data in one or more processors from at least one inertial sensor mounted by the movable object during the movement of the movable object, and a step of being mounted by the movable object during the movement of the movable object. One or more based on multiple initial values, inertial data, and image data using an iterative optimization algorithm between the step of receiving image data from at least two image sensors in one or more processors and the movement of a moving object. It may include the step of determining a plurality of estimates for a plurality of external parameters with the assistance of one or more processors.

In some embodiments, the movable object is an unmanned aerial vehicle (UAV). The operation of the UAV may include that the UAV is in flight and / or is powered.

In some embodiments, the spatial relationship comprises relative positions and orientations of at least two image sensors. The relative positions and relative orientations of at least two image sensors can be determined in relation to at least one inertial sensor.

In some embodiments, a plurality of initial values for one or more external parameters are determined prior to the movement of the moving object. Multiple initial values can be determined using an iterative optimization algorithm prior to the movement of the moving object. Multiple initial values may be measured by the user prior to the movement of the moving object. Multiple initial values for one or more external parameters may be received from the memory device associated with the moving object. In some embodiments, at least two image sensors are coupled to a moving object in one or more fixed locations, with multiple initial values for one or more external parameters being one or more fixed locations. It is decided based on. The plurality of initial values for one or more external parameters may be separated by approximately 2 cm or less from the plurality of actual values for the one or more external parameters. The plurality of initial values for one or more external parameters may deviate from the plurality of actual values for one or more external parameters by approximately 2 degrees or less.

In some embodiments, the inertial data includes one or more measurements that indicate the 3D acceleration and 3D angular velocity of the moving object. Inertia data may include one or more measurements acquired by at least one inertial sensor over at least two different time points.

In some embodiments, the image data includes one or more images of the environment surrounding the moving object. The image data may include one or more images acquired by each of at least two image sensors over at least two different time points. In some embodiments, the determining step comprises processing one or more images using a feature point detection algorithm. The determining step may include processing one or more images using an optical flow algorithm. The determining step may include processing one or more images using a feature matching algorithm. The step of determining is to compare one or more images acquired by the first image sensor of at least two image sensors with one or more images acquired by the second image sensor of at least two image sensors. May have steps.

In some embodiments, at least two image sensors are configured to synchronously acquire a plurality of images. Alternatively, at least two image sensors may be configured to acquire multiple images asynchronously. At least two image sensors include a plurality of image sensor subsets switchably coupled to one or more processors via a switching mechanism, the switching mechanism including a single image sensor subset at one time to one or more processors. It is configured to bind to. The steps of receiving image data include a step of coupling the first image sensor subset to one or more processors via a switching mechanism, a step of receiving image data from the first image sensor subset, and a step of receiving image data via a switching mechanism. It may have a step of combining the second image sensor subset into one or more processors and a step of receiving image data from the second image sensor subset.

In some embodiments, the iterative optimization algorithm is a non-linear optimization algorithm. Iterative optimization algorithms may include calculating maximum posteriori probability (MAP) estimates for one or more external parameters based on multiple initial values, inertial data, and image data. In some embodiments, the inertial data and the image data are the only sensor data used to determine multiple estimates for one or more external parameters.

In some embodiments, the method further comprises a step of determining the state of the moving object based on a plurality of estimates for one or more external parameters. The state may include one or more of the position, orientation, or velocity of the moving object. The state of the moving object can be determined in relation to the previous state of the moving object at a previous point in time during the movement of the moving object. The previous time point can be the time point at which initialization or reinitialization occurred.

In another aspect, a system is provided for calibrating one or more external parameters of a moving object using multiple sensors during the movement of the moving object. The system may include at least one inertial sensor mounted by a moving object, at least two image sensors mounted by a moving object, and one or more processors, each of which may be individually or. Collectively, it receives multiple initial values for one or more external parameters, including multiple spatial relationships between at least two image sensors, and receives inertial data from at least one inertial sensor during the movement of a moving object. It receives image data from at least two image sensors during the movement of the moving object and uses iterative optimization algorithms during the movement of the moving object based on multiple initial values, inertial data, and image data. Alternatively, it may be configured to determine multiple estimates for multiple external parameters.

In other embodiments, one or more non-temporary computer-readable storage media are provided. One or more non-temporary computer-readable storage media may store multiple executable instructions, one or more of the computer systems for calibrating one or more external parameters of a moving object. When executed by a processor, the computer system receives at least multiple initial values for one or more external parameters, including multiple spatial relationships between at least two image sensors mounted by the moving object, of the moving object. Inertial data is received from at least one inertial sensor mounted by the moving object during movement, image data is received from at least two image sensors during movement of the moving object, and iteratively optimized during movement of the moving object. A computerization algorithm is used to determine multiple estimates for one or more external parameters based on multiple initial values, inertial data, and image data.

In another aspect, a method is provided for calibrating one or more external parameters of a moving object having multiple sensors with initial configuration. The method is between the step of detecting that the initial configuration of multiple sensors, including at least one inertial sensor and at least two image sensors, has been modified with the assistance of one or more processors, and the movement of the moving object. Initial configuration: receiving inertial data from at least one inertial sensor on one or more processors, and receiving image data from at least two image sensors on one or more processors during the movement of a moving object. Can comprise one or more external parameters being estimated with the assistance of one or more processors based on inertial data and image data in response to detection of the modification. The parameters are estimated using iterative optimization algorithms during the movement of the moving object, and one or more external parameters include multiple spatial relationships between multiple sensors with modified configurations.

In some embodiments, the movable object is an unmanned aerial vehicle (UAV). The operation of the UAV may include that the UAV is in flight and / or is powered.

In some embodiments, the spatial relationship comprises relative positions and orientations of at least two image sensors. The relative positions and relative orientations of at least two image sensors can be determined in relation to at least one inertial sensor.

In some embodiments, a plurality of initial values for one or more external parameters are determined prior to the movement of the moving object. Multiple initial values can be determined using an iterative optimization algorithm prior to the movement of the moving object. Multiple initial values may be measured by the user prior to the movement of the moving object. Multiple initial values for one or more external parameters may be received from the memory device associated with the moving object. In some embodiments, at least two image sensors are coupled to a moving object in one or more fixed locations, with multiple initial values for one or more external parameters being one or more fixed locations. It is decided based on.

In some embodiments, the initial configuration is modified by removing at least one sensor from the plurality of sensors. The initial configuration can be modified by adding at least one sensor to the plurality of sensors. The initial configuration can be modified by changing at least one of the positions or orientations of at least one of the sensors. The initial configuration can be modified prior to the movement of the moving object.

In some embodiments, the inertial data includes one or more measurements that indicate the 3D acceleration and 3D angular velocity of the moving object. Inertia data may include one or more measurements taken by at least one inertial sensor over at least two different time points.

In some embodiments, the image data includes one or more images of the environment surrounding the moving object. The image data may include one or more images acquired by each of at least two image sensors over at least two different time points. In some embodiments, the determining step comprises processing one or more images using a feature point detection algorithm. The determining step may include processing one or more images using an optical flow algorithm. The determining step may include processing one or more images using a feature matching algorithm. The step of determining is to compare one or more images acquired by the first image sensor of at least two image sensors with one or more images acquired by the second image sensor of at least two image sensors. May have steps.

In some embodiments, at least two image sensors are configured to synchronously acquire a plurality of images. Alternatively, at least two image sensors may be configured to acquire multiple images asynchronously. At least two image sensors include a plurality of image sensor subsets switchably coupled to one or more processors via a switching mechanism, the switching mechanism including a single image sensor subset at one time to one or more processors. It is configured to bind to. The steps of receiving image data include a step of coupling the first image sensor subset to one or more processors via a switching mechanism, a step of receiving image data from the first image sensor subset, and a step of receiving image data via a switching mechanism. It may have a step of combining the second image sensor subset into one or more processors and a step of receiving image data from the second image sensor subset.

In some embodiments, the iterative optimization algorithm is a non-linear optimization algorithm. Iterative optimization algorithms may include calculating maximum posteriori probability (MAP) estimates for one or more external parameters based on multiple initial values, inertial data, and image data. In some embodiments, the inertial data and the image data are the only sensor data used to estimate one or more external parameters.

In some embodiments, the method further comprises a step of determining the state of the moving object based on one or more estimated external parameters. The state may include one or more of the position, orientation, or velocity of the moving object.

In another aspect, a system is provided for calibrating one or more external parameters of a moving object having multiple sensors with an initial configuration. The system may be carried by a moving object and comprises a plurality of sensors having at least one inertial sensor and at least two image sensors and one or more processors, the one or more processors individually or collectively. Detects that the initial configuration of multiple sensors has been modified, receives inertial data from at least one inertial sensor during the movement of the moving object, and images from at least two image sensors during the movement of the moving object. It is configured to receive data and estimate one or more external parameters based on inertial data and image data in response to detection that the initial configuration has been modified, with one or more external parameters being movable objects. Estimated using an iterative optimization algorithm during the operation of, one or more external parameters include multiple spatial relationships between multiple sensors with modified configurations.

In other embodiments, one or more non-temporary computer-readable storage media are provided. One or more non-temporary computer-readable storage media may store multiple executable instructions, which calibrate one or more external parameters of a mobile object with multiple sensors with initial configuration. To cause the computer system to detect that the initial configuration of a plurality of sensors, including at least one inertial sensor and at least two image sensors, has been modified when executed by one or more processors of the computer system. Responds to detection that the initial configuration has been modified by receiving inertial data from at least one inertial sensor during the movement of the moving object and image data from at least two image sensors during the movement of the moving object. Then, one or more external parameters are estimated based on inertial data and image data, and one or more external parameters are estimated using an iterative optimization algorithm during the movement of the moving object. External parameters include multiple spatial relationships between multiple sensors with modified configurations.

In another embodiment, a method is provided for estimating state information about a moving object using a plurality of sensors during the movement of the moving object. The method comprises receiving information on the previous state of the moving object on one or more processors and receiving inertial data on one or more processors from at least one inertial sensor mounted on the moving object. Using an iterative optimization algorithm between the step of receiving image data in one or more processors from at least two image sensors mounted by the moving object and the movement of the moving object, previous state information, inertial data, and It may include steps to determine updated state information about the moving object based on the image data with the assistance of one or more processors, and the inertial data is at least two different time points between the movements of the moving object. Including inertial measurement data acquired by at least one inertial sensor over, the image data was acquired by each of the at least two image sensors over at least two different time points during the movement of the moving object. Contains multiple images.

In some embodiments, the movable object is an unmanned aerial vehicle (UAV). The operation of the UAV may include that the UAV is in flight and / or is powered.

In some embodiments, the previous state information includes the position, orientation, and velocity of the movable object at a previous time point during the movement of the movable object. Information on the previous state can be obtained using an iterative optimization algorithm. The updated state information may include the position, velocity, and orientation of the moving object.

In some embodiments, the inertial data includes one or more measurements that indicate the 3D acceleration and 3D angular velocity of the moving object.

In some embodiments, the image data includes one or more images of the environment surrounding the moving object. In some embodiments, the determining step comprises processing one or more images using a feature point detection algorithm. The determining step may include processing one or more images using an optical flow algorithm. The determining step may include processing one or more images using a feature matching algorithm. The step of determining is to compare one or more images acquired by the first image sensor of at least two image sensors with one or more images acquired by the second image sensor of at least two image sensors. May have steps.

In some embodiments, at least two image sensors are configured to synchronously acquire a plurality of images. Alternatively, at least two image sensors may be configured to acquire multiple images asynchronously. At least two image sensors include a plurality of image sensor subsets switchably coupled to one or more processors via a switching mechanism, the switching mechanism including a single image sensor subset at one time to one or more processors. It is configured to bind to. The steps of receiving image data include a step of coupling the first image sensor subset to one or more processors via a switching mechanism, a step of receiving image data from the first image sensor subset, and a step of receiving image data via a switching mechanism. It may have a step of combining the second image sensor subset into one or more processors and a step of receiving image data from the second image sensor subset.

In some embodiments, the iterative optimization algorithm is a non-linear optimization algorithm. The iterative optimization algorithm may include calculating the maximum posteriori probability (MAP) estimate of the updated state information based on multiple initial values, inertial data, and image data. In some embodiments, the inertial data and the image data are the only sensor data used to determine the updated state information.

In some embodiments, the method further comprises a step of outputting updated state information to the control module in order to control the movement of the moving object.

In another aspect, a system is provided for estimating state information about a moving object using a plurality of sensors during the movement of the moving object. The system may include at least one inertial sensor mounted by a moving object, at least two image sensors mounted by a moving object, and one or more processors, each of which may be individually or. Collectively, it receives previous state information about the moving object, receives inertial data from at least one inertial sensor, receives image data from at least two image sensors, and iteratively optimizes during the movement of the moving object. An algorithm is used to determine updated state information about a moving object based on previous state information, inertial data, and image data, and the inertial data is at least two different time points between the movements of the moving object. Includes inertial measurement data acquired by at least one inertial sensor over, and the image data is acquired by each of the at least two image sensors over at least two different time points during the movement of the moving object. Includes multiple images.

In other embodiments, one or more non-temporary computer-readable storage media are provided. One or more non-temporary computer-readable storage media may store multiple executable instructions, which are executed by one or more processors of the computer system for estimating state information about the moving object. At that time, the computer system is made to receive at least information on the previous state regarding the movable object, inertial data is received from at least one inertial sensor mounted by the movable object, and at least two image sensors mounted by the movable object. Receives image data from, and uses an iterative optimization algorithm during the movement of the moving object to determine previous state information, inertial data, and updated state information about the moving object based on the image data, inertial data. Includes inertial measurement data acquired by at least one inertial sensor over at least two different time points during the movement of the moving object, and the image data spans at least two different time points during the movement of the moving object. It includes a plurality of images acquired by each image sensor out of at least two image sensors.

It will be appreciated that different aspects of the present invention may be recognized individually, collectively or in combination with each other. Various aspects of the invention described herein apply to any of the particular application examples set forth below, or for any other type of moving object. Can be done. Any description of an aircraft herein may apply to and be used for any moving object, such as any transporter. In addition, the systems, devices, and methods disclosed herein in the context of air movement (eg, flight) include ground or water movement, underwater movement, or outer space. It can also be applied in the context of multiple other types of movements, such as movements in. Further, any description of a rotor or rotor assembly herein may apply to any propulsion system, device, or mechanism (eg, propeller, wheel, axle) configured to generate propulsion by rotation. Can be used for them.

Other purposes and features of the present invention will become apparent by reviewing the specification, claims, and the accompanying drawings.
Built-in by reference

All publications, patents, and patent applications referred to herein are the same as shown to indicate that individual publications, patents, or patent applications are specifically and individually incorporated by reference. To a degree, incorporated herein by reference.

The novel features of the present invention will be demonstrated in detail in the accompanying claims. For a better understanding of the features and advantages of the present invention, refer to the following detailed description and the accompanying drawings to clarify exemplary embodiments in which the principles of the present invention are utilized. Will be obtained by.

The state estimation regarding the UAV according to a plurality of embodiments is shown.

External parameter calibration for UAVs for a plurality of embodiments is shown.

The initialization of the UAV in the horizontal orientation according to the plurality of embodiments is shown.

Demonstrates the initialization of a tilted oriented UAV for a plurality of embodiments.

A method for operating a UAV according to a plurality of embodiments is shown.

A system for processing inertial data and image data according to a plurality of embodiments is shown.

An algorithm for performing feature point matching between a plurality of images captured by a single image sensor at a plurality of different time points according to a plurality of embodiments is shown.

An algorithm for performing feature point matching between a plurality of images captured by a plurality of different image sensors according to a plurality of embodiments is shown.

A method for determining initialization information about a UAV using a plurality of sensors according to a plurality of embodiments is shown.

A method for error recovery relating to a UAV using a plurality of sensors according to a plurality of embodiments is shown.

A method for calibrating one or more external parameters of a UAV, according to a plurality of embodiments, is shown.

A method for calibrating one or more external parameters of a UAV with a plurality of sensors having an initial configuration according to a plurality of embodiments is shown.

A method for estimating state information about a UAV using a plurality of sensors during the operation of the UAV according to a plurality of embodiments is shown.

A sliding window filter for selecting information in the previous state according to a plurality of embodiments is shown.

A method for performing state estimation and / or parameter calibration for a UAV, according to a plurality of embodiments, is shown.

A system for controlling a UAV using a plurality of sensors according to a plurality of embodiments is shown.

The UAV according to a plurality of embodiments is shown.

A movable object including a support mechanism and a load according to a plurality of embodiments is shown.

A system for controlling a movable object according to a plurality of embodiments is shown.

A synchronous image data collection scheme according to a plurality of embodiments is shown.

An asynchronous image data collection scheme according to a plurality of embodiments is shown.

A system having a plurality of switchably coupled image sensors according to a plurality of embodiments is shown.

The systems, methods, and devices of the present disclosure allow the determination of information related to the operation of multiple moving objects, such as unmanned aerial vehicles (UAVs). In some embodiments, the present disclosure is used to determine various types of information useful for UAV operation, such as information for state estimation, initialization, error recovery, and / or parameter calibration. Utilize multiple sensor fusion technologies that combine sensor data from multiple different sensor types. For example, a UAV may include at least one inertial sensor and at least two image sensors. Various methods, such as iterative optimization algorithms, can be used to combine data from different types of sensors. In some embodiments, the iterative optimization algorithm involves iteratively linearizing and solving a nonlinear function (eg, a nonlinear objective function). The multiple sensor fusion techniques presented herein can be used to improve the accuracy and robustness of UAV state estimation, initialization, error recovery, and / or parameter calibration, and are therefore "intelligent." Extends multiple capabilities of multiple UAVs for autonomous or semi-autonomous operation.

In some embodiments, the present disclosure relates to the use of multiple imaging sensors (eg, multiple cameras) in combination with inertial sensors to perform initialization, error recovery, parameter calibration, and / or state estimation. Provides multiple systems, methods, and devices. The use of multiple imaging sensors offers various advantages over multiple approaches that utilize a single imaging sensor. These advantages include improved accuracy (eg, due to more image data being available for estimation), and increased stability and robustness (eg, multiple image sensors, multiple image sensors). It can provide redundancy when one of the image sensors malfunctions). Multiple sensor fusion algorithms for combining data from multiple image sensors can differ from multiple algorithms for multiple approaches of a single image sensor. As an example, multiple algorithms for multiple image sensors may take into account multiple spatial relationships (eg, relative position and / or orientation) between different image sensors, such multiple spaces. The relationship would not apply to multiple approaches for a single image sensor.

Although some embodiments herein are presented in the context of multiple UAVs, it will be appreciated that the disclosure may apply to other types of moving objects such as ground transporters. A plurality of examples of a plurality of moving objects suitable for use in the plurality of systems, methods, and devices provided herein are described in more detail below.

The UAVs described herein are fully autonomous (eg, by a suitable computing system such as an on-board controller), completely autonomously, or manually (eg, by a user who is a person). ) Can be operated. The UAV may respond to such commands by receiving multiple commands from the appropriate entity (eg, a human user or autonomous control system) and performing one or more actions. For example, a UAV can take off from the ground, move in the air (eg, with up to three translational degrees of freedom, and up to three rotational degrees of freedom), move to a target location or set of target locations, hover in the air, and reach the ground. It can be controlled to land, etc. As another example, the UAV can be controlled to move at a specified speed and / or acceleration (eg, up to 3 translational degrees of freedom, and up to 3 rotational degrees of freedom) or along a specified path of motion. .. In addition, the commands can be used to control one or more UAV components such as the components described herein (eg, sensors, actuators, propulsion units, loads, etc.). As an example, some commands can be used to control the position, orientation and / or movement of a UAV load such as a camera.

In some embodiments, the UAV may be configured to perform more complex functions that may involve higher autonomy, or to perform entirely autonomously without the need for any user input. Multiple examples of such features include maintaining a specific position (eg, hovering in a fixed position), navigation (eg, planning a route to reach a target destination, etc.). And / or along), obstacle avoidance, and environment mapping, but not limited to these. Implementations of these features may be based on information about the UAV and / or the surrounding environment.

For example, in some embodiments, it is useful to determine state information that indicates past, present, and / or expected future UAV states. The state information may include information about the spatial arrangement of the UAV (eg, location or location information such as longitude, latitude, and / or altitude; orientation or attitude information such as roll, pitch and / or yaw). The state information may also include information about the movement of the UAV (eg, translational velocity, translational acceleration, angular velocity, angular acceleration, etc.). The state information may include information about the spatial arrangement and / or movement of the UAV with respect to up to 6 degrees of freedom (eg, 3 degrees of freedom in position and / or translation, 3 degrees of freedom in orientation and / or rotation). State information may be provided relative to the global coordinate system or relative to the local coordinate system (eg, relative to the UAV or other entity). In some embodiments, state information is provided relative to the state before the UAV, such as the spatial arrangement of the UAV when it begins operation (eg, being powered, taking off, starting flight). Will be done. The determination of state information may be referred to herein as "state estimation." Optionally, state estimation can be performed throughout the operation of the UAV to provide updated state information (eg, continuously or at multiple predetermined time intervals).

FIG. 1 shows state estimation for the UAV 100 according to a plurality of embodiments. The UAV 100 is depicted in FIG. 1 as being in the first state 102 at the first time point k and in the second state 104 at the subsequent second time point k + 1. A local coordinate system or reference system 106 indicating the position and orientation of the UAV 100 can be defined relative to the UAV 100. The UAV 100 may have a first position and orientation when in the first state 102 and a second position and orientation when in the second state 104. The change in state can be represented as a translation T from the first position to the second position and / or a rotation θ from the first orientation to the second orientation. When the position and orientation of the UAV 100 in the previous state 102 is known, T and θ can be used to determine the position and orientation of the UAV 100 in the subsequent state 104. Furthermore, if the length of the time interval [k, k + 1] is known, the translational and angular velocities of the UAV 100 in the second state 104 can also be estimated based on T and θ, respectively. Therefore, in some embodiments, the state estimation uses state change information and information about the first state 102 to determine the state changes T, θ and then to determine the second state 104. Accompany.

In some embodiments, state estimation is performed based on sensor data acquired by one or more sensors. Exemplary plurality of sensors suitable for use in the plurality of embodiments disclosed herein are location sensors (eg, Global Positioning System (GPS) sensors, mobiles that allow location trisection. Device transmitter), image or vision sensor (eg, imaging device such as a camera capable of detecting visible, infrared, or ultraviolet light), proximity or distance sensor (eg, ultrasonic sensor, rider, time of Flight or depth cameras), inertial sensors (eg accelerometers, gyroscopes, inertial measurement units (IMUs)), altitude sensors, attitude sensors (eg compass), pressure sensors (eg barometers), audio sensors (eg metric) , Mike), or field sensors (eg, magnetic field sensors, electromagnetic sensors). Any suitable number and combination of sensors, such as one, two, three, four, five, or more sensors, may be used. Optionally, data can be received from different types of sensors (eg, two, three, four, five, or more types). Different types of sensors can measure different types of signals or information (eg, position, orientation, velocity, acceleration, proximity, pressure, etc.) and / or obtain different data. A type of measurement technique can be used. As an example, multiple sensors include some appropriate combination of active sensors (eg, sensors that generate and measure energy from their own energy source) and passive sensors (eg, sensors that detect available energy). obtain. As another example, some sensors may generate absolute measurement data provided with respect to the global coordinate system (eg, position data provided by a GPS sensor, attitude data provided by a compass or accelerometer). On the other hand, other sensors provide relative measurement data with respect to the local coordinate system (eg, relative angular velocity provided by the gyroscope, relative translational acceleration provided by the accelerometer, image). It may generate a projection of a particular ambient environment provided by a sensor, relative distance information provided by an accelerometer, a rider, or a time of flight camera). In some examples, the local coordinate system can be a body coordinate system that is defined relative to the UAV.

The plurality of sensors described herein can be carried by a UAV. The sensor may be positioned at any suitable part of the UAV, such as above, below, aside, or inside the body of the UAV transporter. In some embodiments, the sensor may be enclosed within the housing of the UAV, positioned outside the housing, coupled to a surface of the housing (eg, inner or outer surface), or housing. It can form part of the body. Some sensors may be mechanically coupled to the UAV so that the spatial arrangement and / or movement of the UAV corresponds to the spatial arrangement and / or movement of the plurality of sensors. The sensor can be coupled to the UAV via a rigid coupling so that it does not move relative to the portion of the UAV to which the sensor is mounted. Alternatively, the coupling between the sensor and the UAV may allow movement of the sensor relative to the UAV. The bond can be a permanent bond or a non-permanent (eg, releasable) bond. Suitable multiple bonding methods may include adhesion, bonding, welding, and / or fasteners (eg, screws, bows, pins, etc.). In some embodiments, the coupling between the sensor and the UAV reduces multiple vibrations, or other undesired mechanical movements, from being transmitted from the UAV body to the sensor. Or equipped with a damper. Optionally, the sensor can be formed integrally with a part of the UAV. Further, the sensor is such that the data collected by the sensor, such as the plurality of embodiments described herein, can be used for various functions of the UAV (eg, navigation, control, propulsion, communication with the user or other device, etc.) It can be electrically coupled to parts of the UAV (eg, processing units, control systems, data storage) to allow it to be used for.

In some embodiments, the plurality of sensing results is generated by a combination of sensor data acquired by the plurality of sensors, also known as "sensor fusion". As an example, sensor fusion can be used to combine sensory data acquired by multiple different sensor types, including GPS sensors, inertial sensors, image sensors, lidar, ultrasonic sensors, and others. As another example, sensor fusion is an absolute measurement data (eg, data provided relative to the global coordinate system, such as GPS data) and relative measurement data (eg, vision sensing data, lidar data, or ultrasound sensing). It can be used to combine different types of sensing data (data provided relative to the local coordinate system, such as data). Sensor fusion can be used to compensate for the limitations or inaccuracies associated with individual sensor types, thereby improving the accuracy and reliability of the final sensory result.

In multiple embodiments where multiple sensors are used, each sensor may be positioned on the UAV in its respective position and orientation. Information about multiple spatial relationships of multiple sensors relative to each other (eg, relative positions and orientations) can be used as a reference for fusing data from multiple sensors. These spatial relationships may be referred to herein as "external parameters" of the plurality of sensors, and the process of determining a plurality of spatial relationships may be referred to herein as "external parameter calibration". In some embodiments, a plurality of sensors are installed on the UAV at a plurality of predetermined locations so that a plurality of initial values for the plurality of external parameters are known. In addition, multiple external parameters can change during the operation of the UAV, for example due to vibrations, collisions, or other events that change the relative position and / or orientation of the sensors. Therefore, to ensure that the results of multiple sensor fusions are accurate and robust, the entire operation of the UAV to provide updated parameter information (eg, continuous or predetermined multiples). It may be helpful to perform external parameter calibration (at time intervals).

FIG. 2 shows external parameter calibration for UAVs for a plurality of embodiments. The UAV includes a reference sensor 200 and a plurality of additional sensors 202. Each of the sensors 200, 202 is associated with a local coordinate system or reference system that indicates the position and orientation of the sensor. In some embodiments, each sensor 202 is in a different position and orientation relative to the coordinate system of the reference sensor 200. The plurality of external parameters of each sensor 202 may be represented as their respective translation T and / or their respective rotation θ from the coordinate system of the sensor 202 to the coordinate system of the reference sensor 200. Therefore, the external parameter calibration is performed by a set (T ₁ , θ ₁ ), (T ₁ , θ ₁ ), ..., (T _m ,) consisting of a plurality of external parameters relating each of the plurality of sensors 202 to the reference sensor 200. It may involve determining θ _m ). Alternatively, or in addition, external parameter calibration may involve determining a set of external parameters that relate each of the sensors 202 to each other rather than to the coordinate system of a single reference sensor 200. .. Those skilled in the art will appreciate that determining a plurality of spatial relationships of a plurality of sensors 202 to a reference sensor 200 is equivalent to determining a plurality of spatial relationships of a plurality of sensors 202 to each other.

The plurality of UAVs described herein may implement various algorithms to perform state estimation and external parameter calibration. In some embodiments, the same algorithm is used to estimate state information and multiple external parameter values simultaneously. In other embodiments, different algorithms are used to evaluate state information and parameter values separately. Multiple examples of such algorithms are further described herein. In some embodiments, these algorithms are used before first performing state estimation and / or parameter calibration (eg, when the UAV is powered, in operation, or in flight). , May be necessary or informative to initialize with specific information. This information, which may be referred to herein as "initialization information," is used at the first time point (eg, when the UAV is powered) to be used to initialize state estimation and / or multiple parameter calibration algorithms. It may contain multiple initial values for the UAV state and / or multiple external parameters (when the operation is started or when the flight is started). The accuracy of the initialization information can affect the accuracy of subsequent state estimation and / or parameter calibration procedures. The process of determining initialization information for a UAV may be referred to herein as "initialization."

For example, in some embodiments, the initialization information includes information indicating the direction of the UAV relative to the direction of gravity. This information can be particularly useful for adjusting data from multiple sensors affected by gravity, such as multiple inertial sensors. In some embodiments, it may be beneficial or necessary to subtract the effects of gravity from multiple acceleration measurements obtained by the inertial sensor so that the resulting inertial sensor data show UAV-only acceleration. Therefore, the initialization procedure may involve determining the initial orientation of the UAV relative to the direction of gravity to allow correction of the inertial sensor data. Alternatively or in combination, the initialization information may include the position, orientation, velocity, acceleration, and / or multiple external parameters of the UAV before or during the operation of the UAV at the first time point. Initialization information can be provided relative to the UAV's local coordinate system, global coordinate system, and / or the coordinate system of other entity (eg, a remote controller for the UAV).

3 and 4 show UAV initialization according to a plurality of embodiments. FIG. 3 shows the initialization of the UAV300 in a horizontal orientation. The "horizontal orientation" can mean that the UAV 300 has a horizontal axis 302 (eg, an axis that passes through opposite lateral sides of the UAV) that is substantially orthogonal to the direction of the gravity vector g. The UAV300 can be in a horizontal orientation, for example, when taking off from a horizontal plane. FIG. 4 shows the initialization of the UAV 400 in a tilted orientation. The "tilted orientation" can mean that the UAV 400 has a horizontal axis 402 that is not orthogonal to the direction of the gravity vector g (eg, an axis that passes through opposite lateral sides of the UAV). The UAV400 can be in a tilted orientation, for example, when taking off from an inclined surface or when launched from a non-resting state (eg, from the air or after being thrown into the air by the user). In some embodiments, the plurality of initialization methods described herein determine whether the UAV is initially in a horizontal or tilted orientation and / or relative to a gravity vector. It can be used to determine the degree of inclination of a typical UAV.

In some embodiments, if the state estimation and / or external parameter calibration algorithm fails, it may be necessary to reinitialize and restart the algorithm. The error is an algorithm in which one or more sensors are malfunctioning (eg, failing to provide sufficient data), or failing to produce a result (eg, to a result within a specified time period). It may involve providing sensor data for the algorithm (which fails to converge). Reinitialization is described herein, except that the reinitialization information is obtained at or around the time of the error rather than at the beginning of the UAV's operation. It can be substantially similar to the chemical procedure. In some embodiments, reinitialization is performed after detecting an error, and the process of reinitialization following an error may be referred to herein as "error recovery." Alternatively, or in combination, the reinitialization can be performed at any time following the initialization as desired, eg, at a plurality of predetermined time intervals.

FIG. 5 shows a method 500 for operating a UAV, according to a plurality of embodiments. In step 502, the UAV begins operation (eg, powering, taking off, etc.). In step 504, initialization is performed to determine initialization information about the UAV (eg, the direction relative to the direction of gravity as described above). In step 506, external parameter calibration and state estimation with respect to the current time is performed, for example, to determine the position, orientation, velocity of the UAV, relative positions and orientations of the plurality of UAV sensors, and the like. As described above, parameter calibration and state estimation can be performed simultaneously or separately. In step 508 it is determined whether an error, such as a sensor malfunction or failure in the parameter calibration and / or state estimation procedure, occurred during the operation of the UAV. If an error occurs, Method 500 returns to step 504 and reinitializes the UAV to recover from the error. If no errors have occurred, in step 510, a plurality of results of parameter calibration and state estimation are output to, for example, a flight control module, remote terminal, or controller for subsequent storage and / or use. For example, the flight control module may use multiple parameter values and / or state information determined to facilitate UAV navigation, mapping, obstacle avoidance, and more. Method 500 then returns to step 506 and repeats the parameter calibration and state estimation procedure for the next time point. Method 500 can be repeated at any rate, such as at least once every 0.1 seconds, to provide updated state information and parameter values during the operation of the UAV.

In some embodiments, this approach does not require any assumptions about the initial state of the UAV (eg, position, orientation, velocity, acceleration, etc.) and therefore UAV initialization as described herein. It is advantageous to perform and / or error recovery. As an example, multiple approaches herein may allow UAV initialization and / or error recovery without assuming that the UAV is initially stationary (eg, velocity and acceleration equal to zero). This assumption may be appropriate for certain situations (eg, when the UAV is taking off from the ground or another plane), and in such embodiments, the direction of the gravity vector is the inertial sensor data. Can be obtained directly from. However, this assumption relates to several other situations (eg, when the UAV first slides down an inclined surface when an error occurs, the UAV is thrown into the air by the user, the UAV is in the air, etc.) It may not be appropriate. In such embodiments, it may not be possible to determine the direction of the gravity vector from inertial sensor data alone, as there may be other acceleration values affecting the sensor results. Absent. Thus, the methods herein allow the determination of the gravity vector regardless of the initial state of the UAV (eg, whether the UAV is stationary or moving) and thus the flexibility of initialization and error recovery. And improve accuracy.

The plurality of UAVs described herein may utilize data from multiple sensors to perform the initialization, state estimation, and external parameter calibration methods provided herein. Multiple sensors of various types and combinations may be used. In some embodiments, the UAV utilizes at least one inertial sensor and at least one image sensor. Optionally, the UAV has at least one inertial sensor, and two or more, three or more, four or more, five or more, six or more, seven or more. Multiple image sensors may be utilized, such as more, 8 or more, 9 or more, 10 or more image sensors.

As used herein, inertial sensors are motion sensors (eg, speed sensors, accelerometers and other acceleration sensors), orientation sensors (eg, gyroscopes, tilt meters), or one or more integrated motion sensors and / or It can be used to refer to an IMU with one or more integrated orientation sensors. Inertia sensors can provide sensing data relative to a single axis of motion. The axis of motion may correspond to the axis of the inertial sensor (eg, the vertical axis). Multiple inertial sensors may be used, and each inertial sensor provides multiple measurements along different axes of motion. For example, three accelerometers can be used to provide acceleration data along three different axes of motion. The three directions of movement can be multiple orthogonal axes. One or more of the accelerometers may be a plurality of linear accelerometers configured to measure acceleration along a translational axis. Conversely, one or more of the accelerometers may be angular accelerometers configured to measure angular acceleration around the axis of rotation. As another example, three gyroscopes can be used to provide orientation data around three different axes of rotation. The three rotation axes can be a plurality of orthogonal axes (eg, roll axis, pitch axis, yaw axis). Alternatively, at least some or all of the multiple inertial sensors may provide multiple axes of the same motion and relative measurements. Such redundancy can be implemented, for example, to improve the accuracy of the measurements. Optionally, a single inertial sensor may be able to provide sensing data relative to multiple axes. For example, an IMU containing multiple integrated accelerometers and gyroscopes can be used to generate acceleration and orientation data for up to six axes of motion. Alternatively, a single accelerometer can be used to detect acceleration along multiple axes, and a single gyroscope can be used to detect rotations around multiple axes.

The image sensor can be any device configured to detect electromagnetic radiation (eg, visible light, infrared light, and / or ultraviolet light) and generate image data based on the detected electromagnetic radiation. The image data generated by the image sensor may include a plurality of still images (eg, photographs), a plurality of moving images (eg, video), or one or more images which may be an appropriate combination thereof. The image data can be multicolored (eg RGB, CMYK, HSV) or monochromatic (eg grayscale, black and white, sepia). In some embodiments, the image sensor can be a camera. Although the particular embodiments provided herein are described in the context of cameras, the disclosure may apply to any suitable image sensor and is relevant to the cameras herein. It will be appreciated that any of these statements may also apply to multiple other types of image sensors. The camera can be used to generate multiple 2D images of a 3D scene (eg, environment, one or more objects, etc.). Multiple images generated by the camera may represent the projection of a 3D scene onto a 2D image plane. Therefore, each point in the 2D image corresponds to the 3D spatial coordinates in the scene. Multiple 2D images of a 3D scene to allow multiple cameras to reconstruct 3D spatial information of the scene (eg, depth information indicating the distance between an object in the scene and the UAV). Can be used to image. Optionally, a single image sensor can be used, for example, to acquire 3D spatial information using structures from multiple motion techniques. The 3D spatial information can be processed to determine the UAV state (eg, position, orientation, velocity, etc.).

The combination of the plurality of inertial sensors and the plurality of image sensors can provide various benefits regarding the operation of the UAV. As an example, the accuracy of inertial data from multiple inertial sensors, such as multiple IMUs, can decline over time due to noise and drift and can be affected by acceleration due to gravity. This challenge can be mitigated or overcome by correcting inertial data with image data from one or more image sensors and / or combining them. As another example, by using multiple image sensors, if some of the image sensors are disturbed and / or malfunction, the remaining image sensors still collect data. It may be possible for the UAV to continue its operation as it is available. Therefore, inertial data from one or more inertial sensors and image data from one or more image sensors use sensor fusion algorithms to provide more robust and accurate sensing results for the operation of the UAV. Can be processed and combined.

FIG. 6 shows a system 600 for processing inertial data and image data according to a plurality of embodiments. In some embodiments, the system 600 includes a data acquisition module 610, an image processing module 620, a sensor fusion module 630, and a flight control module 640. The various modules of the system 600 can be implemented using any suitable combination of hardware and software components, as further described herein. For example, each module is suitable for multiple hardware, such as one or more processors and a memory containing multiple instructions that can be executed by one or more processors to perform the functions described herein. Can include components. Alternatively or in combination, two or more modules may be implemented using the same set of multiple hardware components, eg, the same processor.

The data acquisition module 610 can be used to acquire inertial data and image data from one or more inertial sensors and one or more image sensors, respectively. In some embodiments, inertial data and image data are collected at substantially the same frequency. In other embodiments, inertial data and image data are collected at different frequencies (eg, inertial data is collected more frequently than image data, and vice versa). For example, an inertial sensor may output inertial data at frequencies higher than, equal to, or even higher than approximately 50 Hz, 100 Hz, 150 Hz, 200 Hz, 250 Hz, 300 Hz, 350 Hz, 400 Hz, 450 Hz, 500 Hz. The image sensor may output image data at a frequency higher than, or equal to, approximately 1 Hz, 5 Hz, 10 Hz, 15 Hz, 20 Hz, 25 Hz, 30 Hz, 40 Hz, 50 Hz, or 100 Hz. In a plurality of embodiments in which a plurality of image sensors are used, the data acquisition module 610 may synchronously collect image data from each of the plurality of image sensors at the same plurality of time points.

The image processing module 620 can be used to process the image data received from the data acquisition module 610. In some embodiments, the image processing module 620 implements a feature point algorithm that detects and / or extracts one or more feature points from one or more images. A feature point (also referred to herein as a "feature") is a portion of an image (eg, an edge) that is uniquely distinguishable from the remaining portions of the image and / or from other features of the image. , Corners, points of interest, blobs, bumps, etc.). Optionally, feature points are relatively variable for multiple transformations of the imaged object (eg, translation, rotation, scaling) and / or for changes in multiple characteristics of the image (eg, brightness, exposure). It can be nothing. Feature points can be detected in multiple parts of an image that are rich in terms of information content (eg, large 2D textures, textures that exceed thresholds). Feature points can be detected in multiple parts of a stable image even in multiple confusions (eg, when the illuminance and brightness of the image change). The feature detection described herein can be achieved using various algorithms that can extract one or more feature points from the image data. The algorithm can be an edge detection algorithm, a corner detection algorithm, a blob detection algorithm, or a bump detection algorithm. In some embodiments, the corner detection algorithm can be a "Features from accelerated segment test" (FAST). In some embodiments, the feature detector may extract a plurality of feature points and use FAST to perform a plurality of calculations for the plurality of feature points. In some embodiments, the feature detectors are a Canny edge detector, a Sobel operator, a Harris & Stephens / Plussy / Shi-Tomasi corner detection algorithm, a SUSAN corner detector, a Level curve curvature approach, a Laplacian determinant of the Gas. , Curvature of Hessian, MSER, PCBR, or Grey-level blobs, ORB, FREAK, or any suitable combination thereof.

Optionally, the image processing module 620 may match the same feature points across different images and thus generate a set of feature points that have been matched. Feature point matching is performed by different image sensors (eg, the same time point) between multiple images captured by the same image sensor (eg, multiple images taken at different time points such as consecutive time points). It may involve matching of multiple feature points between multiple images captured (or at different time points) or in a combination thereof. Feature point matching can be performed using multiple corner detection algorithms, multiple optical flow algorithms, and / or multiple feature matching algorithms. For example, optical flow can be used to determine movement between successive images, thereby predicting the location of feature points in subsequent images. Alternatively or in combination, multiple feature point descriptors (eg, multiple characteristics that can be used to uniquely identify a feature point) provide location of the same feature point in multiple other images. Can be used for. Illustrative algorithms suitable for use in multiple embodiments herein are described in more detail herein.

In some embodiments, the image processing module 620 is configured to meet certain performance criteria to ensure that image processing occurs at a sufficiently high speed and with sufficient accuracy. For example, the image processing module 620 may be configured to handle three or more data channels at a real-time processing frequency of approximately 20 Hz. As another example, the image processing module 620 is configured to perform feature tracking and matching so that, for example, the number of inner layers of the RANSAC algorithm is greater than or equal to 70% when the reprojection error is 2. Can be done.

The sensor fusion module 630 may acquire inertial data from the data acquisition module 610 and processed image data (eg, a plurality of matched feature points) from the image processing module 620. In some embodiments, the sensor fusion module 630 processes inertial data and processed image data to determine information related to UAV operation, such as initialization information, state information, or multiple external parameters. Or implement multiple sensor fusion algorithms. For example, a sensor fusion algorithm can be used to calculate UAV position and / or motion information such as UAV position, attitude, and / or velocity. An exemplary sensor fusion algorithm suitable for use in multiple embodiments herein is described in more detail below.

The flight control module 640 may use a plurality of sensor fusion results generated by the sensor fusion module 630 to determine a plurality of control signals for the UAV. Multiple control signals are used to control the operation of one or more propulsion units, one or more sensors, one or more loads, one or more communication modules, and one or more UAV components. Can be. In some embodiments, multiple control signals are used, for example, for navigation, obstacle avoidance, route planning, environment mapping, and others to enable autonomous or semi-autonomous UAV operation. ..

FIG. 7 is for performing feature point matching (also known as “single channel processing”) between a plurality of images captured by a single image sensor at different time points according to the plurality of embodiments. The algorithm 700 is shown. Like all other methods described herein, method 700 can be performed using any embodiment of the plurality of systems and devices described herein. In some embodiments, one or more steps of method 700 are performed by one or more processors associated with the image processing module 620 of system 600.

The input 702 to the algorithm 700 may include a series of images acquired by a single image sensor at multiple time points. In step 704, one of a series of images (“current image”) is received. In step 706, a set of one or more feature points can be calculated from the current image (“temporary points”) using, for example, a corner detection algorithm or some other suitable feature point detection algorithm. Step 706 may involve calculating a plurality of feature points using only the current image and not other images.

In step 708, it is determined whether the current image is the first image in a series of images (eg, the image captured at the earliest time). If there is at least one previous image captured at an earlier point in time than the first image, the algorithm proceeds to step 710. In step 710, the current image is compared to one or more previous images using an optical flow algorithm. In some embodiments, the optical flow algorithm tracks one or more feature points (previous points) detected in a previous image to perform a plurality of previous points in the current image. Match with feature points. Alternatively or in combination, multiple other types of tracking algorithms, such as multiple tracking algorithms that perform matching based on multiple feature point descriptors, are used to match multiple feature points. obtain. If a tracking error occurs during the tracking of a previous point, for example, if the tracking is inaccurate or cannot be performed, that point can be discarded from the analysis in step 712. A set of feature points that have been matched over time for the rest of the time is considered to be the "current point" for analysis.

In step 714, temporary points are added to the current set of points to produce the final set of feature points that is the output of the algorithm 718. A set of is modified (eg, with multiple external parameters). Alternatively, if in step 708 it is determined that the current image is not the first image, steps 710 and 712 may be omitted and the final set of feature points will be a plurality of temporary features determined in step 706. Obtained only from points. In step 720, the current image is set to the previous image, and the current plurality of points are set as the previous plurality of points. The algorithm 700 then returns to step 704 for the next iteration and the next image in the series of images is received for processing.

FIG. 8 shows an algorithm 800 for performing feature point matching (also known as “multi-channel processing”) between a plurality of images captured by different plurality of image sensors according to a plurality of embodiments. In some embodiments, one or more steps of method 800 are performed by one or more processors associated with the image processing module 620 of system 600.

The input 802 to the algorithm 800 may include a plurality of images acquired by a plurality of different image sensors. In some embodiments, each image sensor provides a series of images acquired over multiple time points. Image data acquisition can be synchronous such that multiple image sensors are acquiring multiple images at substantially the same multiple time points. Alternatively, image data acquisition can be asynchronous such that different image sensors acquire multiple images at different time points. Additional examples of asynchronous image data collection are further described herein.

Algorithm 800 is a plurality of temporally matched images from each series of images from each image sensor using a plurality of techniques of single channel processing described herein (eg, Algorithm 700). You can get feature points. For example, a plurality of parallel processes 804a ... 804n can be used to acquire a plurality of temporally matched feature points from a plurality of images of a corresponding number of image sensors. Similar to the multiple steps of Algorithm 700 described herein, single channel processing involves receiving the current image from an image sequence generated by a single image sensor (step 806) and now. Using an optical flow algorithm that tracks multiple feature points in one or more previous images and matches them with multiple feature points in one or more previous images (step 808), and multiple feature points that could not be tracked. Obtaining a set of multiple feature points matched by discarding (step 810) and modifying multiple feature points using multiple external calibration parameters for the image sensor (step 812). Can be accompanied by. The resulting set of feature points can be tracked and matched with multiple feature points in subsequent images, as described herein. The single channel matching procedure can be repeated until all of the multiple images in the image sequence have been processed.

In step 814, once a set of temporally matched feature points is acquired for each image sequence from each image sensor, the feature points from different image sequences are Spatial matching can be done with each other using a spatial matching algorithm. Since different image sensors can be configured to capture multiple images in different fields of view, the same feature points in the scene will appear in different spatial locations in multiple images from different sensors. obtain. At least some of the image sensors have overlapping fields of view to ensure that at least some of the feature points are present in multiple image sequences from more than one image sensor. Can have. In some embodiments, the spatial matching algorithm is acquired by different image sensors (eg, at the same or different time points) in order to identify and match multiple feature points across different image sequences. Analyze multiple images that have been created. The final output 816 of Algorithm 800 is spatially matched over temporally matched feature points and different image sequences within each of the plurality of image sequences generated by each image sensor. It can be a set consisting of a plurality of feature points including a plurality of feature points.

In a plurality of embodiments in which a plurality of image sensors are used, the collection and / or processing of image data from each of the plurality of image sensors can occur synchronously or asynchronously. For example, a synchronous image data acquisition scheme may involve each image sensor acquiring image data at substantially the same time point, and the image data may be transmitted to the processor at the same time. In contrast, asynchronous image data acquisition schemes can involve different image sensors acquiring image data at different time points, and the image data is delivered to the processor at different times (eg, sequentially). Can be sent. In an asynchronous scheme, some of the plurality of image sensors may acquire a plurality of images at the same plurality of time points, and the other plurality of image sensors may acquire a plurality of images at a plurality of different time points.

FIG. 20 shows a synchronous image data acquisition scheme 2000 according to a plurality of embodiments. Scheme 2000 captures data from any number of image sensors, such as 2, 3, 4, 5, 6, 7, 8, 9, 10, 15, 20, 30, 40, 50, or more. Can be used to obtain and process. In the description of FIG. 20, the image data is synchronously received from n image sensors. For example, at the first time point k, each of the plurality of image sensors generates its own image data. Image data from each of the plurality of sensors is simultaneously transmitted to one or more processors (eg, in an image processing module). The image data may be transmitted with a time stamp indicating when the data was generated. Similarly, at the second time point k + 1, image data is acquired by each image sensor and simultaneously transmitted to the processor. This process can be repeated during the operation of the UAV. Synchronous image data acquisition is advantageous in that it improves the ease and accuracy of feature point matching across multiple images from different sensors. As an example, multiple images captured at the same time point may exhibit less variability in exposure time, brightness, or other image characteristics that can affect the ease of feature point matching.

FIG. 21 shows an asynchronous image data acquisition scheme 2100 for a plurality of embodiments. Scheme 2100 captures data from any number of image sensors, such as 2, 3, 4, 5, 6, 7, 8, 9, 10, 15, 20, 30, 40, 50, or more. Can be used to obtain and process. In the description of FIG. 21, the image data is received asynchronously from n image sensors. For example, at the first time point k, the first image sensor acquires image data and transmits it to one or more processors (eg, of an image processing module). At the second time point k + 1, the second image sensor acquires the image data and transmits it to one or more processors. At the third time point k + 2, the nth image sensor acquires the image data and transmits it to one or more processors. Each image data may be transmitted, for example, with a time stamp indicating the time when the image data was acquired in order to facilitate downstream image processing. The time interval between different time points can be constant or variable. In some embodiments, the time interval between different time points is in the range of about 0.02 seconds, or about 0.05 seconds to about 0.2 seconds. This process can be repeated during the operation of the UAV until each of the image data is received from each of the plurality of image sensors. The order in which the image data is acquired and received can be changed as desired. Additionally, FIG. 21 shows that image data is obtained from a single image sensor at each time point, while image data can be received from multiple sensors at some or all of the time points. Let's be aware of that. In some embodiments, asynchronous image data collection selectively transfers different subsets of multiple image sensors to one or more processors via a switching mechanism, as further described herein. It is realized by combining. Asynchronous image data collection can offer various advantages such as relative ease of implementation, compatibility with a wider range of hardware platforms, and reduced computing load.

As described herein, multiple systems and devices of the present disclosure are sensors that process image and inertial data to obtain state information, initialization information, and / or multiple external parameters. It may include a sensor fusion module configured to implement the fusion algorithm (eg, sensor fusion module 630 of system 600). In some embodiments, the sensor fusion algorithm is performed throughout the operation of the UAV (eg, continuously or at a plurality of predetermined time intervals) to provide real-time updates. Various type sensor fusion algorithms, such as Kalman filter-based algorithms or optimization algorithms, are suitable for use in the plurality of embodiments presented herein. In some embodiments, the multiple optimization methods presented herein can be considered as a type of bundle-based algorithm. Kalman filter-based algorithms can also be considered bundle-based algorithms in special cases. In some embodiments, the main distinguishing feature between the Kalman filter-based algorithm and the bundle-based algorithm presented herein is the number of states that will be optimized. The Kalman filter-based algorithm may utilize one or two states, while the bundle-based algorithm herein has more than three states (eg, at least 5 states, at least 10 states, at least 20). States, at least 30 states, at least 40 states, or at least 50 states) may be utilized. In some embodiments, the bundle-based algorithms presented herein can provide higher accuracy as the optimization procedure utilizes more information. In some embodiments, Kalman filter-based algorithms may provide faster speed and higher stability.

In some embodiments, the sensor fusion algorithm involves an optimization algorithm. Optimization algorithms can be used to determine a set of resolution parameters that minimizes or maximizes the value of the objective function. In some embodiments, the optimization algorithm is an iterative optimization algorithm that iterates until the algorithm converges to a solution or stops (eg, exceeds a time threshold) to generate multiple estimates. is there. In some embodiments, the optimization algorithm is linear, while in some other embodiments, the optimization algorithm is non-linear. In some embodiments, the optimization algorithm involves iteratively linearizing and solving a nonlinear function. A plurality of embodiments herein may utilize a single optimization algorithm or different types of optimization algorithms to estimate different types of UAV information. For example, the methods described herein use a linear optimization algorithm to estimate a particular value, and a non-linear optimization algorithm to estimate other values. Can accompany.

In some embodiments, the present disclosure discloses UAV state information (eg, position, orientation, velocity) and / or multiple external parameters (eg, inertial sensors and / or) at one or more time points during UAV operation. Provided is an iterative optimization algorithm for using inertial data from at least one inertial sensor and image data from at least two image sensors to estimate a plurality of spatial relationships between multiple image sensors). Optionally, the iterative optimization algorithm may also determine multiple estimates for state information and / or multiple external parameters based on multiple initial values for state information and / or multiple external parameters. The iterative optimization algorithm may involve calculating, for example, an estimate of the maximum posteriori probability (MAP) of state information and / or multiple external parameters based on inertial data, image data and / or multiple initial values. In some embodiments, the objective function for the iterative optimization algorithm contains state information, initialization information and / or multiple actual values of multiple external parameters, inertial data, image data and / or multiple initial values. Associates with state information, initialization information and / or multiple estimates of multiple external parameters calculated based on. The objective function can be a linear function or a non-linear function. Multiple iterative solution techniques can be used to minimize or maximize the objective function to obtain solutions for state information, initialization information and / or multiple actual values of multiple external parameters. For example, the nonlinear objective function can be iteratively linearized and solved as further described herein.

FIG. 9 shows a method 900 for determining initialization information about a UAV (or some other moving object) using a plurality of sensors, according to a plurality of embodiments. The plurality of steps of Method 900 can be performed using any embodiment of the plurality of systems and devices described herein. For example, some or all of the plurality of steps of Method 900 may be performed using one or more processors mounted by the UAV. Method 900 can be performed in combination with any embodiment of the various methods described herein.

As described herein, the initialization information that will be determined using method 900 is one of the UAV's (eg, gravity direction) orientation, the UAV's position, or the UAV's velocity. Can include more than one. For example, the initialization information may include the orientation of the UAV (eg, relative to the direction of gravity), the position of the UAV, and the velocity of the UAV. In some embodiments, the initialization information is determined with respect to the approximate time point at which the UAV begins operation. For example, the time point is approximately 50 ms, 100 ms, 200 ms, 300 ms, 400 ms, 500 ms, 600 ms, 700 ms, 800 ms, etc. after the UAV started operation. It may be after 900 milliseconds, 1 second, 2 seconds, 3 seconds, 4 seconds, 5 seconds, 6 seconds, 7 seconds, 8 seconds, 9 seconds, 10 seconds, 30 seconds, 60 seconds or less. As another example, the time point is approximately 50ms, 100ms, 200ms, 300ms, 400ms, 500ms, 600ms, 700ms, 800ms when the UAV begins to operate. , 900 milliseconds, 1 second, 2 seconds, 3 seconds, 4 seconds, 5 seconds, 6 seconds, 7 seconds, 8 seconds, 9 seconds, 10 seconds, 30 seconds, 60 seconds or less.

In step 910, it is detected that the UAV has started operating. Step 910 may involve detecting one or more of the UAV being powered, the UAV taking off from the surface, or the UAV starting flight. The UAV can take off from an inclined or non-inclined (non-inclined) surface. The UAV may start flying from a surface (eg, on the ground), from a free fall state (eg, in the air), from a state launched by a launcher, or from a state thrown into the air by a user. Optionally, in step 910, multiple propulsion units of the UAV have been activated, the output of the multiple propulsion units is greater than or equal to the threshold, and the altitude of the UAV is greater than or equal to the threshold. , The speed of the UAV may be faster than or equal to the threshold, or may involve detecting one or more of the acceleration of the UAV being faster than or equal to the threshold.

In step 920, inertial data is received from at least one inertial sensor mounted by the UAV. Inertia data may include one or more measurements that indicate the 3D acceleration and 3D angular velocity of the UAV. In some embodiments, the inertial data is acquired by at least one inertial sensor over a time interval starting from the time the UAV starts operating and / or from the time the UAV is in flight. Includes measurements of.

In step 930, image data is received from at least two image sensors mounted by the UAV. The image data may include one or more images of the environment surrounding the UAV. In some embodiments, image data is obtained from each of at least two image sensors over a time interval starting from the time the UAV starts operating and / or from the time the UAV is in flight. It may include one or more images.

In step 940, initialization information about the UAV is determined based on inertial data and image data. The initialization information may include the orientation of the UAV with respect to its position, velocity and / or direction of gravity when the UAV begins operation. In some embodiments, step 940 comprises processing the image data according to a plurality of image processing algorithms described herein. For example, the image data may include one or more images acquired by each of at least two image sensors over a time interval starting from the time the movable object starts moving. It can be processed using feature matching based on feature point detection algorithms, optical flow algorithms, and / or descriptor algorithms. As described herein, multiple images obtained from different image sensors can be compared to each other, for example, to perform feature point matching. For example, one or more images acquired by the first image sensor can be compared with one or more images acquired by the second image sensor. Optionally, the initialization information can be determined using only inertial data and image data without requiring any other data such as sensor data from multiple other types of sensors. In some embodiments, the initialization information is determined directly from the inertial data and the image data without requiring any plurality of initial estimates or initial values for the initialization information. Initialization information does not rely on any data acquired prior to the operation of the UAV (eg, before the UAV is powered and / or flies), while the UAV is in operation (eg, the UAV). It can be determined using only the data acquired during the flight of the UAV after being powered. As an example, initialization information can be determined during the operation of the UAV without the need for prior initialization performed before the UAV begins operation.

In some embodiments, step 940 involves generating multiple estimates of initialization information using inertial data and image data. For example, step 940 can be performed by using inertial data to generate a first estimate of initialization information and image data to generate a second estimate of initialization information. The first and second estimates can be combined to obtain initialization information about the UAV.

In some embodiments, the initialization information is determined using an optimization algorithm such as a linear optimization algorithm, a non-linear optimization algorithm, or an iterative non-linear optimization algorithm. Exemplary algorithms suitable for use in multiple embodiments herein are presented below.

The UAV may have a sensing system that includes m cameras (or other image sensor types) and one IMU (or other inertial sensor type). The IMU may be configured to output angular velocities around the three axes and linear acceleration values along the three axes. The output frequency of the IMU may be higher than the output frequency of a plurality of cameras. For example, sample rates for multiple cameras can be assumed to be f _cam Hz and N = f _cam × T. The system may receive images of (N + 1) × m at time intervals T after it has started operation (eg, after being powered). The time interval T is at multiple time points t ₀ , t ₁ , t ₂ , t ₃ , t ₄ , t ₅ , ... t _N and multiple UAV states.
Can correspond to.
Represents the current position of the UAV (relative to its position at t ₀ , when the operation started).
Represents the current velocity of the UAV (relative to the UAV's body coordinate system), and g ^k represents the acceleration due to gravity (relative to the UAV's body coordinate system). The initial situation is
Can be.

The number of feature points observed in the received (N + 1) × m image can be M + 1. i-th characteristic point is first observed by the j th camera at time _{t k (0 ≦ k ≦ N} ), λ i is the j-th feature point in the plane perpendicular direction of the camera at time t _k It can be assumed to be depth. All UAV states
And feature depths λ ₀ , λ ₁ , λ ₂ , λ ₃ , ... λ _M can form the overall state X.

A plurality of IMU data, the respectively example the time interval t may be received in the corresponding time _{(t k, t k + 1} ). The following formulas can be defined.
here,
Represents the rotation from time t to time k and is obtained by integrating the angular velocity from the IMU.
Represents the acceleration relative to the UAV body coordinate system at time t, and α and β represent the integral of the raw IMU data. Estimated value from IMU data
Can be determined as follows, where the covariance matrix
Represents an error caused by noise in the raw IMU data.
here,
Is the matrix on the left side of the above formula, X is the overall state,
Is additive noise.
and
Can be calculated using a plurality of pre-integral techniques known to those of skill in the art.

For a plurality of cameras, initially observed by the camera c _n feature points l is at time t _i, assuming that observed by the camera c _m at time t _j, the following estimates
Can be obtained.
X is the overall state,
and
Is a plurality of image coordinates for the feature points l in an image acquired by the i-th camera at time t _j,
Represents the i-th camera rotation associated with the IMU,
Represents the translation of the i-th camera associated with the IMU.
Is about IMU data
Can be derived from the above formula as well.

The objectives may be to optimize UAV state estimates using (1) estimates from the IMU and (2) geometric constraints between multiple image sequences. The objective equation minimizes errors from the IMU and geometric constraints between multiple image sequences.

In addition, the following can be defined.
here,
Is a diagonal matrix that measures the uncertainty of camera observation data.

In some embodiments, the solution procedure is to store an estimate from the IMU in Λ _D , store a geometric constraint between multiple image sequences in Λ _C , and solve the following linear equations: including. (Λ _D + Λ _C ) X = (b _D + b _C )

Method 900 may offer various advantages with respect to the operation of the UAV. For example, Method 900 directs the UAV to gravity directions for various takeoff types, such as takeoff from an inclined surface, takeoff from the air or from free fall, takeoff by manual launch or by being thrown by a user. Can be used to determine. Therefore, the method 900 may further include a step of correcting subsequent inertial data acquired from the inertial sensor using this orientation information. In addition, Method 900 has been used to provide automatic initialization after the UAV has begun operation, even in multiple situations where the initial UAV state (eg, position, orientation, velocity, etc.) is completely unknown. obtain.

In addition to allowing automatic initialization of the UAV, the methods of the present disclosure can also be used to perform reinitialization or error recovery after an error has occurred in the UAV system. Multiple methods for reinitialization are described herein, except that the reinitialization is performed after the UAV has started operating, rather than after an error has been detected. Substantially similar to multiple methods for.

FIG. 10 shows a method 1000 for error recovery on a UAV (or some other moving object) using a plurality of sensors, according to a plurality of embodiments. The plurality of steps of Method 1000 can be performed using any embodiment of the plurality of systems and devices described herein. For example, some or all of the plurality of steps of Method 1000 may be performed using one or more processors mounted by the UAV. Method 1000 can be performed in combination with any embodiment of the various methods described herein.

Method 1000 can be used to determine the reinitialization information used to reinitialize the UAV so that normal operation resumes after an error has occurred. Similar to the initialization information described herein, the reinitialization information may include one or more of the UAV's orientation (eg, in the direction of gravity), the position of the UAV, or the velocity of the UAV. For example, the reinitialization information may include the orientation of the UAV (eg, relative to the direction of gravity), the position of the UAV, and the velocity of the UAV. In some embodiments, the reinitialization information is determined with respect to the approximate time point at which the error occurred. For example, the time point is approximately 50 ms, 100 ms, 200 ms, 300 ms, 400 ms, 500 ms, 600 ms, 700 ms, 800 ms, 900 ms, when the error occurred. It may be after 1 second, 2 seconds, 3 seconds, 4 seconds, 5 seconds, 6 seconds, 7 seconds, 8 seconds, 9 seconds, 10 seconds, 30 seconds, 60 seconds or less. As another example, the time point is approximately 50ms, 100ms, 200ms, 300ms, 400ms, 500ms, 600ms, 700ms, 800ms, 900ms where an error occurs. Seconds, 1 second, 2 seconds, 3 seconds, 4 seconds, 5 seconds, 6 seconds, 7 seconds, 8 seconds, 9 seconds, 10 seconds, 30 seconds, 60 seconds or less may be before.

In step 1010, it is detected that an error has occurred during the operation of the UAV. The operation of the UAV may involve the UAV being powered, the UAV taking off from the surface, and the UAV starting flight. The error can be accompanied by a malfunction in one or more sensors, such as a malfunction in one or more of at least one inertial sensor or at least two image sensors. Errors can be associated with malfunctions in UAV components such as data acquisition modules, image processing modules, sensor fusion modules, or flight control modules. For example, a UAV may include a sensor fusion module that uses an iterative optimization algorithm, and malfunctions may involve the iterative optimization estimation algorithm failing to converge to a solution. Optionally, the UAV may include a state estimation module as further described herein, which may use an iterative state estimation algorithm. Malfunctions can involve the iterative state estimation algorithm failing to converge to a solution.

In step 1020, inertial data is received from at least one inertial sensor mounted by the UAV. Inertia data may include one or more measurements that indicate the 3D acceleration and 3D angular velocity of the UAV. In some embodiments, the inertial data comprises one or more measurements taken by at least one inertial sensor over a time interval starting from the time the error occurs.

In step 1030, image data is received from at least two image sensors mounted by the UAV. The image data may include one or more images of the environment surrounding the UAV. In some embodiments, the image data may include one or more images acquired by each of at least two image sensors over a time interval starting from the time the error occurs.

In step 1040, reinitialization information about the UAV is determined based on inertial data and image data. The reinitialization information may include the position, velocity, and / or orientation of the UAV with respect to the direction of gravity when the error occurred. In some embodiments, step 1040 comprises processing image data according to a plurality of image processing algorithms described herein. For example, the image data may include one or more images acquired by each of at least two image sensors over a time interval starting from the time of the error, where the one or more images are feature point detection. It can be processed using algorithms, optical flow algorithms, and / or feature matching algorithms. As described herein, multiple images obtained from different image sensors can be compared to each other, for example, to perform feature point matching. For example, one or more images acquired by the first image sensor can be compared with one or more images acquired by the second image sensor. Optionally, the reinitialization information can be determined using only inertial data and image data without requiring any other data such as sensor data from other types of sensors. In some embodiments, the reinitialization information is determined directly from the inertial data and the image data without requiring any plurality of initial estimates or initial values for the reinitialization information.

In some embodiments, step 1040 involves generating multiple estimates of reinitialization information using inertial data and image data. For example, step 1040 can be performed by using inertial data to generate a first estimate of reinitialization information and image data to generate a second estimate of reinitialization information. The first and second estimates can be combined to obtain reinitialization information about the UAV.

In some embodiments, the reinitialization information is determined using an optimization algorithm such as a linear optimization algorithm, a nonlinear optimization algorithm, an iterative optimization algorithm, an iterative linear optimization algorithm, or an iterative nonlinear optimization algorithm. .. The optimization algorithm used to evaluate the reinitialization information is, except that the relevant time points are included in the time interval following the occurrence of the error, rather than in the time interval after the UAV has started operating. , It can be substantially similar to the algorithm presented herein for evaluating initialization information.

Method 1000 may offer various advantages with respect to the operation of the UAV. For example, method 1000 can be used to determine the orientation of the UAV with respect to the direction of gravity in various situations where errors can occur, such as when the UAV is in the air during flight. Therefore, Method 1000 may involve correcting subsequent inertial data acquired from the inertial sensor using this orientation information. In addition, Method 1000 provides automatic reinitialization after the UAV has begun operation, even in multiple situations where the initial UAV state (eg, position, orientation, velocity, etc.) is completely unknown after the error has occurred. Can be used to provide. For example, method 1000 may involve using determined reinitialization information to reinitialize the iterative state estimation algorithm implemented by the state estimation module. Advantageously, the multiple reinitialization techniques described herein can be used to detect and address multiple errors in real time during the operation of the UAV, and thus the operation of the UAV. Improve the reliability of.

FIG. 11 shows a method 1100 for calibrating one or more external parameters of a UAV with a plurality of sensors during the operation of the UAV (or some other movable object) according to the plurality of embodiments. The plurality of steps of method 1100 can be performed using any embodiment of the plurality of systems and devices described herein. For example, some or all of the plurality of steps of method 1100 may be performed using one or more processors mounted by the UAV. Method 1100 can be performed in combination with any embodiment of the various methods described herein.

Method 1100 is used to determine multiple external parameters for a UAV during operation (eg, when the UAV is powered, in flight, etc.), which may be referred to herein as "online" calibration. Can be. In some embodiments, online calibration is performed continuously or at a plurality of predetermined time intervals during the operation of the UAV to allow real-time updates of multiple external parameters. For example, method 1100 may be performed once every 0.1 seconds during the operation of the UAV.

In step 1110, a plurality of initial values for one or more external parameters are received. In some embodiments, the plurality of external parameters include a plurality of spatial relationships between at least two image sensors mounted by the UAV. For example, a plurality of spatial relationships can include a plurality of relative positions and a plurality of relative orientations of a plurality of image sensors. The relative positions and orientations of at least two image sensors are determined relative to each other and / or in relation to the position and orientation of at least one inertial sensor mounted by the UAV. obtain.

Various methods can be used to obtain multiple initial values. In some embodiments, the plurality of initial values may be received from a memory device associated with the UAV (eg, mounted on the UAV). Multiple initial values may be determined prior to the operation of the UAV. As an example, multiple initial values may be determined using multiple iterative optimization algorithms described herein prior to UAV operation. As another example, multiple initial values may be measured by the user prior to operation. Optionally, the plurality of initial values can be multiple factory calibration values determined when the UAV is manufactured. In some embodiments, multiple initial values may be determined based on knowledge of the configuration of the UAV. For example, multiple image sensors and / or multiple inertial sensors may be coupled to a UAV at a particular fixed location (eg, a selected location set available for mounting multiple sensors). , Multiple initial values can be determined based on information about multiple fixed locations.

In some embodiments, the plurality of initial values are intended to provide a rough approximation of the plurality of actual values of the plurality of external parameters and are not intended to be so accurate. In contrast to other calibration methods, the methods provided herein do not require accurate multiple initial values for multiple external parameters to perform online calibration. For example, multiple initial values for multiple external parameters (eg, multiple relative positions) are approximately 0.1 cm, 0.25 cm, 0.5 cm, 0.75 cm, 1 cm, 1.25 cm, 1.5 cm, It may be separated from a plurality of actual values of a plurality of external parameters by 1.75 cm, 2 cm, 2.25 cm, 2.5 cm, 2.75 cm, 3 cm, or 5 cm or less. Alternatively, multiple initial values for multiple external parameters (eg, multiple relative positions) are at least approximately 0.1 cm, 0.25 cm, 0.5 cm, 0.75 cm, 1 cm, 1.25 cm, 1 It may be separated from the actual values of the plurality of external parameters by .5 cm, 1.75 cm, 2 cm, 2.25 cm, 2.5 cm, 2.75 cm, 3 cm, or 5 cm. As another example, multiple initial values for multiple external parameters (eg, multiple relative orientations) are approximately 0.1 degrees, 0.25 degrees, 0.5 degrees, 0.75 degrees, 1 degree, and so on. Multiple actuals of multiple external parameters, only 1.25 degrees, 1.5 degrees, 1.75 degrees, 2 degrees, 2.25 degrees, 2.5 degrees, 2.75 degrees, 3 degrees, or 5 degrees or less. It may be far from the value. Alternatively, multiple initial values for multiple external parameters (eg, relative multiple positions) are at least approximately 0.1 degrees, 0.25 degrees, 0.5 degrees, 0.75 degrees, 1 degree, and so on. Multiple actual values of multiple external parameters, only 1.25 degrees, 1.5 degrees, 1.75 degrees, 2 degrees, 2.25 degrees, 2.5 degrees, 2.75 degrees, 3 degrees, or 5 degrees. It may be separated from.

In step 1120, inertial data is received from at least one inertial sensor mounted by the UAV during the operation of the UAV. Inertia data may include one or more measurements that indicate the 3D acceleration and 3D angular velocity of the UAV. In some embodiments, inertial data is acquired by at least one inertial sensor over at least 2, 3, 4, 5, 6, 7, 8, 9, 10, 20, 30, 40, or 50 time points. Includes one or more measurements made.

In step 1130, image data is received from at least two image sensors mounted by the UAV during the operation of the UAV. The image data may include one or more images of the environment around the moving object. In some embodiments, the image data is of at least two image sensors over a time point of at least 2, 3, 4, 5, 6, 7, 8, 9, 10, 20, 30, 40, or 50. It may include one or more images obtained by each.

In step 1140, a plurality of estimates for one or more external parameters are determined based on the plurality of initial values, inertial data, and image data. Multiple estimates can be determined during the operation of the UAV. Optionally, inertial data and image data can be the only sensor data used to determine multiple estimates of multiple external parameters. Step 1140 may involve processing one or more images acquired by multiple image sensors using a feature point detection algorithm, an optical flow algorithm, and / or a feature matching algorithm. Optionally, step 1140 may involve comparing one or more images acquired by the first sensor with one or more images acquired by the second sensor.

In some embodiments, the estimates are determined using an optimization algorithm, such as a nonlinear optimization algorithm, a linear optimization algorithm, an iterative optimization algorithm, or an iterative nonlinear optimization algorithm. Iterative optimization algorithms may include calculating maximum posteriori probability (MAP) estimates for one or more external parameters based on multiple initial values, inertial data, and image data. An exemplary algorithm suitable for use in multiple embodiments herein is presented below.

The UAV may have a sensing system that includes m cameras (or other image sensor types) and one IMU (or other inertial sensor type). Similar to the notation previously described herein, multiple external parameters of multiple cameras
and
It can be 1 ≦ i ≦ m. Similar to the other optimization algorithms provided herein, an optimization objective function may be established that includes integral estimates from the IMU and responses from multiple cameras.

The UAV state over the period from t ₀ to t _N ([t ₀ , t ₁ , t ₂ , t ₃ , t ₄ , t ₅ , ... t _N ]) is
Can be assumed to be. here,
Represents the current position of the UAV (relative to its position at t ₀ when the operation starts).
Represents the current velocity of the UAV (relative to the UAV's body coordinate system) at time k.
Represents the current orientation of the UAV (relative to the orientation at t ₀ ) at time k. The initial situation is
and
Can be. Time i-th feature point in _{t k (0 ≦ k ≦ N} ) are first observed by the j th camera, lambda _i is the j-th feature point in the plane perpendicular direction of the camera at time t _k It can be assumed to be depth. The multiple parameters of the camera that will be estimated are:
Is. here,
Represents the i-th camera rotation associated with the IMU,
Represents the translation of the i-th camera associated with the IMU, and 1 ≦ i ≦ m. The unknown that will be presumed is multiple UAV states
, Multiple external calibration parameters
, And feature depths λ ₀ , λ ₁ , λ ₂ , λ ₃ ... λ _M. All of them form a vector X, referred to herein as the overall state.

In some embodiments, the objective equation is
Is defined as. here,
Is the integral estimate of the overall state X from the IMU
Associated with,
Is the overall state X estimated from the image sensor
Associate with.
Encodes the previous information about X.
and
Is non-linear with respect to X and can therefore be difficult to solve, so the algorithm can process the error state representation δX and multiple functions can be linearized through a first-order Taylor expansion.
Is an estimate of the overall state X within some error terms δX. Error state expression δX
Is
not,
Notice that it can be (minimum expression).

Considering the actual physical model of UAV dynamics and multiple principles of camera geometry, the remainder
and
Is
Can be defined as. here,
Is the coordinates of the feature l in the frame j estimated through the attitude information.
There are three elements. Other notations in multiple formulas are defined above.

and
Is about δX
and
It is a derivative of, and there are two cases as explained below.

In one case, it is assumed that feature l was first observed by camera ci at time i and then by camera cj at the same time.
here,
Is the normalized coordinates. The following formulas can be obtained.

In other cases, it is assumed that feature l was first observed by camera ci at time i and then by camera cj at time j.

When ci = cj, the following multiple mathematical formulas can be obtained.

When ci ≠ cj, the following multiple mathematical formulas can be obtained.

By substituting the above plurality of equations into the objective equation, the following equations can be obtained.

In some embodiments, initial values may be provided and the equation of interest may be iteratively solved. Initial values can be obtained by adding an integral estimate from the IMU to the optimized estimate from the last point in time, using an initialization or reinitialization algorithm as described herein, or.
It can be obtained by initializing as. The objective equation gets δX and
Until
According to
Can be solved iteratively to update. Finally, after convergence
Is the UAV state output by the system.

In some embodiments, if δX does not approach zero after multiple iterations and the parameter calibration algorithm fails to converge to the solution, this can be considered an error in the system and is described herein. Reinitialization can be performed to recover from the error.

Optimal estimates of multiple external parameters after factory calibration and / or multiple parameters from the last point in time can be input to the objective equation as part of the UAV state, and some error in estimating multiple external parameters. The equation can be iteratively solved and updated to correct and minimize the equation of interest. The system can detect (solve δX) errors in multiple external parameters in real time and correct the errors (
Update) get.

Optionally, following step 1140, the state of the UAV (eg, position, orientation, and / or velocity) can be determined based on multiple estimates for multiple external parameters. For example, multiple external parameters can be used to fuse sensor data from multiple image sensors and multiple inertial sensors to calculate state information. In some embodiments, the state information is determined in relation to the previous UAV state at the previous time point during the operation of the UAV. The previous time point can be the first time point when the state information is available, for example, the time point when the UAV has started operation, initialization has occurred, or reinitialization has occurred. Alternatively, the state information can be determined in relation to the global coordinate system rather than relative to the previous UAV state.

FIG. 12 shows a method 1200 for calibrating one or more external parameters of a UAV (or some other moving object) with multiple sensors having an initial configuration, according to a plurality of embodiments. The plurality of steps of method 1200 may be performed using any embodiment of the plurality of systems and devices described herein. For example, some or all of the plurality of steps of method 1200 may be performed using one or more processors mounted by the UAV. Method 1200 can be performed in combination with any embodiment of the various methods described herein. Similar to Method 1100, Method 1200 can be used to perform online calibration during UAV operation.

In step 1210, it is detected that the initial configurations of the plurality of sensors mounted by the UAV have been modified. The plurality of sensors may include at least one inertial sensor and at least two image sensors. The initial configuration is to remove at least one of the sensors, add at least one to the sensors, change the position and / or orientation of one of the sensors, or combine them. It may have been fixed. In some embodiments, the initial configuration is modified prior to the operation of the UAV (eg, before the UAV flies and / or is powered), and the modifications are detected during the operation of the UAV. In a plurality of other embodiments, the configuration is modified during the operation of the UAV. Various methods can be used to detect the correction. Modifications can be detected, for example, if the values of δθ and δT in δX in the iterations described herein are not close to zero, which means that additional updates must be performed. ..

In step 1220, inertial data is received from at least one inertial sensor mounted by the UAV during the operation of the UAV. Inertia data may include one or more measurements that indicate the 3D acceleration and 3D angular velocity of the UAV. In some embodiments, inertial data is acquired by at least one inertial sensor over at least 2, 3, 4, 5, 6, 7, 8, 9, 10, 20, 30, 40, or 50 time points. Includes one or more measurements made.

In step 1230, image data is received from at least two image sensors mounted by the UAV during the operation of the UAV. The image data may include one or more images of the environment around the moving object. In some embodiments, the image data is of at least two image sensors over a time point of at least 2, 3, 4, 5, 6, 7, 8, 9, 10, 20, 30, 40, or 50. It may include one or more images obtained by each.

At step 1240, one or more external parameters are estimated in response to detection that the initial configuration has been modified. One or more external parameters may include multiple spatial relationships between multiple sensors with modified configurations. In some embodiments, multiple spatial relationships can be determined relative to each other and / or in relation to inertial sensors, relative multiple positions and relative orientations of the image sensors. including. Multiple external parameters can be estimated in different ways. In some embodiments, the one or more external parameters are estimated based on the inertial and image data received in steps 1220 and 1230. Optionally, inertial data and image data can be the only sensor data used to determine multiple estimates of multiple external parameters. Step 1240 may involve processing one or more images acquired by multiple image sensors using a feature point detection algorithm, an optical flow algorithm, and / or a feature matching algorithm. Optionally, step 1240 may involve comparing one or more images acquired by the first sensor with one or more images acquired by the second sensor.

In some embodiments, the plurality of external parameters are estimated based on a plurality of initial values for one or more external parameters. Various methods can be used to obtain multiple initial values. In some embodiments, the plurality of initial values may be received from a memory device associated with the UAV (eg, mounted on the UAV). Multiple initial values may be determined prior to the operation of the UAV. As an example, multiple initial values may be determined using multiple iterative optimization algorithms described herein prior to UAV operation. As another example, multiple initial values may be measured by the user prior to operation. Optionally, the plurality of initial values can be multiple factory calibration values determined when the UAV is manufactured. In some embodiments, multiple initial values may be determined based on knowledge of the configuration of the UAV. For example, multiple image sensors and / or multiple inertial sensors may be coupled to the UAV at a particular fixed location (eg, a selected location set available for mounting multiple sensors). The initial value of can be determined based on information about multiple fixed locations.

In some embodiments, the plurality of external parameters are estimated using an optimization algorithm such as a nonlinear optimization algorithm, a linear optimization algorithm, an iterative optimization algorithm, an iterative nonlinear optimization algorithm, or an iterative linear optimization algorithm. .. Iterative optimization algorithms may include calculating maximum posteriori probability (MAP) estimates for one or more external parameters based on multiple initial values, inertial data, and image data. The iterative optimization algorithm may be similar to the plurality of algorithms previously described herein for Method 1100.

Optionally, following step 1240, the state of the UAV (eg, position, orientation, and / or velocity) can be determined based on a plurality of estimated external parameters. For example, multiple external parameters can be used to fuse sensor data from multiple image sensors and multiple inertial sensors to calculate state information. In some embodiments, the state information is determined in relation to the previous UAV state at the previous time point during the operation of the UAV. The previous time point can be the first time point when the state information is available, for example, the time point when the UAV has started operation, initialization has occurred, or reinitialization has occurred. Alternatively, the state information can be determined in relation to the global coordinate system rather than relative to the previous UAV state.

The automatic parameter calibration methods described herein (eg, methods 1100, 1200) are, for example, vibrations, collisions, or other multiple events that can change the spatial relationship of multiple sensors relative to each other. Due to this, multiple external parameters can change during the operation of the UAV, which can be beneficial in ensuring the accuracy and reliability of UAV sensor data processing. For example, the methods described herein detect and correct multiple errors in multiple estimates of multiple spatial relationships during operation, as well as during UAV operation. It can be used to continuously estimate multiple spatial relationships (eg, multiple relative positions and orientations) between multiple image sensors and multiple inertial sensors. An additional implementation of the online calibration described herein can advantageously eliminate the need for accurate calibration prior to the operation of the UAV, which may be referred to herein as "offline" calibration. This can be modified by the UAV's sensor configuration (eg, by adding one or more sensors, excluding one or more sensors, and moving one or more sensors), and a number of new sensor configurations. It may enable a "plug and play" approach in which the UAV can be operated immediately after modification without requiring lengthy offline calibration procedures to determine external parameters. In some embodiments, the plurality of parameter calibration methods provided herein determine a plurality of external parameters during the operation of the UAV without performing any parameter calibration prior to the operation of the UAV. It is possible to do.

FIG. 13 shows method 1300 for estimating state information about a UAV using a plurality of sensors during the operation of the UAV (or some other movable object) according to the plurality of embodiments. The plurality of steps of Method 1300 may be performed using any embodiment of the plurality of systems and devices described herein. For example, some or all of the plurality of steps of method 1300 may be performed using one or more processors mounted by the UAV. Method 1300 can be performed in combination with any embodiment of the various methods described herein.

Method 1300 can be used to estimate the current state of the UAV during operation (eg, when the UAV is powered, in flight, etc.). State estimation can involve determining various types of state information such as UAV position, orientation, velocity, and / or acceleration. The state information can be determined in relation to the state prior to the UAV at an earlier point in time, or from an absolute point (eg, relative to the global coordinate system). In some embodiments, state estimation is performed continuously or at a plurality of predetermined time intervals during the operation of the UAV to allow real-time updates of the plurality of external parameters. For example, method 1300 may be performed once every 0.1 seconds during the operation of the UAV.

At step 1310, information about the previous state regarding the UAV is received. Information on the previous state may include the position, orientation, velocity, and / or acceleration of the UAV at the previous time point during the operation of the UAV. In some embodiments, the previous state information is obtained using an iterative optimization algorithm, eg, the same algorithm used in step 1340 of method 1300. The state information from one or more previous time points may be stored in the memory device associated with the UAV to facilitate the estimation of the updated state information for the subsequent points in time.

In step 1320, inertial data is received from at least one inertial sensor mounted by the UAV. Inertia data is inertial measurement data acquired by the inertial sensor over at least 2, 3, 4, 5, 6, 7, 8, 9, 10, 20, 30, 40 or 50 during the operation of the UAV. May include. Inertia data may indicate the 3D acceleration and 3D angular velocity of the UAV.

In step 1330, image data is received from at least two image sensors mounted by the UAV. The image data is a plurality of image data acquired by each image sensor over at least 2, 3, 4, 5, 6, 7, 8, 9, 10, 20, 30, 40 or 50 during the operation of the UAV. May include images. The images can be from the environment surrounding the UAV.

At step 1340, updated state information about the UAV is determined based on previous state information, inertial data and / or image data during the operation of the UAV. The updated state information may include the position, orientation, velocity, and / or acceleration of the UAV. The updated status information can be the current status information regarding the UAV at the present time. Optionally, inertial data and image data may be the only sensor data used to determine the updated state information. Step 1340 may involve processing a plurality of images acquired by each image sensor using a feature point detection algorithm, an optical flow algorithm, and / or a feature matching algorithm. Optionally, step 1340 may involve comparing one or more images acquired by the first sensor with one or more images acquired by the second sensor.

In some embodiments, the estimates are determined using an optimization algorithm such as a nonlinear optimization algorithm, a linear optimization algorithm, an iterative optimization algorithm, an iterative nonlinear optimization algorithm, or an iterative linear optimization algorithm. .. Iterative optimization algorithms may include calculating maximum posteriori probability (MAP) estimates for one or more external parameters based on multiple initial values, inertial data, and image data. An exemplary algorithm suitable for use in multiple embodiments herein is presented below.

The UAV may have a sensing system that includes m cameras (or other image sensor types) and one IMU (or other inertial sensor type). The output frequency of the IMU can be higher than the output frequency of multiple cameras. Multiple external parameters for multiple cameras
and
Can be. here,
Represents the i-th camera rotation associated with the IMU,
Represents the translation of the i-th camera associated with the IMU, and 1 ≦ i ≦ m. Similar to the other optimization algorithms provided herein, an optimization objective function may be established that includes integral estimates from the IMU and responses from multiple cameras.

The UAV state over the period from t ₀ to t _N ([t ₀ , t ₁ , t ₂ , t ₃ , t ₄ , t ₅ , ... t _N ]) is
Can be assumed to be. here,
Represents the current position of the UAV (relative to its position at t ₀ when the operation starts).
Represents the current velocity of the UAV (relative to the UAV's body coordinate system) at time k.
Represents the current orientation of the UAV (relative to the orientation at t ₀ ) at time k. The initial situation is
and
Can be. The number of feature points observed in the (N + 1) × m image can be M. i-th characteristic point is first observed by the j th camera at time _{t k (0 ≦ k ≦ N} ), λ i is the j-th feature point in the plane perpendicular direction of the camera at time t _k It can be assumed to be depth. The unknown that will be presumed is multiple UAV states
, Multiple external calibration parameters
, And feature depths λ ₀ , λ ₁ , λ ₂ , λ ₃ ... λ _M. All of them form a vector X, referred to herein as the overall state.

In some embodiments, the objective equation is
Is defined as. here,
Stores previous information (representing an estimate for X)
Is an integral estimate from the overall state X and IMU
Is the relationship with
Is the overall state X and multiple image sensors
The relationship with the estimated value from. Formula
and
Can be derived as described herein.
and
Are non-linear, so they are expanded by a first-order Taylor expansion.
and
Can be obtained.

By substituting the above two equations into the objective equation, the following equation can be obtained.

In some embodiments, initial values may be provided and the equation of interest may be iteratively solved using the Gauss-Newton algorithm. Initial values can be obtained by adding an integral estimate from the IMU to the optimized estimate from the last point in time, using an initialization or reinitialization algorithm as described herein, or.
Can be obtained by initializing as. The objective equation should get δX and minimize the objective equation
Until
According to
Can be solved iteratively to update. Therefore,
Is the UAV state output by the system.

In some embodiments, if δX does not approach zero after multiple iterations and the state estimation algorithm fails to converge to the solution, this can be considered an error in the system and is described herein. Reinitialization can be performed to recover from the error.

In some embodiments, sensor data from one or more previous time points can be used to estimate current state information. For example, at least 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 20, 30, 40 or 50 previous inertial data and / or image data from a previous point in time is in the current state. It can be used to estimate information. Various methods can be used to determine the amount of information in the previous state used to estimate the current state of the UAV, such as a sliding window filter.

FIG. 14 shows a sliding window filter for selecting information in the previous state according to a plurality of embodiments. The size of the sliding window can be K such that the sensor data from the point in time of K (eg, inertial data and / or image data) is used to estimate the current state. Although K = 4 in FIG. 14, it can be understood that K can be some suitable value, such as 20. The time point of K may include the time point before K-1 and the current time point. When new sensor data is acquired at subsequent points in time, new data may be added to the sliding window and data from the previous point in time may be removed from the sliding window to maintain a constant window size K. For example, in the description of FIG. 14, the first sliding window 1400a holds the data from the time points 1-4, the second sliding window 1400a holds the data from the time points 2-5, and the third The sliding window 1400c holds the data from the time points 3-6, and the fourth sliding window 1400d holds the data from the time points 4-7. In some embodiments, the time of destruction is the earliest time in the sliding window (eg, first-in first-out method (FIFO). In some other embodiments, the time of destruction is not the earliest. Often, other than FIFO, multiple other marginalization approaches, such as first-in, last-out (FILO), or a mixture of FIFO and FILO to determine which point in time to maintain and which point in time to discard. Can be used. In some embodiments, multiple time points are discarded based on checking the parallax between a particular time point and a number of points around it. An arithmetic solution with stable parallax. If not large enough for, for example, the time points can be marginalized and discarded using the Schur complement matrix merging method.

In some embodiments, b _p and lambda _p may store information from a plurality of time before used to estimate the current state. For example, b _p and Λ _p are sensor data (eg, inertial data and / or image data) at multiple previous time points since the UAV started operating (or after being reinitialized due to an error). Can be stored. Optionally, b _p and Λ _p may exclude data from time K that is already contained in the sliding windows described herein. In such multiple embodiments, when data is discarded from the sliding window, it can be used to update the b _p and lambda _p.

In some embodiments, the algorithm can be used to detect errors in the input sensor data in real time. For example, an error can occur if one of the image sensors malfunctions or is disturbed. As another example, the error can occur when there is excessive noise in the image data acquired by the image sensor. Error detection solves δX
Detects whether δX corresponding to image data and / or inertial data is converging (eg, δX is smaller over multiple iterations and / or is approaching zero) when updating Can be done by doing. If δX is converged, the sensor data can be considered error-free. However, if δX does not converge, the sensor data can be considered error and the sensor data causing the convergence problem can be excluded from the optimization process.

Optionally, following step 1340, updated state information may be output to, for example, a control module for controlling the movement of the UAV. For example, the control module provides updated state information to enable, for example, autonomous or semi-autonomous navigation, obstacle avoidance, route planning, environment mapping, and others to control one or more propulsion units of a UAV. Can be used.

Method 1300 may provide some advantages with respect to the operation of the UAV. For example, the plurality of state estimation methods described herein may be more accurate than other types of state estimation methods while conserving computing resources. The estimated state information can be more accurate than the state information estimated using only inertial data or only image data. In some embodiments, the plurality of estimates for state information (eg, location information) are approximately 0.1 cm, 0.25 cm, 0.5 cm, 0.75 cm, 1 cm, 1.25 cm, 1.5 cm, 1 It is deviated from multiple actual values of the state information by .75 cm, 2 cm, 2.25 cm, 2.5 cm, 2.75 cm, 3 cm, or 5 cm or less. Accuracy can be further enhanced by using more image sensors. In addition, the plurality of embodiments herein are suitable for use with multiple image sensors, and thus one or more of the plurality of image sensors has malfunctioned or been disturbed. Allows the system to continue operating even in some cases. In some embodiments, the state estimation method reduces noise and thus improves the robustness and stability of the UAV system.

FIG. 15 shows method 1500 for performing state estimation and / or parameter calibration for a UAV (or any other moving object) according to a plurality of embodiments. In method 1500, inertial data 1502 and image data 1504 are received from at least one inertial sensor and at least two image sensors mounted by the UAV, as described herein. The inertial data 1502 and the image data 1504 can be input to the linear sliding window filter 1506 and the non-linear sliding window filter 1508. The linear sliding window filter 1506 can utilize a FIFO approach to discard data from multiple previous time points when new data is received. The nonlinear sliding window filter 1508 may utilize a mergerization approach to discard data from multiple previous time points when new data is received.

In some embodiments, state estimation and / or parameter calibration inputs inertial data and image data selected by the nonlinear sliding widow filter 1508 into the nonlinear solver 1510 to input state information and / or multiple external parameters. It is performed by generating a solution that provides an estimate of. The non-linear solver 1510 can be a non-linear optimization algorithm, such as the plurality of embodiments described herein. Method 1500 may include detecting whether the nonlinear solver 1510 has converged to generate solution 1512. If the nonlinear solver 1510 converges and produces a solution, the solution may be output 1514 to another UAV component, such as a flight control module. The method then proceeds to the reception of new sensor data, removing the previous sensor data by marginalization via a nonlinear sliding window filter 1508 1516, so that the process continues to generate multiple estimates for the updated state. Can be repeated.

If the nonlinear solver 1510 fails to converge, method 1500 may proceed to use the linear solver 1518, generate a solution that provides multiple estimates for state information, and output solution 1514. The linear solver 1518 can be a linear optimization algorithm, such as the plurality of embodiments described herein. In some embodiments, the failure of the nonlinear solver 1510 to converge is considered an error, and the linear solver 1518 may implement a reinitialization algorithm to recover from the error. The solution provided by the linear solver 1518 (eg, reinitialization information) can be used as multiple initial values for multiple estimates for subsequent states performed by the nonlinear solver 1510.

The various methods described herein can be implemented by any suitable system or device. In some embodiments, the methods herein are performed by a hardware platform that includes a computing platform, flight control modules, sensor modules, and data acquisition subsystems. The computing platform may include a sensor fusion module that receives and processes sensor data according to various sensor fusion algorithms described herein. Computing platforms can be implemented using any suitable combination of hardware and software components. For example, in some embodiments, the computing platform utilizes NVIDIA Tegra K1 as a criterion for designing CPU and GPU computing platforms in ARM architecture. Computing platforms can be designed to meet specific performance requirements such as stability and scalability. Table 1 provides some exemplary requirements and test methods for computing platforms.

The flight control module may include any suitable combination of hardware and software components for controlling the operation of the UAV, such as multiple components for controlling the position, orientation, velocity, and / or acceleration of the UAV. In some embodiments, the flight control module may include, for example, one or more components for controlling the operation of multiple propulsion units of the UAV in order to achieve the desired position, orientation, speed, etc. .. The flight control module may be communicably coupled to a computing platform to receive data (eg, state information) from the platform and use the data as input to one or more flight control algorithms. For example, a flight control module may include multiple lower APIs to receive data from a computing platform, eg, at a frequency of at least 50 Hz.

The sensor module may include one or more sensors, such as at least one inertial sensor and at least two image sensors as described herein. In some embodiments, the sensor module is configured with two cameras set forward (eg, along the direction of movement of the UAV) in a binocular configuration and a downward facing monocular configuration. Includes three global shutter cameras, including one camera. Multiple cameras may have a sampling frequency of approximately 20 Hz. In some embodiments, the sensor module comprises one IMU configured to acquire the angular velocity and linear acceleration of the UAV at a frequency of at least 100 Hz. The IMU can also be configured to provide angular values using, for example, uncorrected integrals from the gyroscope. Optionally, the sensor module can also include other types of sensors such as compasses, barometers, GPS, and / or ultrasonic sensors (eg, multiple ultrasonic sensors pointing forward and downward). Can include. The plurality of sensors for the sensor module may be coupled to the body using a plurality of rigid connections that limit the movement of the plurality of sensors relative to the UAV body. Optionally, multiple vibration reduction systems, such as dampers or shock absorbers, may be installed between the sensors and the UAV module to reduce unwanted movements of the sensors.

The data acquisition subsystem may be operably coupled to the sensor module to send sensor data from various sensors to the computing platform and / or flight control module. The data acquisition system may utilize some suitable type of communication interface, such as a USB interface, a serial port, or a combination thereof. In some embodiments, the plurality of image sensors and the plurality of ultrasonic sensors are coupled via the plurality of USB interfaces, and the IMU and other sensor types are coupled via the plurality of serial ports.

FIG. 16 shows a system 1600 for controlling a UAV using a plurality of sensors according to a plurality of embodiments. System 1600 can be used to implement any of the methods described herein. At least some or all of the multiple components of the system 1600 may be carried by the UAV. The system 1600 can be considered to be divided into two different functional units: the sensor fusion unit 1601a (several components above the dashed line) and the flight control unit 1601b (several components below the dashed line). The sensor fusion unit 1601a and the flight control unit 1601b may be communicably coupled to each other for data, multiple control signals, and other exchanges. In some embodiments, the sensor fusion unit 1601a and the flight control unit 1601b are configured to operate independently of each other so that if one unit fails, the other can continue to operate. For example, if the sensor fusion unit 1601a experiences a malfunction, the flight control unit 1601b may be configured to continue functioning independently, for example, to perform an emergency landing operation.

In some embodiments, the system 1600 includes a sensor fusion module 1602 operably coupled to the flight control module 1604. The sensor fusion module 1602 may also be coupled to multiple sensors, such as one or more inertial sensors 1606 and one or more image sensors 1608. The sensor fusion module 1602 uses inertial and image data from the inertial sensor 1606 and the image sensor 1608 to perform the initialization, error recovery, state estimation, and external parameter calibration methods described herein. It may include one or more processors. Optionally, the sensor fusion module 1602 may be coupled to or include an image processing module for processing image data, as described above and herein. Multiple results produced by the sensor fusion module 1602 may be transmitted to the flight control module 1604 to facilitate various flight operations. Multiple flight operations are multiple results from the sensor fusion module, multiple user commands received from the remote terminal 1610, and sensor data received from multiple other sensors 1612 (eg, GPS sensor, magnetometer, ultrasonic sensor). , Or a combination of these. For example, the flight control module 1604 may control one or more propulsion units 1614 to control the position, orientation, velocity, and / or acceleration of the UAV, for example for navigation, obstacle avoidance, and other UAV operations. Multiple control signals transmitted to (eg, multiple rotors) may be determined.

Multiple components of the system 1600 may be implemented using various types and combinations of hardware elements. For example, the sensor fusion module 1602 can be implemented using any suitable hardware platform that includes one or more processors and a memory that stores multiple instructions that can be executed by one or more processors. Multiple connections between different components of the system 1600 are implemented using different types of communication interfaces, such as analog or digital interfaces such as USB interfaces, serial ports, pulse width modulation channels, or combinations thereof. obtain. Such multiple interfaces may utilize multiple wired or wireless communication methods. For example, the image sensor 1608 may be coupled to the sensor fusion module 1602 via a plurality of USB interfaces, and the inertial sensor 1606 may be coupled to the sensor fusion module 1602 via a plurality of serial ports. The sensor fusion module 1602 may be coupled to the flight control module 1604 via multiple serial ports. The flight control module 1604 can be coupled to the remote terminal 1610 via a radio communication interface such as a radio control (RC) interface. The flight control module 1604 may send multiple commands to the propulsion unit 1614 via pulse width modulation.

In some embodiments, the sensor fusion module 1602 is used to combine the inertial data from the inertial sensor 1606 with the image data from the image sensor 1608 as described herein. Optionally, the sensor fusion module 1602 may combine internal data from the flight control module 1604 with sensor data from the sensors 1606, 1608 to estimate state information about the UAV (eg, motion information such as speed or acceleration). .. The sensor fusion module 1602 may use any suitable computing platform, such as the Jetson TK1 (TK1) platform. The image sensor 1608 may be communicably coupled to the TK1 platform via a USB interface. The TK1 may run any operating system, such as the Ubuntu 14.04 operating system, with multiple suitable drivers. In some embodiments, a robot operating system (ROS) can be used for data transmission. Table 2 provides some exemplary hardware and software components for the sensor fusion module 1602.

Multiple systems, devices, and methods herein include 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 15, 20, 30, 40, 50, or more images. Any suitable number of image sensors, such as sensors, may be utilized. The plurality of image sensors may be one or more processors of the imaging module, for example, to receive and analyze image data via a corresponding number of communication interfaces (eg, to perform feature point detection and matching). Can be combined with at the same time. However, in some embodiments, it may not be possible for a particular hardware platform to support simultaneous coupling to multiple image sensors (eg, more than 6 image sensors, 10 more image sensors). unknown. For example, some image processing modules may not include a sufficient number of communication interfaces so that all of the multiple image sensors can be coupled at the same time. In such multiple embodiments, the plurality of image sensors may not all be coupled to the module at the same time at the same time. As an example, a switching mechanism can be used to selectively combine a particular subset of image sensors into a module at different times.

FIG. 22 shows a system 2200 with a plurality of switchably coupled image sensors according to a plurality of embodiments. The various components of System 2200 may be combined with and / or replaced with any of the other multiple systems and devices presented herein. System 2200 may be used in combination with any of the methods described herein, eg, asynchronous data collection scheme 2100. The system 2200 may include multiple image sensor subsets 2202 (eg, different subsets of n as depicted herein). Any number of image sensor subsets can be used, such as 2, 3, 4, 5, 6, 7, 8, 9, 10, 15, 20, 30, 40, 50, or more. Each image sensor subset is an arbitrary number of individual, such as 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 15, 20, 30, 40, 50, or more image sensors. It may include an image sensor. Each image sensor subset may include the same number of image sensors. Alternatively, some or all of the plurality of image sensor subsets may include a different number of image sensors. Each image sensor subset may be coupled with a switching mechanism 2204 coupled to the image processing module 2206. The switching mechanism 2204 may include any suitable combination of hardware and software components for selectively coupling multiple subsets of the sensor to the module 2206 (eg, switches, repeaters, etc.). The image processing module 2206 can be used to process image data to perform feature point detection and / or matching, as described herein. In some embodiments, the image processing module 2206 is coupled to and / or a component of the sensor fusion module to perform initialization, error recovery, parameter calibration, and / or state estimation.

The switching mechanism 2204 may be configured to combine only a single image sensor subset to the image processing module 2206 at one time so that the image processing module 2206 receives and processes image data from a single subset at one time. In order to obtain data from all of the plurality of image sensors in the system 2200, the switching mechanism 2204 may be controlled to alternate subsets coupled to the image processing module 2206. For example, the image data may combine a first image sensor subset with module 2206 and receive image data from the first subset, combine a second image sensor subset with module 2206 and receive image data from a second subset. It can be received by continuing this until multiple images from all subsets are received. The order and frequency with which the switching mechanism 2204 switches between different subsets can be changed as desired. This approach allows image data to be received from a relatively large number of image sensors without requiring the image processing module 2206 to maintain multiple simultaneous connections to each of the sensors at all times. .. This reduces the computing load associated with processing image data from multiple sensors at once, while adapting the system to any number of image sensors (eg, for plug and play). It can be advantageous in terms of increasing flexibility.

The plurality of systems, devices, and methods described herein can be applied to a wide variety of moving objects. As mentioned earlier, any description of an aircraft herein may apply to, and may be used for, any moving object. Movable objects of the present invention are in the air (eg, fixed-wing aircraft, rotorcraft, or aircraft without fixed-wing or rotorcraft), underwater (eg, ships or submarines), and on the ground (eg, cars, trucks, buses). , Vans, motorbikes and other vehicles, rods, fishing rods and other movable structures or frames, or trains), underground (eg, subways), space (eg, spacecraft, satellites, or space exploration equipment), or these environments. It can be configured to move in some suitable environment, such as some combination of. The movable object can be a transporter, such as the transporter described elsewhere herein. In some embodiments, the movable object can be mounted on a living body such as a human or animal. Suitable animals may include avines, canines, felines, horses, bovids, sheep, pigs, dolphins, rodents, or insects.

Movable objects may be free to move in an environment with respect to 6 degrees of freedom (eg, 3 translational degrees of freedom, and 3 rotational degrees of freedom). Alternatively, the movement of a moving object can be constrained with respect to one or more degrees of freedom, such as by a predetermined path, trajectory, or orientation. The movement can be driven by some suitable actuating mechanism such as an engine or motor. The working mechanism of a moving object can be powered by any suitable energy source, such as electrical energy, magnetic energy, solar energy, wind energy, gravity energy, chemical energy, nuclear energy, or any suitable combination thereof. Movable objects can be self-propelled through a propulsion system, as described elsewhere herein. Optionally, the propulsion system can operate on energy sources such as electrical energy, magnetic energy, solar energy, wind energy, gravity energy, chemical energy, nuclear energy, or any suitable combination thereof. Alternatively, movable objects can be carried by living things.

In some examples, the moving object can be a transporter. Suitable transporters may include water transporters, aircraft, space transporters, or ground transporters. For example, multiple aircraft may be fixed-wing aircraft (eg, airplanes, gliders), rotorcraft (eg, helicopters, rotorcraft), aircraft with both fixed-wing and rotorcraft, or none (eg, aircraft). , Small blimp, hot aircraft). Transporters can be self-propelled, such as in the air, on the water, underwater, in outer space, above ground, or underground. Self-propelled transporters may utilize propulsion systems such as propulsion systems that include one or more engines, motors, wheels, axles, magnets, rotors, propellers, blades, nozzles, or any suitable combination thereof. In some examples, movable objects can take off from a surface, land on a surface, maintain their current position and / or orientation (eg, hover), orient, and / or reposition. Propulsion system can be used.

The movable object may be controlled remotely by the user, or may be controlled within or closer to the occupant in or on the movable object. In some embodiments, the movable object can be an unmanned movable object such as a UAV. An unmanned movable object such as a UAV does not have to have a occupant on board the movable object. Movable objects can be controlled by humans or autonomous control systems (eg, computer control systems), or by any suitable combination thereof. The movable object can be an autonomous or semi-autonomous robot such as a robot constructed by using artificial intelligence.

Movable objects can have some suitable size and / or multiple dimensions. In some embodiments, the movable object can be of a size and / or multiple dimensions for having a occupant who is a person in or on the transport body. Alternatively, the movable object can be smaller in size and / or multiple dimensions capable of having a occupant who is a person in or on the transport body. The movable object can be of a size and / or multiple dimensions suitable for being lifted or mounted by a person. Alternatively, the movable object may be larger than a size and / or multiple dimensions suitable for being lifted or mounted by a person. In some examples, the movable object is approximately 2 cm, 5 cm, 10 cm, 50 cm, 1 m, 2 m, 5 m, or a maximum dimension equal to or less than 10 m (eg, length, width, height, diameter, diagonal). ) Can have. The maximum dimensions may be greater than or equal to approximately 2 cm, 5 cm, 10 cm, 50 cm, 1 m, 2 m, 5 m, or 10 m. For example, the distance between a plurality of shafts of a plurality of rotors facing each other of a movable object may be shorter than or equal to approximately 2 cm, 5 cm, 10 cm, 50 cm, 1 m, 2 m, 5 m, or 10 m. Good. Alternatively, the distance between the shafts of the facing rotors may be greater than or equal to approximately 2 cm, 5 cm, 10 cm, 50 cm, 1 m, 2 m, 5 m, or 10 m.

In some embodiments, the movable object may have a volume of less than 100 cm x 100 cm x 100 cm, less than 50 cm x 50 cm x 30 cm, or less than 5 cm x 5 cm x 3 cm. The total volume of movable objects is approximately 1 cm ³ , 2 cm ³ , 5 cm ³ , 10 cm ³ , 20 cm ³ , 30 cm ³ , 40 cm ³ , 50 cm ³ , 60 cm ³ , 70 cm ³ , 80 cm ³ , 90 cm ³ , 100 cm ³ , 150 cm ³ , ^{200cm 3, 300cm 3, 500cm 3} , 750cm 3, 1000cm 3, 5000cm 3, 10,000cm 3, 100,000cm 3, 1m 3 or may be smaller than 10 m ^3,, or it may be equal to them. On the contrary, the total volume of movable objects is about 1 cm ³ , 2 cm ³ , 5 cm ³ , 10 cm ³ , 20 cm ³ , 30 cm ³ , 40 cm ³ , 50 cm ³ , 60 cm ³ , 70 cm ³ , 80 cm ³ , 90 cm ³ , 100 cm ³ , ^{150cm 3, 200cm 3, 300cm 3} , 500cm 3, 750cm 3, 1000cm 3, 5000cm 3, 10,000cm 3, 100,000cm 3, 1m 3 or may be greater than 10 m ^3,, or may be equal to those ..

In some embodiments, the movable object is approximately ^{32,000cm 2, 20,000cm 2, 10,000cm 2} , 1,000cm 2, 500cm 2, 100cm 2, 50cm 2, 10cm 2, or 5 cm ² less than Or they may have an equal footprint (which may refer to the lateral cross-sectional area surrounded by moving objects). Conversely, the footprint is approximately ^{32,000cm 2, 20,000cm 2, 10,000cm 2} , 1,000cm 2, 500cm 2, 100cm 2, 50cm 2, 10cm 2, or may be greater than 5 cm ^2, or It may be equal to them.

In some examples, the movable object can weigh less than 1000 kg. The weights of movable objects are approximately 1000 kg, 750 kg, 500 kg, 200 kg, 150 kg, 100 kg, 80 kg, 70 kg, 60 kg, 50 kg, 45 kg, 40 kg, 35 kg, 30 kg, 25 kg, 20 kg, 15 kg, 12 kg, 10 kg, 9 kg, 8 kg, 7 kg. , 6 kg, 5 kg, 4 kg, 3 kg, 2 kg, 1 kg, 0.5 kg, 0.1 kg, 0.05 kg, or 0.01 kg may be lighter or equal to them. On the contrary, the weights are about 1000 kg, 750 kg, 500 kg, 200 kg, 150 kg, 100 kg, 80 kg, 70 kg, 60 kg, 50 kg, 45 kg, 40 kg, 35 kg, 30 kg, 25 kg, 20 kg, 15 kg, 12 kg, 10 kg, 9 kg, 8 kg, 7 kg. , 6 kg, 5 kg, 4 kg, 3 kg, 2 kg, 1 kg, 0.5 kg, 0.1 kg, 0.05 kg, or 0.01 kg may be heavier or equal to them.

In some embodiments, the movable object may be relatively small relative to the payload mounted by the movable object. The load may include a load and / or a support mechanism as described in more detail below. In some examples, the ratio of the weight of the moving object to the load may be greater than, less than, or equal to approximately 1: 1. In some examples, the ratio of the weight of the moving object to the load may be greater than, less than, or equal to approximately 1: 1. Optionally, the weight ratio of the support mechanism to the load may be greater than, less than, or equal to approximately 1: 1. When desired, the ratio of the weight of the moving object to the load may be less than, or equal to, or even more than 1: 2, 1: 3, 1: 4, 1: 5, 1:10. It may be small. Conversely, the ratio of the weight of a moving object to a load may be greater than, equal to, or greater than 2: 1, 3: 1, 4: 1, 5: 1, 10: 1. But it is possible.

In some embodiments, the moving object can be of low energy consumption. For example, the movable object may use a smaller amount than about 5 W / h, 4 W / h, 3 W / h, 2 W / h, and 1 W / h, or even a smaller amount. In some examples, the supporting mechanism of the moving object can be of low energy consumption. For example, the support mechanism may use a smaller amount than approximately 5 W / h, 4 W / h, 3 W / h, 2 W / h, and 1 W / h, or even a smaller amount. Optionally, the load of movable objects may be of low energy consumption, such as approximately 5 W / h, 4 W / h, 3 W / h, 2 W / h, less than 1 W / h, or even less.

FIG. 17 illustrates an unmanned aerial vehicle (UAV) 1700 according to a plurality of embodiments of the present invention. The UAV can be an example of a moving object as described herein. The UAV1700 may include a propulsion system with four rotors 1702, 1704, 1706 and 1708. Any number of rotors (eg, 1, 2, 3, 4, 5, 6, or more) may be provided. Multiple rotors, rotor assemblies, or other propulsion systems of an unmanned aerial vehicle may allow the unmanned aerial vehicle to hover / maintain position, change orientation, and / or change location. The distance between the shafts of the rotors facing each other can be any suitable length 1710. For example, the length 1710 may be shorter than, or equal to, less than 2 m, less than 5 m, or equal. In some embodiments, the length 1710 can be in the range of 40 cm to 1 m, 10 cm to 2 m, or 5 cm to 5 m. Any description of a UAV herein may apply to moving objects such as different types of moving objects, and vice versa.

In some embodiments, the movable object may be configured to carry a load. The load may include one or more of passengers, cargo, equipment, equipment, and others. The load can be provided in the housing. The housing can be separate from the housing of the movable object and can be part of the housing for the movable object. Alternatively, the payload may be provided with a housing, while the movable object does not have a housing. Alternatively, multiple parts of the load, or the entire load, can be provided without a housing. The load can be firmly fixed to a moving object. Optionally, the payload can be movable relative to the moving object (eg, translatable or rotatable relative to the moving object).

In some embodiments, the load includes a load. The load may be configured to perform no action or function. Alternatively, the load can be a load that is configured to perform an operation or function also known as a functional load. For example, the load may include one or more sensors for investigating one or more targets. Some suitable sensor, such as an imaging device (eg, a camera), an audio capture device (eg, a parabolic microphone), an infrared imaging device, or an ultraviolet imaging device, may be incorporated into the load. The sensor may provide static sensing data (eg, photo) or dynamic sensing data (eg, video). In some embodiments, the sensor provides sensing data about the target of the load. Alternatively or in combination, the load may include one or more emitters for delivering multiple signals to one or more targets. Any suitable emitter, such as an illumination light source or sound source, may be used. In some embodiments, the load comprises one or more transmitters and receivers, such as for communication with a module that is remote from the moving object. Optionally, the load may be configured to interact with the environment or target. For example, the load may include a tool, instrument, or mechanism such as a robot arm capable of manipulating multiple objects.

Optionally, the payload may include a support mechanism. The support mechanism can be provided for the load, which is either directly (eg, in direct contact with a moving object) or indirectly (eg, without contact with a movable object) via the support mechanism. Can be combined with a moving object. Conversely, the load can be mounted on a moving object without the need for a support mechanism. The load can be formed integrally with the support mechanism. Alternatively, the load may be releasably coupled to the support mechanism. In some embodiments, the load may include one or more load elements, one or more of the plurality of load elements with a moving object and / or support mechanism as described above. It may be possible to move relatively.

The support mechanism can be formed integrally with the movable object. Alternatively, the support mechanism can be releasably coupled to the moving object. The support mechanism can be directly or indirectly coupled to the moving object. The support mechanism may provide support for the load (eg, carry at least a portion of the weight of the load). The support mechanism may include a suitable mounting structure (eg, a gimbal platform) capable of stabilizing and / or directing the movement of the load. In some embodiments, the support mechanism may be adapted to control the state of the load relative to the moving object (eg, position and / or orientation). For example, the support mechanism is relative to the moving object (eg, 1, 2, or 3) so that the load maintains its position and / or orientation relative to the appropriate frame of reference regardless of the movement of the moving object. It can be configured to move in translation (with respect to translation and / or 1, 2, or 3 rotations). The reference system can be a fixed reference system (eg, the surrounding environment). Alternatively, the frame of reference can be a moving frame of reference (eg, a moving object, a load target).

In some embodiments, the support mechanism may be configured to allow movement of the support mechanism and / or moving objects and relative loads. The movement is translation for up to 3 degrees of freedom (eg, along 1, 2, or 3 axes), rotation for up to 3 degrees of freedom (eg, around 1, 2, or 3 axes), or any of these. It can be the right combination.

In some examples, the support mechanism may include a support mechanism frame assembly and a support mechanism actuating assembly. The support mechanism frame assembly may provide structural support for the load. The support mechanism frame assembly may include multiple separate support mechanism frame components, some of which may be capable of moving relative to each other. The support mechanism actuating assembly may include one or more actuators (eg, motors) that actuate the movement of a plurality of individual support mechanism frame components. The plurality of actuators may allow the movement of the plurality of support mechanism frame components at the same time, or may be configured to allow the movement of a single support mechanism frame component at one time. The movement of multiple support mechanism frame components can result in the corresponding movement of the load. For example, a support mechanism actuating assembly may actuate the rotation of one or more support mechanism frame components around one or more rotation axes (eg, roll axis, pitch axis, or yaw axis). Rotation of one or more support mechanism frame components may rotate the load around one or more axes of rotation relative to the moving object. Alternatively or in combination, the support mechanism actuating assembly may actuate the translation of one or more support mechanism frame components along one or more translational axes, whereby one or more relative to the moving object. Translation of the load along multiple corresponding axes can occur.

In some embodiments, the movement of moving objects, support mechanisms, and loads relative to a fixed frame of reference (eg, the surrounding environment) and / or relative to each other can be controlled by a terminal. The terminal can be a remote control device at a location away from moving objects, support mechanisms, and / or loads. The terminal can be placed on or attached to a supporting platform. Alternatively, the terminal can be a handheld device or a wearable device. For example, the terminal may include a smartphone, tablet, laptop, computer, eyeglasses, gloves, helmet, microphone, or any suitable combination thereof. The terminal may include a user interface such as a keyboard, mouse, joystick, touch screen, or display. Some appropriate user input, such as manually entered commands, voice control, gesture control, position control (eg, via terminal movement, location, or tilt), can be used to interact with the terminal.

The terminal can be used to control any suitable state of the moving object, support mechanism, and / or load. For example, terminals can be used to control positions and / or orientations relative to and / or relative to a fixed reference system of moving objects, support mechanisms, and / or loads. In some embodiments, the terminal is used to control multiple individual elements of the load, such as moving objects, support mechanisms, and / or working assemblies of the support mechanism, load sensors, load emitters, and the like. Can be used. The terminal may include a wireless communication device adapted to communicate with one or more of a moving object, a support mechanism, or a load.

The terminal may include a suitable display unit for viewing information on movable objects, support mechanisms, and / or loads. For example, the terminal may be configured to display information on moving objects, support mechanisms, and / or loads with respect to position, translational velocity, translational acceleration, orientation, angular velocity, angular acceleration, or any suitable combination thereof. In some embodiments, the terminal may display information provided by the load, such as data provided by the functional load (eg, multiple images recorded by a camera or other imaging device).

Optionally, the same terminal controls the state of the movable object, support mechanism, and / or load, or movable object, support mechanism, and / or load, and in addition, the movable object, support mechanism, and / or load. Can receive and / or display information from objects. For example, the terminal may control the positioning of the load relative to the environment while displaying the image data captured by the load, or information about the position of the load. Alternatively, different terminals may be used for different functions. For example, a first terminal may control the movement or state of a movable object, support mechanism, and / or load, while a second terminal receives and / or receives information from a movable object, support mechanism, and / or load. Or can be displayed. For example, a first terminal can be used to control the positioning of the load relative to the environment, while a second terminal can display image data captured by the load. Between a movable object and an integrated terminal that both controls the movable object and receives data, or between a movable object and multiple terminals that both control the movable object and receive data. Therefore, various communication modes can be used. For example, at least two different modes of communication may be formed between a moving object and a terminal that both controls the moving object and receives data from the moving object.

FIG. 18 illustrates a movable object 1800 including a support mechanism 1802 and a load 1804 according to a plurality of embodiments. Although the movable object 1800 is described as an aircraft, this description is not intended to be limited and any suitable type of movable object may be used as previously described herein. Those skilled in the art will appreciate that any of the embodiments described herein in the context of multiple aircraft systems may be applied to some suitable movable object (eg, UAV). In some examples, the load 1804 may be provided on the movable object 1800 without the need for a support mechanism 1802. The movable object 1800 may include a plurality of propulsion mechanisms 1806, a sensing system 1808, and a communication system 1810.

Multiple propulsion mechanisms 1806 include multiple rotors, multiple propellers, multiple blades, multiple engines, multiple motors, multiple wheels, multiple axles, multiple magnets, multiple nozzles, as previously described. It may include one or more of them. For example, the plurality of propulsion mechanisms 1806 can be a plurality of self-tightening rotors, a plurality of rotor assemblies, or a plurality of other rotary propulsion units, as disclosed elsewhere herein. The movable object may have one or more, two or more, three or more, four or more propulsion mechanisms. Multiple propulsion mechanisms can all be of the same type. Alternatively, the one or more propulsion mechanisms can be different types of propulsion mechanisms. The plurality of propulsion mechanisms 1806 may be mounted on the movable object 1800 using some suitable means such as support elements (eg, drive shafts) as described elsewhere herein. The plurality of propulsion mechanisms 1806 may be mounted on any suitable part of the movable object 1800, such as a top, a bottom, a front, a rear, a plurality of sides, or any suitable combination thereof.

In some embodiments, the plurality of propulsion mechanisms 1806 take off the movable object 1800 vertically from the surface without requiring any horizontal movement of the movable object 1800 (eg, without gliding on the runway). Or it may be possible to land vertically on the surface. Optionally, the plurality of propulsion mechanisms 1806 may be operational to allow the movable object 1800 to hover in the air at a particular position and / or orientation. One or more of the plurality of propulsion mechanisms 1800 may be controlled independently of the other plurality of propulsion mechanisms. Alternatively, the plurality of propulsion mechanisms 1800 may be configured to be controlled simultaneously. For example, the movable object 1800 may have a plurality of horizontally oriented rotors capable of providing lift and / or thrust to the movable object. A plurality of horizontally oriented rotors may be actuated to provide vertical takeoff, vertical landing, and hovering capabilities to the movable object 1800. In some embodiments, one or more of the horizontally oriented rotors rotate clockwise while one or more of the horizontal rotors may rotate counterclockwise. obtain. For example, the number of clockwise rotors may be equal to the number of counterclockwise rotors. The rotational speed of each of the horizontally oriented rotors is to control the lift and / or thrust produced by each rotor, and thereby (eg, up to 3 translations, and up to 3). It can be independently varied to adjust the spatial arrangement, velocity, and / or acceleration of the movable object 1800 (with respect to one degree of rotation).

The sensing system 1808 may include one or more sensors capable of sensing the spatial arrangement, velocity and / or acceleration of the moving object 1800 (eg, with respect to up to 3 translations and up to 3 rotations). One or more sensors may include a Global Positioning System (GPS) sensor, a motion sensor, an inertial sensor, a proximity sensor, or an image sensor. The sensing data provided by the sensing system 1808 is the spatial placement, velocity, and / or orientation of the movable object 1800 (eg, using the appropriate processing unit and / or control module, as described below). Can be used to control. Alternatively, the sensing system 1808 describes the environment surrounding moving objects, such as weather conditions, proximity to possible obstacles, locations of multiple geographic features, locations of multiple man-made structures, and more. It can be used to provide data.

The communication system 1810 enables communication with a terminal 1812 having the communication system 1814 via a plurality of radio signals 1816. Communication systems 1810, 1814 may include any number of transmitters, receivers, and / or transmitters and receivers suitable for wireless communication. Communication can be one-way communication such that data can be transmitted in only one direction. For example, one-way communication can involve only the movable object 1800 transmitting data to the terminal 1812, and vice versa. Data may be transmitted from one or more transmitters of communication system 1810 to one or more receivers of communication system 1812, and vice versa. Alternatively, the communication can be bidirectional so that data can be transmitted between the movable object 1800 and the terminal 1812 in both directions. Bidirectional communication can involve transmitting data from one or more transmitters of communication system 1810 to one or more receivers of communication system 1814, and vice versa.

In some embodiments, the terminal 1812 may provide control data for one or more of the movable object 1800, the support mechanism 1802, and the load 1804, the movable object 1800, the support mechanism 1802, and the load 1804. Information may be received from one or more of the loads (eg, data sensed by the load, such as moving object, support mechanism or load position and / or motion information, image data captured by the load's camera). In some examples, the control data from the terminal is a plurality of relative positions of moving objects, support mechanisms, and / or loads, multiple movements, multiple operations, or multiple instructions for multiple controls. May include. For example, control data can modify the location and / or orientation of the movable object (eg, through the control of multiple propulsion mechanisms 1806), or the load relative to the movable object (eg, support). It can move (via the control of mechanism 1802). Control data from the terminal controls the behavior of the camera or other imaging device (eg, capturing still or moving images, zooming in or out, powering on / off, switching between multiple imaging modes, changing image resolution, changing focus). , Changes in depth of field, changes in exposure time, changes in viewing angle or visibility), etc. can result in control of the load. In some examples, multiple communications from movable objects, support mechanisms, and / or loads may include information from one or more sensors (eg, of sensing systems 1808 or of loads 1804). The communication may include information sensed from one or more different types of sensors (eg, GPS sensors, motion sensors, inertial sensors, proximity sensors, or image sensors). Such information may relate to the position (eg, location, orientation), movement, or acceleration of moving objects, support mechanisms, and / or loads. Such information from the load may include data captured by the load, or the perceived state of the load. The control data transmitted and provided by the terminal 1812 may be configured to control one or more states of the movable object 1800, the support mechanism 1802, or the load 1804. Alternatively or in combination, the support mechanism 1802 and the load 1804 control each of the terminals 1812 so that they can communicate independently of the movable object 1800, the support mechanism 1802, and the load 1804, respectively. It may include a communication module configured to communicate with the terminal as much as possible.

In some embodiments, the movable object 1800 may be configured to communicate with other remote devices in addition to or on behalf of the terminal 1812. The terminal 1812 may also be configured to communicate not only with the movable object 1800 but also with other remote devices. For example, the movable object 1800 and / or the terminal 1812 may communicate with another movable object, or a support mechanism or load of the other movable object. When desired, the remote device can be a second terminal or other computing device (eg, a computer, laptop, tablet, smartphone, or other mobile device). The remote device may be configured to transmit data to the movable object 1800, receive data from the movable object 1800, transmit data to terminal 1812, and / or receive data from terminal 1812. Optionally, the remote device may be connected to the Internet or other telecommunications network so that data received from the movable object 1800 and / or terminal 1812 can be uploaded to a website or server.

FIG. 19 is a schematic view using a block diagram of a system 1900 for controlling a movable object according to a plurality of embodiments. System 1900 can be used in combination with any suitable embodiment of the plurality of systems, devices, and methods disclosed herein. The system 1900 may include a sensing module 1902, a processing unit 1904, a non-temporary computer readable medium 1906, a control module 1908, and a communication module 1910.

Sensing module 1902 may utilize different types of sensors that collect information related to multiple moving objects in different ways. Different types of sensors may sense different types of signals, or signals from different sources. For example, the plurality of sensors may include inertial sensors, GPS sensors, proximity sensors (eg, riders), or vision / image sensors (eg, cameras). The sensing module 1902 may be operably coupled to a processing unit 1904 having multiple processors. In some embodiments, the sensing module can be operated on a transmitting module 1912 (eg, a Wi-Fi® image transmitting module) configured to transmit the sensing data directly to the appropriate external device or system. Can be combined with. For example, the transmission module 1912 can be used to transmit a plurality of images captured by the camera of the sensing module 1902 to a remote terminal.

The processing unit 1904 may have one or more processors such as a programmable processor (eg, a central processing unit (CPU)). The processing unit 1904 may be operably coupled to a non-transitory computer-readable medium 1906. Non-temporary. A computer-readable medium 1906 may store logic, code, and / or program instructions that can be executed by the processing unit 1904 to perform one or more steps. A non-temporary computer-readable medium is one. Alternatively, it may include multiple memory units (eg, a removable medium such as an SD card or random access memory (RAM) or external storage). In some embodiments, the data from the sensing module 1902 is non-transient computer readable. The plurality of memory units of the non-transitory computer-readable medium 1906 can be transmitted directly to and stored in the plurality of memory units of the medium 1906. The processing unit 1904 may store logic, code, and / or a plurality of program instructions that can be executed by the processing unit 1904 to execute the embodiment. For example, the processing unit 1904 may be stored in one or more processors of the processing unit 1904 by a sensing module. It may be configured to execute multiple instructions to analyze the generated sensed data. Multiple memory units may store sensed data from a sensed module that will be processed by the processing unit 1904. Some implementations. In the embodiment, the plurality of memory units of the non-temporary computer-readable medium 1906 can be used to store the plurality of processing results generated by the processing unit 1904.

In some embodiments, the processing unit 1904 may be operably coupled to a control module 1908 configured to control the state of a moving object. For example, the control module 1908 may be configured to control a plurality of propulsion mechanisms of a movable object to adjust the spatial arrangement, velocity, and / or acceleration of the movable object with respect to 6 degrees of freedom. Alternatively or in combination, the control module 1908 may control one or more of the states of the support mechanism, load, or sensing module.

The processing unit 1904 can operate on a communication module 1910 configured to send and / or receive data from one or more external devices (eg, a terminal, display device, or other remote controller). Can be combined with. Any suitable means of communication, such as wired or wireless communication, may be used. For example, the communication module 1910 can be a local area network (LAN), wide area network (WAN), infrared, wireless, WiFi®, point-to-point (P2P) network, long-distance communication network, cloud communication, and others. One or more of them may be used. Optionally, multiple relay stations such as towers, satellites, and mobile stations can be used. Multiple radio communications can be accessibility-dependent or accessibility-independent. In some embodiments, line-of-sight may or may not be required for multiple communications. The communication module 1910 may include sensing data from sensing module 1902, multiple processing results generated by processing unit 1904, predetermined control data, multiple user commands from a terminal or remote controller, and one or more of others. Can be sent and / or they can be received.

Multiple components of system 1900 may be arranged in some suitable configuration. For example, one or more of the components of System 1900 may be on a moving object, support mechanism, load, terminal, sensing system, or additional external device communicating with one or more of the above. Can be placed in. Additionally, FIG. 19 depicts a single processing unit 1904 and a single non-transient computer-readable medium 1906, but this is not intended to be limiting, and the system 1900 has multiple processes. Those skilled in the art will appreciate that units and / or non-transitory computer-readable media may be included. In some embodiments, one or more of the plurality of processing units and / or non-transitory computer-readable media are movable objects, supporting any suitable aspect of the processing and / or memory function performed by the system 1900. Occurs at one or more of different locations, such as on mechanisms, loads, terminals, sensing modules, additional external devices communicating with one or more of the above, or on appropriate combinations of these. It can be located in multiple locations as described above to obtain.

As used herein, A and / or B includes one or more of A or B, combinations thereof, such as A and B.

Although preferred embodiments of the present invention have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. One of ordinary skill in the art can come up with a number of modifications, modifications, and substitutions without departing from the invention of the present application. It should be understood that various alternatives of the plurality of embodiments of the present invention described herein may be employed in carrying out the present invention. Many different combinations of the plurality of embodiments described herein are possible and such combinations are considered part of this disclosure. In addition, all features described in the context of any one embodiment herein may be readily adapted for use in a plurality of other embodiments herein. It is intended that the following claims define the scope of the present invention, and that a plurality of methods and structures within the scope of these claims and their equivalents are covered by them.
(Item 1)
A method for determining initialization information about moving objects using multiple sensors.
A step of detecting that the movable object has started operation with the assistance of one or more processors, and
A step of receiving inertial data from at least one inertial sensor mounted by the movable object in the one or more processors.
A step of receiving image data from at least two image sensors mounted by the movable object in the one or more processors.
It comprises a step of determining the initialization information about the movable object based on the inertial data and the image data with the assistance of the one or more processors.
The initialization information includes one or more of the position of the movable object, the velocity of the movable object, or the orientation of the movable object with respect to the direction of gravity when the movable object starts to move.
(Item 2)
The method of item 1, wherein the movable object is an unmanned aerial vehicle (UAV).
(Item 3)
The method according to item 2, wherein the step of detecting that the UAV has started operation comprises detecting that the UAV has started flight.
(Item 4)
The method according to item 3, wherein the UAV starts flight from a free fall state.
(Item 5)
The method according to item 3, wherein the UAV starts a flight by being thrown by a user.
(Item 6)
The method according to item 2, wherein the step of detecting that the UAV has started operation includes a step of detecting that the UAV has taken off from a surface.
(Item 7)
The method according to item 6, wherein the surface is an inclined surface.
(Item 8)
The method according to item 6, wherein the surface is a non-sloping surface.
(Item 9)
The method according to item 1, wherein the inertial data includes one or more measured values indicating the three-dimensional acceleration and the three-dimensional angular velocity of the movable object.
(Item 10)
The method according to item 1, wherein the inertial data includes one or more measured values acquired by the at least one inertial sensor over a time interval starting from the time the movable object starts moving.
(Item 11)
The method according to item 1, wherein the image data includes one or more images of the environment around the movable object.
(Item 12)
The method according to item 1, wherein the image data includes one or more images acquired by each of the at least two image sensors over a time interval starting from the time the movable object starts moving.
(Item 13)
The method of item 12, wherein the step of determining comprises processing the one or more images using a feature point detection algorithm.
(Item 14)
The method of item 12, wherein the step of determining comprises processing the one or more images using an optical flow algorithm.
(Item 15)
12. The method of item 12, wherein the step of determining comprises processing the one or more images using a feature matching algorithm.
(Item 16)
The step of determining is one or more images acquired by the first image sensor of the at least two image sensors and one or more images acquired by the second image sensor of the at least two image sensors. 12. The method of item 12, comprising a step of comparing with.
(Item 17)
The method according to item 1, wherein the at least two image sensors are configured to synchronously acquire a plurality of images.
(Item 18)
The method according to item 1, wherein the at least two image sensors are configured to asynchronously acquire a plurality of images.
(Item 19)
The at least two image sensors include a plurality of image sensor subsets that are switchably coupled to the one or more processors via a switching mechanism, the switching mechanism simultaneously combining a single image sensor subset with the above one. Or the method of item 1, which is configured to be coupled to a plurality of processors.
(Item 20)
The above step of receiving the above image data is
The step of coupling the first image sensor subset to the one or more processors via the switching mechanism, and
The step of receiving image data from the first image sensor subset and
The step of coupling the second image sensor subset to the one or more processors via the switching mechanism, and
19. The method of item 19, comprising the step of receiving image data from the second image sensor subset.
(Item 21)
The method according to item 1, wherein the initialization information further includes the position and speed of the movable object when the movable object starts to move.
(Item 22)
The above steps to determine
The step of generating the first estimated value of the initialization information using the inertial data, and
A step of generating a second estimated value of the initialization information using the image data, and
The method according to item 1, further comprising a step of acquiring the initialization information regarding the movable object by combining the first and second estimated values.
(Item 23)
The method according to item 1, wherein the initialization information is determined using a linear optimization algorithm.
(Item 24)
The method according to item 1, wherein the initialization information is determined based only on the inertial data and the image data.
(Item 25)
The method of item 1, further comprising a step of correcting subsequent inertial data acquired from the at least one inertial sensor using the orientation of the movable object with respect to the direction of gravity.
(Item 26)
It is a system for determining initialization information about moving objects using multiple sensors.
With at least one inertial sensor mounted by the movable object,
At least two image sensors mounted by the movable object,
Equipped with one or more of the above processors
The one or more processors mentioned above, individually or collectively,
Detecting that the above movable object has started to move,
Inertia data is received from at least one of the above inertial sensors,
Image data is received from at least the above two image sensors,
It is configured to determine the initialization information about the movable object based on the inertial data and the image data.
The initialization information includes one or more of the position of the movable object, the velocity of the movable object, or the orientation of the movable object with respect to the direction of gravity when the movable object starts operation.
(Item 27)
One or more non-temporary computer-readable storage media containing multiple executable instructions.
The plurality of executable instructions to the computer system, at least when executed by the one or more processors of the computer system for determining initialization information about a moving object,
Detect that the above movable object has started to move,
Inertia data is received from at least one inertial sensor mounted by the movable object.
Image data is received from at least two image sensors mounted by the movable object.
The initialization information regarding the movable object is determined based on the inertial data and the image data.
The initialization information includes one or more of the position of the movable object, the velocity of the movable object, or the direction of the movable object with respect to the direction of gravity when the movable object starts to move. Non-temporary computer-readable storage medium.
(Item 28)
A method for performing error recovery on moving objects using multiple sensors.
A step of detecting that an error has occurred during the operation of the movable object with the assistance of one or more processors, and
A step of receiving inertial data from at least one inertial sensor mounted by the movable object in the one or more processors.
A step of receiving image data from at least two image sensors mounted by the movable object in the one or more processors.
A step of determining reinitialization information about the movable object based on the inertial data and the image data with the assistance of the one or more processors is provided.
The method, wherein the reinitialization information includes one or more of the position of the movable object, the velocity of the movable object, or the orientation of the movable object with respect to the direction of gravity when the error occurs.
(Item 29)
28. The method of item 28, wherein the movable object is an unmanned aerial vehicle (UAV).
(Item 30)
28. The method of item 28, wherein the error includes a malfunction in one or more of the at least one inertial sensor or the at least two image sensors.
(Item 31)
28. The method of item 28, wherein the error includes a malfunction in the state estimation module used by the movable object.
(Item 32)
The method according to item 31, wherein the state estimation module uses an iterative state estimation algorithm, and the malfunction includes the failure of the iterative state estimation algorithm to converge to a solution.
(Item 33)
32. The method of item 32, further comprising a step of reinitializing the iterative state estimation algorithm with the reinitialization information.
(Item 34)
28. The method of item 28, wherein the inertial data comprises one or more measurements indicating the three-dimensional acceleration and the three-dimensional angular velocity of the moving object.
(Item 35)
28. The method of item 28, wherein the inertial data comprises one or more measurements acquired by the at least one inertial sensor over a time interval starting from the time the error occurs.
(Item 36)
28. The method of item 28, wherein the image data includes one or more images of the environment surrounding the movable object.
(Item 37)
28. The method of item 28, wherein the image data comprises one or more images acquired by each of the at least two image sensors over a time interval starting from the time the error occurs.
(Item 38)
37. The method of item 37, wherein the step of determining comprises processing the one or more images using a feature point detection algorithm.
(Item 39)
37. The method of item 37, wherein the step of determining comprises processing the one or more images using an optical flow algorithm.
(Item 40)
37. The method of item 37, wherein the step of determining comprises processing the one or more images using a feature matching algorithm.
(Item 41)
The step of determining is one or more images acquired by the first image sensor of the at least two image sensors and one or more images acquired by the second image sensor of the at least two image sensors. 37. The method of item 37, comprising a step of comparing with.
(Item 42)
28. The method of item 28, wherein the at least two image sensors are configured to synchronously acquire a plurality of images.
(Item 43)
28. The method of item 28, wherein the at least two image sensors are configured to asynchronously acquire a plurality of images.
(Item 44)
The at least two image sensors include a plurality of image sensor subsets that are switchably coupled to the one or more processors via a switching mechanism, the switching mechanism simultaneously combining a single image sensor subset with the above one. 28. The method of item 28, which is configured to be coupled to multiple processors.
(Item 45)
The above step of receiving the above image data is
The step of coupling the first image sensor subset to the one or more processors via the switching mechanism, and
The step of receiving image data from the first image sensor subset and
The step of coupling the second image sensor subset to the one or more processors via the switching mechanism, and
It has a step of receiving image data from the second image sensor subset.
The method according to item 44.
(Item 46)
28. The method of item 28, wherein the reinitialization information further includes the position and velocity of the movable object when the error occurs.
(Item 47)
The above steps to determine
The step of generating the first estimated value of the reinitialization information using the inertial data, and
A step of generating a second estimated value of the reinitialization information using the image data, and
28. The method of item 28, comprising combining the first and second estimates to obtain the reinitialization information for the movable object.
(Item 48)
28. The method of item 28, wherein the reinitialization information is evaluated using a linear optimization algorithm.
(Item 49)
28. The method of item 28, wherein the reinitialization information is evaluated based solely on the inertial data and the image data.
(Item 50)
28. The method of item 28, further comprising a step of correcting subsequent inertial data acquired from the at least one inertial sensor using the orientation of the movable object with respect to the direction of gravity.
(Item 51)
A system for performing error recovery on moving objects using multiple sensors.
With at least one inertial sensor mounted by the movable object,
At least two image sensors mounted by the movable object,
Equipped with one or more processors
The one or more processors mentioned above, individually or collectively,
Detecting that an error has occurred during the movement of the above movable object,
Inertia data is received from at least one of the above inertial sensors,
Image data is received from at least the above two image sensors,
It is configured to determine the reinitialization information for the movable object based on the inertial data and the image data.
The reinitialization information includes one or more of the position of the movable object, the velocity of the movable object, or the orientation of the movable object with respect to the direction of gravity when an error occurs.
(Item 52)
One or more non-temporary computer-readable storage media containing multiple executable instructions.
The plurality of executable instructions are at least in the computer system when executed by the one or more processors of the computer system for performing error recovery on a moving object.
Detect that an error has occurred during the operation of the above movable object,
Inertia data is received from at least one inertial sensor mounted by the movable object.
Image data is received from at least two image sensors mounted by the movable object.
Reinitialization information about the movable object is determined based on the inertial data and the image data.
The reinitialization information includes one or more non-temporary objects including one or more of the position of the movable object, the velocity of the movable object, or the direction of the movable object with respect to the direction of gravity when the error occurs. Computer-readable storage medium.

Claims (15)

A method for performing error recovery on moving objects that are unmanned aerial vehicles (UAVs) using multiple sensors.
A step of detecting that an error has occurred during the flight of the moving object with the assistance of one or more processors.
A step of receiving inertial data from at least one inertial sensor mounted by the movable object over a time interval starting from the free fall state after the error has occurred.
A step of receiving image data from at least two image sensors carried by the movable object in the one or more processors over a time interval starting from the time the error occurs.
Based on the inertial data and the image data received over a time interval starting from the free fall state after the error occurred, during the movement of the movable object after the error occurred. Including the step of determining an estimate of the reinitialization information for the moving object using an iterative optimization algorithm.
The method, wherein the estimated value of the reinitialization information includes the position of the movable object, the velocity of the movable object, and the orientation of the movable object with respect to the direction of gravity, indicating the state of the movable object when the error occurs. The method of claim 1, wherein the error comprises a malfunction in one or more of the at least one inertial sensor and the at least two image sensors. The method of claim 1 or 2 , wherein the error includes a malfunction in the state estimation module used by the movable object. The method according to claim 3 , wherein the state estimation module uses the iterative optimization algorithm, and the malfunction includes the failure of the iterative optimization algorithm to converge to a solution. The method of claim 4 , further comprising a step of reinitializing the iterative optimization algorithm using the reinitialization information. The method according to any one of claims 1 to 5 , wherein the inertial data includes one or more measured values indicating a three-dimensional acceleration and a three-dimensional angular velocity of the movable object. The method according to any one of claims 1 to 6 , wherein the image data includes one or more images of the environment around the movable object. The step feature point detection algorithm, optical flow algorithm or with the step of processing the one or more images using a feature matching algorithm A method according to either one of claims 1 to 7, the determining. The step of determining is one or more images acquired by the first image sensor of the at least two image sensors and one or more images acquired by the second image sensor of the at least two image sensors. The method according to any one of claims 1 to 8 , which comprises a step of comparison with. The step of receiving the image data is
The step of coupling the first image sensor subset to the one or more processors via a switching mechanism.
A step of receiving image data from the first image sensor subset,
The step of coupling the second image sensor subset to the one or more processors via the switching mechanism.
It comprises a step of receiving image data from the second image sensor subset.
The method according to any one of claims 1 to 9 . The step to determine is
The step of generating the first estimated value of the reinitialization information using the inertial data, and
A step of generating a second estimated value of the reinitialization information using the image data, and
The method according to any one of claims 1 to 10 , comprising a step of combining the first estimated value and the second estimated value to obtain an estimated value of the reinitialization information regarding the movable object. The step of determining is based solely on the inertial data and the image data received over a time interval starting from the free fall state after the error has occurred, based on the estimated value of the reinitialization information. The method according to any one of claims 1 to 11 , comprising the step of determining using the iterative optimization algorithm. The method according to any one of claims 1 to 12 , further comprising a step of correcting inertial data acquired from the at least one inertial sensor using the orientation of the movable object with respect to the direction of gravity. A system for performing error recovery on moving objects using multiple sensors.
With at least one inertial sensor carried by the movable object,
At least two image sensors carried by the movable object,
Equipped with one or more processors
The one or more processors performs a method according to any one of claims 1 13, system. A program for causing a computer to execute the method according to any one of claims 1 to 13 .