JP6487010B2 - Method for controlling an unmanned aerial vehicle in a certain environment, method for generating a map of a certain environment, system, program, and communication terminal - Google Patents

Description

  Unmanned aerial vehicles, such as unmanned aerial vehicles (UAVs), can be used to perform surveillance, reconnaissance, and search missions in a wide variety of military and civilian environments. The UAV can be manually controlled by a remote user, or can operate in a semi-autonomous manner, or in a fully autonomous manner. Such a UAV may include a sensor configured to collect data from the surrounding environment.

  Existing approaches to obtaining environmental data may not be the best in some cases. For example, the accuracy of environmental data can be limited based on the performance of the particular sensor type used to collect the data. Inaccurate environmental data can adversely affect UAV functionality.

  The embodiments disclosed herein provide improved approaches for controlling multiple movable objects, such as multiple UAVs, within an environment. In many embodiments, the UAV includes a number of different sensor types that are used to gather information about the surrounding environment. Data obtained from each of a plurality of different sensor types can be combined to generate a representation of multiple surrounding environments, such as a two-dimensional (2D) or three-dimensional (3D) environment map, Can be used to facilitate navigation, object recognition and obstacle avoidance. Advantageously, the multiple approaches described herein can be used to improve UAV functionality in various environmental types and operating situations.

  As described below, in one aspect, a method for controlling a movable object within an environment is provided. The method includes determining an initial location of the movable object using at least one of a plurality of sensors carried by the movable object, and generating a first signal that causes the movable object to navigate in the environment. Receiving sensing data associated with the environment using at least one of the plurality of sensors, generating an environment map representing at least a portion of the environment based on the sensing data, and an initial location Receiving an instruction to go back to, and generating a second signal that causes the movable object to return to the initial location based on the environment map.

  In some embodiments, the movable object is an unmanned aerial vehicle. An unmanned aerial vehicle may have a weight of 10 kg or less. The maximum size of the unmanned aerial vehicle may be 1.5 m or less. Unmanned aerial vehicles can be configured to fly at a height of 400 meters or less. Optionally, the unmanned aerial vehicle can be configured to detect the presence of a flight restriction area and not fly within a predetermined distance of the flight restriction area. The flight restriction area may be an airport. The unmanned aerial vehicle may be a multi-rotor aircraft.

  In some embodiments, the environment may be an indoor environment or a low altitude environment.

  In some embodiments, the plurality of sensors may include a global positioning system (GPS) sensor, a vision sensor, or a proximity sensor. The proximity sensor may include at least one of the following lidar sensor, ultrasonic sensor, or in-flight camera sensor. The plurality of sensors may include a plurality of different sensor types.

  In some embodiments, the environment map may include a topological map or a metric map. The metric map may include at least one of the following point cloud, 3D grid map, 2D grid map, or 2.5D grid map. Optionally, the metric map may include an occupancy grid map.

  In some embodiments, generating the second signal includes determining the current location of the movable object using at least one of the plurality of sensors, and initializing from the current location based on the environment map. Determining a route to the location of the device, and generating a signal that causes the movable object to move along the route back to the initial location. Determining the route may include determining the shortest route from the current location to the initial location. Alternatively, or in combination, determining the route can include determining a route from the current location to the initial location that avoids one or more obstacles in the environment. The path can include one or more portions through which the movable object has previously passed. The path may be different from the path through which the movable object has previously passed. Optionally, the path may be a flight path of a movable object. The path can include the spatial position and direction of the movable object.

  In some embodiments, the route includes a plurality of passage points corresponding to a plurality of positions where the movable object has previously passed, the plurality of passage points being recorded as the movable object navigates within the environment. A plurality of passing points can be recorded in real time as the movable object navigates through the environment. Alternatively, the plurality of passing points can be recorded at predetermined time intervals as the movable object navigates within the environment. Multiple passage points can be stored in the list data structure.

  In some embodiments, generating a signal that causes the movable object to move along the path includes detecting an obstacle positioned along the path in the environment and avoiding the obstacle. Therefore, the method can include changing the route and generating a signal for moving along the changed route to the movable object.

  In another aspect, a system for controlling a movable object within the environment is provided. The system can include a plurality of sensors carried by the movable object and one or more processors. One or more processors, individually or collectively, determine an initial location of the movable object using at least one of the plurality of sensors and generate a first signal that causes the movable object to navigate within the environment. And receiving sensing data related to the environment using at least one of the plurality of sensors, generating an environment map representing at least a part of the environment based on the sensing data, and returning to an initial location. A second signal may be configured to receive the instruction and cause the movable object to navigate to return to the initial location based on the environment map.

  In another aspect, a method for controlling an unmanned aerial vehicle in an environment is provided. The method includes generating a first signal that causes an unmanned aerial vehicle to navigate within the environment, and using a plurality of sensors carried by the unmanned aerial vehicle to receive sensing data associated with at least a portion of the environment. Generating an environment map representing at least part of the environment based on the sensing data; detecting one or more obstacles located in the part of the environment using the environment map; and environment map Generating a second signal that causes the unmanned aircraft to navigate to avoid one or more obstacles.

  In some embodiments, the unmanned aerial vehicle is a rotorcraft. An unmanned aerial vehicle may have a weight of 10 kg or less. The maximum size of the unmanned aerial vehicle may be 1.5 m or less. Unmanned aerial vehicles can be configured to fly at a height of 400 meters or less. Optionally, the unmanned aerial vehicle can be configured to detect the presence of a flight restriction area and not fly within a predetermined distance of the flight restriction area. The flight restriction area may be an airport.

  In some embodiments, the environment may be an indoor environment or a low altitude environment.

  In some embodiments, the plurality of sensors may include a global positioning system (GPS) sensor, a vision sensor, or a proximity sensor. The proximity sensor may include at least one of the following lidar sensor, ultrasonic sensor, or in-flight camera sensor. The plurality of sensors may include a plurality of different sensor types. Sensing data can include data for a plurality of different coordinate systems, and generating the environment map includes mapping the data to a single coordinate system.

  In some embodiments, the environment map may include a topological map or a metric map. The metric map may include at least one of the following point cloud, 3D grid map, 2D grid map, or 2.5D grid map. Optionally, the metric map may include an occupancy grid map.

  In some embodiments, the first signal is generated based on a plurality of instructions received from a remote terminal that communicates with the unmanned aerial vehicle. Multiple instructions may be entered into the remote terminal by the user. Alternatively, the first signal can be generated autonomously by an unmanned aerial vehicle.

  In some embodiments, the sensing data includes data for a plurality of different coordinate systems, and generating the environment map includes mapping the data to a single coordinate system. Generating the first signal can include generating a flight path for an unmanned aerial vehicle, and generating the second signal is based on an environment map to avoid one or more obstacles. The step of changing the flight path can be included. The flight path can be configured to direct the unmanned aircraft from a current position to a previous position.

  In another aspect, a system for controlling an unmanned aerial vehicle within an environment is provided. The system includes a plurality of sensors carried by an unmanned aerial vehicle and one or more processors. The one or more processors, individually or collectively, generate a first signal that causes the unmanned aircraft to navigate within the environment and relate to at least a portion of the environment using a plurality of sensors carried by the unmanned aircraft. Sensing the sensing data, generating an environment map representing at least a part of the environment based on the sensing data, detecting one or more obstacles located in the part of the environment using the environment map, The environment map can be used to generate a second signal that causes the unmanned aerial vehicle to navigate to avoid one or more obstacles.

  In another aspect, a method for controlling an unmanned aerial vehicle in an environment is provided. The method includes receiving a first sensing signal associated with the environment from the first sensor, receiving a second sensing signal associated with the environment from the second sensor, and a first using the first sensing signal. Generating an environment map and a second environment map using the second sensing signal; and combining the first environment map and the second environment map, wherein the first sensor and the second sensor are a plurality of different sensors. Type and carried by an unmanned aerial vehicle, each of the first environment map and the second environment map includes obstacle occupancy information for the environment, thereby generating a final environment map including obstacle occupancy information for the environment .

  In some embodiments, the method further includes generating a signal that causes the unmanned aircraft to navigate within the environment based at least in part on the obstacle occupancy information in the final environment map.

  In some embodiments, the unmanned aerial vehicle is a rotorcraft. An unmanned aerial vehicle may have a weight of 10 kg or less. The maximum size of the unmanned aerial vehicle may be 1.5 m or less. Unmanned aerial vehicles can be configured to fly at a height of 400 meters or less. Optionally, the unmanned aerial vehicle can be configured to detect the presence of a flight restriction area and not fly within a predetermined distance of the flight restriction area. The flight restriction area may be an airport.

  In some embodiments, the environment may be an indoor environment or a low altitude environment.

  In some embodiments, the plurality of sensors may include a global positioning system (GPS) sensor, a vision sensor, or a proximity sensor. The proximity sensor may include at least one of the following lidar sensor, ultrasonic sensor, or in-flight camera sensor. The plurality of sensors may include a plurality of different sensor types.

  In some embodiments, the environment map may include a topological map or a metric map. The metric map may include at least one of the following point cloud, 3D grid map, 2D grid map, or 2.5D grid map. Optionally, the metric map may include an occupancy grid map.

  In some embodiments, the first environment map is provided for a first coordinate system and the second environment map is provided for a second coordinate system that is different from the first coordinate system. The first coordinate system can be a global coordinate system and the second coordinate system can be a local coordinate system. Combining the first and second environment maps may include converting the first and second environment maps into a single coordinate system.

  In some embodiments, the sensing range of the first sensor is different from the sensing range of the second sensor.

  In another aspect, a system for controlling an unmanned aerial vehicle within an environment is provided. The system carries a first sensing signal carried by an unmanned aerial vehicle and configured to generate a first sensing signal associated with the environment and a second sensing signal carried by the unmanned aerial vehicle and associated with the environment. A second sensor configured as described above and one or more processors, wherein the second sensor and the first sensor are of a plurality of different sensor types. The one or more processors individually or collectively receive the first and second sensing signals and have a first environment map that uses the first sensing signal and a second environment map that uses the second sensing signal. Generating and combining the first and second environment maps, each of the first and second environment maps including obstacle occupancy information for the environment, thereby including an obstacle occupancy information for the environment Generate an environment map.

  In another aspect, a method for controlling an unmanned aerial vehicle in an environment is provided. The method includes determining an initial location of the unmanned aerial vehicle using at least one of a plurality of sensors carried by the unmanned aircraft, and generating a first signal that causes the unmanned aircraft to navigate within the environment. Receiving sensing data related to the environment using at least one of the plurality of sensors, receiving an instruction to return to the initial location, and unmanned aircraft along the route to the initial location. Generating a second signal to be returned, and when at least one of the plurality of sensors detects an obstacle positioned along the route in the environment, the route avoids the obstacle It will be changed accordingly.

  In some embodiments, the unmanned aerial vehicle is a rotorcraft. An unmanned aerial vehicle may have a weight of 10 kg or less. The maximum size of the unmanned aerial vehicle may be 1.5 m or less. Unmanned aerial vehicles can be configured to fly at a height of 400 meters or less. Optionally, the unmanned aerial vehicle can be configured to detect the presence of a flight restriction area and not fly within a predetermined distance of the flight restriction area. The flight restriction area may be an airport.

  In some embodiments, the environment may be an indoor environment or a low altitude environment.

  In some embodiments, the plurality of sensors may include a global positioning system (GPS) sensor, a vision sensor, or a proximity sensor. The proximity sensor may include at least one of the following lidar sensor, ultrasonic sensor, or in-flight camera sensor. The plurality of sensors may include a plurality of different sensor types.

  In some embodiments, the environment map may include a topological map or a metric map. The metric map may include at least one of the following point cloud, 3D grid map, 2D grid map, or 2.5D grid map. Optionally, the metric map may include an occupancy grid map.

  In some embodiments, the path includes one or more portions through which the movable object has previously passed. The path may be different from the path through which the movable object has previously passed. Optionally, the path may be a flight path of a movable object. The path can include the spatial position and direction of the movable object.

  In another aspect, a system for controlling an unmanned aerial vehicle within an environment is provided. The system can include a plurality of sensors carried by the movable object and one or more processors. The one or more processors determine the initial location of the movable object using at least one of a plurality of sensors carried by the unmanned aircraft, individually or collectively, and cause the unmanned aircraft to navigate within the environment. Generating a first signal, receiving sensing data associated with the environment using at least one of the plurality of sensors, receiving an instruction to return to the initial location, and The second signal is configured to generate a second signal to be returned to the place, and when at least one of the plurality of sensors detects an obstacle positioned along the route in the environment, the route is the obstacle It is changed to avoid.

  In another aspect, a method for generating a map of an environment is provided. The method includes receiving first sensing data including depth information for the environment from one or more vision sensors carried by the unmanned aerial vehicle, and from one or more proximity sensors carried by the unmanned aerial vehicle for the environment. Receiving second sensing data including depth information; and generating an environment map including depth information for the environment using the first sensing data and the second sensing data.

  In some embodiments, each of the first sensing data and the second sensing data includes at least one image having a plurality of pixels, each pixel of the plurality of pixels being associated with a two-dimensional image coordinate and a depth value. ing. Each pixel of the plurality of pixels can be associated with a color value. Each of the first sensing data and the second sensing data includes silhouette information for one or more objects in the environment.

  In some embodiments, the method further includes generating a signal that causes the unmanned aircraft to navigate through the environment based at least in part on the depth information in the environment map.

  In some embodiments, the unmanned aerial vehicle is a rotorcraft. An unmanned aerial vehicle may have a weight of 10 kg or less. The maximum size of the unmanned aerial vehicle may be 1.5 m or less. Unmanned aerial vehicles can be configured to fly at a height of 400 meters or less. Optionally, the unmanned aerial vehicle can be configured to detect the presence of a flight restriction area and not fly within a predetermined distance of the flight restriction area. The flight restriction area may be an airport.

  In some embodiments, the environment may be an indoor environment or a low altitude environment.

  In some embodiments, the one or more vision sensors include only one camera. Alternatively, the one or more vision sensors can include two or more cameras. The one or more proximity sensors can include at least one ultrasound or at least one lidar sensor. The first sensing data can include a first set of depth images, and the second sensing data can include a second set of depth images. Generating the environment map includes identifying a first plurality of feature points present in the first set of depth images and identifying a second plurality of feature points present in the second set of depth images. Determining a correspondence between the first plurality of feature points and the second plurality of feature points, and combining the first set of depth images and the second set of depth images based on the correspondences Generating an environment map, wherein each of the second plurality of feature points corresponds to one of the first plurality of feature points.

  In some embodiments, the first sensing data is provided for a first coordinate system and the second sensing data is provided for a second coordinate system that is different from the first coordinate system. Generating the environment map may include expressing the first and second sensing data with respect to the third coordinate system. The third coordinate system can be either the first coordinate system or the second coordinate system. Alternatively, the third coordinate system can be different from the first coordinate system and the second coordinate system.

  In some embodiments, the environment map may include a topological map or a metric map. The metric map may include at least one of the following point cloud, 3D grid map, 2D grid map, or 2.5D grid map. Optionally, the metric map may include an occupancy grid map.

  In another aspect, a system for generating a map of an environment is provided. The system is carried by an unmanned aerial vehicle and includes one or more vision sensors configured to generate first sensing data that includes depth information for the environment, and includes depth information for the environment carried by the unmanned aerial vehicle. One or more proximity sensors configured to generate the second sensing data may include one or more processors. The one or more processors receive the first and second sensing data, individually or collectively, and use the first and second sensing data to generate an environment map that includes depth information for the environment. Can be configured.

  It should be understood that multiple different aspects of the present invention can be evaluated individually, collectively, or in combination with each other. Various aspects of the invention described herein may be applied to any of a plurality of specific applications identified below, or any other plurality of types of movable objects. You may apply to. Any description herein for an aircraft may be applied to and used with any movable object, such as any airframe. In addition, systems, devices, and methods disclosed herein in connection with air movement (e.g., flight) can also be used for ground or water movement, underwater movement, or space. It may be applied in connection with several other types of movement, such as movement in space. Further, any description herein of a rotor blade or rotor assembly includes any propulsion system, device or mechanism (eg, propeller, wheel, axle) configured to generate propulsion by rotation. Can be used applied.

  Other objects and features of the present invention will become apparent upon review of the specification, the claims and the accompanying drawings.

[Incorporation by reference]
All publications, patents, and patent applications mentioned in this specification are as if each individual publication, patent, and patent application was specifically shown to be incorporated by reference, Which is incorporated herein by reference.

The novel features of the invention are set forth with particularity in the appended claims. A better understanding of the features and advantages of the present invention will be obtained by reference to the following detailed description that sets forth illustrative embodiments, in which the principles of the invention are utilized, and the accompanying drawings of which: Will be obtained.
FIG. 3 illustrates a UAV operating in an outdoor environment, according to embodiments. FIG. 4 illustrates a UAV operating in an indoor environment, according to embodiments. FIG. 4 illustrates a scheme for estimating UAV pose using sensor fusion, according to embodiments. FIG. 6 illustrates a scheme for estimating UAV position and velocity using sensor fusion, according to embodiments. FIG. 4 illustrates a scheme for environment mapping using sensor fusion, according to embodiments. FIG. 4 illustrates a method for generating a map of an environment using sensor fusion, according to embodiments. FIG. 6 illustrates a method for environment mapping using a plurality of different sensor types, according to embodiments. FIG. 6 illustrates a method for controlling a UAV to avoid multiple obstacles, according to embodiments. FIG. 4 illustrates a method for controlling a UAV to return to an initial location according to embodiments. FIG. 6 illustrates a method for controlling a UAV to return to an initial location while avoiding multiple obstacles, according to embodiments. FIG. 6 illustrates an algorithm for controlling a UAV to return to an initial location using a plurality of passing points according to embodiments. FIG. 6 illustrates an algorithm for controlling a UAV to return to an initial location using a plurality of passing points according to embodiments. FIG. 6 illustrates an algorithm for controlling a UAV to return to a target location using a topology map, according to embodiments. FIG. 6 illustrates an algorithm for controlling a UAV to return to a target location using a topology map, according to embodiments. FIG. 6 illustrates a UAV according to embodiments. FIG. 4 illustrates a movable object including a support mechanism and a load according to embodiments. 1 illustrates a system for controlling a movable object, according to embodiments.

  The present disclosure provides multiple systems and methods for controlling multiple movable objects, such as unmanned aerial vehicles (UAVs). In some embodiments, the UAV may be adapted to carry multiple sensors that collect environmental data. Some of the plurality of sensors may be of different types (eg vision sensors used in combination with proximity sensors). Data acquired by multiple sensors can be combined to generate an environment map that represents the surrounding environment. In some embodiments, the environment map may include information regarding the location of multiple objects within the environment, such as multiple objects or multiple obstacles. UAVs can use maps generated to perform various operations, some of which may be semi-automated or fully automated. For example, in some embodiments, the environment map can be used to automatically determine a flight path for a UAV that navigates from its current location to a target location. As another example, the environment map can be used to determine the spatial arrangement of one or more obstacles, which can allow the UAV to perform multiple obstacle avoidance operations. Advantageously, by using multiple sensor types that collect environmental data as disclosed herein, the accuracy of environmental mapping can be improved even in various environments and operating situations. This can enhance the robustness and flexibility of multiple UAV functionalities such as navigation and obstacle avoidance.

  The embodiments provided herein can be applied to various types of UAVs. For example, the UAV may be a small UAV having a weight of 10 kg or less and / or having a maximum dimension of 1.5 m or less. In some embodiments, the UAV may be a rotorcraft such as a multi-rotor aircraft that is propelled to move in the air by a plurality of propellers (eg, quadcopters). Other examples of multiple UAVs and other multiple movable objects suitable for use with the multiple embodiments presented herein are described in further detail below.

  The multiple UAVs described herein may be fully autonomous (eg, by a suitable computing system such as an onboard controller), semi-autonomously, or manually (eg, by a human user) Can be manipulated. A UAV can respond to such multiple commands by receiving multiple commands from an appropriate entity (eg, a human user or an autonomous control system) and performing one or more operations. For example, a UAV takes off from the ground, moves in the air (eg up to 3 degrees of freedom of translation and up to 3 degrees of freedom of rotation), moves to a target location or to a series of target locations, It can be controlled to hover in the air, land on the ground, etc. As another example, a UAV may move at a certain speed and / or acceleration (eg, up to 3 degrees of freedom of translation and up to 3 degrees of freedom of rotation) or along a particular movement path, Can be controlled. In addition, multiple commands are intended to control one or more UAV components, such as multiple components described herein (eg, multiple sensors, multiple actuators, multiple propulsion units, loads, etc.). Can be used. For example, some commands can be used to control the position, orientation and / or movement of a UAV payload such as a camera. Optionally, the UAV can be configured to operate according to one or more predetermined operating rules. The plurality of rules of operation may include a UAV position (eg, latitude, longitude, altitude), direction (eg, roll, pitch, yaw), speed (eg, translational velocity and / or angular velocity), and / or acceleration (eg, It may be used to control any suitable aspect of the UAV, such as translational acceleration and / or angular acceleration). For example, multiple rules of operation can be designed so that the UAV is not allowed to fly above a threshold height, for example, the UAV can be configured to fly at a height of 400 m or less from the ground. In some embodiments, the plurality of operating rules may be adapted to provide automated mechanisms for improving UAV safety and preventing safety related accidents. For example, the UAV can be configured to detect a flight restriction area (eg, an airport) and not fly within a predetermined distance of the flight restriction area, thereby allowing multiple potentials with aircraft and other obstacles. Can avoid common collisions.

  Turning now to the drawings, FIG. 1A illustrates a UAV 102 operating within an outdoor environment 100, according to embodiments. The outdoor environment 100 may be an urban environment, a suburban environment, or a rural environment, or any other environment that is not at least partially within a building. The UAV 102 may be operated relatively near the ground 104 (eg, at a low altitude) or relatively far from the ground 104 (eg, at a high altitude). For example, a UAV 102 that operates at a height equal to or less than approximately 10 m from the ground may be considered low altitude, while operating at a height equal to or greater than approximately 10 m from the ground. The UAV 102 may be considered high altitude.

  In some embodiments, the outdoor environment 100 includes one or more obstacles 108a-d. Obstacles may include any object or entity that may interfere with UAV 102 movement. Some obstacles include, for example, multiple buildings, multiple ground vehicles (eg, multiple cars, multiple motorcycles, multiple trucks, multiple bicycles), multiple humans, multiple animals, multiple plants ( For example, multiple trees, multiple bushes) and other artificial or natural structures may be located on the ground 104 (e.g., obstacles 108a, 108d). Some obstacles may be in contact with and / or supported by the ground 104, water, multiple artificial structures, or multiple natural structures. Alternatively, several obstacles, including multiple aircraft (e.g., multiple airplanes, multiple helicopters, multiple hot air balloons, other multiple UAVs), or multiple birds are completely in the air 106 It may be arranged (eg obstacles 108b, 108c). Airborne obstacles may not be supported by the ground 104 or water, or may be not supported by any of multiple natural structures or artificial structures. Obstacles disposed on the ground 104 may include multiple portions that substantially extend into the air 106 (eg, multiple towers, multiple skyscrapers, multiple external light poles, multiple wireless poles). Towers, multiple power lines, multiple high-rise structures such as trees).

  FIG. 1B illustrates a UAV 152 operating within an indoor environment 150, according to embodiments. The indoor environment 150 is within a building 154 having a floor 156, one or more walls 158, and / or a ceiling or roof 160. Exemplary buildings include residential, commercial, or industrial buildings such as, for example, multiple homes, multiple apartments, multiple offices, multiple manufacturing facilities, multiple storage facilities, etc. . The interior of the building 154 may be completely surrounded by the floor 156, the plurality of walls 158, and the ceiling 160 so that the UAV 152 is limited to the internal space. Conversely, at least one of the floor 156, the plurality of walls 158, or the ceiling 160 may be absent, thereby allowing the UAV 152 to fly from inside to outside or vice versa. Can be. Alternatively or in combination, one or more openings 164 may be formed in the floor 156, the plurality of walls 158, or the ceiling 160 (eg, doors, windows, skylights).

  Similar to the outdoor environment 100, the indoor environment 150 may include one or more obstacles 162a-d. Some obstacles are located on the floor 156, such as furniture, electrical appliances, humans, animals, plants, or other artificial or natural objects, for example. It is also possible (for example, the obstacle 162a). Conversely, some obstacles may be placed in the air, such as multiple birds or other multiple UAVs (eg, obstacle 162b). Some obstacles in the indoor environment 150 may be supported by other structures or objects. The multiple obstacles may also be attached to the ceiling 160, such as multiple lighting fixtures, multiple ceiling fans, multiple beams, or other multiple ceiling mounted appliances or structures (eg, obstacles). Product 162c). In some embodiments, multiple obstacles attach to the wall 158, such as multiple luminaires, multiple shelves, multiple cabinets, and other multiple wall mounted appliances or structures. (E.g., obstacle 162d). Obviously, multiple structural components of building 154 may also be considered as multiple obstacles, including floor 156, multiple walls 158 and ceiling 160.

  The plurality of obstacles described herein may be substantially stationary (eg, multiple buildings, multiple plants, multiple structures), or may be substantially movable. Good (eg, multiple humans, multiple animals, multiple airframes, or other movable objects). Some obstacles may include a combination of fixed and movable components (eg, a windmill). The plurality of moving obstacles or the plurality of obstacle components may move according to a predetermined or predictable route or pattern. For example, vehicle movement may be relatively predictable (eg, according to road shape). Alternatively, some moving obstacles or obstacle components may move along a random trajectory, otherwise they may move along an unpredictable trajectory. For example, an organism such as an animal may move in a relatively unpredictable manner.

  In order to ensure safe and efficient operation, it may be beneficial to provide UAVs with multiple mechanisms to detect and identify multiple environmental objects, such as multiple obstacles. In addition, recognizing multiple environmental objects such as multiple landmarks and multiple features can facilitate navigation, especially when the UAV is operating in a semi-autonomous or fully autonomous manner. . Furthermore, familiarity with the exact location of the UAV within the environment, as well as its own spatial relationship to the surrounding environmental objects, can be beneficial to a wide variety of UAV functionalities.

  Accordingly, the plurality of UAVs described herein include one or more sensors configured to collect relative data, such as information about UAV status, the surrounding environment, or a plurality of objects within the environment. Can be included. Exemplary sensors suitable for use in the embodiments disclosed herein include multiple position sensors (eg, multiple global positioning system (GPS) sensors, capable of triangulating locations). Multiple mobile device transmitters), multiple vision sensors (eg, multiple imaging devices capable of detecting visible infrared or ultraviolet light, such as multiple cameras), multiple proximity or range sensors (eg, multiple Acoustic sensors, multiple rider flight or depth cameras), multiple inertial sensors (eg, multiple accelerometers, multiple gyroscopes, multiple inertial measurement units (IMU)), multiple altitude sensors, multiple attitude sensors ( For example, a compass), multiple pressure sensors (eg, multiple barometers), multiple audio sensors (eg, multiple microphones), or multiple field sensors (eg, In includes a plurality of magnetometers, a plurality of electromagnetic field sensors). Any suitable number and combination of multiple sensors can be used, such as one, two, three, four, five, or more sensors. Optionally, the data can be received from a plurality of different types of sensors (eg, 2, 3, 4, 5, or more types). The plurality of different types of sensors may measure a plurality of different types of signals or information (eg, position, direction, velocity, acceleration, proximity, pressure, etc.) and / or a plurality of data acquisition data Different types of measurement techniques may be utilized. For example, the plurality of sensors may include a plurality of active sensors (eg, a plurality of sensors that generate and measure energy from their own energy source) and a plurality of passive sensors (eg, a plurality of sensors that detect available energy). May be included in any suitable combination. As another example, some sensors generate absolute measurement data provided from points in the global coordinate system (eg, position data provided by a GPS sensor, attitude data provided by a compass or magnetometer). On the other hand, the other sensors may include relative measurement data provided from points in the local coordinate system (eg, relative angular velocity provided by a gyroscope, relative translational acceleration provided by an accelerometer, vision sensor Relative attitude information provided by, an ultrasonic sensor, a rider or a relative distance information provided by a flight camera). In some cases, the local coordinate system may be a body coordinate system defined for the UAV.

  Multiple sensors described herein may be carried by a UAV. The sensor can be located in any suitable part of the UAV, such as above, below, on the side (s) of the UAV's transport body, or inside it. Some sensors may be mechanically coupled to the UAV such that the spatial arrangement and / or movement of the UAV corresponds to the spatial arrangement and / or movement of multiple sensors. The sensor can be connected to the UAV by a rigid connection so that the sensor does not move relative to the portion of the UAV to which the sensor is attached. Alternatively, the connection between the sensor and the UAV may allow movement of the sensor relative to the UAV. The connection can be a permanent connection or a non-permanent (eg, releasable) connection. Suitable multiple connection methods can include multiple bonds, bonding, welding, and / or multiple fasteners (eg, multiple screws, nails, pins, etc.). Optionally, the sensor can be integrally formed with a portion of the UAV. In addition, the sensor, as in the embodiments discussed herein, allows data collected by the sensor to communicate with various UAV functions (eg, navigation, control, propulsion, communication with a user or other device). Etc.) may be electrically coupled to a portion of the UAV (eg, processing unit, control system, data storage).

  The plurality of sensors can be configured to collect various types of data, such as data about the UAV, the surrounding environment, or a plurality of objects in the environment. For example, at least some of the plurality of sensors may be configured to provide data regarding the status of the UAV. The state information provided by the sensor includes information about the spatial arrangement of the UAV (eg location or position information such as longitude, latitude and / or altitude, direction or attitude information such as roll, pitch and / or yaw). Can do. The state information can also include information about the movement of the UAV (eg, translation speed, translation acceleration, angular speed, angular acceleration, etc.). The sensor can be configured, for example, to determine the spatial arrangement and / or movement of the UAV with respect to up to 6 degrees of freedom (eg, 3 degrees of freedom of position and / or translation, 3 degrees of freedom of direction and / or rotation). . State information may be provided for the global coordinate system or for a local coordinate system (eg, for a UAV or other entity). For example, the sensor can be configured to determine the distance between the UAV and the user controlling the UAV, or the distance between the UAV and the flight start point for the UAV.

  Data acquired by multiple sensors may provide various types of environmental information. For example, the sensor data may indicate an environment type such as an indoor environment, an outdoor environment, a low altitude environment, or a high altitude environment. The sensor data may also provide information regarding multiple current environmental conditions, including weather (eg, clear, rain, snow), multiple visibility conditions, wind speed, time of day, and the like. Further, the environmental information collected by the sensor may include information regarding multiple objects in the environment, such as multiple obstacles described herein. The obstacle information may include information regarding the number, density, geometry, and / or spatial arrangement of a plurality of obstacles in the environment.

  In some embodiments, multiple sensing results are generated by combining sensor data acquired by multiple sensors, also known as “sensor fusion”. For example, sensor fusion can be used to combine sensing data acquired by multiple different sensor types, including multiple GPS sensors, inertial sensors, vision sensors, lidars, multiple ultrasonic sensors, and the like. As another example, sensor fusion may include absolute measurement data (eg, data provided for a global coordinate system such as GPS data) and relative measurement data (eg, vision sensing data, lidar data, or super It can be used to combine different types of sensing data, such as data provided for a local coordinate system, such as sonic sensing data. Sensor fusion can be used to compensate for multiple limitations or multiple errors associated with individual sensor types, which can improve the accuracy and reliability of the final sensing result.

  FIG. 2A illustrates a scheme 200 for estimating UAV pose using sensor fusion, according to a number of embodiments. Although the embodiment of FIG. 2A is directed to estimating the UAV yaw angle, it is understood that the approach described in Scheme 200 can also be applied to estimate the roll angle or pitch angle of the UAV. It should be. Scheme 200 utilizes IMU 202, at least one relative direction sensor 204, and magnetometer 206. The IMU 202 and magnetometer 206 can be used to provide respective absolute estimates 208, 210 of the UAV yaw angle. Relative direction sensor 204 can be any sensor that provides attitude information about a local coordinate system (eg, a UAV body coordinate system) rather than a global coordinate system. Exemplary multiple relative orientation sensors include multiple vision sensors, lidars, multiple ultrasonic sensors, and multiple in-flight or depth cameras. The relative direction sensor data can be analyzed for the purpose of providing estimates of the yaw rate 212 and the relative yaw angle 214.

  In embodiments where the relative direction sensor 204 is a vision sensor configured to capture a series of images (“multiple frames”), the yaw rate 212 and yaw angle 214 are selected from one or more sequences. Can be determined using an image ("multiple keyframes"). Multiple key frames can be selected using any suitable method. For example, the plurality of key frames include a plurality of predetermined time intervals, a plurality of predetermined distance intervals, a plurality of predetermined arrangement intervals, and a plurality of predetermined posture intervals between the plurality of key frames. Or, it can be selected at a plurality of predetermined intervals such as a plurality of predetermined average parallax intervals. As another example, a plurality of key frames can be selected based on a plurality of relationships between a plurality of consecutive key frames, for example, an amount of region overlapping between a plurality of key frames. Optionally, multiple key frames can be selected based on multiple combinations of multiple parameters provided herein. In some embodiments, at least some of the plurality of key frames may include a plurality of consecutive image frames. Alternatively, the plurality of key frames may not include any plurality of consecutive image frames. The latter approach may be advantageous in that it reduces the spread of multiple estimation errors present in multiple consecutive key frames.

  The yaw rate 212 and yaw angle 214 at a given time can be estimated using any suitable number of key frames, such as one, two, three, four, or more key frames. . The yaw rate 212 and yaw angle 214 can be determined using any suitable approach, such as image analysis. In some embodiments, two key frames that capture the same scene at different yaw angles may be assumed to be different. The yaw angle between different key frames can be determined by multiple mathematical modeling techniques. For example, the difference between a plurality of key frames can be determined by identifying and matching a plurality of feature points present in both of the plurality of key frames. Based on the coordinates of the feature points in both of the key frames, a similarity transformation matrix for the two key frames can be determined, and the yaw angle 212 can be obtained from the similarity transformation matrix. Using the yaw angle 214 and the time interval between two key frames, the yaw rate 212 can then be calculated.

  The absolute and relative estimates provided by the sensors 202, 204, 206 can be combined (eg, using an extended Kalman filter 216 or other type of Kalman filter) for the purpose of providing a final yaw angle result 218. . Any suitable method can be implemented for the purpose of combining multiple estimates. For example, the integration of the yaw rate estimate 212 can be combined with the relative yaw angle estimate 214 for the purpose of updating the relative yaw angle estimate 214. Relative yaw angle estimate 214 can be combined with absolute yaw angle estimates 208, 210 from IMU 202 and magnetometer 206, respectively, to determine the UAV yaw angle. The final yaw angle result 218 can be provided as a relative yaw angle (eg, relative to a UAV body coordinate system) or as an absolute yaw angle. As previously described, the accuracy of the final result 218 may be improved by combining multiple different types of measurement data. For example, by using data from IMU 202 and / or relative direction sensor 204 in multiple situations where the data provided by magnetometer 206 is insufficient (eg, due to multiple external magnetic fields), The extent to which the meter error affects the final result can be reduced.

  FIG. 2B illustrates a scheme 250 for estimating UAV position and velocity using sensor fusion, according to embodiments. Scheme 250 utilizes IMU 252, at least one relative position and / or velocity sensor 254, and GPS sensor 256. The IMU 252 can provide an estimate of the UAV acceleration 258. The GPS sensor 256 can provide an estimate of the absolute position of the UAV 260. Relative position and / or speed sensor 254 (e.g., vision sensor, lidar, ultrasonic sensor, flight or depth camera, or any other sensor that provides relative position and / or speed information) may include UAV speed 262 and relative It can be used to obtain an estimate of the UAV position 264. Similar to the relative direction sensor 204 of FIG. 2A, in embodiments where the relative position and / or speed sensor 254 is a vision sensor, the speed and relative position estimates 262, 264 are one or more obtained by the vision sensor. Can be determined based on the keyframes. For example, image analysis and mathematical modeling can be used to calculate multiple differences between multiple consecutive key frames, thereby determining the multiple translations of the UAV that will produce these differences. Next, the UAV speed can be estimated based on the time elapsed between a plurality of key frames.

  The multiple estimates provided by sensors 252, 254, and 256 can be combined using extended Kalman filter 266, or other suitable Kalman filter type, to obtain final position and velocity results 268 for UAV. . For example, the integration of the acceleration estimate 258 can be combined with the UAV velocity estimate 262 for the purpose of determining the UAV velocity. The integral of the determined UAV velocity can be combined with the relative position estimate 264 and the absolute position estimate 260 for the purpose of determining the UAV position. The position and velocity results 268 can be expressed with respect to a local or global coordinate system.

  Optionally, sensor fusion with the Kalman filter 266 can also be used to determine calibration data for conversion between the relative coordinate system of the relative velocity and / or position sensor 254 and the global coordinate system of the IMU 252 and GPS 256. . The calibration data can be used to map the absolute sensing data obtained by the IMU 252 and / or GPS 256 onto the relative sensing data obtained by the sensor 254, or vice versa. For example, the IMU 252 and / or the GPS 256 can be used to determine the path that the UAV moves in a plurality of absolute coordinates, and the relative sensor 254 can be used to obtain sensing data that indicates the path that the UAV moves in a plurality of relative coordinates. it can. Apply multiple sensor fusion methods provided herein to determine the multiple calibration parameters (eg, scaling factor, rotation, translation) required to project absolute path data onto relative path data Yes, and vice versa. For example, the relationship between a plurality of calibration parameters, absolute path data, and relative path data can be represented by mathematical modeling. Next, for the purpose of obtaining a plurality of calibration parameters, absolute path data and relative path data acquired by a plurality of sensors can be input into the model. The combination of absolute route data and relative route data described herein can be used to increase the accuracy of route determination, thereby improving UAV navigation.

The multiple sensor fusion approach described herein can be applied to provide multiple more accurate estimates of UAV conditions as well as environmental information. In some embodiments, the UAV state information and the environment information estimate may be dependent on each other so that the UAV state information is used to estimate the environment information and vice versa. For example, some sensors have absolute environmental information
While other multiple sensors can be used to obtain relative environmental information
Can be used to get Absolute environmental information is related
Can be projected onto relative environment information (or vice versa). here,
Represents the rotation matrix between the global coordinate system and the local coordinate system,
Represents a translation vector from the origin of the global coordinate system to the origin of the local coordinate system. In some embodiments,
as well as
Can be determined based on the UAV position and direction respectively.
Are provided for UAV body coordinate systems. Thus, the UAV position and direction information can be used to combine absolute and relative environment data, thereby generating an estimate of the environment information. Conversely, absolute environment data and relative environment data can be used to estimate UAV position and orientation using the relationships described above. This approach can be repeated with the goal of providing updated estimates of UAV status and environmental information.

  In some embodiments, the sensor data can be used to generate a representation of the environment, or at least a portion thereof. Such multiple representations may be referred to herein as “multiple environmental maps”. The environment map may represent multiple parts of the environment very close to the UAV (local map) or may represent multiple parts relatively far from the UAV (overall map). For example, the local map may represent portions of the environment within a radius of about 2 m, 5 m, 10 m, 15 m, 20 m, 25 m, or 50 m from the UAV. The overall map may represent multiple portions within a radius of about 25m, 50m, 75m, 100m, 125m, 150m, 200m, 225m, 250m, 300m, 400m, 500m, 1000m, 2000m, or 5000m from the UAV. The size of the environment map may be determined based on the effective range of multiple sensors used to generate the map. The effective range of the sensor may vary based on the sensor type. For example, the vision sensor can be used to detect multiple objects within a radius of about 10 m, 15 m, 20 m, or 25 m from the UAV. The lidar sensor can be used to detect multiple objects within a radius of about 2 m, 5 m, 8 m, 10 m, or 15 m from the UAV. Ultrasonic sensors can be used to detect multiple objects within a radius of about 2 m, 5 m, 8 m, 10 m, or 15 m from the UAV. In some embodiments, the environment map may only represent multiple portions that the UAV has previously passed. Alternatively, in other embodiments, the environment map may also only represent portions of the environment where the UAV is not moving.

  The environment map can be used to represent various types of environment information. For example, depending on the map, the geometry (eg, length, width, height, thickness, shape, surface area, etc.) of multiple environmental objects such as multiple obstacles, multiple structures, multiple landmarks, or multiple features. Information indicating volume), spatial arrangement (eg, position, orientation), and type. As another example, a map can provide information indicating that parts of the environment are blocked (eg, UAVs cannot pass) and that parts are not blocked (eg, UAVs can pass). In addition, the map can provide information about the current location of the UAV and / or the spatial arrangement of the UAV with respect to various environmental objects.

  Any suitable type of environment map can be used, such as a metric map or a topological map. A topological map may depict connectivity between multiple locations within the environment, while a metric map may depict the geometry of multiple objects within the environment. Optionally, the environment map may include a combination of metric information and topological information. For example, an environment map can depict connectivity as well as multiple absolute spatial relationships (eg, multiple distances) between multiple environmental objects and / or locations. Alternatively or in combination, some parts of the map may be depicted topologically while other parts may be depicted metrically.

  The environment map can be provided in any suitable format. For example, the topological environment map can be provided as a graph having a plurality of vertices representing a plurality of places and a plurality of edges representing a plurality of paths between the plurality of places. Multiple edges of the topological environment map may be associated with distance information for the corresponding route. A metric environment map may be any representation that depicts multiple environmental locations and object spatial coordinates as a point cloud, terrain map, 2D grid map, 2.5D grid map, or 3D grid map. The plurality of spatial coordinates may be a plurality of three-dimensional coordinates (for example, an x coordinate, a y coordinate, and a z coordinate) representing the spatial positions of a plurality of points on a plurality of surfaces of a plurality of environmental objects. In some embodiments, the metric environment map may be an occupancy grid map. The occupancy grid map can represent the environment as multiple volumes. The plurality of volumes may be a plurality of equal sizes or a plurality of different sizes. The occupancy grid map may indicate, for each volume, whether the volume is substantially occupied by an obstruction, so that the exact amount of disturbed and unobstructed space in the environment Can provide simple expressions.

  The multiple environment maps described herein can be generated using any suitable method. For example, one or more portions of the environment map can be generated during UAV operation (eg, in flight), for example, by using simultaneous positioning mapping (SLAM) or other robotic mapping techniques. Alternatively, one or more portions of the map can be generated prior to UAV operation and provided to the UAV prior to or during flight (eg, can be transmitted from a remote computing system). . Such multiple maps may be modified based on data acquired by the UAV during the flight, which can provide further refinement of the map data.

  In some embodiments, environment mapping can be performed using sensor fusion that combines environmental information collected by multiple sensor types. This approach may be advantageous in order to compensate for multiple limitations of individual sensor types. For example, if the UAV is in an indoor environment, GPS sensing data may be inaccurate or unavailable. The multiple vision sensors may not be optimized for detecting multiple transparent objects (eg, glasses) or for multiple relatively dark environments (eg, nighttime). In some embodiments, the plurality of lidar sensors may have a relatively short detection range compared to other sensor types. By using the multi-sensor fusion described herein, precise mapping under a variety of environmental types and operating conditions can be provided, thus improving the robustness and flexibility of UAV operation.

  FIG. 3A illustrates a scheme 300 for environment mapping using sensor fusion, according to embodiments. Scheme 300 can be used to generate any embodiment of the multiple environment maps described herein. In scheme 300, sensing data is received from multiple sensors 302. The plurality of sensors 302 may include one or more different sensor types. The plurality of sensors 302 may be carried by the UAV, eg, connected to the UAV's transport body. Optionally, sensing data from each of the plurality of sensors 302 can be processed in advance to improve data quality by filtering, noise reduction, image distortion correction, and the like. In some embodiments, the sensing data provided by each of the plurality of sensors 302 may be represented for each coordinate system (eg, based on the position and orientation of the sensor relative to the UAV's transport body). . Accordingly, the sensing data from each sensor can be combined by converting 304 all sensing data into a single coordinate system. For example, sensing data provided for a local coordinate system may be converted to a global coordinate system and vice versa. Coordinate system transformation 304 can be achieved based on sensor calibration data 306. Sensor calibration can be performed using any suitable technique and can be performed prior to UAV operation (offline calibration) or during UAV operation (online calibration). In some embodiments, the sensor calibration includes determining a plurality of incidental parameters for the plurality of sensors 302, such as a plurality of spatial relationships (eg, relative positions and orientations) between the plurality of sensors 302. . Next, a plurality of conversion calculations for converting the sensing data into a single coordinate system can be determined based on the determined plurality of sensor parameters.

  Following the coordinate system transformation 304, the transformed sensing data can then be combined using a sensor fusion 308 that obtains a single sensing result. Various techniques can be used to perform sensor fusion, such as Kalman filtering (eg, Kalman filter, extended Kalman filter, unscented Kalman filter), particle filtering, or other filtering techniques known to those skilled in the art. The method used may vary depending on the particular combination and types of sensors used. In some embodiments, sensor fusion 308 may utilize sensing data from all sensors 302. Conversely, sensor fusion 308 may utilize data from only a subset of multiple sensors 302. The latter approach may be advantageous for the purpose of removing insufficient or unreliable sensor data (eg, GPS sensing data when the UAV is indoors). The combined sensing data can then be used for environment map generation 310.

Environmental mapping can be performed based on sensing data from any suitable combination of multiple sensors. For example, in some embodiments, the multiple mapping methods described herein can be implemented by a UAV carrying a lidar sensor and a vision sensor (eg, a monocular sensor such as a single camera). The lidar sensor can be used to acquire distance data of a plurality of environmental objects relative to the UAV, while the vision sensor can be used to capture image data of a plurality of surrounding environmental objects. Sensor calibration data for the lidar and vision sensor may include sensor parameters that indicate a spatial relationship between the lidar and the vision sensor. For example, distance from UAV by lidar sensor
Coordinates in the image data determined and also captured by the vision sensor
For environmental objects corresponding to objects,
When
The relationship between
Can be represented by here,
Is a rotation matrix,
Is the transformation matrix,
Is an unknown scalar quantity,
Is an internal matrix for the vision sensor (a unique parameter determined by pre-calibration). In the above equation,
When,
When,
Is known, but on the other hand,
When
Is unknown. Number of multiple measurements
Is used,
as well as
Is fixed (the relative position and direction of the lidar sensor and the vision sensor are fixed), the problem is the perspective n-point (PNP) problem as described below. Parameter,
as well as
Can be solved by using a plurality of techniques known to those skilled in the art.

Once the sensor calibration data is obtained, the lidar sensing data and the vision sensing data can be converted into the same coordinate system (“world” coordinate system) according to Equation 14 below.
here,
Is data for the world coordinate system (corresponding to the vision sensor coordinate system in this embodiment),
Is vision sensing data for the world coordinate system,
Is vision sensing data for the vision sensor coordinate system,
Is lidar sensing data for the world coordinate system,
Is lidar sensing data for the lidar sensor coordinate system. And
as well as
Is a rotation matrix and transformation matrix for transformation between the vision sensor coordinate system and the lidar sensor coordinate system (described above,
as well as
Corresponding to).
Is a diagonal matrix
It is. here,
Is an unknown scale factor. The converted sensing data can then be combined using multiple sensor fusion techniques. For example, the vision coordinate system
Vision sensing data and world coordinate system
The relationship between vision sensing data for
Can be represented by Using the lidar sensing data, the closest point (point) to the UAV along the sensing direction of the lidar sensor
Indicated as)
Can be obtained. Therefore,
When,
as well as
Is known (because
So),
Can be determined. Therefore, it is local sensor data,
as well as
Each world coordinate
as well as
Single result
Can be combined to produce The combined data can then be used to generate a map of the environment surrounding the UAV.

  FIG. 3B illustrates a method 350 for generating a map of an environment using sensor fusion, according to embodiments. The method 350 can be performed by any of the systems and devices provided herein, for example, by one or more processors of a UAV.

  In step 360, first sensing data including depth information for the environment is received from one or more vision sensors. For example, a vision sensor can include only one camera (monocular sensor). Alternatively, the vision sensor can include two (binocular vision sensors) or more cameras. The multiple vision sensors can be carried by the UAV, for example by the UAV transport body. In embodiments where multiple vision sensors are used, each sensor can be located in a different part of the UAV, and the parallax between the image data collected by each sensor can be used to provide depth information for the environment. . Depth information can be used herein to refer to information regarding the distance of one or more objects from a UAV and / or sensor. In embodiments where a single vision sensor is used, the depth information captures image data for a plurality of different positions and orientations of the vision sensor and then recovers the depth information from a suitable plurality of image analysis techniques ( For example, it can be obtained by using a structure from motion.

  In step 370, second sensing data including depth information for the environment is received from one or more proximity sensors. The plurality of proximity sensors can include at least one ultrasonic sensor (eg, wide angle sensor, array sensor) and / or at least one lidar sensor. In some embodiments, an ultrasonic array sensor may provide improved detection accuracy compared to other types of ultrasonic sensors. Multiple proximity sensors can also be carried by the UAV. The plurality of proximity sensors can be disposed near the plurality of vision sensors. Alternatively, the plurality of proximity sensors can be located on different portions of the UAV that are used to carry the plurality of vision sensors.

  In step 380, an environment map including depth information for the environment is generated using the first and second sensing data. As described herein, each of the first and second sensing data can include depth information for the environment. Optionally, the first and second sensing data can also include silhouette information for one or more objects in the environment. Silhouette information may be used herein to refer to information about multiple outlines, multiple outlines, or multiple edges of environmental objects. Sensing data generated by one or more proximity sensors can be provided as a pixel depth map, while depth information is derived from image data collected by one or more vision sensors from a structure from motion. Extraction using multiple techniques such as motion, structured light, sheet of light, time-of-flight, or stereovision disparity mapping it can. In some embodiments, each of the first and second sensing data includes at least one image having a plurality of pixels, each pixel having two-dimensional image coordinates (eg, x and y coordinates), a depth value. (Eg, the distance between the environmental object corresponding to the pixel and the UAV and / or sensor) and / or a color value (eg, RGB color value). Such multiple images may be referred to herein as multiple depth images.

  Sensing data for the purpose of generating a map containing depth information (eg, a three-dimensional environmental representation such as an occupancy grid map) taking into account the relative reliability and accuracy of each type of sensing data during the combining process. The depth information associated with each set of can be spatially arranged and combined (eg, using a suitable multiple sensor fusion method such as Kalman filtering). In some embodiments, generating an environment map identifies a plurality of feature points present in a set of depth images provided by a plurality of vision sensors, a set provided by a plurality of proximity sensors. Identifying corresponding feature points present in the depth image and determining correspondence between both feature points. The correspondence includes information about one or more transformations (eg, translation, rotation, scaling) that can be applied to map multiple proximity depth images onto multiple vision depth images, or vice versa. But you can. In order to generate an environment map, multiple depth images can then be combined based on the correspondence. Alternatively or in combination, the first and second sensing data may be provided for a plurality of different coordinate systems, and the environment map is represented by representing the first and second sensing data in the same coordinate system. Can be generated. This coordinate system may be a coordinate system associated with the first sensing data, a coordinate system associated with the second sensing data, or an entirely different coordinate system. Once the environment map is generated, the UAV can navigate in the environment based on the depth information included in the environment map.

  The combination of proximity sensing and vision sensing described herein can compensate for multiple limitations of multiple individual sensor types, thereby improving map generation accuracy. For example, a plurality of vision sensors can generate a plurality of color images with relatively high resolution, but when a monocular camera is used or when a binocular camera distance is relatively small (a plurality of cameras are small UAVs). It may be relatively difficult to obtain accurate path data from image data. In addition, multiple vision sensors cannot provide sufficient image data when the lighting is bright or when the lighting has high contrast, or when there are multiple adverse environmental conditions such as rain, fog, smog, etc. It may be. Conversely, multiple proximity sensors, such as multiple ultrasonic sensors, may provide accurate path data, but may have lower resolution when compared to multiple vision sensors. Also, in some cases, multiple ultrasonic sensors and other multiple proximity sensor types may include multiple objects with multiple small reflective surfaces (eg, multiple branches, multiple corners, multiple fences), Alternatively, multiple absorbent objects (eg, rugs) may not be detected, or multiple distances may not be analyzed within multiple complex environments (eg, multiple indoor environments) with multiple objects. However, vision sensing data is generally complementary to proximity sensing data in that in some situations where proximity sensors produce sub-optimal data, vision sensors can produce reliable data and vice versa. It may be. Thus, the vision sensor and proximity sensor can be used in combination to generate multiple accurate environment maps in a wide variety of operating situations and for various types of environments.

  FIG. 4 illustrates a method 400 for environment mapping using a plurality of different sensor types, according to embodiments. As with all methods disclosed herein, method 400 can be performed using any of the multiple system and multiple device embodiments presented herein, for example, onboard a UAV. It can be executed by one or more processors that are carried around. In addition, as with all methods disclosed herein, any of the steps of method 400 can be combined with any of the steps of other methods herein, or These can be replaced.

  In step 410, a first sensing signal is received from a first sensor carried by the UAV. The first sensing signal may include information related to the environment in which the UAV is operating, such as environmental data indicating the location and geometry of multiple environmental objects (eg, multiple obstacles). Similarly, in step 420, a second sensing signal is received from a second sensor carried by the UAV. In some embodiments, the first and second sensors can be a plurality of different sensor types (eg, lidar and vision sensors, ultrasound, including any of the plurality of sensor types previously described herein). Sensors and vision sensors, etc.).

  In step 430, a first environment map including occupation information for the environment is generated using the first sensing signal. In step 440, a second environment map including occupancy information for the environment is generated using the second sensing signal. Each of the first and second environment maps can include obstacle occupancy information for the environment so that the environment map can be used to determine multiple locations of multiple obstacles for the UAV. For example, the first and second environment maps may be a plurality of occupancy grid maps or any other plurality of mapping types that contain information about blocked and unblocked spaces in the environment. Also good.

  Optionally, the first environment map may represent a different part of the environment from the second environment map. For example, the first environment map may represent a portion of the environment that is relatively close to the UAV, while the second environment map may represent a portion of the environment that is relatively distant from the UAV. . In some embodiments, different sensing signals can be used to generate maps that span different parts of the environment. The selection of the signal used to generate the different maps is based on any suitable criteria, such as the relative signal quality and / or accuracy of the first and second sensing signals for a particular part of the environment. obtain. The quality and accuracy of the sensing data may depend on the specific characteristics of each sensor, and the characteristics of the environmental objects to be sensed (eg, transparency, reflectivity, absorption, shape, size, As well as environmental type (eg indoor, outdoor, low altitude, high altitude), multiple weather conditions (eg clear, rainy, foggy), perceived environmental objects May be different based on their relative positions (eg, short distance, long distance). For example, the first sensor may be more accurate than the second sensor at multiple short distances, while the second sensor may be more accurate than the first sensor at multiple long distances. Good. Thus, the first sensor can be used to generate a map for a plurality of environmental parts that are relatively close to the UAV, while the second sensor can generate a map for a plurality of environmental parts that are relatively far from the UAV. Can be used to generate. Alternatively or in combination, the selection of the signal may be based on whether the environmental part is within the sensing range of the corresponding sensor. In embodiments where the first and second sensors have different sensing ranges, this approach may be advantageous. For example, a short range sensor can be used to generate a map of multiple environmental parts that are relatively close to the UAV, while a long range sensor is relative to the UAV that is outside the range of the short range sensor. Can be used to generate maps of multiple environmental parts that are far apart.

  In step 450, the first and second environment maps can be combined to generate a final environment map that includes occupancy information for the environment. Multiple environment maps can be combined using multiple sensor fusion techniques as described herein. In embodiments where multiple sensing signals are provided for multiple different coordinate systems (eg, local and global coordinate systems), generating multiple map portions aligns each environmental data Converting both sensing signals to a single coordinate system for purposes of generating a final environment map. Subsequently, the UAV can navigate through the environment based on the final environment map. For example, the UAV can navigate based at least in part on obstacle occupancy information represented in the final environment map for the purpose of avoiding multiple collisions. The UAV can be navigated by the user, by an automated control system, or by a plurality of suitable combinations thereof. Several other examples of UAV navigation based on environmental map data are described in more detail below.

  Although the above steps illustrate a method 400 of environment mapping according to embodiments, those skilled in the art will recognize numerous variations based on the teachings described herein. Some steps may include multiple substeps. In some embodiments, as desired, as the UAV navigates through the environment, the resulting environment map is updated and refined as desired by performing multiple steps of the method 400 (eg, continuously). Or at predetermined time intervals). This real-time mapping approach can enable the UAV to quickly detect multiple environmental objects and adapt to changing operating conditions.

  The multiple sensor fusion approaches described herein are applicable to various types of multiple UAV functionality, including navigation, object recognition and obstacle avoidance. In some embodiments, environmental data obtained using multiple sensor fusion results provides robust location and safety for UAV operation by providing accurate location information as well as information about multiple potential obstacles. Can be used to improve performance and flexibility. Environmental data can be provided to the user (eg, via a remote controller, remote terminal, mobile device, or other user device) for the purpose of informing user manual control of the UAV. Alternatively or in combination, environmental data can be used for a semi-autonomous control system or a fully autonomous control system that directs UAV automated flight.

  For example, a plurality of embodiments disclosed herein can be used to perform a plurality of obstacle avoidance operations for the purpose of preventing a UAV from colliding with a plurality of environmental objects. In some embodiments, obstacle detection and avoidance can be automated, thereby improving UAV safety and reducing user responsibility for avoiding multiple collisions. This approach may be advantageous not only in situations where the user cannot easily recognize the presence of multiple obstacles near the UAV, but also for inexperienced UAV operators. Good. In addition, by implementing automated obstacle avoidance, multiple safety hazards associated with semi-autonomous or fully autonomous UAV navigation can be reduced. In addition, multiple multi-sensor fusion techniques described herein can be used to generate more accurate multiple environmental representations, thus improving the reliability of such multiple automated anti-collision mechanisms. can do.

  FIG. 5 illustrates a method 500 for controlling a UAV to avoid multiple obstacles, according to multiple embodiments. Method 500 may be performed by one or more processors carried by the UAV. As previously described herein, the method 500 can be fully or at least partially automated, thereby providing automatic obstacle detection performance and automatic obstacle avoidance performance.

  In step 510, a first signal is generated that causes the UAV to navigate within the environment. The first signal may include a plurality of control signals for a UAV propulsion system (eg, a plurality of rotor blades). The first signal can be generated based on a plurality of user commands that are input to a remote terminal or other user device and subsequently transmitted to the UAV. Alternatively, the first signal can be generated autonomously by a UAV (eg, an automated onboard control device). In some cases, the first signal can be generated semi-autonomously with multiple contributions from user input as well as from multiple automated routing mechanisms. For example, a user can indicate a series of passing points for a UAV, and the UAV can automatically calculate a flight path to traverse multiple passing points.

  In some embodiments, the flight path can indicate a series of desired multiple positions and / or multiple directions for the UAV (eg, for up to 6 degrees of freedom). For example, the flight path can include at least an initial location for the UAV and a target location. Optionally, the flight path can be configured to direct the UAV from the current position to the previous position. As another example, the flight path can include a target flight direction for the UAV. In some embodiments, the plurality of instructions can specify a speed and / or acceleration for the UAV to move the UAV along the flight path (eg, up to 6 degrees of freedom).

  At step 520, sensing data associated with at least a portion of the environment is received using the plurality of sensors. The plurality of sensors can include a plurality of sensors of different types (eg, vision, lidar, ultrasound, GPS, etc.). Sensing data may include information regarding location and characteristics of multiple obstacles and / or other environmental objects. For example, the sensing data may include distance information indicating the proximity of the UAV to a plurality of nearby obstacles. Optionally, the sensing data may include information about a portion of the environment along the UAV flight path, such as a portion that overlaps the UAV flight path or a portion near the UAV flight path.

  In step 530, an environment map representing at least a part of the environment is generated based on the sensing data. The environment map may be a local map that represents a portion of the environment that directly surrounds the UAV (eg, within a radius of about 2 m, 5 m, 10 m, 15 m, 20 m, 25 m, or 50 m from the UAV). Alternatively, the environment map may be a global map that also represents a portion of the environment relatively far from the UAV (eg, about 25 m, 50 m, 75 m, 100 m, 125 m, 150 m, 200 m, 225 m, radius from the UAV). 250 m, 300 m, 400 m, 500 m, 1000 m, 2000 m, or 5000 m). The size of the environment map may be determined based on the effective range of multiple sensors used in step 510, as previously described. In some embodiments, the environment map can be generated using a plurality of sensor fusion-based methods (eg, scheme 300 and / or method 400) previously described herein. For example, sensing data can include data for a plurality of different coordinate systems, and environment map generation can include mapping the data onto a single coordinate system to facilitate combining the sensing data.

  In step 540, the environment map is used to detect one or more obstacles located in a portion of the environment. Obstacle detection from the map information can be performed using various strategies, for example, by feature extraction techniques or pattern recognition techniques. Optionally, a number of suitable machine learning algorithms can be implemented to perform obstacle detection. In some embodiments, if the environment map is an occupancy grid map, multiple obstacles can be identified by detecting multiple volumes in the continuously occupied occupancy grid map. Multiple obstacle detection results may provide information regarding the location, orientation, size, proximity and / or type of each obstacle, as well as corresponding confidence information for the results. The map can be analyzed to identify multiple obstacles that pose a collision risk to the UAV (eg, located along or near the flight path).

  At step 550, the environment map is used to generate a second signal that causes the UAV to navigate to avoid one or more obstacles. Analyze the environmental map for the purpose of determining the location of one or more obstacles to the UAV, as well as any multiple unobstructed spatial locations that the UAV can move to avoid collisions with obstacles it can. Thus, the second signal can provide an appropriate plurality of control signals (eg, for a UAV propulsion system) that cause the UAV to navigate through unobstructed spaces. In embodiments where the UAV navigates according to a flight path, the flight path can be changed based on the environment map to avoid one or more obstacles, and the UAV can navigate according to the changed flight path. Multiple flight path changes can instruct the UAV to traverse only through multiple unobstructed spaces. For example, the flight path can be changed to cause the UAV to fly around the obstacle (eg, up, down, or side), to fly away from the obstacle, or to maintain a specific distance from the obstacle. In multiple situations where multiple modified flight paths are possible, for example, minimizing distance traveled, minimizing travel time, minimizing the amount of energy required to travel the path A preferred plurality of flight paths can be selected based on any suitable criteria, such as minimizing the amount of change from the original flight path. Optionally, if an appropriate flight path cannot be determined, the UAV may simply hover in place and wait for the obstacle to move off that path, or the user may take manual control You may wait.

  In other exemplary applications of multi-sensor fusion for UAV operation, the embodiments presented herein can be implemented as part of an “auto-return” function, in which the UAV is a specific plurality of Under circumstances, you will automatically navigate from your current location to your “home” location. The home location may be the initial location where the UAV has previously passed, such as the location where the UAV first took off and started flying. Alternatively, the home location may not be the location where the UAV has previously passed. The home location can be determined automatically or can be specified by the user. Examples of situations in which the auto-return function can be triggered include receiving a return indication from the user, communicating with the user's remote controller, or other indication that the user can no longer control the UAV , Detecting a low UAV battery level, or a UAV failure, or other emergency situations.

  FIG. 6 illustrates a method 600 for controlling a UAV to return to an initial location, according to embodiments. In some embodiments, one or more processors associated with the automated control system implement the method 600 so that little user input is required to perform the method 600.

  In step 610, at least one of the plurality of sensors is used to determine the initial location of the UAV. In some embodiments, multiple locations having multiple different sensor types can be used to determine the location. The location can be determined for a global coordinate system (eg, multiple GPS coordinates) or for a local coordinate system (eg, for multiple local environmental landmarks or features).

  In step 620, a first signal is generated that causes the UAV to navigate within the environment. The first signal can be generated manually, semi-autonomously, or fully autonomously. For example, the user can input a plurality of instructions transmitted to the UAV for the purpose of controlling the movement of the UAV to the remote controller. As another example, the UAV can move according to a predetermined flight path or can move toward a predetermined location. The UAV can navigate towards a location in the environment that is different from the initial location (eg, for longitude, latitude, or altitude).

  In step 630, sensing data associated with the environment is received from at least one of the plurality of sensors. Sensing data can be received from the same sensors used in step 610 or from different sensors. Similarly, sensing data can be obtained from a plurality of different sensor types. Such sensing data may include information regarding the location and characteristics of multiple environmental objects, as previously described herein.

  In step 640, an environment map representing at least a part of the environment is generated based on the sensing data. In embodiments where the sensing data includes data from multiple sensor types, the sensor fusion-based methods (eg, scheme 300 and / or method 400) previously described herein may be used to A map can be generated. The resulting map can be provided in any suitable format and may include information related to obstacle occupancy (eg, an obstacle grid map).

  In step 650, a second signal is generated that causes the UAV to return to the initial location based on the environment map. The second signal can be generated in response to an auto return command. The auto-return command can be input by the user or can be automatically generated in response to an emergency situation, for example, loss of communication with a remote controller that controls the UAV. In some embodiments, once the UAV receives an instruction to auto-return, the UAV can determine its current location, the spatial relationship between the current location and the initial location, and / or any An environmental map can be used to determine multiple locations for multiple environmental obstacles. Based on the obstacle occupancy information in the environment map, the UAV can then determine an appropriate route (eg, flight route) for navigating from the current location to the initial location. Various approaches can be used to determine an appropriate path for the UAV. For example, a route can be determined using an environment map, such as a topology map, as described in more detail herein with respect to FIGS. 9A and 9B. In some embodiments, the path can be configured to avoid one or more environmental obstacles that may interfere with the flight of the UAV. Alternatively or in combination, the route may be the shortest route between the current location and the initial location. The path may include one or more parts that the UAV has previously passed, and one or more parts that are different from the path that the UAV has previously passed. For example, the entire path may have been previously passed by the UAV. As another example, the path may include a plurality of passage points corresponding to a plurality of locations that the UAV has previously passed, as described in further detail herein with respect to FIGS. 8A and 8B. Conversely, the entire path may not have been previously passed by the UAV. In some embodiments, multiple potential paths can be generated and based on appropriate criteria (eg, minimizing total flight time, minimizing total flight distance, minimizing energy consumption) , Minimizing the number of obstacles encountered, maintaining a predetermined distance from multiple obstacles, maintaining a predetermined altitude range, etc.), multiple potentials One route can be selected from the routes. The environment map may include flight path information for the UAV (eg, previously routed paths, potential paths, selected paths).

  Once the route is determined, the route information can be processed to generate a plurality of instructions for controlling the UAV propulsion system for the purpose of moving the UAV along the route to the initial location. For example, the path may include a series of spatial location information and spatial direction information for the UAV that can convert the flight of the UAV into a plurality of control signals that direct the current location to the initial location. In some embodiments, the UAV may detect one or more obstacles along the path that interfere with its flight when navigating along the path. In order to allow the UAV to continue navigating to the initial location in such situations, for example, using the techniques previously described herein for method 500, The route can be changed to avoid obstacles.

  FIG. 7 illustrates a method 700 for controlling a UAV to return to an initial location while avoiding multiple obstacles, according to a number of embodiments. Similar to method 500, method 700 can be performed in a semi-automated or fully automated manner as part of the UAV auto return function. At least some of the steps of method 700 may be performed by one or more processors associated with the UAV.

  In step 710, at least one of the plurality of sensors is used to determine the initial location of the UAV. Optionally, a plurality of different sensor types can be used to determine the initial location. The location information may include information regarding the location (eg, altitude, latitude, longitude) and / or direction (eg, roll, pitch, yaw) of the UAV. The location information can be provided for a global coordinate system (eg, a geographic coordinate system) or a local coordinate system (eg, for a UAV).

  In step 720, a first signal is generated that causes the UAV to navigate within the environment, eg, to a location different from the initial location. As previously described herein, the first signal can be generated based on user input, multiple instructions provided by the autonomous control system, or a suitable combination thereof.

  In step 730, sensing data associated with the environment is received using at least one of the plurality of sensors. Optionally, sensing data can be received from a plurality of sensors different from the plurality of sensors used in step 710. Sensing data can be generated by combining multiple sensing signals from multiple different sensor types using appropriate sensor fusion-based multiple approaches, as described above. In some embodiments, the combined sensor data can be used to generate a representative environment map, as in embodiments previously discussed herein.

  In step 740, an instruction to return to the initial location is received. The return instruction can be entered by the user into the remote controller and subsequently sent to the UAV. Alternatively, the instructions can be automatically generated by the UAV, for example, automatically by an onboard processor and / or controller. In some embodiments, the instructions may include any user, such as in situations where the UAV detects that communication between the user device and the UAV has been disabled in the event of a failure or emergency. It can be generated independently from the input.

  In step 750, a second signal is generated that causes the UAV to return to an initial location along the path, and if at least one of the plurality of sensors detects an obstacle along the path, the obstacle should be avoided. Change the route. Optionally, the plurality of sensors used to detect the obstacle may be different from the plurality of sensors used in step 710 and / or step 730. Obstacles can be detected based on sensing data generated by a plurality of different sensor types. In some embodiments, obstacles can be detected using an environmental map (eg, an occupancy grid map) that is generated based on sensor data. The environment map may have been previously generated (eg, based on sensing data collected at step 730) or may be generated in real time as the UAV navigates along the route. Various approaches can be used to change the path in order to prevent the UAV from colliding with multiple detected obstacles, such as those previously described herein with respect to the method 500. The determination of the corrected route can be performed based on the obtained sensing data and / or obstacle information represented in the environment map.

  FIGS. 8A and 8B illustrate an algorithm 800 for controlling a UAV to return to an initial location using multiple pass points, according to multiple embodiments. The algorithm 800 can be implemented by a suitable processor and / or controller used to control the operation of the UAV. One or more steps of the algorithm 800 can be performed automatically, thereby providing auto-return functionality to the UAV.

  Following the start of UAV flight in step 802 (eg, takeoff), a determination is made in step 804 as to whether the UAV flight has ended. The determination may be based on whether the UAV has landed (eg, on a surface such as the ground), whether the UAV's propulsion system has been turned off, whether the user has provided an instruction to end the flight, etc. If the flight is over, the algorithm 800 is complete at step 806. Otherwise, the algorithm 800 proceeds to step 808 with the goal of determining the current location (eg, latitude, longitude, altitude) of the UAV. The current position can be determined using one or more sensors, at least some of which have a plurality of different types, as previously described herein. For example, a UAV can include GPS sensors and other sensor types such as vision sensors. Multiple situations in which the GPS sensor cannot provide reliable sensing data (for example, in indoor and / or low altitude environments where communication with multiple GPS satellites may be suboptimal, the GPS sensor is not compatible with multiple GPS satellites) Vision sensing data can be used in combination with or in place of GPS sensing data to compensate for (when the flight has not yet been established). Therefore, positioning accuracy can be improved by using a combination of GPS sensing data and vision sensing data. Optionally, the current position can be determined using an environment map that allows the UAV to be located relative to the global or local coordinate system. The environment map may be generated using multiple sensor fusion techniques as described above.

  In step 810, the current position information is stored in an appropriate data structure, such as a list of position information. The location list can be used to store a set of locations that the UAV has previously passed through, which may be referred to herein as “multiple passing points”. A plurality of passing points can be stored sequentially, thereby providing a representation of the flight path traveled by the UAV. In such embodiments, the first passing point stored in the position list corresponds to the initial position of the UAV, while the last passing point in the position list corresponds to the most recent position of the UAV. Therefore, as will be described below, a plurality of stored passing points can be moved in reverse order (from last to first) for the purpose of causing the UAV to return to the initial location.

  In step 812, a determination is made whether the UAV should initiate an auto return function with the goal of returning to the initial location corresponding to the first passing point stored in the position list. As previously described herein, the determination of auto-return can be made automatically or based on user input. If the auto return function is not initiated, the algorithm 800 returns to step 804 to begin the next iteration of pass point determination and storage. In some embodiments, steps 804, 808, 810, and 812 are performed at a desired frequency (e.g., continuously, for the purpose of generating and storing a series of previously passed points during a UAV flight. It can be repeated continuously (at predetermined time intervals) such as once every 0.5 seconds, 1 second, 2 seconds, 5 seconds, 10 seconds, 20 seconds, or 30 seconds. Once it is determined at step 812 that the UAV should autoreturn, an autoreturn function is performed at step 814, as described in more detail below with respect to FIG. 8B. After the auto return function is complete, at step 806, the algorithm 800 is complete.

  FIG. 8B illustrates multiple sub-steps of step 814 of algorithm 800, according to multiple embodiments. Following the start of the auto-return function in step 816, in step 818, a determination is made whether the position list (list of multiple passing points) is empty. If the list is empty, this indicates that the UAV has reached its initial location, and in step 820 the auto return function is complete. If the list is not empty, this indicates that the UAV has not yet reached its initial location. In such examples, the next target position (passing point) is then retrieved from the position list in step 822.

  Next, in step 824, the current position of the UAV is determined. The current position can be determined using any of a number of approaches described herein, such as using multi-sensor fusion and / or environment mapping. In some embodiments, the spatial relationship between the current position and the retrieved target position is determined in order to generate a flight path between the current position and the target position. As previously described, a flight path can be generated based on environmental mapping data to avoid multiple potential flight obstacles.

  In step 826, the UAV navigates from the current position to the target position using, for example, the flight path generated in step 824. As the UAV moves toward the target position, the UAV can acquire and process sensor data to determine if there are any obstacles that interfere with its flight. For example, the sensor data can be used to provide a local environment map that includes obstacle occupancy information, as discussed above. Exemplary sensors that can be used to provide obstacle information include vision sensors, lidar sensors, ultrasonic sensors, and the like. Based on the sensor data, a determination is made in step 828 as to whether an obstacle has been detected. If no obstacle is detected, the UAV completes its navigation towards the target position and the auto return function proceeds to step 818 to begin the next iteration. If an obstacle is detected, an appropriate obstacle avoidance operation is determined and executed as shown in step 830. For example, the flight path can be changed (eg, based on environmental map data) so that the UAV navigates to the target location without colliding with the detected obstacle.

  FIGS. 9A and 9B illustrate an algorithm 900 for controlling a UAV to return to a target location using a topology map, according to embodiments. Similar to algorithm 800, algorithm 900 can be implemented by a suitable processor and / or controller used to control the operation of the UAV. One or more steps of the algorithm 900 can be performed automatically, thereby providing auto-return functionality to the UAV.

  Following the start of UAV flight in step 902 (eg, takeoff), as in step 802 of algorithm 800, a determination is made in step 904 as to whether the UAV flight has ended. If the flight is over, the algorithm 900 is complete at step 906. Otherwise, the algorithm 900 proceeds to step 908 for the purpose of determining not only the environment information related to the environment surrounding the UAV, but also the current location of the UAV. In some embodiments, as previously described herein, environmental information includes information regarding the spatial arrangement, geometry, and / or characteristics of one or more environmental objects. Current location and / or environmental information can be determined using sensing data obtained from one or more sensors. Optionally, sensing data from multiple different sensor types can be processed and combined using multiple sensor fusion techniques to improve the reliability and accuracy of the sensing results. For example, sensing data can be obtained from a GPS sensor and at least one other sensor type (eg, a vision sensor, a lidar sensor and / or an ultrasonic sensor).

  In step 910, the current position and environment information are added to the topology map representing the operating environment. In alternative embodiments, other types of maps can also be used (eg, multiple metric maps such as grid maps). The topology map can be used to store location information corresponding to multiple locations that the UAV has previously passed through, thereby providing a record of the flight path of the UAV. In addition, the topology map can be used to depict connectivity between various environmental locations according to embodiments previously described herein. Optionally, the topology map may include metric information such as information indicating multiple distances between the represented locations as well as information indicating the spatial location and geometry of multiple environmental objects. At least some portions of the topology map can be generated during a UAV flight using, for example, multi-sensor mapping. Alternatively or in combination, one or more portions of the topology map can be generated prior to UAV operation and provided to the UAV (eg, transmitted from a remote controller and / or stored in on-board memory). In the latter embodiment, the topology map can be refined or modified during UAV operation based on the collected sensor data to reflect current environmental information.

  In step 912, as in step 812 of algorithm 800, a determination is made as to whether the UAV should initiate an auto-return function for the purpose of returning to the target position. If the auto return function is not started, the algorithm 900 returns to step 904. As the UAV navigates within the environment, steps 904, 908, 910 and 912 are performed at a desired frequency (eg, continuously, with the goal of updating the topology map with the most recent location and environment information). It can be repeated continuously (at predetermined time intervals) such as once every 0.5 seconds, 1 second, 2 seconds, 5 seconds, 10 seconds, 20 seconds, or 30 seconds. Once it is determined at step 912 that the UAV should be autoreturned, an autoreturn function is performed at step 914 as described in more detail below with respect to FIG. 9B. After the auto return function is complete, in step 906, the algorithm 900 is complete.

  FIG. 9B illustrates multiple sub-steps of step 914 of algorithm 900, according to multiple embodiments. Following the start of the auto return function in step 916, in step 918, a determination is made whether the UAV has reached the target position. The target position may be a position specified by the user or a position automatically determined by the UAV. The target position may be a position where the UAV has previously passed, such as the initial position of the UAV at the start of flight (eg, the location of takeoff). Alternatively, the target location may be a location where no UAV has previously passed. If it is determined that the UAV has reached the target position, at step 920, the auto return function is complete. Otherwise, auto return proceeds to step 922.

  In step 922, the current location of the UAV is determined using any of a number of approaches described herein, such as using multi-sensor fusion and / or environment mapping. In step 924, the topology map position corresponding to the current UAV position is determined, thereby allowing the UAV to be placed relative to the environmental information represented in the topology map. In step 926, a flight path from the current position to the target position is determined based on the topology map. In some embodiments, one or more portions of the returning flight path may correspond to the flight path that the UAV has previously passed. Alternatively, one or more portions of the flight path may be different from the previous flight path of the UAV. The flight path can be determined based on any suitable criteria. For example, the flight path can be configured to avoid multiple environmental obstacles. As another example, the flight path can be configured to minimize the total distance traveled by the UAV to reach the target position. Subsequently, in step 928, a plurality of UAV flight controls are adjusted based on the determined flight path to cause the UAV to move along the flight path. For example, the flight path can identify a series of positions and / or directions for the UAV, and generate appropriate control signals to allow the UAV to take a specific position and / or direction to multiple UAV propulsion systems. Can be sent.

  As the UAV moves toward the target position, the UAV can detect multiple obstacles and perform multiple obstacle avoidance operations in an appropriate manner, as depicted in steps 930 and 932. . The techniques for performing obstacle detection and avoidance may each be similar to that described for steps 828 and 830 of algorithm 800. Optionally, obstacle detection and / or avoidance may be based on the generated topology map. For example, once an obstacle is identified, the environmental information represented in the topology map can be used to determine appropriate multiple changes to the flight path that avoid multiple potential collisions.

  The multiple systems, devices and methods described herein can be applied to a wide variety of movable objects. As previously mentioned, any description herein of an aircraft may be applied and used with any movable object. The movable object of the present invention may be, for example, in the air (for example, a fixed-wing aircraft, a rotary-wing aircraft, or an aircraft that does not have a plurality of fixed wings or a plurality of rotary wings), underwater (for example, a ship or a submarine), and the ground (for example, Cars, trucks, buses, vans, motor vehicles such as motorcycles, movable structures or frames such as rods or fishing rods, or trains), underground (for example, subways), outer space (for example, spaceships, satellites, or space exploration) Or any combination of these environments can be configured to run in any suitable environment. The movable object may be a fuselage such as a fuselage described elsewhere herein. In some embodiments, the movable object can be attached to a living body such as a human or animal. Suitable animals may include birds, dogs, cats, horses, cows, sheep, pigs, delphine, rodents or insects.

  The movable object may be freely movable within the environment for six degrees of freedom (eg, three degrees of freedom for translation and three degrees of freedom for rotation). Alternatively, the movement of the movable object can be limited for one or more degrees of freedom, such as by a predetermined path, passage or direction. The movement can be actuated by any suitable drive mechanism such as an engine or motor. The drive mechanism of the movable object can be powered by any suitable energy source, such as electrical energy, magnetic energy, solar energy, wind energy, gravity energy, chemical energy, nuclear energy, or any suitable combination thereof. Can be supplied. The movable object may be self-propelled via a propulsion system, as described elsewhere herein. The propulsion system may optionally operate with an energy source such as electrical energy, magnetic energy, solar energy, wind energy, gravity energy, chemical energy, nuclear energy, or any suitable combination thereof. Alternatively, the movable object may be carried by an organism.

  In some cases, the movable object may be a fuselage. Suitable multiple aircraft may include multiple water vehicles, multiple aircraft, multiple space aircraft, or multiple ground vehicles. For example, an aircraft may be a fixed wing aircraft (eg, an airplane, a glider), a rotary wing aircraft (eg, a helicopter, a rotary wing aircraft), an aircraft having both fixed and rotary wings, or an aircraft that has neither (eg, an airship) , Hot air balloon). The airframe may be self-propelled, for example, self-propelled in the air, water, underwater, outer space, ground, or underground. The self-propelled airframe can utilize a propulsion system, such as a propulsion system including one or more engines, motors, wheels, axles, magnets, rotor blades, propellers, wings, nozzles, or any suitable combination thereof. . In some cases, moving objects take off from a surface, land on a surface, maintain their current position and / or direction (eg, hover), change direction, and / or change position A propulsion system can be used to make this possible.

  The movable object can be remotely controlled by the user, or can be controlled on the spot by a passenger in or on the movable object. In some embodiments, the movable object is an unmanned movable object such as a UAV. An unmanned movable object such as a UAV may not have a passenger on the movable object. The movable object can be controlled by a human or autonomous control system (eg, a computer control system), or any suitable combination thereof. The movable object can be an autonomous or semi-autonomous robot, such as a robot configured with artificial intelligence.

  The movable object can have any suitable size and / or dimensions. In some embodiments, the movable object may be sized and / or dimensioned to carry a human occupant in or on the fuselage. Alternatively, the movable object may be smaller than the size and / or dimensions that allow a human occupant to be placed in or on the aircraft. The movable object may be of a size and / or dimensions suitable for being lifted or carried by a human. Alternatively, the movable object may be larger than the size and / or dimensions suitable for being lifted or carried by a human. In some cases, the movable object is approximately 2 cm, 5 cm, 10 cm, 50 cm, 1 m, 2 m, 5 m, or a maximum dimension (eg, length, width, height, diameter, diagonal, less than or equal to 10 m). ). The maximum dimension may be approximately equal to or greater than 2 cm, 5 cm, 10 cm, 50 cm, 1 m, 2 m, 5 m, or 10 m. For example, the distance between the shafts of the opposed rotor blades of the movable object may be approximately equal to or less than 2 cm, 5 cm, 10 cm, 50 cm, 1 m, 2 m, 5 m, or 10 m. Alternatively, the distance between the shafts of opposing rotor blades may be approximately equal to or greater than 2 cm, 5 cm, 10 cm, 50 cm, 1 m, 2 m, 5 m, or 10 m.

In some embodiments, the movable object may have a volume of less than 100 cm × 100 cm × 100 cm, a volume of less than 50 cm × 50 cm × 30 cm, or a volume of less than 5 cm × 5 cm × 3 cm. The total volume of the movable object is approximately 1 cm ³ , 2 cm ³ , 5 cm ³ , 10 cm ³ , 20 cm ³ , 30 cm ³ , 40 cm ³ , 50 cm ³ , 60 cm ³ , 70 cm ³ , 80 cm ³ , 90 cm ³ , 100 cm ³ , 150 cm ^{3. , 200cm 3, 300cm 3, 500cm} 3, 750cm 3, 1000cm 3, 5000cm 3, 10,000cm 3, 100,000cm 3, 1m 3, or equal to or 10 m ^3, may be less than it. Conversely, the total volume of the movable object is approximately 1 cm ³ , 2 cm ³ , 5 cm ³ , 10 cm ³ , 20 cm ³ , 30 cm ³ , 40 cm ³ , 50 cm ³ , 60 cm ³ , 70 cm ³ , 80 cm ³ , 90 cm ³ , 100 cm ^{3. , 150cm 3, 200cm 3, 300cm} 3, 500cm 3, 750cm 3, 1000cm 3, 5000cm 3, 10,000cm 3, 100,000cm 3, 1m 3, or equal to or 10 m ^3, may be greater than that.

In some embodiments, the movable object is ^{approximately, 32,000cm 2, 20,000cm 2, 10,000cm} 2, 1,000cm 2, 500cm 2, 100cm 2, 50cm 2, 10cm 2, or a 5 cm ² It may have an installation area equal to or less than that (which may refer to the lateral cross-sectional area covered by the movable object). Conversely, the footprint is ^{approximately, 32,000cm 2, 20,000cm 2, 10,000cm} 2, 1,000cm 2, 500cm 2, 100cm 2, 50cm 2, 10cm 2, or, or equal to 5 cm ^2, it It may be larger.

  In some cases, the movable object may have a weight of 1000 kg or less. The weight of the movable object is approximately 1000 kg, 750 kg, 500 kg, 200 kg, 150 kg, 100 kg, 80 kg, 70 kg, 60 kg, 50 kg, 45 kg, 40 kg, 35 kg, 30 kg, 25 kg, 20 kg, 15 kg, 12 kg, 10 kg, 9 kg, 8 kg, It may be equal to or less than 7 kg, 6 kg, 5 kg, 4 kg, 3 kg, 2 kg, 1 kg, 0.5 kg, 0.1 kg, 0.05 kg, or 0.01 kg. Conversely, the weight is approximately 1000 kg, 750 kg, 500 kg, 200 kg, 150 kg, 100 kg, 80 kg, 70 kg, 60 kg, 50 kg, 45 kg, 40 kg, 35 kg, 30 kg, 25 kg, 20 kg, 15 kg, 12 kg, 10 kg, 9 kg, 8 kg, It may be equal to or heavier than 7 kg, 6 kg, 5 kg, 4 kg, 3 kg, 2 kg, 1 kg, 0.5 kg, 0.1 kg, 0.05 kg, or 0.01 kg.

  In some embodiments, the movable object may be small relative to the luggage carried by the movable object. The load may include loads and / or support mechanisms, as will be described in more detail below. In some examples, the ratio of the weight of the movable object to the weight of the load may be approximately equal to, greater than, or less than approximately 1: 1. In some cases, the ratio of the weight of the movable object to the weight of the load may be approximately equal to, greater than, or less than approximately 1: 1. Optionally, the ratio of the weight of the support mechanism to the weight of the load may be approximately equal to, greater than, or less than approximately 1: 1. If desired, the ratio of the weight of the movable object to the weight of the load is equal to or less than 1: 2, 1: 3, 1: 4, 1: 5, 1:10, or a smaller ratio. There may be. Conversely, the ratio of the weight of the moving object to the weight of the load is equal to or greater than 1: 2, 1: 3, 1: 4, 1: 5, 1:10, or even smaller ratios. obtain.

  In some embodiments, the movable object may have a low energy consumption. For example, the movable object may have a usage amount approximately equal to or less than 5 W / h, 4 W / h, 3 W / h, 2 W / h, 1 W / h, or a smaller numerical value. In some cases, the support mechanism of the movable object may have a low energy consumption. For example, the support mechanism may be approximately less than 5 W / h, 4 W / h, 3 W / h, 2 W / h, 1 W / h, or even less than the amount used. Optionally, the load of movable objects may have a low energy consumption, such as approximately 5 W / h, 4 W / h, 3 W / h, 2 W / h, 1 W / h, or even smaller values. Good.

  FIG. 10 illustrates an unmanned aerial vehicle (UAV) 1000 in accordance with embodiments of the present invention. A UAV may be an example of a movable object described herein. The UAV 1000 can include a propulsion system having four rotor blades 1002, 1004, 1006, and 1008. Any number of rotor blades may be provided (eg, one, two, three, four, five, six, or more). The plurality of rotor blades can be a plurality of embodiments of a plurality of self-tightening rotor blades described elsewhere herein. The unmanned aerial vehicle's multiple rotors, multiple rotor assembly, or other propulsion systems may be used to maintain / change the direction and / or location of the unmanned aircraft May be changed. The distance between the shafts of opposing rotor blades can be any suitable length 1010. For example, the length 1010 can be equal to or less than 2 m, or can be equal to or less than 5 m. In some embodiments, the length 1010 can be in the range of 40 cm to 1 m, 10 cm to 2 m, or 5 cm to 5 m. Any description herein for UAVs may apply to movable objects, such as different types of movable objects, and vice versa.

  In some embodiments, the movable object can be configured to carry luggage. The luggage can include one or more of a plurality of passengers, cargo, equipment, equipment, and the like. The luggage can be provided in the housing. The casing may be separate from the casing of the movable object, or may be a part of the casing for the movable object. Alternatively, the luggage may be provided with a housing, while the movable object does not have a housing. Alternatively, multiple pieces of luggage or the entire luggage can be provided in the housing. The load can be firmly fixed to the movable object. Optionally, the load can be movable relative to the movable object (eg, can be translated or rotated relative to the movable object).

  In some embodiments, the load includes a load. The load can be configured not to perform any action or function. Alternatively, the load may be a load configured to perform an action or function, also known as a functional load. For example, the load can include one or more sensors that survey one or more targets. Any suitable sensor can be incorporated into the load, such as an image capture device (eg, a camera), an audio capture device (eg, a radiation surface microphone), an infrared imaging device, or an ultraviolet imaging device. Sensors can provide static sensing data (eg, photographs) or dynamic sensing data (eg, video). In some embodiments, the sensor provides sensing data for a load target. Alternatively or in combination, the load can include one or more emitters that provide multiple signals to one or more targets. Any suitable emitter can be used, such as an illumination source or a sound source. In some embodiments, the load includes one or more transceivers, such as for communication with a module remote from the movable object. Optionally, the load can be configured to interact with the environment or goals. For example, the load can include a mechanism capable of manipulating multiple objects, such as a tool, equipment, or robot arm.

  Optionally, the luggage may include a support mechanism. A support mechanism can be provided for the load, the load being either directly (eg, directly contacting the movable object) or indirectly (eg, not contacting the movable object), the movable object via the support mechanism. Can be linked to. Conversely, the load can be mounted on a movable object without the need for a support mechanism. The load can be formed integrally with the support mechanism. Alternatively, the load can be removably coupled to the support mechanism. In some embodiments, the load can include one or more load elements, one or more of the load elements being on the movable object and / or the support mechanism described above. In contrast, it may be movable.

  The support mechanism can be formed integrally with the movable object. Alternatively, the support mechanism can be detachably connected to the movable object. The support mechanism can be directly or indirectly connected to the movable object. The support mechanism can provide support to the load (eg, can carry at least a portion of the weight of the load). The support mechanism can include a suitable mounting structure (eg, a gimbal platform) that can stabilize and / or direct the movement of the load. In some embodiments, the support mechanism can be adapted to control the state (eg, position and / or orientation) of the load relative to the movable object. For example, the support mechanism can be moved relative to the movable object (eg, 1, 2 or 3 degrees of translation) so that the load maintains its position and / or orientation relative to the appropriate reference frame regardless of the movement of the movable object. , And / or about 1, 2 or 3 degrees of rotation). The reference frame may be a fixed reference frame (eg, the ambient environment). Alternatively, the reference frame can be a moving reference frame (eg, movable object, load target).

  In some embodiments, the support mechanism can be configured to allow movement of the load relative to the support mechanism and / or the movable object. The movement can be translational up to 3 degrees of freedom (eg along 1, 2 or 3 axes) or up to 3 degrees of freedom (eg 1, 2 or 3 axes). (Around) rotational movement, or any suitable combination thereof.

  In some cases, the support mechanism can include a support mechanism frame assembly and a support mechanism actuation assembly. The support mechanism frame assembly can provide structural support to the load. The support mechanism frame assembly may include individual support mechanism frame components, some of which may be movable relative to each other. The support mechanism actuation assembly can include one or more actuators (e.g., multiple motors) that actuate the movement of individual support mechanism frame components. The plurality of actuators may allow multiple support mechanism frame components to move simultaneously, or may be configured to allow one support mechanism frame component to move at a time. The movement of the support mechanism frame component can generate a corresponding movement of the load. For example, the support mechanism actuation assembly can actuate rotation of one or more support mechanism frame components about one or more rotation axes (eg, roll axis, pitch axis, or yaw axis). By rotating one or more support mechanism frame components, the load can be rotated about one or more axes of rotation relative to the movable object. Alternatively or in combination, the support mechanism actuation assembly can actuate the translation of one or more support mechanism frame components along one or more translation axes, thereby providing one or more to the movable object. A translational movement of the load along the corresponding axis can be generated.

  In some embodiments, the terminal can control the movement of the movable object, the support mechanism, and the load relative to a fixed reference frame (eg, the surrounding environment) and / or relative to each other. The terminal may be a remote control device that is remote from the movable object, the support mechanism and / or the load. The terminal can be located on the support platform or can be attached to the support platform. Alternatively, the terminal can be a handheld device or a wearable device. For example, the terminal can include a smartphone, tablet, laptop, computer, glasses, glove, helmet, microphone, or any combination thereof. The terminal may include a user interface such as a keyboard, mouse, joystick, touch screen, or display. Any suitable user input, such as manually entered commands, voice control, gesture control, or position control (eg, via terminal movement, location, or tilt) interacts with the terminal Can be used as much as possible.

  The terminal may be used to control any suitable state of the movable object, support mechanism, and / or load. For example, the terminals can be used to control the position and / or orientation of movable objects, support mechanisms and / or loads from each other and / or with respect to a fixed reference. In some embodiments, the terminal is used to control a movable object, support mechanism, and / or individual elements of the load, such as a support mechanism actuation assembly, load sensor, or load emitter. Can be done. The terminal may include a wireless communication device adapted to communicate with one or more of a movable object, a support mechanism, or a load.

  The terminal may include a display unit suitable for viewing information on movable objects, support mechanisms and / or loads. For example, the terminal can be configured to display information about movable objects, support mechanisms and / or loads regarding position, translation speed, translation acceleration, direction, angular velocity, angular acceleration, or any suitable combination thereof. . In some embodiments, the terminal displays information provided by the load, such as data provided by the functional load (eg, multiple images recorded by a camera or other imaging device). Can be displayed.

  Optionally, the same terminal receives and / or displays information from the movable object, support mechanism and / or load, and the movable object, support mechanism and / or load, or movable object, support mechanism and / or Or you may perform both with controlling the state of a load. For example, the terminal may control the positioning of the load with respect to a certain environment while displaying image data captured by the load or information about the position of the load. Alternatively, different terminals may be used for different functions. For example, the first terminal may control the movement or state of the movable object, support mechanism and / or load, while the second terminal may receive information from the movable object, support mechanism and / or load. May be received and / or displayed. For example, a first terminal can be used to control the positioning of a load relative to an environment while a second terminal displays image data captured by the load. Between a movable object and an integrated terminal that both controls the movable object and receives data, or between a movable object and multiple terminals that both control the movable object and receive data, Various communication modes may be used. For example, at least two different communication modes may be formed between a movable object and a terminal that performs both control of the movable object and data reception from the movable object.

  FIG. 11 illustrates a movable object 1100 including a support mechanism 1102 and a load 1104 according to multiple embodiments. Although the movable object 1100 is depicted as an aircraft, this depiction is not intended to be limiting and any suitable type of movable object may be used, as previously described herein. Those skilled in the art will appreciate that any of the embodiments described herein in connection with multiple aircraft systems can be applied to any suitable movable object (eg, UAV). right. In some cases, the load 1104 may be provided on the movable object 1100 without the need for the support mechanism 1102. The movable object 1100 may include a propulsion mechanism 1106, a sensing system 1108, and a communication system 1110.

  As described above, the propulsion mechanism 1106 includes a plurality of rotor blades, a plurality of propellers, a plurality of blades, a plurality of engines, a plurality of motors, a plurality of wheels, a plurality of axles, a plurality of magnets, or a plurality of nozzles. One or more of them. For example, the propulsion mechanism 1106 may be a plurality of self-tightening rotor blades, a plurality of rotor blade assemblies, or other plurality of rotary propulsion units, as disclosed elsewhere herein. . The movable object may have one or more, two or more, three or more, three or four or more, four or five or more propulsion mechanisms. The plurality of propulsion mechanisms may all be the same type. Alternatively, the one or more propulsion mechanisms may be different types of propulsion mechanisms. The plurality of propulsion mechanisms 1106 can be mounted on the movable object 1100 using any suitable means, such as a support element (eg, a drive shaft), as described elsewhere herein. . The propulsion mechanism 1106 can be mounted on any suitable portion of the movable object 1100, such as top, bottom, front, back, multiple sides, or any combination thereof.

  In some embodiments, the multiple propulsion mechanisms 1106 do not require any horizontal movement of the movable object 1100 (eg, without traveling on the runway), and the movable object 1100 takes off vertically from a surface. Or it can be possible to land vertically on a surface. Optionally, the plurality of propulsion mechanisms 1106 are operable to allow the movable object 1100 to hover in the air to a particular position and / or direction. One or more of the plurality of propulsion mechanisms 1100 may be controlled independently of the other plurality of propulsion mechanisms. Alternatively, multiple propulsion mechanisms 1100 can be configured to be controlled simultaneously. For example, the movable object 1100 can have a plurality of horizontally oriented rotors that can provide lift and / or propulsion to the movable object. The plurality of horizontally oriented rotors may be operated to provide the movable object 1100 with vertical takeoff performance, vertical landing performance, and hover performance. In some embodiments, one or more of the horizontally oriented rotors may spin in a clockwise direction, while one or more of the horizontally rotors are You may spin in a counterclockwise direction. For example, the number of clockwise rotating blades may be equal to the number of counterclockwise rotating blades. The rotational speed of each of the plurality of horizontally oriented rotors can be independently varied to control the lift and / or propulsion generated by each rotor, thereby allowing (for example, up to 3 Adjust the spatial arrangement, speed and / or acceleration of the movable object 1100 (for translation up to degrees and rotational movement up to 3 degrees).

  Sensing system 1108 includes one or more sensors that can sense the spatial arrangement, velocity, and / or acceleration of movable object 1100 (eg, for translational movements up to 3 degrees and rotational movements up to 3 degrees). Can be included. The one or more sensors can include a plurality of global positioning system (GPS) sensors, motion sensors, inertial sensors, proximity sensors, or image sensors. Sensing data provided by the sensing system 1108 controls the spatial arrangement, velocity and / or direction of the movable object 1100 (eg, using an appropriate processing unit and / or control module as described below). Can be used as much as possible. Alternatively, the sensing system 1108 may provide data about the environment surrounding the movable object, such as multiple weather conditions, proximity to multiple potential obstacles, multiple geographical feature locations, multiple artificial building locations, etc. Can be used to provide.

  The communication system 1110 can communicate with a terminal 1112 having a communication system 1114 via a wireless signal 1116. The communication systems 1110, 1114 may include any number of transmitters, receivers, and / or transceivers suitable for wireless communication. The communication may be unidirectional so that data can be transmitted in only one direction. For example, unidirectional communication may include only a movable object 1100 that transmits data to the terminal 1112 or vice versa. Data may be transmitted from one or more transmitters of communication system 1110 to one or more receivers of communication system 1112 or vice versa. Alternatively, the communication may be two-way communication so that data can be transmitted in both directions between the movable object 1100 and the terminal 1112. Bi-directional communication can include transmitting data from one or more transmitters of communication system 1110 to one or more receivers of communication system 1114 or vice versa.

  In some embodiments, terminal 1112 provides control data for one or more of movable object 1100, support mechanism 1102 and load 1104, and moveable object 1100, support mechanism 1102 and load 1104. Receive information from one or more of the above (eg, data sensed by the load, such as position information and / or movement information of a movable object, support mechanism or load, image data captured by a load camera) can do. In some cases, the control data from the terminal may include multiple relative positions, multiple movements, multiple actuations, or multiple instructions for multiple controls of the movable object, support mechanism and / or load. For example, the control data may result in a change in the location and / or orientation of the movable object (eg, by control of a plurality of propulsion mechanisms 1106) or movement of the load relative to the movable object (eg, by control of the support mechanism 1102). May be provided. Control of the operation of the camera or other imaging device (for example, taking a plurality of still images or moving images, zooming in or out, turning on or off, switching a plurality of imaging modes, and the like according to control data from the terminal, image This may result in control of the load, such as changing resolution, changing focus, changing depth of field, changing exposure time, changing viewing angle or field of view. In some cases, multiple communications from movable objects, support mechanisms, and / or loads may include information from one or more sensors (eg, sensing system 1108 or load 1104). The plurality of communications may include sensed information from one or more different types of sensors (eg, multiple GPS sensors, multiple motion sensors, multiple inertial sensors, multiple proximity sensors, or multiple image sensors). Good. Such information may relate to the position (eg, location, direction, movement, or acceleration) of the movable object, support mechanism, and / or load. Such information from the load may include data captured by the load or a sensed state about the load. The control data transmitted and provided by the terminal 1112 can be configured to control one or more states of the movable object 1100, the support mechanism 1102, or the load 1104. Alternatively or in combination, the support mechanism 1102 and the load 1104 may also each include a communication module configured to communicate with the terminal 1112, whereby the terminal independently includes the movable object 1100, the support mechanism. Communication with each of 1102 and the load 1104 and control of these can be performed.

  In some embodiments, the movable object 1100 can be configured to communicate with other remote devices in addition to or in place of the terminal 1112. Terminal 1112 may also be configured to communicate with other remote devices as well as movable object 1100. For example, the movable object 1100 and / or the terminal 1112 may communicate with other movable objects or other movable object support mechanisms or loads. If desired, the remote device may be a second terminal or other computing device (eg, a computer, laptop, tablet, smartphone, or other mobile device). The remote device can be configured to send data to the movable object 1100, receive data from the movable object 1100, send data to the terminal 1112, and / or receive data from the terminal 1112. Optionally, the remote device can be connected to the Internet or other communication network so that data received from movable object 1100 and / or terminal 1112 can be uploaded to a website or server.

  FIG. 12 is a schematic diagram as a block diagram of a system 1200 for controlling a movable object, according to multiple embodiments. System 1200 can be used in combination with any suitable embodiment of the systems, devices, and methods disclosed herein. System 1200 can include a sensing module 1202, a processing unit 1204, a non-transitory computer readable medium 1206, a control module 1208, and a communication module 1210.

  The sensing module 1202 can utilize a plurality of different types of sensors that collect information about a plurality of movable objects in a plurality of different ways. The plurality of different types of sensors may sense a plurality of different types of signals or a plurality of signals from a plurality of different sources. For example, the plurality of sensors can include a plurality of inertial sensors, a plurality of GPS sensors, a plurality of proximity sensors (eg, a lidar), or a plurality of vision / image sensors (eg, a camera). The sensing module 1202 can be operatively coupled to a processing unit 1204 having a plurality of processors. In some embodiments, the sensing module can be operably coupled to a transmission module 1212 (eg, a Wi-Fi image transmission module) configured to transmit sensing data directly to an appropriate external device or system. For example, the transmission module 1212 can be used to transmit multiple images captured by the camera of the sensing module 1202 to a remote terminal.

  The processing unit 1204 can include one or more processors, such as a programmable processor (eg, a central processing unit (CPU)). The processing unit 1204 can be operatively coupled to a non-transitory computer readable medium 1206. The non-transitory computer readable medium 1206 can store logic, code, and / or multiple program instructions that can be executed by the processing unit 1204 for the purpose of performing one or more steps. Non-transitory computer readable media may include one or more memory units (eg, removable media such as an SD card or random access memory (RAM) or an external storage device). In some embodiments, data from the sensing module 1202 can be communicated directly to and stored in multiple memory units of the non-transitory computer readable medium 1206. The plurality of memory units of the non-transitory computer readable medium 1206 may include logic, code, and / or code that can be executed by the processing unit 1204 to perform any suitable embodiment of the methods described herein. Or a plurality of program instructions can be stored. For example, the processing unit 1204 can be configured to execute a plurality of instructions that cause one or more processors of the processing unit 1204 to analyze sensing data generated by the sensing module. The plurality of memory units can store sensing data from the sensing module that is to be processed by the processing unit 1204. In some embodiments, multiple memory units of non-transitory computer readable media 1206 can be used to store multiple processing results generated by processing unit 1204.

  In some embodiments, the processing unit 1204 can be operatively coupled to a control module 1208 that controls the state of the movable object. For example, the control module 1208 can be configured to control multiple propulsion mechanisms of the movable object to adjust the spatial arrangement, speed and / or acceleration of the movable object with respect to six degrees of freedom. Alternatively or in combination, the control module 1208 can control one or more of the state of the support mechanism, the state of the load, or the state of the sensing module.

  The processing unit 1204 can be operatively coupled to a communication module 1210 that is configured to transmit and / or receive data from one or more external devices (eg, a terminal, display device, or remote controller). Any suitable communication means such as wired communication or wireless communication can be used. For example, the communication module 1210 uses one or more of a local area network (LAN), a wide area network (WAN), an infrared ray, a wireless device, WiFi, a point-to-point (P2P) network, a telecommunication network, cloud communication, and the like. it can. Optionally, multiple relay stations such as multiple towers, multiple satellites, or multiple mobile stations can be used. The plurality of wireless communications may be proximity dependent or proximity independent. In some embodiments, a line of sight may or may not be required for communication. The communication module 1210 receives one or more of sensing data from the sensing module 1202, a plurality of processing results generated by the processing unit 1204, predetermined control data, a plurality of user commands from a terminal or a remote controller, and the like. Can transmit and / or receive.

  The multiple components of system 1200 can be arranged in any suitable configuration. For example, one or more of the components of system 1200 can be located on a movable object, a support mechanism, a load, a terminal, a sensing system, or an additional external device that communicates with one or more of the above. In addition, FIG. 12 depicts a single processing unit 1204 and a single non-transitory computer readable medium 1206, which is not intended to be limiting by those skilled in the art. It will be appreciated that the system 1200 can include multiple processing units and / or non-transitory computer readable media. In some embodiments, one or more of the plurality of processing units and / or non-transitory computer readable media is on a movable object, on a support mechanism, on a load, on a terminal, on a sensing module, or as described above. It can be located on a number of different locations, such as on an additional external device that communicates with one or more, or a suitable combination thereof. This allows any suitable aspect of the processing functions and / or memory functions performed by the system 1200 to occur at one or more of the aforementioned locations.

  As used herein, A and / or B includes one or more of A or B and combinations thereof such as A and B.

While preferred embodiments of the present invention have been shown and described herein, it should be understood by those skilled in the art that such embodiments are provided for illustrative purposes only. It will be clear to you. Numerous variations, modifications, and substitutions will now occur to those skilled in the art without departing from the invention. It should be understood that various alternatives to the embodiments of the invention described herein may be employed when practicing the invention. The following claims define the scope of the present invention, and are intended to encompass the methods and structures and their equivalents within the scope of these claims. Has been.
[Item 1]
A method for controlling a movable object in an environment,
Determining an initial location of the movable object using at least one of a plurality of sensors carried by the movable object;
Generating a first signal that causes the movable object to navigate within the environment;
Receiving sensing data associated with the environment using at least one of the plurality of sensors;
Generating an environment map representing at least a portion of the environment based on the sensing data;
Receiving an instruction to return to the initial location;
Generating a second signal that causes the movable object to return to the initial location based on the environment map.
[Item 2]
The method of claim 1, wherein the movable object is configured to detect the presence of a flight restriction region and not fly within a predetermined distance of the flight restriction region.
[Item 3]
The method of claim 1, wherein the plurality of sensors comprises a global positioning system (GPS) sensor.
[Item 4]
The method of claim 1, wherein the plurality of sensors comprises a vision sensor.
[Item 5]
The method of claim 1, wherein the plurality of sensors includes proximity sensors.
[Item 6]
The method of claim 5, wherein the proximity sensor comprises at least one of the following: a lidar sensor, an ultrasonic sensor or a flight camera sensor.
[Item 7]
The method of claim 1, wherein the environment map comprises a topological map.
[Item 8]
The method of claim 1, wherein the environment map comprises a metric map.
[Item 9]
9. The method of claim 8, wherein the metric map comprises at least one of the following point cloud, 3D grid map, 2D grid map, or 2.5D grid map.
[Item 10]
The method of claim 8, wherein the metric map comprises an occupancy grid map.
[Item 11]
Generating the second signal comprises:
Using at least one of the plurality of sensors to determine a current location of the movable object;
Determining a route from the current location to the initial location based on the environment map;
Generating a signal that causes the movable object to move along a path back to the initial location.
[Item 12]
The method of claim 11, wherein determining the route includes determining a shortest route from the current location to the initial location.
[Item 13]
12. The method of claim 11, wherein determining the path comprises determining a path from the current location to the initial location that avoids one or more obstacles in the environment.
[Item 14]
The method of claim 11, wherein the path includes one or more portions through which the movable object has previously passed.
[Item 15]
The method of claim 11, wherein the path is different from a path through which the movable object has previously passed.
[Item 16]
The method of claim 11, wherein the path is a flight path of the movable object.
[Item 17]
The method of claim 16, wherein the path includes a spatial position and direction of the movable object.
[Item 18]
The route includes a plurality of passage points corresponding to a plurality of positions where the movable object has previously passed, and the plurality of passage points are recorded when the movable object navigates in the environment. The method described.
[Item 19]
The method of claim 18, wherein the plurality of passing points are recorded in real time as the movable object navigates within the environment.
[Item 20]
The method of claim 18, wherein the plurality of passing points are recorded at predetermined time intervals as the movable object navigates within the environment.
[Item 21]
The method of claim 18, wherein the plurality of passage points are stored in a list data structure.
[Item 22]
Generating the signal to move the movable object along the path comprises:
Detecting obstacles located along the path in the environment;
Changing the route to avoid the obstacle;
Generating a signal to cause the movable object to move along the altered path.
[Item 23]
A system for controlling a moving object in an environment,
A plurality of sensors carried by the movable object;
With one or more processors,
The one or more processors may be individually or collectively
Using at least one of the plurality of sensors to determine an initial location of the movable object;
Generating a first signal that causes the movable object to navigate within the environment;
Receiving sensing data associated with the environment using at least one of the plurality of sensors;
Generating an environment map representing at least a portion of the environment based on the sensing data;
Receiving instructions to return to the initial location,
Based on the environment map, configured to generate a second signal that causes the movable object to navigate back to the initial location;
system.
[Item 24]
A method for controlling an unmanned aerial vehicle in an environment,
Generating a first signal that causes the unmanned aircraft to navigate within the environment;
Receiving sensing data related to at least a portion of the environment using a plurality of sensors carried by the unmanned aerial vehicle;
Generating an environment map representing at least a portion of the environment based on the sensing data;
Detecting one or more obstacles located in a part of the environment using the environment map;
Generating a second signal using the environmental map to cause the unmanned aircraft to navigate to avoid the one or more obstacles.
[Item 25]
25. The method of claim 24, wherein the unmanned aerial vehicle is configured to detect the presence of a flight restriction area and not fly within a predetermined distance of the flight restriction area.
[Item 26]
25. The method of claim 24, wherein the plurality of sensors includes a global positioning system (GPS) sensor.
[Item 27]
The method of claim 24, wherein the plurality of sensors comprises a vision sensor.
[Item 28]
25. The method of claim 24, wherein the plurality of sensors includes proximity sensors.
[Item 29]
29. The method of claim 28, wherein the proximity sensor comprises at least one of the following lidar sensor, ultrasonic sensor or in-flight camera sensor.
[Item 30]
The method of claim 24, wherein the environment map comprises a topological map.
[Item 31]
The method of claim 24, wherein the environment map comprises a metric map.
[Item 32]
32. The method of claim 31, wherein the metric map comprises at least one of the following point cloud, 3D grid map, 2D grid map, or 2.5D grid map.
[Item 33]
32. The method of claim 31, wherein the metric map comprises an occupancy grid map.
[Item 34]
25. The method of claim 24, wherein the first signal is generated based on a plurality of instructions received from a remote terminal that communicates with the unmanned aircraft.
[Item 35]
35. The method of claim 34, wherein the plurality of instructions are input to the remote terminal by a user.
[Item 36]
25. The method of claim 24, wherein the first signal is generated autonomously by the unmanned aerial vehicle.
[Item 37]
25. The method of claim 24, wherein the sensing data includes data for a plurality of different coordinate systems, and generating the environment map includes mapping the data to a single coordinate system.
[Item 38]
The step of generating the first signal includes the step of generating a flight path for the unmanned aerial vehicle, and the step of generating the second signal is performed on the environment map to avoid the one or more obstacles. 25. The method of claim 24, comprising changing the flight path based on.
[Item 39]
40. The method of claim 38, wherein the flight path is configured to direct the unmanned aircraft from a current position to a previous position.
[Item 40]
A system for controlling an unmanned aerial vehicle in an environment,
A plurality of sensors carried by the unmanned aerial vehicle;
With one or more processors,
The one or more processors may be individually or collectively
Generating a first signal that causes the unmanned aircraft to navigate within the environment;
Using a plurality of sensors carried by the unmanned aerial vehicle to receive sensing data related to at least a portion of the environment;
Generating an environment map representing at least a portion of the environment based on the sensing data;
Using the environment map to detect one or more obstacles located in the part of the environment;
Configured to generate a second signal using the environmental map to cause the unmanned aircraft to navigate to avoid the one or more obstacles;
system.
[Item 41]
A method for controlling an unmanned aerial vehicle in an environment,
Receiving a first sensing signal associated with the environment from a first sensor and receiving a second sensing signal associated with the environment from a second sensor;
Generating a first environment map using the first sensing signal and a second environment map using the second sensing signal;
Combining the first environment map and the second environment map;
The first sensor and the second sensor are a plurality of different sensor types and are carried by the unmanned aerial vehicle,
Each of the first environment map and the second environment map includes obstacle occupancy information for the environment,
This generates a final environment map containing obstacle occupancy information for the environment.
Method.
[Item 42]
42. The method of claim 41, further comprising generating a signal that causes the unmanned aircraft to navigate within the environment based at least in part on the obstacle occupancy information in the final environment map.
[Item 43]
42. The method of claim 41, wherein the unmanned aerial vehicle is configured to detect the presence of a flight restriction region and not fly within a predetermined distance of the flight restriction region.
[Item 44]
42. The method of claim 41, wherein at least one of the first sensor or the second sensor comprises a global positioning system (GPS) sensor.
[Item 45]
42. The method of claim 41, wherein at least one of the first sensor or the second sensor comprises a vision sensor.
[Item 46]
42. The method of claim 41, wherein at least one of the first sensor or the second sensor comprises a proximity sensor.
[Item 47]
47. The method of claim 46, wherein the proximity sensor comprises at least one of the following lidar sensor, ultrasonic sensor or in-flight camera sensor.
[Item 48]
42. The method of claim 41, wherein the final environment map comprises a topological map.
[Item 49]
42. The method of claim 41, wherein the final environment map comprises a metric map.
[Item 50]
50. The method of claim 49, wherein the metric map comprises at least one of the following point cloud, 3D grid map, 2D grid map, or 2.5D grid map.
[Item 51]
50. The method of claim 49, wherein the metric map comprises an occupancy grid map.
[Item 52]
42. The method of claim 41, wherein the first environment map is provided for a first coordinate system and the second environment map is provided for a second coordinate system that is different from the first coordinate system.
[Item 53]
53. The method of claim 52, wherein the first coordinate system is a global coordinate system and the second coordinate system is a local coordinate system.
[Item 54]
53. The method of claim 52, wherein combining the first environment map and the second environment map includes transforming the first environment map and the second environment map into a single coordinate system.
[Item 55]
42. The method of claim 41, wherein a sensing range of the first sensor is different from a sensing range of the second sensor.
[Item 56]
A system for controlling an unmanned aerial vehicle in an environment,
A first sensor carried by the unmanned aerial vehicle and configured to generate a first sensing signal associated with the environment;
A second sensor carried by the unmanned aerial vehicle and configured to generate a second sensing signal associated with the environment;
With one or more processors,
The second sensor and the first sensor are a plurality of different sensor types,
The one or more processors may be individually or collectively
Receiving the first sensing signal and the second sensing signal;
Generating a first environment map using the first sensing signal, generating a second environment map using the second sensing signal;
Configured to combine the first environment map and the second environment map;
Each of the first environment map and the second environment map includes obstacle occupancy information for the environment,
This generates a final environment map containing obstacle occupancy information for the environment.
system.
[Item 57]
A method for controlling an unmanned aerial vehicle in an environment,
Determining an initial location of the unmanned aerial vehicle using at least one of a plurality of sensors carried by the unmanned aerial vehicle;
Generating a first signal that causes the unmanned aircraft to navigate within the environment;
Receiving sensing data associated with the environment using at least one of the plurality of sensors;
Receiving an instruction to return to the initial location;
Generating a second signal that causes the unmanned aircraft to return along the path to the initial location;
If at least one of the plurality of sensors detects an obstacle located along the path in the environment, the path is changed to avoid the obstacle;
Method.
[Item 58]
58. The method of claim 57, wherein the unmanned aerial vehicle is configured to detect the presence of a flight restriction region and not fly within a predetermined distance of the flight restriction region.
[Item 59]
58. The method of claim 57, wherein the plurality of sensors includes a global positioning system (GPS) sensor.
[Item 60]
58. The method of claim 57, wherein the plurality of sensors comprises a vision sensor.
[Item 61]
58. The method of claim 57, wherein the plurality of sensors includes proximity sensors.
[Item 62]
62. The method of claim 61, wherein the proximity sensor comprises at least one of the following: a lidar sensor, an ultrasonic sensor, or a flight camera sensor.
[Item 63]
58. The method of claim 57, wherein the environment map comprises a topological map.
[Item 64]
58. The method of claim 57, wherein the environment map comprises a metric map.
[Item 65]
The method of claim 64, wherein the metric map comprises at least one of the following point cloud, 3D grid map, 2D grid map, or 2.5D grid map.
[Item 66]
The method of claim 64, wherein the metric map comprises an occupancy grid map.
[Item 67]
58. The method of claim 57, wherein the path includes one or more portions that the unmanned aircraft has previously passed.
[Item 68]
58. The method of claim 57, wherein the route is different from a route that the unmanned aircraft has previously traveled.
[Item 69]
58. The method of claim 57, wherein the path is a flight path of the unmanned aircraft.
[Item 70]
70. The method of claim 69, wherein the path includes a spatial position and orientation of the unmanned aerial vehicle.
[Item 71]
A system for controlling an unmanned aerial vehicle in an environment,
A plurality of sensors carried by the unmanned aerial vehicle;
With one or more processors,
The one or more processors may be individually or collectively
Using at least one of a plurality of sensors carried by the unmanned aerial vehicle to determine an initial location of the unmanned aerial vehicle;
Generating a first signal that causes the unmanned aircraft to navigate within the environment;
Receiving sensing data associated with the environment using at least one of the plurality of sensors;
Receiving instructions to return to the initial location,
Configured to generate a second signal that causes the unmanned aircraft to return along the path to the initial location;
If at least one of the plurality of sensors detects an obstacle located along the path in the environment, the path is changed to avoid the obstacle;
system.
[Item 72]
A method for generating a map of an environment,
Receiving first sensing data including depth information for the environment from one or more vision sensors carried by an unmanned aerial vehicle;
Receiving second sensing data including depth information for the environment from one or more proximity sensors carried by the unmanned aerial vehicle;
Generating an environment map that includes depth information for the environment using the first sensing data and the second sensing data.
[Item 73]
Each of the first sensing data and the second sensing data includes at least one image having a plurality of pixels, and each pixel of the plurality of pixels is associated with a two-dimensional image coordinate and a depth value. Item 72. The method according to Item 72.
[Item 74]
74. The method of claim 73, wherein each pixel of the plurality of pixels is associated with a color value.
[Item 75]
The method of claim 72, wherein each of the first sensing data and the second sensing data includes silhouette information for one or more objects in the environment.
[Item 76]
73. The method of claim 72, further comprising generating a signal that causes the unmanned aircraft to navigate through the environment based at least in part on the depth information in the environment map.
[Item 77]
73. The method of claim 72, wherein the unmanned aerial vehicle is configured to detect the presence of a flight restriction region and not fly within a predetermined distance of the flight restriction region.
[Item 78]
The method of claim 72, wherein the one or more vision sensors include a single camera.
[Item 79]
73. The method of claim 72, wherein the one or more vision sensors include two or more cameras.
[Item 80]
75. The method of claim 72, wherein the one or more proximity sensors include at least one ultrasonic sensor.
[Item 81]
75. The method of claim 72, wherein the one or more proximity sensors include at least one lidar sensor.
[Item 82]
73. The method of claim 72, wherein the first sensing data includes a first set of depth images and the second sensing data includes a second set of depth images.
[Item 83]
The step of generating the environment map includes:
Identifying a first plurality of feature points present in the first set of depth images;
Identifying a second plurality of feature points present in the second set of depth images;
Determining a correspondence between the first plurality of feature points and the second plurality of feature points;
Generating the environment map based on the correspondence by combining the first set of depth images and the second set of depth images; and
83. The method of claim 82, wherein each of the second plurality of feature points corresponds to one of the first plurality of feature points.
[Item 84]
73. The method of claim 72, wherein the first sensing data is provided for a first coordinate system and the second sensing data is provided for a second coordinate system that is different from the first coordinate system.
[Item 85]
85. The method of claim 84, wherein generating the environment map comprises expressing the first sensing data and the second sensing data with respect to a third coordinate system.
[Item 86]
86. The method of claim 85, wherein the third coordinate system is either the first coordinate system or the second coordinate system.
[Item 87]
86. The method of claim 85, wherein the third coordinate system is different from the first coordinate system and the second coordinate system.
[Item 88]
The method of claim 72, wherein the environment map comprises a topological map.
[Item 89]
The method of claim 72, wherein the environment map comprises a metric map.
[Item 90]
90. The method of claim 89, wherein the metric map comprises at least one of the following point cloud, 3D grid map, 2D grid map, or 2.5D grid map.
[Item 91]
90. The method of claim 89, wherein the metric map comprises an occupancy grid map.
[Item 92]
A system that generates a map of an environment,
One or more vision sensors carried by an unmanned aerial vehicle and configured to generate first sensing data including depth information for the environment;
One or more proximity sensors carried by the unmanned aerial vehicle and configured to generate second sensing data including depth information for the environment;
With one or more processors,
The one or more processors may be individually or collectively
Receiving the first sensing data and the second sensing data;
Configured to generate an environment map including depth information for the environment using the first sensing data and the second sensing data;
system.

Claims (29)

A method for controlling an unmanned aerial vehicle in an environment,
Receiving a first sensing signal associated with the environment from a first sensor and receiving a second sensing signal associated with the environment from a second sensor;
Generating a first environment map using the first sensing signal and a second environment map using the second sensing signal;
Combining the first environment map and the second environment map;
The first sensor and the second sensor are a plurality of different sensor types, mounted on the unmanned aerial vehicle,
The first environment map and the second environment map each include obstacle information in the environment,
Generating a final environment map including obstacle information in the environment;
Method.   The method of claim 1, further comprising generating a signal to navigate in the environment based at least in part on the obstacle information in the final environment map.   The method according to claim 1 or 2, wherein the unmanned aerial vehicle detects the presence of a flight restriction area and does not fly within a predetermined distance of the flight restriction area.   4. The device according to claim 1, wherein at least one of the first sensor or the second sensor includes at least one of a global positioning system (GPS) sensor, a vision sensor, and a proximity sensor. The method described.   The method according to any one of claims 1 to 4, wherein the final environment map comprises a topological map or a metric map.   The first environment map is provided for a first coordinate system, and the second environment map is provided for a second coordinate system different from the first coordinate system. The method according to one item.   The method of claim 6, wherein the first coordinate system is a global coordinate system and the second coordinate system is a local coordinate system.   The method according to claim 6 or 7, wherein the step of generating the final environment map further includes the step of converting the first environment map and the second environment map into a single coordinate system.   The method according to any one of claims 1 to 8, wherein a sensing range of the first sensor is different from a sensing range of the second sensor. A system for controlling an unmanned aerial vehicle in an environment,
A first sensor mounted on the unmanned aerial vehicle and generating a first sensing signal related to the environment;
A second sensor mounted on the unmanned aerial vehicle and generating a second sensing signal related to the environment;
One or more processors,
The second sensor and the first sensor are a plurality of different sensor types,
The one or more processors are:
Receiving the first sensing signal and the second sensing signal;
A first environment map including first obstacle occupation information is generated using the first sensing signal, and a second environment map including second obstacle occupation information is generated using the second sensing signal. ,
Generating a final environment map based on the first environment map and the second environment map;
system.   The program for making the said unmanned aircraft perform the method as described in any one of Claim 1 to 9. A communication terminal that controls an unmanned aerial vehicle in an environment,
One or more processors,
The one or more processors are:
Receiving a first sensing signal and a second sensing signal related to the environment from each of a first sensor and a second sensor of different sensor types mounted on the unmanned aerial vehicle;
A first environment map including first obstacle occupation information is generated using the first sensing signal, and a second environment map including second obstacle occupation information is generated using the second sensing signal. ,
Generating a final environment map based on the first environment map and the second environment map;
Communication terminal. A method for generating a map of an environment,
Receiving first sensing data including depth information for the environment from one or more vision sensors mounted on an unmanned aerial vehicle;
Receiving second sensing data including depth information for the environment from one or more proximity sensors mounted on the unmanned aerial vehicle;
Generating an environment map including depth information for the environment using the first sensing data and the second sensing data.   Each of the first sensing data and the second sensing data includes at least one image having a plurality of pixels, and each pixel of the plurality of pixels is associated with a two-dimensional image coordinate and a depth value. Item 14. The method according to Item 13.   The method of claim 14, wherein each pixel of the plurality of pixels is associated with a color value.   The method according to any one of claims 13 to 15, wherein each of the first sensing data and the second sensing data includes silhouette information for one or more objects in the environment.   17. A method according to any one of claims 13 to 16, further comprising generating a signal for causing the unmanned aircraft to navigate through the environment based at least in part on the depth information in the environment map.   18. A method according to any one of claims 13 to 17, wherein the unmanned aerial vehicle is configured to detect the presence of a flight restriction area and not fly within a predetermined distance of the flight restriction area.   The method according to any one of claims 13 to 18, wherein the one or more proximity sensors comprise at least one ultrasonic sensor or at least one lidar sensor.   20. The method according to any one of claims 13 to 19, wherein the first sensing data includes a first set of depth images and the second sensing data includes a second set of depth images. The step of generating the environment map includes:
Identifying a first plurality of feature points present in the first set of depth images;
Identifying a second plurality of feature points present in the second set of depth images;
Determining a correspondence between the first plurality of feature points and the second plurality of feature points;
Generating the environment map based on the correspondence using the first set of depth images and the second set of depth images;
21. The method of claim 20, wherein each of the second plurality of feature points corresponds to one of the first plurality of feature points.   The first sensing data is provided for a first coordinate system, and the second sensing data is provided for a second coordinate system different from the first coordinate system. The method according to one item.   23. The method of claim 22, wherein generating the environment map includes expressing the first sensing data and the second sensing data with respect to a third coordinate system.   24. The method of claim 23, wherein the third coordinate system is either the first coordinate system or the second coordinate system.   24. The method of claim 23, wherein the third coordinate system is different from the first coordinate system and the second coordinate system.   26. A method according to any one of claims 13 to 25, wherein the environment map comprises a topological map or a metric map. A system that generates a map of an environment,
One or more vision sensors mounted on an unmanned aerial vehicle and generating first sensing data including depth information for the environment;
One or more proximity sensors mounted on the unmanned aerial vehicle to generate second sensing data including depth information for the environment;
One or more processors,
The one or more processors are:
Receiving the first sensing data and the second sensing data;
Generating an environment map including depth information for the environment using the first sensing data and the second sensing data;
system.   A program for causing the unmanned aircraft to execute the method according to any one of claims 13 to 26. A communication terminal that communicates with an unmanned aerial vehicle and generates a map of an environment,
One or more processors,
The one or more processors are:
Receiving first sensing data and second sensing data including depth information for the environment from each of one or more vision sensors and one or more proximity sensors mounted on the unmanned aerial vehicle;
Generating an environment map including depth information for the environment using the first sensing data and the second sensing data;
Communication terminal.