JP2019050007A - Method and device for determining position of mobile body and computer readable medium - Google Patents

Abstract

To provide a device and a method for determining a position of a mobile body and a computer readable medium.SOLUTION: There is provided a method 100 for determining a position of a mobile body, in which an imaging device is connected to such a movable object as an unmanned aircraft (UAV). The imaging device, for example, a camera captures one or more images. Significant features in the one or more images are identified. Distances from the mobile body to the significant features are calculated so that the position of the mobile body is determined.SELECTED DRAWING: Figure 1

Description

  Mobile objects such as unmanned aerial vehicles (UAVs) can be used to perform surveillance, reconnaissance and exploration tasks in a wide variety of environments for military and civilian applications. Such movable objects may include sensors configured to assess the position of the movable object in the environment. Accurate and precise determination of the position of the moveable object may be important, for example, when the moveable object operates in a semi-autonomous or fully autonomous manner.

  Existing approaches to evaluating the position of movable objects may not be optimal in some instances. For example, approaches utilizing GPS sensors may be limited by the environment in which the moveable object (e.g., UAV) operates, and may require an on-board auxiliary assistance sensor. For example, techniques that use SLAM (Simultaneous Localization and Mapping) may lack accuracy and precision, and may accumulate errors over time.

  Embodiments disclosed herein provide an apparatus and method for evaluating the position of a movable object in an environment. In many embodiments, an imaging device may be used to gather information about the environment surrounding the moveable object. Image data obtained from the imaging device may be processed, for example, to determine salient features in the image. The identified salient features may further be utilized to determine the position of the moveable object. Advantageously, the techniques described herein may provide an improved and efficient assessment of object position based on image data, and may be used to improve the autonomous navigation or control of a vehicle.

  Thus, in one aspect, a method is provided for determining the position of a vehicle. The method comprises the steps of receiving a plurality of images captured by the one or more field sensors and a plurality of stationary salient features substantially stationary in the environment from the plurality of images with the aid of one or more processors. And calculating the distance from the vehicle to each of the plurality of stationary salient features with the help of one or more processors; and based on the distance from the vehicle to each of the plurality of stationary salient features Determining the location of the data with the help of one or more processors.

  In another aspect, an apparatus is provided for determining the position of a vehicle. The apparatus identifies one or more view sensors configured to capture a plurality of images, a plurality of stationary salient features that are substantially stationary in the environment from the plurality of images, and a plurality of stationary salient features from the vehicle And one or more processors individually or collectively configured to calculate the distance to each of and to determine the position of the vehicle based on the distance from the vehicle to each of the plurality of stationary salient features.

  In another aspect, there is provided a non-transitory computer readable medium comprising program instructions for determining the position of a vehicle. The computer readable medium comprises program instructions for receiving a plurality of images captured by one or more field of view sensors and a program for identifying a plurality of stationary stationary salient features within the environment from the plurality of images. Instructions, program instructions for calculating the distances from the vehicle to each of the plurality of stationary salient features, and program instructions for determining the position of the vehicle based on the distances from the vehicle to each of the plurality of stationary salient features Including.

  In another aspect, a method is provided for determining the position of a vehicle. The method comprises the steps of: receiving a plurality of images captured by the one or more vision sensors; identifying a plurality of salient features from the plurality of images with the aid of one or more processors; Calculating the distance from the vehicle to each of the plurality of salient features with the aid of one or more processors, and the position of the vehicle on the basis of the distance from the vehicle to each of the plurality of salient features. Determining with the help of, wherein the location is in at least two sets of concentric zones of intersection, each set being centered on at least one of the plurality of salient features.

  In another aspect, an apparatus is provided for determining the position of a vehicle. The apparatus identifies a plurality of salient features from the plurality of images, one or more view sensors configured to capture a plurality of images, and calculates a distance from the vehicle to each of the plurality of salient features. One or more processors individually or collectively configured to determine the position of the vehicle based on the distance from the vehicle to each of the plurality of salient features, the positions being at least two sets of concentric circles And a processor within the intersection zone, each set centered on at least one of the plurality of salient features.

  In another aspect, there is provided a non-transitory computer readable medium comprising program instructions for determining the position of a vehicle. The computer readable medium comprises program instructions for receiving a plurality of images captured by the one or more vision sensors, program instructions for identifying a plurality of salient features from the plurality of images, and a plurality of programs from the vehicle. Program instructions for calculating the distance to each of the salient features and program instructions for determining the position of the vehicle based on the distances to the salient features from the vehicle, the positions being at least two sets And program instructions within concentric zones of intersection, each set centered on at least one of the plurality of salient features.

  In another aspect, a method is provided for determining the position of a vehicle. The method comprises the steps of receiving a plurality of images captured by the one or more vision sensors, and identifying a plurality of salient feature candidates from the plurality of images with the aid of one or more processors. Selecting two or more salient features with the help of one or more processors, wherein the selected salient features are a subset of the plurality of salient feature candidates, and the salient selected from the vehicle Calculating the distance to each of the features with the help of one or more processors, and the position of the vehicle on the basis of the distance of each of the salient features selected from the vehicle to one or more processors. And a step of borrowing and judging.

  In another aspect, an apparatus is provided for determining the position of a vehicle. The apparatus identifies one or more view sensors configured to capture a plurality of images, identifies a plurality of salient feature candidates from the plurality of images, and is a subset of the plurality of salient feature candidates. Select the salient features, calculate the distance from the vehicle to each of the salient features selected, and individually or collectively to determine the position of the vehicle based on the distance to the salient features selected from the vehicle And one or more configured processors.

  In another aspect, there is provided a non-transitory computer readable medium comprising program instructions for determining the position of a vehicle. The computer readable medium comprises program instructions for receiving a plurality of images captured by one or more view sensors, program instructions for identifying a plurality of salient feature candidates from the plurality of images, and a plurality of salient features. Program instructions for selecting two or more salient features that are a subset of the feature candidate, program instructions for calculating the distance from the vehicle to each of the salient features selected, and salient features selected from the car Program instructions for determining the position of the vehicle based on the distances to each.

  In another aspect, a method is provided for determining the position of a vehicle. The method comprises the steps of receiving a plurality of images captured by the one or more field sensors, and a plurality of salient regions corresponding to one or more recognizable objects from the plurality of images. The steps of identifying with the aid of the step of calculating the distance from the vehicle to each of the plurality of saliency areas with the help of one or more processors, and the distance from the vehicle to each of the plurality of saliency areas Determining the position of the vehicle with the help of one or more processors.

  In another aspect, an apparatus is provided for determining the position of a vehicle. The apparatus identifies a plurality of salient regions corresponding to one or more recognizable objects from the one or more view sensors configured to capture the plurality of images, and the plurality of images, from the vehicle One or more processors individually or collectively configured to calculate the distance to each of the plurality of salient regions and to determine the position of the vehicle based on the distance from the vehicle to each of the plurality of salient regions including.

  In another aspect, there is provided a non-transitory computer readable medium comprising program instructions for determining the position of a vehicle. The computer readable medium identifies program instructions for receiving a plurality of images captured by the one or more vision sensors, and identifies a plurality of salient regions corresponding to one or more recognizable objects from the plurality of images Program instructions for determining the position of the vehicle based on the program instructions for calculating the distance from the vehicle to each of the plurality of salient regions And.

  It should be understood that the various aspects of the invention may be understood individually, collectively or in combination with one another. The various aspects of the invention described herein may be applied to any of the specific applications described below. Other objects and features of the present invention will become apparent upon review of the specification, claims and accompanying drawings.

Incorporation by Reference All publications, patents, and patent applications mentioned herein are specifically and specifically indicated to be incorporated by reference as if each individual publication, patent or patent application was incorporated by reference. , Incorporated herein by reference.

  The novel features of the present invention are set forth by the peculiarities in the appended claims. A further understanding of the features and advantages of the present invention can be obtained by reference to the following detailed description that sets forth illustrative embodiments, in which the principles of the invention are utilized, and the accompanying drawings.

According to some embodiments, a method of determining the position of a vehicle is provided. 5 shows a saliency map of an image captured by an imaging device and an image according to some embodiments. 7 illustrates the position of the moveable object determined based on the distance from the moveable object to the salient feature, according to some embodiments. 7 illustrates a UAV operating in an outdoor environment, according to some embodiments. 7 illustrates a point cloud calculated with SLAM, according to some embodiments. 7 illustrates a UAV operating in an indoor environment, according to some embodiments. 1 illustrates an unmanned aerial vehicle (UAV), according to some embodiments. 1 illustrates a movable object including a carrier and a payload, according to some embodiments. FIG. 1 shows a schematic diagram according to a block diagram of a system for controlling movable objects according to some embodiments.

  Systems and methods are provided for evaluating the position of a moveable object. As used herein, a moveable object may refer to any object that may be moved as done elsewhere herein. For example, the movable object may be a mobile phone, a watch, a UAV, a car, a boat, a computer, a PDA, a tablet, and the like. Although many embodiments herein are described with reference to a UAV, it is understood that the references are non-limiting and that the embodiments are equally applicable to any moveable object. It should. In some instances, the position of the moveable object may be determined with the aid of one or more sensors. For example, one or more vision sensors may capture an image of the environment. Salient regions in the image may be identified (e.g., determined) as described further elsewhere. The distance from the movable object to each of the salient regions may for example be calculated based on the parallax, while the position of the movable object may be determined using, for example, a multi-later approach. Information regarding the position (e.g., placement) of movable objects within the environment may further be utilized in applications such as autonomous navigation and control (e.g., in UAVs).

  In some embodiments, a UAV or other movable object may be made to carry one or more sensors. The one or more sensors may be configured to collect relevant data, such as information related to the state of the UAV, the surrounding environment, or objects and obstacles in the environment. Relevant data may be analyzed, processed or used in other applications. For example, based on the relevant data collected, it is possible to generate control signals to control UAV navigation. Exemplary sensors suitable for use with the embodiments disclosed herein include position sensors (eg, global positioning system (GPS) sensors, portable device transmitters that enable position triangulation) Vision sensor (for example, an imaging device such as a camera capable of detecting visible light, infrared light, or ultraviolet light), proximity or distance measurement sensor (for example, ultrasonic sensor, lidar, time of flight measurement) Formula or depth camera), inertial sensor (eg, accelerometer, gyroscope, inertial measurement unit (IMU)), altitude sensor, attitude sensor (eg, compass), pressure sensor (eg, barometer), voice Sensor (eg, microphone) or electric field Magnetic field sensor (e.g., magnetometer (electromagnetic sensor)) including.

  Any suitable number and combination of sensors may be used, such as 1, 2, 3, 4, 5, 6, 7, 8, or more sensors. Optionally, data may be received from various types of sensors (eg, 2, 3, 4, 5, 6, 7, 8, or more types). Different types of sensors may measure different types of signals or information (eg, position, orientation, velocity, acceleration, proximity, pressure, etc.) and / or utilize different types of measurement techniques to acquire data It can. For example, the sensor may include any suitable combination of an active sensor (eg, a sensor that generates and measures energy from its own energy source) and a passive sensor (eg, a sensor that detects effective energy). As another example, some sensors may generate absolute measurement data (eg, position data provided by a GPS sensor, attitude information provided by a compass or magnetometer) provided in a global coordinate system, while , Other sensors, relative measurement data provided in the local coordinate system (eg, relative angular velocity provided by the gyroscope, relative translational acceleration provided by the accelerometer, relative attitude information provided by the visual sensor Relative distance information provided by an ultrasonic sensor, a rider, or a time-of-flight camera. In some instances, the local coordinate system may be a body coordinate system defined for the UAV.

  The sensors may be configured to collect various types of data, such as data related to UAVs, the surrounding environment, or objects in the environment. For example, at least some of the sensors may be configured to provide data regarding the status of the UAV. The state information provided by the sensor may include information on the spatial arrangement of the UAV (eg, position information such as position, longitude, latitude and / or altitude, orientation or attitude information such as roll, pitch and / or yaw) . The state information may also include information regarding the motion of the UAV (eg, translational velocity, translational acceleration, angular velocity, angular acceleration, etc.). The sensor is configured to determine, for example, the spatial arrangement and / or movement of the UAV with respect to up to six degrees of freedom (eg, three degrees of position and / or translation, three degrees of orientation and / or rotation) It can be done. State information may be provided for a global coordinate system or a local coordinate system. A global coordinate system may refer to a coordinate system that is independent of the location of the UAV or another entity. A local coordinate system may refer to a coordinate system to a UAV or another entity. For example, the sensor may be configured to determine the distance between the UAV and the user controlling the UAV or the distance between the UAV and the departure point of the flight of the UAV. In some instances, the sensor may be configured to determine the distance between the UAV and an object near the UAV.

  The data obtained by the sensors can provide various types of environmental information. For example, sensor data may indicate an environment type, such as an indoor environment, an outdoor environment, a low altitude environment, or a high altitude environment. Sensor data may also provide information regarding current environmental conditions, including weather (eg, sunny, rain, snow), visibility conditions, wind speed, time of day, and so on. In addition, environmental information collected by the sensors may include information regarding objects in the environment, such as obstacles or landmarks recognizable by the processor as described herein. Obstacle information may include information regarding the number, density, geometry, spatial arrangement, movement, trajectory, and / or speed of obstacles in the environment.

  In some embodiments, the UAV may include one or more vision sensors, also referred to herein as an "imaging device." Although many embodiments are described herein as having one imaging device coupled to a UAV, any number of imaging devices, such as 1, 2, 3, 4, 5 or more may be imaging devices, etc. It should be understood that it can be coupled to the UAV. The imaging device may be configured to detect electromagnetic radiation (eg, visible light, infrared light and / or ultraviolet light) and generate image data based on the detected electromagnetic radiation. For example, the imaging device may include a charge coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor that generates an electrical signal depending on the wavelength of light. . The resulting electrical signal may be processed to generate image data. The image data generated by the imaging device may include one or more images that may be still images (eg, photographs), moving images (eg, video), or a suitable combination thereof. The image data may be multicolor (e.g., RGB, CMYK, HSV) or single color (e.g., grayscale, black and white, sepia).

  In some embodiments, the imaging device may be a camera. The camera may be a movie camera or video camera that may capture moving image data (e.g., video). The camera may be a still camera that captures still images (eg, photographs). The camera may be a binocular camera. As used herein, a binocular camera may refer to a stereo or stereo vision camera. A stereo camera may include two cameras. The camera may be a monocular camera. Although some embodiments provided herein are described in the context of a camera, it should be understood that the present disclosure may be applied to any suitable imaging device. Any description herein relating to a camera may apply to any suitable imaging device or other type of imaging device. A camera may be used to generate 2D images of 3D scenes (e.g., an environment, one or more objects, etc.). The image generated by the camera may represent a projection of the 3D scene onto a 2D image plane. Thus, each point in the 2D image corresponds to 3D space coordinates in the scene. The camera may include optical elements (eg, lenses, mirrors, filters, etc.). The camera may capture color images, tone scale images, infrared images, and the like.

  The imaging device may capture an image or a series of images at a particular image resolution. In some embodiments, the image resolution may be defined by the number of pixels in the image. In some embodiments, the image resolution is approximately 352 × 420 pixels, 480 × 320 pixels, 720 × 480 pixels, 1280 × 720 pixels, 1440 × 1080 pixels, 1920 × 1080 pixels, 2048 × 1080 pixels, 3840 × 2160. It may be more than a pixel, 4096 × 2160 pixels, 7680 × 4320 pixels, or 15360 × 8640 pixels. The camera may be a 4K camera or a camera with higher resolution.

  The imaging device may have adjustable parameters. Under different parameters, different images may be captured by the imaging device under the same external conditions (e.g. position, illumination). Adjustable parameters may include exposure (eg, exposure time, shutter speed, aperture, film sensitivity), gain, gamma rays, region of interest, binning / subsampling, pixel clock, offset, trigger, ISO, etc. Parameters related to exposure may control the amount of light reaching the image sensor in the imaging device. For example, the shutter speed may control the amount of time of light reaching the image sensor, and the aperture may control the amount of light reaching the image sensor in a given time. Parameters related to gain may control the amplification of the signal from the light sensor. ISO can control the level of sensitivity of the camera to natural light. Parameter control of exposure and gain may be considered collectively and may be referred to herein as EXPO.

  Each of the imaging devices may have a field of view. The field of view of the imaging device may be a spread of the environment that is detectable (eg, visible) by the imaging device. The field of view may relate to the angle of view that may be measured by the angular range of a given scene imaged by the imaging device. The angle of view of the imaging device may be less than about 360 °, 300 °, 240 °, 180 °, 150 °, 120 °, 90 °, 60 °, 30 °, 20 ° or 10 °. The field of view can be described by the relative orientation of the imaging device relative to the movable object. For example, the field of view may be perpendicular, horizontal, above, below, sides, etc. with respect to the moveable object (eg, UAV). The imaging devices may each have an optical axis. The optical axis (sometimes referred to as the "principal axis") of the imaging device may be a line along which there is a degree of rotational symmetry within the imaging device. In some embodiments, the optical axis of the imaging device passes through the center of the optics (eg, lens, light sensor) of the imaging device.

  The imaging device of the present disclosure may be located on any suitable part of the movable object, such as above, below, to the side of, or in the body of the movable object. Some imaging devices may be mechanically coupled to the UAV such that the spatial arrangement and / or movement of the movable object corresponds to the spatial arrangement and / or movement of the imaging device. The imaging device may be coupled to the movable object via a fixed joint so that the imaging device does not move relative to the part of the movable object to be attached. Alternatively, the coupling between the imaging device and the movable object may allow movement of the imaging device relative to the movable object (e.g. translational or rotational movement relative to the UAV). For example, coupling between the imaging device and the movable object via a carrier such as a gimbal may allow movement of the imaging device relative to the movable object. The movement of the imaging device relative to the moveable object may be translational (e.g. horizontal, vertical) and / or rotation (e.g. around pitch, yaw and / or roll axes). The movement of the imaging device relative to the movable object may be a predetermined or known amount. One or more sensors may detect movement of the imaging device relative to the vehicle. The movement of the imaging device relative to the moveable object may be controlled remotely, autonomously or semi-autonomously by user input. The joint may be a permanent joint or a non-permanent (eg, releasable) joint. Suitable bonding methods may include adhesives, bonding, welding and / or fasteners (eg, screws, nails, pins). Optionally, the imaging device may be integrally formed with a part of the movable object. Further, as in the embodiments discussed herein, the imaging device may be configured to communicate data collected by the imaging device to various functions of the UAV (eg, navigation, control, propulsion, communication with users or other devices, etc.) Can be electrically coupled to a portion of the movable object (e.g., a processing unit, control system, data storage). The imaging device may be operatively coupled to a portion of the UAV (e.g., a processing unit, control system, data storage).

  One or more images may be captured by an imaging device. Two or more sequences of images may be captured by the imaging device. For example, sequences of about 2, 3, 4, 5, 10, 15, 20, 25, 50, 100, 150, 200 or more images may be captured by the imaging device. The imaging device may capture a sequence of images at a particular capture rate. In some embodiments, a series of images may be captured at a standard video frame rate, such as about 24p, 25p, 30p, 48p, 60p, 72p, 90p, 100p, 120p, 300p, 50i or 60i. In some embodiments, a series of images may have 0.0001 second, 0.0002 second, 0.0005 second, 0.001 second, 0.002 second, 0.005 second, 0.002 second,. About 1 image every 05 seconds, 0.01 seconds, 0.02 seconds, 0.05 seconds, 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 2 seconds, 5 seconds, or 10 seconds It can be captured below. In some embodiments, the capture rate may vary depending on user input and / or external conditions (eg, rain, snow, wind, texture of the captured environment).

  The images obtained by the imaging device described herein may be used in a wide variety of applications related to UAV operation. In some embodiments, the image is used to facilitate UAV navigation within the environment (eg, autonomously, semi-autonomously, or manually). In some embodiments, the image is used for obstacle detection and avoidance. In some embodiments, the image may be processed to evaluate or determine UAV status information. State information may include position information, orientation information, velocity information (e.g., angular or translational velocity) and / or acceleration information (e.g., angular or translational acceleration). State information may be evaluated or determined (eg, by processing by one or more processors) using one or more images obtained by one or more imaging devices.

  For example, the image captured by the imaging device may be used to determine the position of the UAV in the environment. FIG. 1 provides a method 100 for determining the position of a vehicle (e.g., a UAV), according to some embodiments. Similarly, non-transitory computer readable media may also be provided that include program instructions for performing the method 100. At step 102, a plurality of images are received by one or more processors. The processor includes field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), application-specific standard product (ASSP), digital signal processor (DSP: digital signal processor), central processing unit (CPU: central processing unit), graphics processing unit (GPU: graphics processing unit), visual field processing unit (VPU: vision processing unit), complex professional It may include a programmable logic device (CPLD) or the like. The processor may be an on-board processor mounted on a moveable object (e.g., a UAV) or an embedded processor carried by an imaging device. The processor may be a non-mounted processor (e.g., communicating with the UAV and / or the camera at the ground station) separate from the UAV and / or the imaging device. The one or more processors referred to herein may be configured individually or collectively to further aid in the steps 104, 106, 108 recited herein.

  In some embodiments, multiple images are captured by a single imager. For example, the plurality of images may include two or more images captured by a single imager over a fixed time interval. The time interval is about 1 second, 2 seconds, 3 seconds, 4 seconds, 5 seconds, 6 seconds, 7 seconds, 8 seconds, 9 seconds, 10 seconds, 15 seconds, 20 seconds, 30 seconds, 45 seconds or 60 seconds or less It can be. In some embodiments, the plurality of images are captured by more than one imaging device. The two or more imaging devices may be the same type or different types of imaging devices. For example, the two or more imaging devices may include a monocular camera, a binocular camera, a still camera, a video camera. In some embodiments, the plurality of images may include one or more images captured by two or more imagers each having a different field of view at a single time or over a fixed time interval. For example, the plurality of images may include two or more images captured over a fixed time interval by two or more imaging devices. The time interval is about 1 second, 2 seconds, 3 seconds, 4 seconds, 5 seconds, 6 seconds, 7 seconds, 8 seconds, 9 seconds, 10 seconds, 15 seconds, 20 seconds, 30 seconds, 45 seconds or 60 seconds or less It can be. The plurality of images may include about 2, 3, 4, 5, 10, 15, 20, 25, 50, 100, 150, 200, 250, 300, 500, 1000 or more images. Multiple images may be captured while the vehicle is moving (eg, while the UAV is in flight). Multiple images may be captured while the vehicle is stationary. Multiple images may be captured in an indoor or outdoor environment.

  At step 104, a plurality of salient features are identified from the plurality of images using one or more processors. In some embodiments, one or more salient features are identified from the plurality of images using one or more processors. As used herein, salient features may refer to salient areas or features (eg, recognizable) objects in an image. A salient feature may refer to an element in an image that is likely to be noticeable or attentive to a human observer. Prominent features may have semantic meaning. Prominent features may refer to elements that can be consistently identified under computer vision processing. Prominent features may refer to creatures, non-livings, landmarks, marks, logos, obstacles, etc. in the image. Prominent features can be consistently observed under different conditions. For example, the salient features may be images acquired from different viewpoints, such as under different lighting conditions, under different weather conditions, under different image capture settings (eg, under different gains, exposures, etc.) during different times of day It can be consistently identified (eg, by a human observer or by a computer program). For example, salient features may include humans, animals, surfaces, bodies, structures, buildings, vehicles, planes, signs, and the like.

  Prominent features may be identified or determined using any existing saliency calculation (eg, detection) method. In some instances, the saliency detection method may be learning based or image processing based. For example, salient features may be filtered based on contrast (eg, color, intensity, orientation, size, motion, depth, etc.), and frequency-tuned salient detection using spectral residual techniques. Site entropy rate (Region detection) by using context-aware top down approach, through region detection, through binarized normed gradient, for objectness estimation. It can be identified by measuring the visual saliency according to site entropy rate) and so on. For example, salient features may be identified within a saliency map (eg, color, intensity, orientation, etc.) generated by contrast-based filtering one or more images. The saliency map may represent a region with feature contrast. The saliency map may be a human-viewing predictor. The saliency map may include a spatial heat map that represents features or fixations. For example, in a saliency map, salient regions may have higher luminance contrast, color contrast, edge content, intensity, etc. than non- salient regions. In some embodiments, salient features may be identified using object recognition algorithms (eg, feature based methods, appearance based methods, etc.). Optionally, one or more objects or types of patterns, objects, figures, colors, logos, contours etc. may be pre-stored as possible salient features. The image may be analyzed to identify pre-stored salient features (eg, objects or types of objects). The pre-stored salient features may be updated. Alternatively, the salient features may not need to be pre-stored. The salient features may be recognized on a real time basis independently of the prestored information.

  In some embodiments, step 104 may include identifying a plurality of salient features and selecting a subset of the plurality of salient features identified as shown in step 110. The subset may include all or part of the plurality of salient features identified. In some embodiments, step 104 may include identifying a plurality of salient feature candidates and selecting a subset of salient feature candidates for further processing as shown in step 112. The subset may include all or part of the plurality of salient feature candidates. In some instances, a predetermined number of salient features may be selected from the plurality of salient salient features (or identified salient feature candidates). For example, the predetermined number of salient features selected may be less than about 2, 3, 4, 5, 10, 15, 20, 25, 50, 100, 150, 200, 500, 1000, 2000, 3000 or 5000 salient features possible. In some instances, only salient features (or salient feature candidates) that meet certain conditions or meet the criteria may be identified and / or selected. For example, stationary or nearly stationary salient features (or salient feature candidates) may be selected. For example, salient features (or salient feature candidates) that meet or exceed a predetermined distinctiveness (eg, pixel tone level, contrast level, variance such as color, orientation, intensity, complexity, etc.) may be selected. In some instances, a predetermined number of salient features that meet or best match the criteria may be selected. For example, 2, 3, 4, 5, 10, 15, 20, 25, 50, 100, 150, 200, 500, 1000, 2000, 3000 or 5000 salient features having the highest contrast levels in multiple images. It can be selected. As used herein, the steps of identifying multiple salient features (or salient feature candidates) and selecting a subset of salient features (or salient feature candidates) may also be salient features (or salient feature candidates) Sometimes referred to as identifying a subset of For example, the steps of identifying multiple salient features from multiple images and selecting stationary salient features for further processing (e.g., for computer vision) may identify multiple stationary salient features from multiple images. It is sometimes called a step to

  As used herein, stationary salient features may refer to salient features that do not experience positional changes in multiple images. The stationary salient features may be identified from a plurality of images (e.g., time lapse images) taken by a single imaging device and / or from a plurality of images taken by two or more imaging devices. A salient feature may be determined to not experience a change in position if it does not experience any change in position in the plurality of images. For example, salient features may occupy fixed locations in multiple images. For example, a single camera mounted on a stationary or airborne UAV may acquire multiple images including image data of a building over a period of time. The building may maintain fixed positions (e.g., substantially fixed positions) in the plurality of images and may be determined not to experience position changes. Alternatively or in combination, the salient feature may change position if it does not experience position changes in the plurality of images associated with other factors (eg, taking into account other factors and compensating for other factors) It can be determined that you do not experience. For example, while the salient features may occupy different locations in multiple images, the position of the salient features in each image may be the vehicle, the imaging device, movement of the carrier supporting the imaging device, movement / orientation between different imaging devices, etc. If compensated (e.g., offset) by other factors, the salient feature may be determined not to experience position change. For example, a moving UAV may acquire multiple images including image data of a building. The building may occupy different locations in each of the plurality of images. However, once compensated with the movement of the UAV, it may be determined that the building does not experience position changes in multiple images. In some instances, the movement of the UAV may be measured with a sensor, such as an inertial sensor. A near stationary salient feature may refer to a salient feature that experiences a positional change of about 1%, 2%, 5%, 10%, 15%, 20% or 30% or less in multiple images. The salient features may or may not be stationary. Stationary objects, such as buildings, may be identified as stationary salient features in the image. Creatures or non-organisms can be identified as stationary salient features in the image.

  FIG. 2 illustrates an image 202 captured by an imaging device and a saliency map 204 of the image, according to some embodiments. The saliency map 204 may be obtained by filtering the image 202. After filtering, each pixel in the saliency map may explain how the same pixel is noticeable. A portion of the saliency map may be classified as a salient feature candidate. For example, FIG. 2 shows three salient feature candidates 206, 208, 210. The salient feature candidates may be generated randomly or exhaustively, for example, for evaluation of the saliency of the salient feature candidates. In some instances, a box may be generated to cover an object (e.g., salient feature candidate) for evaluation of the characteristics of the bounding box. Although FIG. 2 shows an object surrounded by a box, it is understood that the object can be encompassed by any shape (e.g., a circle, an ellipse, a triangle, etc.) for the assessment of feature characteristics.

  The distinctiveness of the salient feature candidate is, for example, whether the bounding box (eg salient feature candidate) is a salient feature (eg salient object) or not (eg a notable background) by eg threshold or by trained classifier. ) Can be judged to be classified. For example, the variance of pixel tone levels of each salient feature candidate (e.g., each bounding box) may be calculated. If the variance of the salient feature candidates exceeds a certain threshold, it may mean that the discriminability is sufficiently large, and that part of the image may be selected as a salient feature or salient object. Alternatively, if the variance of salient feature candidates does not exceed a certain threshold, it may mean that the discrimination is not large enough, and that part of the image may be dropped as the salient feature candidate or not interested It can be classified as background. In some instances, the distinctiveness of salient feature candidates may be determined via trained classifiers (eg, support vector machines, random forests).

  In FIG. 2, it is determined that only the salient feature candidates 206 and 208 have the variance of the pixel gradation level exceeding the predetermined threshold, and the salient feature candidate 210 is dropped from the salient feature candidates. In some instances, salient feature candidates that have not been dropped may be selected as salient features. In some instances, salient feature candidates that have not been dropped may be selected as salient features if other conditions are satisfied. For example, salient feature candidates may be tracked over successive video frames (e.g., if multiple images are taken by the imaging device over a fixed time interval). If it is determined that the salient feature candidate does not experience a position change in the continuous video frame (eg, after movement of the imaging device has been compensated), the salient feature candidate may be selected as the salient feature. In FIG. 2, salient feature candidates 206, 208 were determined to be stationary over successive image frames (not shown) and were selected as salient features.

  The plurality of salient features (or salient feature candidates) identified (e.g., or selected) may be further processed, for example, for use in computer vision. At step 106, the distances from the vehicle to each of the plurality of salient features are calculated using one or more processors. In some instances, virtual coordinates may be set for each salient feature. The virtual coordinates may be two dimensional coordinates or a three dimensional coordinate system. In some instances, the position of one of the salient features may be set as origin coordinates and UAV coordinates, and / or other salient features may be relative to one of the salient features. In some instances, the position of the vehicle may be set as the origin coordinates, and the coordinates of the salient features may be relative to the vehicle. The distance from the vehicle to the salient feature may be determined by calculating the depth of the scene (eg, the depth of the salient feature) based on the parallax. For example, a vehicle may obtain multiple images from various locations. Objects in close proximity may have greater parallax than objects farther away when observed from different locations, and the parallax may be used to determine the distance to the salient feature. The distance to the salient feature may refer to the distance to the salient feature region. For example, the distance to the salient feature may refer to the distance to the center of the salient feature (eg, the horizontal center, the vertical center, or both), the distance to the centroid of the salient feature, etc. Calculation of the distance to salient feature regions (e.g., not points) may minimize calculation errors that may result from unstable features. The coordinates of the salient feature may point to the coordinates of the salient feature area. In some instances, the coordinates of the salient feature may point to coordinates such as the center of the salient feature (eg, the horizontal center, the vertical center, or both), the coordinates of the centroid of the salient feature, and the like. In some embodiments, it is not necessary to calculate two-dimensional or three-dimensional coordinates of each of the vehicles or salient features. The virtual coordinates of each of the salient features may be stored in a memory operably coupled to the imaging device or a processor mounted or not mounted on the vehicle. The distance to each salient feature may be stored in a memory operably coupled to an imaging device or a processor mounted or not mounted on a vehicle. The distance between the vehicle and the salient feature may be calculated in real time. For example, the distance between the vehicle and the salient feature may be calculated while the vehicle is moving (eg, while the UAV is in flight). Vehicle motion may be determined with the aid of one or more inertial sensors. The position of the salient feature may remain stationary while the distance between the vehicle and the salient feature is calculated. The position of the salient feature may change while the distance between the vehicle and the salient feature is calculated. In some embodiments, a global map of salient features may be maintained (eg, in a memory operably coupled to an imaging device or a processor mounted or not mounted on a vehicle). In some embodiments, there is no need to maintain a global map of the environment or salient features, and the position of the vehicle can be determined. This is because a salient feature is determined in a plurality of images and the distance to the salient feature is calculated.

  At step 108, the position of the vehicle is determined based on the distance from the vehicle to each of the plurality of salient features. The position of the vehicle can be determined in a two-dimensional coordinate system based on two or more salient features. The position of the vehicle can be determined in a three-dimensional coordinate system based on three or more salient features. The position of the vehicle can be determined based on multipoint principles (eg, trilateration, multilateration, etc.).

  FIG. 3 illustrates the position of the moveable object determined based on the distance from the moveable object to the salient feature, according to some embodiments. As shown, the position of the movable object 301 in the two-dimensional coordinate system may be determined based on the distances R1, R2 to the two salient features 303, 305. The distance to each vehicle of the plurality of salient features may be determined based on step 106 of method 100. A circle of radius R1 may be traced around virtual coordinates of salient feature 303, and a circle of radius R2 may be traced around virtual coordinates of salient feature 305. The two points 307, 309 where the circles intersect may be possible positions of the movable object. In some instances, the position of the movable object may be within the intersection zone 312 of a circle. A third salient feature having a known (e.g., predetermined) distance may be utilized to determine the correct placement of the movable object (e.g., between positions 307 and 309).

  The same principle can be applied to the step of determining the three-dimensional position of the movable object based on three or more salient features. For example, a position within the intersection of three circles centered on each of the three salient features and a position within the intersection of a suitable distance (eg, a suitable radius) to each of the three salient features is the movable object It may be determined to be a position. Additional salient features may be considered to improve the determination of the position of the movable object.

  The devices, methods and computer readable media disclosed herein can present improved maneuverability of movable objects. FIG. 4 illustrates a UAV 402 operating in an outdoor environment 400 according to some embodiments. The outdoor environment 400 may be a urban, suburban, rural environment, or any other environment not at least partially present in a building. UAV 402 may be operated relatively near (e.g., low altitude) ground 404 or relatively away (e.g., high altitude) ground 404. For example, a UAV 402 operating at about 10 m or less from the ground may be considered to be present at low altitude, and a UAV 402 operating at about 10 m from the ground may be considered to be present at high altitude.

  In some embodiments, outdoor environment 400 includes one or more objects 408a-408d. Some objects (eg, objects 408a, 408d) such as buildings, ground vehicles (eg, cars, motorcycles, trucks, bicycles), humans, animals, plants (eg, trees, bushes), other artificial or natural structures May be located on the ground 404. Some objects may contact and / or be supported by the ground 404, water, artificial structures, or natural structures. Alternatively, some objects may be completely located in the air 406 (eg, objects 408b, 408c) and movable (eg, an aircraft, plane, helicopter, hot air balloon, UAV or bird). Objects in the air may not be supported by the ground 404 or by water or by any natural or artificial structure. Obstacles on the ground 404 (eg, towers, skyscrapers, streetlights, wireless towers, power lines, tall structures such as trees) may include portions that substantially extend into the air 406.

  In some examples, sensors such as GPS sensors may be utilized to position the UAV 402 within the outdoor environment 400. In some embodiments, the object may interfere with the GPS signal, and the GPS signal may not be available in the outdoor environment. In some embodiments, the accuracy of positioning based on GPS sensors may be on the order of meters, and precise positioning may require auxiliary positioning sensors (eg, IMU sensors, imaging devices). In some instances, SLAM may be utilized to position a UAV in an outdoor environment. FIG. 5 shows a point cloud computed with SLAM, according to some embodiments. Three-dimensional coordinates of each point (eg, 501, 503, 505) in the image may be calculated and a global map of the environment may be maintained. In some instances, unstable features can result in computational errors that can accumulate. A field of view sensor may be utilized as an alternative to, or as a complement to, other sensors or methods (eg, GPS or SLAM) for positioning a UAV in the environment. For example, a field of view sensor may be utilized in areas where the GPS sensor does not work.

  In some examples, one or more vision sensors coupled to UAV 402 may acquire multiple images of outdoor environment 400 (including image data of an object). The image data of the object may be salient features in the image. The image data of the object may be stationary salient features in the image. In some embodiments, only stationary salient features may be considered to determine the position of the movable object. The position of the UAV 402 can be determined by measuring the distance of the UAV to two or more salient features as described earlier herein. The location of the UAV can be determined without maintaining a global map of the environment, and error accumulation may be minimal or not at all. Positioning of movable objects according to the apparatus and method of the present application is the actuality of movable objects for salient features within about 1 cm, 5 cm, 10 cm, 20 cm, 50 cm, 75 cm, 100 cm, 1.5 m, 2 m, 5 m, or 10 m or more. It may be in the position of The determination of the position of the moveable object can be used during another operation. For example, location information may be utilized to navigate UAV 402 in outdoor environment 400. In some instances, one or more vision sensors may be sufficient to navigate the UAV in an outdoor environment. In some instances, one or more field of view sensors may be utilized with one or more other sensors (eg, an IMU sensor) for navigation.

  FIG. 6 illustrates a UAV 652 operating in a room environment 650, according to some embodiments. The indoor environment 650 is the interior of a building 654 having a floor 656, one or more walls 658, and / or a ceiling or roof 660. Exemplary buildings include residential, commercial, or industrial buildings such as homes, apartments, offices, manufacturing facilities, storage facilities, and the like. The interior of the building 654 may be completely surrounded by the floor 656, the wall 658 and the ceiling 660 such that the UAV 652 is confined to the interior space. Conversely, at least one of the floor 656, the wall 658 or the ceiling 660 may not be present, thereby allowing the UAV 652 to fly from the inside to the outside (or vice versa). Alternatively or in combination, one or more stops 864 may be formed in floor 656, wall 658 or ceiling 660 (eg, door, window, skylight).

  Similar to the outdoor environment 400, the indoor environment 650 may include one or more objects 662a-662d. Some objects (eg, obstacle 662 a) such as furniture, appliances, humans, animals, plants, other artificial or natural objects may be located on floor 656. Conversely, some objects (eg, object 662b) such as birds or other UAVs may be located in the air. Some obstacles in the indoor environment 650 may be supported by other structures or objects. Objects such as lighting fixtures, ceiling fans, beams or other ceiling mounts or structures (eg, obstacle 662 c) may also be attached to the ceiling 660. In some embodiments, objects such as lighting fixtures, shelves, cabinets, other wall-mounted fixtures or structures (eg, obstacle 662d) may be attached to the wall 658. In particular, the structural components of the building 654 (including the floor 656, the walls 658, the ceiling 660) may also be considered objects.

  The objects described herein may be substantially stationary (eg, buildings, plants, structures) or substantially movable (eg, humans, animals, vehicles or other movable objects). Some objects may include a combination of stationary and moving parts (e.g., a wind turbine). The moveable object or obstacle part may move according to a predetermined or predictable path or pattern. For example, the movement of a car may be relatively predictable (eg, based on the shape of the road). Alternatively, some moveable objects or parts of interest may move along random or otherwise unpredictable trajectories. For example, organisms such as animals can move in a relatively unpredictable manner.

  In some embodiments, GPS signals may not be available in the indoor environment, so GPS sensors for positioning the UAV 652 in the indoor environment may not be utilized. In some instances, SLAM may be utilized to locate a UAV in an indoor environment. For SLAM-based positioning, three-dimensional coordinates of each point in the image can be calculated and a global map of the environment can be maintained. In some instances, unstable features may result in computational errors that may accumulate. A field of view sensor may be utilized as an alternative to, or as a complement to, other sensors or methods (eg, GPS or SLAM) for positioning a UAV in the environment. For example, a field of view sensor may be utilized within the area where the GPS sensor does not work (e.g., an indoor environment).

  In some examples, one or more vision sensors coupled to UAV 652 may acquire multiple images of indoor environment 650 (including image data of an object). The image data of the object may be salient features in the image. The image data of the object may be stationary salient features in the image. In some embodiments, only stationary salient features may be considered to determine the position of the movable object. In some embodiments, both stationary salient features and movable features that move according to a predictable path or pattern may be considered to determine the position of the movable object. The position of UAV 652 may be determined by measuring the distance of the UAV to two or more salient features as described earlier herein. The location of the UAV can be determined without maintaining a global map of the environment, and error accumulation may be minimal or not at all. Positioning of movable objects according to the apparatus and method of the present application is the actual movement object relative to salient features within about 1 cm, 5 cm, 10 cm, 20 cm, 50 cm, 75 cm, 100 cm, 1.5 m, 2 m, 5 m or 10 m or less. It may be a position. The determination of the position of the moveable object can be used during another operation. For example, location information may be utilized to navigate UAV 652 in indoor environment 650. In some instances, one or more vision sensors may be sufficient to navigate the UAV in the indoor environment. In some instances, one or more field of view sensors may be utilized with one or more other sensors (eg, an IMU sensor) for navigation.

  The embodiments provided herein enable the use of relatively few features (e.g. salient features), such as for autonomous navigation and control, to determine the position of the vehicle, for positioning the vehicle, etc. obtain. The use of less features may require less or less processing power during vehicle navigation. The use of fewer features (e.g., than points) may result in smaller errors (e.g., by minimizing the effects of unstable feature points), enabling precise and accurate navigation and control of the vehicle. Position determination based on distance to features (e.g. salient features) or self-positioning without the need to maintain a global map may reduce error accumulation and allow precise and accurate navigation and control of the vehicle. The use of salient features may reduce the number of features required for position determination, as salient features are often uniquely identifiable (e.g., consistently and repeatedly distinguishable). Prominent features may also be identified on an ad hoc or real time basis without the need for pre-stored patterns or objects. This may reduce the processing time requirements to look for image matches and the like.

  The embodiments provided herein may be applied to various types of UAVs. For example, the UAV may be a compact UAV having a weight of 10 kg or less and / or a maximum dimension of 1.5 m or less. In some embodiments, the UAV may be a rotorcraft, such as a multi-rotor aircraft (e.g., quadcopter) that is propelled to move in the air by multiple propellers. Additional examples of UAVs and other moveable objects suitable for use with the embodiments presented herein are described in further detail below.

The UAVs described herein may be operated completely autonomously (eg, by a suitable computer system such as an on-board controller), semi-autonomously, or manually (eg, by a human user) . A UAV may receive instructions from a suitable entity (eg, a human user or an autonomous control system) and respond to such instructions by performing one or more operations. For example, the UAV may take off from the ground, move in the air (e.g., for a maximum of three degrees of freedom for translation and for a maximum of three degrees of rotation), move to a target position or a series of target positions, and hover , Landing on the ground, etc. can be controlled. As another example, the UAV is controlled to move at a particular velocity and / or acceleration (e.g., for a maximum of three degrees of freedom for translation and for a maximum of three degrees of rotation) or along a particular path of movement It can be done. Further, instructions may be used to control one or more UAV components, such as components described herein (eg, sensors, actuators, propulsion devices, payloads, etc.). For example, some instructions may be used to control the position, orientation, and / or operation of a UAV payload, such as a camera. Optionally, the UAV may be configured to operate according to one or more predetermined driving rules. The driving rules may be UAV position (eg, latitude, longitude, altitude), orientation (eg, roll, pitch, yaw), velocity (eg, translation and / or angle), and / or acceleration (eg, translation and / or angle) Etc. may be used to control any suitable aspect of the UAV. For example, the driving rules may be designed such that the UAV is not allowed to fly above the threshold height, for example the UAV may be configured to fly at a height of 400 m or less from the ground. In some embodiments, driving rules can be made to provide an automatic mechanism to improve UAV safety and prevent safety incidents. For example, the UAV may be configured to detect a limited flight area (e.g., an airport) and not to fly within a predetermined distance of the restricted flight area, thereby allowing the potential with aircraft and other obstacles. Prevent potential conflicts.

  The systems, devices and methods described herein may be applied to a wide variety of moveable objects. As mentioned above, any description herein of a UAV may be applied and used to any moveable object. The movable object of the present invention is in the air (eg, fixed wing aircraft, rotary wing aircraft, or an aircraft having neither fixed wings nor rotary wings), underwater (eg, ship or submarine), and a motor such as a car Vehicles, trucks, buses, vans, motorbikes, rods, movable structures or frames such as rods, or trains, underground (for example, underground roads), in space (for example, space planes, satellites, or search satellites) Or any combination of these environments may be configured to move within any suitable environment. The movable object may be a vehicle, such as the vehicle described elsewhere herein. The movable object may be a self-propelled unmanned vehicle that does not require human input. In some embodiments, the moveable object may be mounted on a living organism such as a human or an animal. Suitable animals include goats, dogs, cats, horses, cows, sheep, pigs, dolphins, rats or insects. In some embodiments, movable objects may be carried.

  A movable object may move freely within the environment with respect to six degrees of freedom (e.g., three degrees of translation and three degrees of rotation). Alternatively, the movement of the moveable object may be constrained with respect to one or more degrees of freedom, such as a predetermined path, track or orientation. The movement may be actuated by any suitable actuation mechanism, such as an engine or a motor. The actuation mechanism of the movable object may be powered by any suitable energy source such as electrical energy, magnetic energy, solar energy, wind energy, gravity energy, chemical energy, nuclear energy or any suitable combination thereof. The moveable object may be self-propelled through the propulsion system as described elsewhere herein. The propulsion system may optionally operate with an energy source such as electrical energy, magnetic energy, solar energy, wind energy, gravity energy, chemical energy, nuclear energy or any suitable combination thereof. Alternatively, the movable object may be carried by a living being.

  In some instances, the moveable object may be a vehicle. Suitable vehicles may include seaplanes, aircraft, spacecraft or ground vehicles. For example, the aircraft may be fixed wing aircraft (eg, airplanes, gliders), rotary wing aircraft (eg, helicopters, rotary wing aircraft), aircraft having both fixed and rotary wings, or aircraft having neither It can be an airship, a hot air balloon). The vehicle can self-promote the air, water or water, space, ground or underground. A self-propelled vehicle may utilize a propulsion system such as a propulsion system including one or more engines, motors, wheels, axles, magnets, rotors, propellers, rotors, nozzles, or any suitable combination thereof. . In some instances, the propulsion system takes off from the surface, lands on the surface, maintains its current position and / or orientation (eg, is hovering), changes orientation, and / or position Can be used to allow changes.

  The moveable object may be controlled remotely by the user or locally by an occupant in or on the moveable object. In some embodiments, the movable object is an unmanned movable object such as a UAV. An unmanned movable object such as a UAV may have no occupant. The moveable object may be controlled by a person, or an autonomous control system (e.g. a computer control system), or any suitable combination thereof. The movable object may be an autonomous or semi-autonomous robot such as a robot configured by artificial intelligence.

  The moveable object may have any suitable size and / or dimension. In some embodiments, the moveable object may be sized and / or dimensioned to have a human occupant in or on the vehicle. Alternatively, the movable object may be of smaller size and / or dimension than can have a human occupant in or on the vehicle. The moveable object may be of suitable size and / or dimension to be lifted or carried by a human. Alternatively, the moveable object may be larger than the preferred size and / or dimension for being lifted or carried by a human. In some examples, the movable object may have maximum dimensions (eg, length, width, height, diameter, diagonal) of about 2 cm, 5 cm, 10 cm, 50 cm, 1 m, 2 m, 5 m, or 10 m or less. The largest dimension may be about 2 cm, 5 cm, 10 cm, 50 cm, 1 m, 2 m, 5 m, or 10 m or more. For example, the distance between the axes of opposing rotors of the movable object may be less than or equal to about 2 cm, 5 cm, 10 cm, 50 cm, 1 m, 2 m, 5 m, or 10 m. Alternatively, the distance between the axes of the opposed rotors may be about 2 cm, 5 cm, 10 cm, 50 cm, 1 m, 2 m, 5 m or more than 10 m.

In some embodiments, the movable object may have a volume of less than 100 cm × 100 cm × 100 cm, 50 cm × 50 cm × 30 cm, 5 cm × 5 cm × 3 cm. The total volume of the movable object is about 1 cm ³ , 2 cm ³ , 5 cm ³ , 10 cm ³ , 20 cm ³ , 30 cm ³ , 40 cm ³ , 50 cm ³ , 60 cm ³ , 70 cm ³ , 80 cm ³ , 90 cm ³ , 100 cm ³ , 150 cm ³ , 200 cm ^{3 3, 300cm 3, 500cm 3,} 750cm 3, 1000cm 3, 5000cm 3, 10,000cm 3, may be 100,000cm ^3, 1m ^3, or 10 m ³ or less. Conversely, the total volume of the movable object is about 1 cm ³ , 2 cm ³ , 5 cm ³ , 10 cm ³ , 20 cm ³ , 30 cm ³ , 40 cm ³ , 50 cm ³ , 60 cm ³ , 70 cm ³ , 80 cm ³ , 90 cm ³ , 100 cm ³ , 150 cm ^{3, 200cm 3, 300cm 3,} 500cm 3, 750cm 3, 1000cm 3, 5000cm 3, 10,000cm 3, it may be 100,000cm ^3, 1m ^3, or 10 m ³ or more.

In some embodiments, the movable object is about ^{32,000cm 2, 20,000cm 2, 10,000cm 2} , 1,000cm 2, 500cm 2, 100cm 2, 50cm 2, 10cm 2, or 5 cm ² or less footprint It may have (sometimes referred to as a transverse cross-sectional area encompassed by a moveable object). Conversely, the footprint of about ^{32,000cm 2, 20,000cm 2, 10,000cm 2} , may be ^{1,000cm 2, 500cm 2, 100cm 2} , 50cm 2, 10cm 2 or 5 cm ² or more.

  In some instances, the moveable object can weigh 1000 kg or less. The weight of the movable object is about 1000 kg, 750 kg, 500 kg, 200 kg, 100 kg, 80 kg, 70 kg, 60 kg, 50 kg, 45 kg, 40 kg, 35 kg, 30 kg, 25 kg, 20 kg, 15 kg, 12 kg, 10 kg, 9 kg, 8 kg, 7 kg It may be less than 6 kg, 5 kg, 4 kg, 3 kg, 2 kg, 1 kg, 0.5 kg, 0.1 kg, 0.05 kg or 0.01 kg. Conversely, the weight is about 1000 kg, 750 kg, 500 kg, 200 kg, 100 kg, 80 kg, 70 kg, 60 kg, 50 kg, 45 kg, 40 kg, 35 kg, 30 kg, 25 kg, 20 kg, 15 kg, 12 kg, 10 kg, 9 kg, 8 kg, 7 kg, It may be 6 kg, 5 kg, 4 kg, 3 kg, 2 kg, 1 kg, 0.5 kg, 0.1 kg, 0.05 kg or more than 0.01 kg.

  In some embodiments, the moveable object may be smaller than the load carried by the moveable object. The load may include the payload and / or the carrier as described in more detail below. In some instances, the ratio of movable object weight to load weight may be greater than, less than, or equal to about 1: 1. In some instances, the ratio of movable object weight to load weight may be greater than, less than, or equal to about 1: 1. Optionally, the ratio of carrier weight to load weight may be greater than, less than, or equal to about 1: 1. Optionally, the ratio of carrier weight to loaded weight may be less than or equal to 1: 2, 1: 3, 1: 4, 1: 5, 1:10 or even less. Conversely, the ratio of movable object weight to load weight may be greater than 1: 2, 1: 3, 1: 4, 1: 5, 1:10 or even more.

  In some embodiments, movable objects may have low energy consumption. For example, the movable object may use less than or about 5 W / h, 4 W / h, 3 W / h, 2 W / h, 1 W / h or less. In some instances, the movable object carrier may have a low energy consumption. For example, the carrier may use about 5 W / h, 4 W / h, 3 W / h, 2 W / h, less than 1 W / h, or less. Optionally, the payload of the movable object may have a low energy consumption, such as about 5 W / h, 4 W / h, 3 W / h, 2 W / h, less than or less than 1 W / h.

  FIG. 7 shows an unmanned aerial vehicle (UAV) 700 according to an embodiment of the present invention. A UAV may be an example of a moveable object as described herein. UAV 1000 may include a propulsion system having four rotors 702, 704, 706, 708. Any number (eg, 1, 2, 3, 4, 5, 6, 7, 8 or more) of rotors may be provided. A rotor, rotor assembly, or other propulsion system of the unmanned aerial vehicle may allow the unmanned aerial vehicle to remain airborne / position maintained, change orientation, and / or change position. The distance between the opposed rotor axes may be any suitable length 710. For example, the length 710 can be 2 m or less or 5 m or less. In some embodiments, the length 710 can be in the range of 40 cm to 1 m, 10 cm to 2 m, or 5 cm to 5 m. Any description herein of a UAV may apply to movable objects, such as different types of movable objects, and vice versa.

  In some embodiments, the moveable object may be configured to carry a load. The load may include one or more passengers, cargo, equipment, equipment, and the like. The load may be provided in the housing. The housing may be separate from the housing of the movable object or part of the housing of the movable object. Alternatively, the load may comprise a housing, but the movable object does not have a housing. Alternatively, part or all of the load may be provided without a housing. The load may be rigidly fixed to the movable object. Optionally, the load may be movable (e.g., movable or rotatable relative to the movable object) relative to the movable object.

  In some embodiments, the load comprises a payload. The payload may be configured to perform no operation or function. Alternatively, the payload may be a payload (also known as a functional payload) configured to perform an operation or function. For example, the payload may include one or more sensors for surveying one or more targets. Any suitable sensor such as an image capture device (e.g., a camera), an audio capture device (e.g., a parabolic microphone), an infrared imager, or an ultraviolet imager may be incorporated into the payload. The sensor may provide static sensing data (eg, a photo) or dynamic sensing data (eg, an image). In some embodiments, the sensor provides sensing data of a target of the payload. Alternatively or in combination, the payload may include one or more launch means for providing a signal to one or more targets. Any suitable launch means may be used, such as an illumination light source or a sound source. In some embodiments, the payload includes one or more transceivers, such as for communication with a module remote from the moveable object. Optionally, the payload can be configured to interact with the environment or target. For example, the payload may include a tool, device, or mechanism capable of manipulating an object, such as a robotic arm.

  Optionally, the load may include a carrier. The carrier may be provided in the payload, and the payload is coupled to the movable object either directly (eg, in direct contact with the movable object) or indirectly (eg, without contact with the movable object) via the carrier It can be done. Conversely, the payload can be mounted on the movable object without the need for a carrier. The payload may be integrally formed with the carrier. Alternatively, the payload may be removably coupled to the carrier. In some embodiments, the payload may include one or more payload elements, and one or more of the payload elements may be movable relative to the moveable object and / or carrier as described above.

  The carrier may be integrally formed with the movable object. Alternatively, the carrier may be removably coupled to the movable object. The carrier may be coupled directly or indirectly to the movable object. The carrier may support the payload (eg, may carry at least a portion of the weight of the payload). The carrier may include a suitable mounting structure (eg, a gimbal platform) that can stabilize and / or direct the movement of the payload. In some embodiments, the carrier may be adapted to control the state (eg, position and / or orientation) of the payload relative to the moveable object. For example, the carrier may be moved relative to the movable object (eg, one, two or three degrees of freedom of translation such that the payload maintains its position and / or orientation relative to the preferred reference coordinate system despite movement of the movable object) And / or with respect to one, two or three degrees of freedom of rotation). The reference coordinate system may be a fixed reference coordinate system (eg, the surrounding environment). Alternatively, the reference coordinate system may be a moving reference coordinate system (e.g. movable object, payload target).

  In some embodiments, the carrier may be configured to allow movement of the payload relative to the carrier and / or the moveable object. The movement may be translated for up to three degrees of freedom (eg, along one, two or three axes), or rotated for up to three degrees of freedom (eg, about one, two or three axes), or any of them It may be a preferred combination.

  In some instances, the carrier can include a carrier frame assembly and a carrier actuation assembly. The carrier frame assembly may provide structural support to the payload. The carrier frame assembly may include individual carrier frame parts, some of which may be movable relative to one another. The carrier actuation assembly may include one or more actuators (eg, motors) that actuate the movement of the individual carrier frame components. The actuator may be configured to allow movement of multiple carrier frame parts simultaneously or to allow movement of one single carrier frame part. Movement of the carrier frame component may result in movement of the payload accordingly. For example, the carrier actuation assembly may actuate rotation of one or more carrier frame components about one or more rotation axes (e.g., roll, pitch, or yaw axes). Rotation of the one or more carrier frame components may cause the payload to rotate about one or more rotation axes relative to the moveable object. Alternatively or in combination, the carrier actuation assembly may actuate the translation of one or more carrier frame parts along one or more translational axes, thus for the moveable object one or more axes of interest. Translation of the payload along can occur.

  In some embodiments, movement of the moveable object, the carrier, and the payload relative to a fixed reference coordinate system (e.g., the surrounding environment) and / or with respect to each other may be controlled by the terminal. The terminal may be a remote control at a location remote from the movable object, the carrier and / or the payload. The terminal may be disposed on or fixed to the support platform. Alternatively, the terminal may be a portable or wearable device. For example, the terminal may include a smartphone, a tablet, a laptop, a computer, glasses, gloves, a helmet, a microphone, or a suitable combination thereof. The terminal may include a user interface such as a keyboard, a mouse, a joystick, a touch screen or a display. Any suitable user input may be utilized, such as manual input instructions, voice control, gesture control, or position control (eg, via movement, position or tilt of the terminal) to interact with the terminal.

  The terminal may be used to control any suitable state of the moveable object, the carrier and / or the payload. For example, terminals may be used to control the position and / or orientation of movable objects, carriers and / or payloads with respect to fixed references and / or with respect to each other. In some embodiments, the terminal may be used to control the movable object, the carrier and / or the individual elements of the payload, such as an actuation assembly of the carrier, a sensor of the payload, or a launch means of the payload. The terminal may include a wireless communication device adapted to communicate with one or more of the movable object, the carrier or the payload.

  The terminal may include a display suitable for viewing movable object, carrier and / or payload information. For example, the terminal may be configured to display information of movable objects, carriers and / or payloads regarding position, translational velocity, translational acceleration, orientation, angular velocity, angular acceleration or any suitable combination thereof. In some embodiments, the terminal may display information provided by the payload, such as data provided by the functional payload (eg, an image recorded by a camera or other imaging device).

  Optionally, the same terminal controls the movable object, the carrier and / or the payload or controls the state of the movable object, the carrier and / or the payload, and receives information from the movable object, the carrier and / or the payload And / or displaying. For example, the terminal may control the position of the payload relative to the environment while displaying information about the position of the image data or payload captured by the payload. Alternatively, different terminals may be used for different functions. For example, a first terminal may control movable object, carrier and / or payload movement or status, while a second terminal may display and / or receive information from the movable object, carrier and / or payload. For example, while the second terminal displays the image data captured by the payload, the first terminal may be used to control the positioning of the payload relative to the environment. Various communication schemes may be used between the movable object and the integrated terminal that controls the movable object and receives data, or between the movable object and a plurality of terminals that controls the movable object and receives data. For example, at least two different communication schemes may be formed between the movable object and a terminal that both controls the movable object and receives data from the movable object.

  FIG. 8 shows a moveable object 800 including a carrier 802 and a payload 804, according to some embodiments. Although the moveable object 800 is depicted as an aircraft, this depiction is not intended to be limiting as described earlier herein, and any suitable type of moveable object may be used. Those skilled in the art will recognize that any of the embodiments described herein in the context of an aircraft system may be applied to any suitable moveable object (e.g., a UAV). In some instances, the payload 804 can be provided on the moveable object 800 without the need for the carrier 802. Movable object 800 may include a propulsion mechanism 806, a sensing system 808, and a communication system 810.

  The propulsion mechanism 806 may include one or more of a rotor, a propeller, a rotor, an engine, a motor, a wheel, an axle, a magnet or a nozzle as described above. For example, the propulsion mechanism 806 can be a rotor assembly or other rotational propulsion device, as described elsewhere herein. The moveable object may have one or more, two or more, three or more, or four or more propulsion mechanisms. The propulsion mechanisms may all be of the same type. Alternatively, one or more propulsion mechanisms may be different types of propulsion mechanisms. The push mechanism 806 may be mounted on the moveable object 800 by using any suitable means, such as a support element (e.g., a drive shaft) as described elsewhere herein. The propulsion mechanism 806 may be mounted on any suitable portion, such as the upper, lower, front, rear, side or any suitable combination thereof of the moveable object 800.

  In some embodiments, the propulsion mechanism 806 can be vertically taken off or on the surface from the surface (e.g., without traveling the runway) without the movable object 800 requiring any horizontal movement of the movable object 800. It may be possible to land vertically. Optionally, the propulsion mechanism 806 may be operable to allow the movable object 800 to hover in the air at a predetermined position and / or orientation. One or more propulsion mechanisms 800 may be controlled independently of other propulsion mechanisms. Alternatively, propulsion mechanism 800 may be configured to be controlled simultaneously. For example, movable object 800 may have a plurality of horizontally oriented rotors that can provide lift and / or thrust to the movable object. A multi-horizontally-oriented rotor can be actuated to give the movable object 800 vertical take-off, vertical landing and hovering capabilities. In some embodiments, one or more of the multi-horizontally oriented rotors may spin in a clockwise direction and one or more of the multi-horizontally oriented rotors may spin in the counterclockwise direction. For example, the number of clockwise rotors may be equal to the number of counterclockwise rotors. The rotational speed of each of the horizontally oriented rotors may be independently varied to control the lift and / or thrust generated by each rotor, thereby causing the spatial arrangement, velocity, and / or acceleration of the movable object 800. (E.g., for a maximum of three degrees of freedom of translation and a maximum of three degrees of rotation).

  The sensing system 808 may sense one or more sensors that may sense the spatial arrangement, velocity, and / or acceleration of the moveable object 800 (e.g., in terms of up to three degrees of freedom for translation and up to three degrees of rotation). May be included. The one or more sensors may include global positioning system (GPS) sensors, motion sensors, inertial sensors, proximity sensors or image sensors. The sensing data provided by the sensing system 808 controls the spatial arrangement, velocity, and / or orientation of the moveable object 800 (eg, by using a suitable processing unit and / or control module as described below) Can be used for Alternatively, sensing system 808 may be used to provide data regarding the environment surrounding the moveable object (weather status, proximity to potential obstacles, location of geographic features, location of artificial structures, etc.).

  Communication system 810 enables communication via wireless signal 816 with terminal 812 having communication system 814. The communication systems 810, 814 may include any number of transmitters, receivers and / or transceivers suitable for wireless communication. The communication may be one-way communication, so data may be transmitted in one direction only. For example, one-way communication may include the step where only movable object 800 transmits data to terminal 812 (or vice versa). Data may be transmitted from one or more transmitters of the communication system 810 to one or more receivers of the communication system 812 (or vice versa). Alternatively, the communication may be bi-directional communication, so data may be transmitted in both directions between the movable object and the terminal 812. Two-way communication may include transmitting data from one or more transmitters of the communication system 810 to one or more receivers of the communication system 814 (and vice versa).

  In some embodiments, terminal 812 may provide control data to one or more of movable object 800, carrier 802, payload 804 and may be one or more of movable object 800, carrier 802, payload 804. Information may be received from more than one (eg, position and / or motion information of the movable object, carrier or payload, data sensed by the payload, such as image data captured by the payload camera). In some instances, control data from the terminal may include relative position, movement, actuation or control of the moveable object, carrier and / or payload. For example, the control data may result in correction of the position and / or orientation of the moveable object (eg, via control of the propulsion mechanism 806) or movement of the payload relative to the moveable object (eg, via control of the carrier 802) . Control data from the terminal controls the payload, such as controlling the operation of a camera or other imaging device (e.g., taking a still or moving image, zooming in or out, turning on or off, switching imaging mode, imaging It is possible to change the depth of field, change the focus, change the image resolution, change the exposure time, change the viewing angle or the field of view). In some examples, communication from the moveable object, carrier and / or payload may include information from one or more sensors (eg, regarding sensing system 808 or payload 804). The communication may include sensed information from one or more different types of sensors (eg, GPS sensors, motion sensors, inertial sensors, proximity sensors or image sensors). Such information may relate to the position (eg, position, orientation), movement or acceleration of the moveable object, carrier and / or payload. Such information from the payload may include the data or payload sensed from the payload. Control data transmitted and provided by terminal 812 may be configured to control the state of one or more of movable object 800, carrier 802 or payload 804. Alternatively or in combination, carrier 802 and payload 804 may also each include a communication module configured to communicate with terminal 812 so that the terminal communicates independently with each of movable object 800, carrier 802, and payload 804. And each can be controlled.

  In some embodiments, movable object 800 may be configured to communicate with another remote device in addition to or in place of terminal 812. Terminal 812 may also be configured to communicate not only with moveable object 800 but also with other remote devices. For example, movable object 800 and / or terminal 812 may communicate with another movable object or with the carrier or payload of another movable object. If desired, the remote device may be a second terminal or other computing device (eg, a computer, laptop, tablet, smartphone or other portable device). The remote device may be configured to transmit data to movable object 800, receive data from movable object 800, transmit data to terminal 812 and / or receive data from terminal 812. Optionally, the remote device may be connected to the Internet or other telecommunication network, so that data received from the mobile object 800 and / or the terminal 812 may be uploaded to a website or server.

  FIG. 9 is a schematic diagram according to a block diagram of a system 900 for controlling movable objects according to some embodiments. System 900 may be used in combination with any of the preferred embodiments of the systems, apparatus and methods disclosed herein. System 900 may include a sensing module 902, a processing unit 904, non-transitory computer readable media 906, a control module 908, and a communication module 910.

  The sensing module 902 may utilize various types of sensors that collect information related to movable objects in various manners. Different types of sensors may sense different types of signals or signals from different sources. For example, the sensor may include an inertial sensor, a GPS sensor, a proximity sensor (eg, a rider) or a field / image sensor (eg, a camera). The sensing module 902 may be operatively coupled to a processing unit 904 having a plurality of processors. In some embodiments, the sensing module may be operatively coupled to a transmission module 912 (e.g., a Wi-Fi video transmission module) configured to transmit sensing data directly to a suitable external device or system. For example, transmission module 912 may be used to transmit an image captured by the camera of sensing module 902 to a remote terminal.

  The processing unit 904 may comprise one or more processors, such as a programmable processor (e.g., a central processing unit (CPU)). Processing unit 904 may be operatively coupled to non-transitory computer readable medium 906. Non-transitory computer readable medium 906 may store logic, code and / or program instructions executable by processing unit 904 for performing one or more steps. Non-transitory computer readable media may include one or more memory units (eg, removable media such as an SD card or random access memory (RAM) or external storage). In some embodiments, data from sensing module 902 may be directly carried to and stored in a memory unit of non-transitory computer readable medium 906. A memory unit of non-transitory computer readable medium 906 stores logic, code, program instructions and / or code executable by processing unit 904 to perform any of the preferred embodiments of the methods described herein. It can. For example, processing unit 904 may be configured to execute instructions that cause one or more processors of processing unit 904 to analyze sensing data generated by the sensing module. The memory unit may store sensing data from the sensing module processed by the processing unit 904. In some embodiments, a memory unit of non-transitory computer readable medium 906 may be used to store processing results generated by processing unit 904.

  In some embodiments, the processing unit 904 may be operatively coupled to a control module 908 configured to control the state of the moveable object. For example, control module 908 may be configured to control the propulsion mechanism of the movable object to adjust the spatial arrangement, velocity, and / or acceleration of the movable object with respect to six degrees of freedom. Alternatively or in combination, control module 908 may control one or more of the carrier, payload or sensing module states.

  The processing unit 904 may be operatively coupled to a communication module 910 configured to transmit and / or receive data from one or more external devices (eg, terminals, displays or other remote control devices). . Any suitable communication means such as wired communication or wireless communication may be used. For example, the communication module 910 may use one or more of a local area network (LAN), wide area network (WAN), infrared, wireless, WiFi, point-to-point (P2P) network, communication network, cloud communication, etc. obtain. Optionally, relay stations such as towers, satellites or mobile stations may be used. Wireless communication may be proximity dependent or proximity independent. In some embodiments, a line of sight for communication may or may not be required. The communication module 910 transmits and / or transmits one or more of sensing data from the sensing module 902, processing results generated by the processing unit 904, predetermined control data, user commands from a terminal or a remote control device, etc. Can receive.

  The components of system 900 may be arranged in any suitable configuration. For example, one or more of the components of system 900 may be disposed on a movable object, a carrier, a payload, a terminal, a sensing system, or an additional external device in communication with one or more of these. In addition, although FIG. 9 depicts a single processing unit 904 and a single non-transitory computer readable medium 906, one of ordinary skill in the art is not intended to be limited thereto, and that multiple systems 900 are provided. It will be appreciated that the processing unit and / or non-transitory computer readable medium may be included. In some embodiments, one or more of the plurality of processing units and / or non-transitory computer readable media communicate with one or more of the movable object, the carrier, the payload, the terminal, the sensing module, There may be various locations, such as on additional external devices, or a suitable combination thereof, and thus any suitable aspect of the processing and / or memory functions performed by the system 900 is in one or more of the aforementioned locations. It can occur.

  As used herein, A and / or B include one or more of A or B, a combination thereof such as A and B, and the like.

  While preferred embodiments of the present invention have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Numerous variations, changes, and substitutions will now occur to those skilled in the art without departing from the invention. It should be understood that various alternatives to the embodiments of the invention described herein may be employed in practicing the invention. It is intended that the following claims define the scope of the invention and that methods and structures within the scope of these claims and their equivalents be covered thereby.

Claims (21)

A method of determining the position of a mobile
Receiving a plurality of images acquired by one or more sensors;
Identifying a plurality of salient features from the plurality of images;
Calculating the distance from the mobile to each of the plurality of salient features;
Based on the calculated distance, the position is in the intersection zone of at least two sets of concentric circles, and each set is determined the position of the mobile body around at least one of the plurality of salient features And a method comprising the steps of:   The method according to claim 1, wherein the mobile body is an unmanned aerial vehicle (UAV).   The method of claim 1, wherein the plurality of salient features are identified using a saliency map generated by filtering the plurality of images.   4. The method of claim 3, wherein the plurality of salient features are identified by measuring the variance of pixel tone levels of one or more portions of the saliency map.   5. The method of claim 4, wherein the plurality of salient features are identified as the variance of the pixel tone level exceeds a variance threshold.   The method of claim 1, wherein the plurality of images are captured by a single focus sensor.   The method according to claim 1, further comprising the step of determining virtual coordinates of the local coordinate system of each salient feature with the mobile object as the origin of the local coordinate system.   The method of claim 7, wherein the virtual coordinates of the salient feature are at the center of the salient feature.   The method according to claim 7, wherein the virtual coordinates of the salient feature are at the center of mass of the stationary salient feature.   The method according to claim 7, wherein the local coordinate system is a two-dimensional coordinate system or a three-dimensional coordinate system.   The method according to claim 1, wherein calculating the distance comprises measuring the depth of the salient feature based on disparity.   The method of claim 11, wherein the disparity is observed in consecutive frames in the plurality of images.   The method according to claim 12, wherein the continuous frame is captured during movement of the mobile.   The method of claim 1, wherein the plurality of images are captured during movement of the mobile.   The method of claim 1, wherein the plurality of images are two or more images captured in different fields of view.   The method according to claim 1, wherein the plurality of salient features are identified by (1) obtaining a saliency map by filtering and (2) obtaining salient feature candidates from the saliency map.   The method of claim 1, wherein the plurality of salient features are identified using a spectral residual approach.   The method of claim 1, wherein the plurality of salient features are identified using frequency tuned salient area detection.   The method of claim 1, wherein the plurality of salient features are identified using a binary norm gradient for objectness estimation. A device for determining the position of a moving body,
One or more vision sensors configured to acquire a plurality of images;
20. An apparatus comprising one or more processors for performing the method of any one of claims 1-19. A non-transitory computer readable medium comprising program instructions for determining the position of a mobile, comprising:
20. A computer readable medium comprising program instructions for performing the method according to any one of the preceding claims.