JP2020042854A - System and method for selecting operation mode of mobile platform - Google Patents

Abstract

To provide a system capable of using different operation modes of a mobile platform and satisfying high accuracy requirements for selecting the operation mode of the mobile platform within a wide height range.SOLUTION: A heigh of a mobile platform and a difference between first and second images are determined from the viewpoint of the mobile platform, and an accurate operation mode of the mobile platform is selected from among a plurality of operation modes of the mobile platform on the basis of the determined height and the difference between the images. These operation modes can be classified based on, for example, the height and/or the image difference. In another mode, a system can operate the mobile platform on the basis of the image acquired at different time points and data acquired via an inertia measurement unit.SELECTED DRAWING: Figure 1

Description

  The disclosed embodiments relate generally to the operation of a mobile platform, and more particularly, without limitation, to a system and method for operating a mobile platform within a wide range of heights.

  Unmanned aerial vehicles ("Unmanned Aerial Vehicles: UAVs") are commonly navigated or otherwise manipulated via visual techniques. However, the performance and accuracy of vision technology is limited and may change according to the height of the UAV.

  Currently available vision techniques can only guarantee their performance and accuracy within a certain height range. Due to the inherent shortcomings of vision technology, the accuracy of operating a mobile platform at relatively small or large heights is limited and cannot be guaranteed.

  In view of the above, there is a need for a system and method for effectively operating a mobile platform within a wide range of heights.

According to a first aspect disclosed herein, a method for selecting an operation mode of a mobile platform is disclosed, the method comprising:
Detecting the height grade of the mobile platform;
Selecting an operation mode of the mobile platform according to a result of the detection.

  In an exemplary embodiment of the disclosed method, the step of detecting the height grade comprises the step of determining the height of the mobile platform and / or the first and second images of the remote object from the perspective of the mobile platform. Determining the difference between the two.

  In another exemplary embodiment of the disclosed method, the determining step obtains a height via a barometer, an ultrasonic detector, and / or a global positioning system ("GPS"). With steps.

  In another exemplary embodiment of the disclosed method, the determining comprises obtaining a difference between the first and second images of the captured object by a binocular imaging system associated with the mobile platform. Have.

  An exemplary embodiment of the disclosed method further comprises classifying the operating mode based on the height and / or difference values.

  An exemplary embodiment of the disclosed method further comprises activating the mobile platform to operate in the first height mode.

  In an exemplary embodiment of the disclosed method, the first height mode comprises a very low altitude monocular mode.

  An exemplary embodiment of the disclosed method further comprises providing a moving platform distance sensor to support an ultra-low altitude monocular mode.

  In an exemplary embodiment of the disclosed method, selecting an operating mode comprises switching the operating mode based on the height grade, and wherein the height grade is determined. The determination is based on at least one of the height and the determined difference.

  In another exemplary embodiment of the disclosed method, selecting the operating mode comprises switching the mobile platform to a second height mode when the height is above a first height threshold. Having.

  In another exemplary embodiment of the disclosed method, selecting an operating mode comprises switching the mobile platform to a second height mode when the difference is less than a first difference threshold.

  In another exemplary embodiment of the disclosed method, the step of selecting an operation mode includes the step of: when the height is above a first height threshold and the difference is below a first difference threshold, the mobile platform is brought into a second position. Switching to a height mode.

  In another exemplary embodiment of the disclosed method, switching the mobile platform to the second height mode comprises selecting a stereo vision mode having a first resolution.

  In another exemplary embodiment of the disclosed method, selecting an operating mode includes switching the mobile platform to a third height mode when the difference is less than or equal to a third difference threshold.

  In another exemplary embodiment of the disclosed method, switching the mobile platform to a third height mode comprises switching from a binocular imaging device to a stereo vision mode with improved resolution.

  In another exemplary embodiment of the disclosed method, selecting the operating mode comprises switching the mobile platform to a fourth height mode when the height is above a third height threshold. Have.

  In another exemplary embodiment of the disclosed method, the step of selecting an operation mode comprises the step of: setting the mobile platform to a fourth when the height is above a third height threshold and the difference is below a fifth difference threshold. Switching to a height mode.

  In another exemplary embodiment of the disclosed method, the step of switching the mobile platform to the fourth height mode includes the step of switching the binocular imaging device to a barometer, GPS, and / or vertical between the mobile platform and ground level. Switching to a high altitude monocular mode in a combination of visual measurement of directional distance.

  In another exemplary embodiment of the disclosed method, selecting an operating mode comprises switching the mobile platform to a third height mode when the height is less than a fourth height threshold. .

  In another exemplary embodiment of the disclosed method, the step of selecting an operation mode comprises the step of: setting the mobile platform to a third when the height is less than a fourth height threshold and the difference is above a sixth difference threshold. Switching to a height mode.

  In another exemplary embodiment of the disclosed method, selecting an operating mode comprises switching the mobile platform to a second height mode when the difference exceeds a fourth difference threshold.

  In another exemplary embodiment of the disclosed method, selecting an operating mode comprises switching the mobile platform to a first height mode when the height is less than a second height threshold. .

  In another exemplary embodiment of the disclosed method, the step of selecting an operation mode comprises the steps of: first, when the height is less than a second height threshold and the difference is greater than a second difference threshold, the mobile platform is first. Switching to a height mode.

In another exemplary embodiment of the disclosed method, the second difference threshold is greater than the first difference threshold;
At least one of the first and second difference thresholds is greater than at least one of the third and fourth difference thresholds;
The third difference threshold is higher than the fourth difference threshold,
At least one of the third and fourth difference thresholds is greater than at least one of the fifth and sixth difference thresholds; and
The sixth difference threshold is higher than the fifth difference.

In another exemplary embodiment of the disclosed method,
The first height threshold is greater than the second height threshold,
At least one of the third and fourth height thresholds is greater than at least one of the first and second height thresholds, and
The third height threshold is higher than the fourth height threshold.

In another exemplary embodiment of the disclosed method, determining the difference between the first and second images comprises:
Selecting a plurality of feature points from the first image;
Matching feature points of the first image with points of the second image.

  In another exemplary embodiment of the disclosed method, the feature points comprise pixels of a first image or a second image.

In another exemplary embodiment of the disclosed method, matching the plurality of feature points comprises:
Scanning the second image to select points in the second image that match selected feature points in the first image;
Calculating the similarity between the selected feature point of the first image and the point of the second image.

In another exemplary embodiment of the disclosed method, calculating the similarity comprises:
By comparing the intensities of each point pair in the region and constructing a first binary string representing the first region around the selected feature point, the first binary stable independent basic feature ("Binary Robust Independent Elementary"). Generating a Feature: BRIEF ") descriptor;
Generating a second BRIFF descriptor by comparing the intensities of each point pair of the second region to construct a second binary string representing the second region around the point of the second image;
Determining that a point in the second image matches the selected feature point when the Hamming distance between the first and second BRIEF descriptors is less than a first Hamming threshold. .

  In another exemplary embodiment of the disclosed method, calculating the similarity comprises selecting selected feature points of the first image from a 3 × 3 pixel area centered about a point on the second image. Comparing with.

  In another exemplary embodiment of the disclosed method, the comparing step comprises summing a difference for each color component of each pixel of the color image or a sum of differences of grayscale values of each pixel of the black and white image. .

  In another exemplary embodiment of the disclosed method, determining the differences comprises obtaining the differences based on an average of the differences of the feature points.

  In another exemplary embodiment of the disclosed method, determining the difference comprises selecting one or more feature points and obtaining the difference based on the difference between the selected feature points. Having.

  According to another aspect disclosed herein, selecting an operating mode of a mobile platform configured to perform a detection process according to any one of the above-described embodiments of the disclosed method. A system for performing the above is disclosed.

  According to another embodiment disclosed herein, a mode of operation of a mobile platform configured to perform a detection process according to any one of the above-described embodiments of the disclosed method is provided. A computer program product having instructions for selecting is disclosed.

According to another aspect disclosed herein, an apparatus for selecting an operation mode of a mobile platform is disclosed, wherein the apparatus comprises:
A binocular imaging device associated with the mobile platform;
A processor,
Detect height grade of mobile platform, and
A processor configured to select an operation mode of the mobile platform according to a result of the detection.

  In an exemplary embodiment of the disclosed apparatus, the processor is configured to determine a height of the mobile platform and / or a difference between the first and second images of the remote object from the perspective of the mobile platform. Have been.

  Example embodiments of the disclosed apparatus further include a barometer associated with the mobile platform to obtain a height of the mobile platform.

  Example embodiments of the disclosed apparatus further include an ultrasonic detector associated with the mobile platform to obtain a height of the mobile platform.

  Example embodiments of the disclosed device further include a GPS associated with the mobile platform to obtain the height and / or location of the mobile platform.

  In an exemplary embodiment of the disclosed apparatus, the processor is configured to obtain a difference between the first and second images of the captured object by a binocular imaging system associated with the mobile platform. I have.

  In an exemplary embodiment of the disclosed device, the operating modes of the mobile platform are categorized based on height values and / or difference values.

  In an exemplary embodiment of the disclosed apparatus, the processor is configured to initialize the mobile platform to operate in the first height mode.

  In an exemplary embodiment of the disclosed apparatus, the first height mode comprises a very low altitude monocular mode.

  In an exemplary embodiment of the disclosed device, a moving platform distance sensor is provided to support ultra-low altitude monocular mode.

  In another exemplary embodiment of the disclosed apparatus, the processor is configured to switch the operating mode based on the height grade, and wherein the height grade is determined. The determination is based on at least one of the height and the determined difference.

  In another exemplary embodiment of the disclosed apparatus, the processor is configured to switch to a second height mode when the height exceeds a first height threshold.

  In another exemplary embodiment of the disclosed apparatus, the processor is configured to switch to the second height when the difference is less than the first difference threshold.

  In another exemplary embodiment of the disclosed apparatus, the processor is configured to switch to a second height mode when the height is above a first height threshold and the difference is below a first difference threshold. It is configured.

  In another exemplary embodiment of the disclosed device, the second height mode comprises a stereo vision mode having a first resolution.

  In another exemplary embodiment of the disclosed apparatus, the processor is configured to switch to a third height mode when the difference is less than or equal to a third difference threshold.

  In another exemplary embodiment of the disclosed device, the third height mode is a stereo vision mode with improved resolution.

  In another exemplary embodiment of the disclosed apparatus, the processor is configured to switch to a fourth height mode when the height exceeds a third height threshold.

  In another exemplary embodiment of the disclosed apparatus, the processor is configured to switch to a fourth height mode when the height is above a third height threshold and the difference is below a fifth difference threshold. It is configured.

  In another exemplary embodiment of the disclosed device, the fourth height mode is a high altitude in a combination of barometer, GPS, and / or a visual measurement of the vertical distance between the mobile platform and ground level. It is a monocular mode.

  In another exemplary embodiment of the disclosed apparatus, the processor is configured to switch to a third height mode when the height is less than a fourth height threshold.

  In another exemplary embodiment of the disclosed apparatus, the processor is configured to switch to a third height mode when the height is less than a fourth height threshold and the difference is above a sixth difference threshold. It is configured.

  In another exemplary embodiment of the disclosed apparatus, the processor is configured to switch to a second height mode when the difference exceeds a fourth difference threshold.

  In another exemplary embodiment of the disclosed apparatus, the processor is configured to switch to a first height mode when the height is less than a second height threshold.

  In another exemplary embodiment of the disclosed apparatus, the processor is configured to switch to a first height mode when the height is less than a second height threshold and the difference exceeds a second difference threshold. It is configured.

In another exemplary embodiment of the disclosed device, the second difference threshold is greater than the first difference threshold;
One or both of the first and second difference thresholds is greater than one or more of the third and fourth difference thresholds;
The third difference threshold is higher than the fourth difference threshold,
One or more of the third and fourth difference thresholds is greater than one or more of the fifth and sixth difference thresholds, and
The sixth difference threshold is higher than the fifth threshold.

In another exemplary embodiment of the disclosed device, the first height threshold is greater than the second height threshold,
Any one of the third and fourth height thresholds is greater than the first and second height thresholds, and
The third height threshold is higher than the fourth threshold.

In another exemplary embodiment of the disclosed apparatus, the processor is configured to determine a height of the unmanned aerial vehicle (“UAV”) and a difference from a UAV perspective, and
Here, the processor is configured to select an operation mode of the UAV according to the determination.

  In another exemplary embodiment of the disclosed apparatus, the processor is configured to select a plurality of feature points on the first image and match the plurality of feature points of the first image with points of the second image. ,It is configured.

  In another exemplary embodiment of the disclosed apparatus, the feature points comprise pixels of a first image or a second image.

  In another exemplary embodiment of the disclosed device, the difference is at least 5 pixels and does not exceed one fifth of the width of the first or second image.

  In another exemplary embodiment of the disclosed apparatus, the processor scans the second image to identify points in the second image that match selected feature points of the first image, and processes the first image. It is configured to calculate the similarity between the selected feature points and the points.

In another exemplary embodiment of the disclosed apparatus, the processor comprises:
A first binary stable independent basic feature ("BRIEF") description by constructing a first binary string representing a first region around the selected feature point by comparing the intensity of each point pair in the region. Create a child,
Generating a second BRIFF descriptor by comparing the intensities of each point pair of the second region to construct a second binary string representing the second region around the point of the second image; and
When the Hamming distance between the first BRIEF descriptor and the second BRIEF descriptor is less than the first Hamming threshold, it is configured to determine that the point of the second image matches the selected feature point. ing.

  In another exemplary embodiment of the disclosed apparatus, the similarity is comparing a selected feature point of the first image with a 3 × 3 pixel area centered about a point on the second image. Is calculated by

  In another exemplary embodiment of the disclosed apparatus, the 3 × 3 pixel area is the sum of the differences for each color component of each pixel of the color image or the grayscale value of each pixel of the black and white image. They are compared by the sum of the differences.

  In another exemplary embodiment of the disclosed apparatus, the differences are determined based on an average of the feature differences.

  In another exemplary embodiment of the disclosed apparatus, the processor is configured to select one or more feature points and obtain a difference based on a feature difference of the selected feature points. I have.

According to another aspect disclosed herein, a method for determining a mode of operation of a mobile platform is disclosed, the method comprising:
Detecting the height grade of the mobile platform;
Selecting one or more sensors based on the result of the detection;
Obtaining a measurement from the selected sensor,
In this case, the height grade is selected from a group of four height grades.

  In another exemplary embodiment of the disclosed method, selecting one or more sensors comprises selecting one image sensor at a first height grade.

  In another exemplary embodiment of the disclosed method, selecting one or more sensors comprises selecting an image sensor and a distance sensor at a first height grade.

  In another exemplary embodiment of the disclosed method, the distance sensor has an ultrasonic detector and / or a laser detection device.

In another exemplary embodiment of the disclosed method, selecting one or more sensors further comprises selecting at least two image sensors at a second height grade.
In this case, at least two image sensors have a first resolution.

  In another exemplary embodiment of the disclosed method, selecting one or more sensors further comprises selecting at least two image sensors having a second resolution at a third height grade. However, the second resolution is an improved resolution.

  In another exemplary embodiment of the disclosed method, the second resolution is greater than the first resolution.

  In another exemplary embodiment of the disclosed method, selecting one or more sensors further comprises selecting one image sensor and one height sensor in a fourth height grade. .

  In another exemplary embodiment of the disclosed method, the height sensor comprises a barometer and / or a global positioning system ("GPS").

According to another aspect disclosed herein, a mobile platform flight operation system is disclosed, the system comprising:
A height sensor that detects the height grade of the mobile platform;
A processor configured to select one or more sensors based on the detected height grade and obtain measurements from the selected sensors;
In this case, the height grade is selected from a group of four height grades.

  In another exemplary embodiment of the disclosed system, the processor is configured to select one image sensor at a first height grade.

  In another exemplary embodiment of the disclosed system, the processor is configured to select one image sensor and one distance sensor at a first height grade.

  In another exemplary embodiment of the disclosed system, the height sensor comprises a distance sensor, and in this case, the distance sensor comprises an ultrasonic detector and / or a laser detection device.

In another exemplary embodiment of the disclosed system, the processor is configured to select at least two image sensors at the second height grade,
In this case, at least two image sensors have a first resolution.

  In another exemplary embodiment of the disclosed system, the processor is configured to select at least two image sensors having a second resolution at a third height grade, wherein the second resolution is improved. Resolution.

  In another exemplary embodiment of the disclosed system, the second resolution is greater than the first resolution.

  In another exemplary embodiment of the disclosed system, the processor is configured to select one image sensor and one height sensor in a fourth height grade.

  In another exemplary embodiment of the disclosed system, the height sensor comprises a barometer and / or a global positioning system ("GPS").

5 is an exemplary top-level flowchart illustrating one embodiment of a method for selecting an operating mode within a wide height range. FIG. 2 is an exemplary schematic diagram illustrating a mobile platform equipped with a device for implementing the method of FIG. 1, wherein such a device includes a barometer, a GPS, an ultrasound detector, and a stereo vision system. . 2 is an exemplary flowchart of an alternative embodiment of the method of FIG. 1, wherein the method includes the step of classifying operating modes for different heights. 4 is an exemplary block diagram of an alternative embodiment of the method of FIG. 3, where the method classifies the four modes of operation based on height. FIG. 2 is an exemplary block diagram of another alternative embodiment of the method of FIG. 1, wherein height and difference information is used to determine an operating mode of operation. FIG. 3 is an exemplary block diagram of an alternative embodiment of the mobile platform of FIG. 2, where the mobile platform uses a barometer, an ultrasonic detector, and / or GPS to obtain height information. . FIG. 3 is an exemplary top-level diagram illustrating another alternative embodiment of the mobile platform of FIG. 2, where a processor is collecting state data and controlling an operating mode. FIG. 4 is an exemplary flowchart of another alternative embodiment of the method of FIG. 1, illustrating switching conditions between four different modes of operation. FIG. 4 is an exemplary detail drawing illustrating one embodiment of a stereoscopic imaging method, in which the differences of the method of FIG. 1 have been determined. FIG. 10 is an exemplary diagram of an alternative embodiment of the method of FIG. 9, illustrating an exemplary method of matching two images with an overlap area. FIG. 10 is an exemplary diagram illustrating the alternative embodiment of FIG. 9, wherein the first image is matched with a second image having a plurality of feature points. FIG. 12 is an exemplary diagram illustrating an alternative embodiment of the method of FIG. 11, wherein each feature point is matched by calculating similarity. 5 is an exemplary top-level flowchart of an alternative embodiment of a method of determining an operating mode, wherein a sensor of a mobile platform has been selected for each of the four operating modes of FIG. FIG. 14 is an exemplary flow chart of an alternative embodiment of the method of FIG. 13, illustrating a method of selecting a sensor based on each of the operating modes. FIG. 4 is an exemplary top-level diagram illustrating yet another alternative embodiment of the mobile platform flight motion system of FIG. 2, wherein the processor selects a mobile platform sensor for each of the four modes of operation. I have.

  It is noted that the accompanying drawings are not to scale, and that elements of similar structure or function are generally denoted by the same reference numerals throughout the drawings for purposes of illustration. It should also be noted that the accompanying drawings are only intended to facilitate a description of the preferred embodiment. The accompanying drawings do not illustrate every aspect of the described embodiments, and do not limit the scope of the disclosure.

  Generally, navigation of an unmanned aerial vehicle ("UAV") that uses stereo vision technology to operate a UAV is performed. However, the accuracy of stereo vision technology is limited and can vary with height.

  Stereo vision systems typically perform navigation by considering the overlap area of the scene viewed by each of the two lenses. The baseline length between the lenses of a conventional stereo vision system is typically between 4 and 20 centimeters. However, the range of applicable heights of the stereo vision technique is limited by the baseline length. In other words, the range of measurable height is limited by the baseline length.

  The UAV is operated depending on the overlap area of the scene. For example, at low altitudes, the distance between the lens of the binocular imaging system and the ground is too short to form a usable overlap area between each scene viewed by the lens of the binocular imaging device. On the other hand, at very high altitudes, the distance between the lens of the stereo vision system and the ground is too long. In such cases, a long distance creates a short baseline between the two lenses of the stereo vision system, which results in inaccurate calculation results.

  Currently available stereo visual navigation systems are limited by baseline length. Thus, a mobile system and method that can meet the requirements for operating a UAV by switching between operating modes at different heights based on the height and differences of the mobile system may be desirable. In addition, the mobile system and method can provide a basis for accurately measuring depth for systems such as UAV systems and other mobile systems. This result can be achieved according to one embodiment disclosed in FIG.

  Referring first to FIG. 1, an embodiment of a method 100 for selecting an operating mode of a mobile platform 200 (shown in FIG. 2) within a wide height range is shown. Referring to FIG. 1, at 120, height 121 and / or difference 122 may be detected as a basis for determining the mode of operation of mobile platform 200. Height 121 and / or difference 122 may represent a height grade. In a preferred embodiment, such detection may be performed in a real-time manner and / or in a time-delayed manner. The height 121 is associated with the height of the flight of the mobile platform 200, that is, altitude information such as altitude. The difference 122 indicates a difference between the positions of the images of the object drawn in the two images or frames. The difference 122 may be determined statistically statically and / or may be determined dynamically in the manner shown and described below with reference to FIGS. . The mobile platform 200 can have any form of aircraft. Height 121 can be determined in the manner shown and described below with reference to FIG.

  At 130, the mobile platform 200 can select some default operating modes 131 by using the obtained height grade, ie, height 121 and / or difference 122; Alternatively, switching can be performed during these modes of operation. The operating mode 131 can have operations with various devices associated with the mobile platform 200, and these devices can be included at any time. Such a device is shown and described below with reference to FIGS. The selection or switching of the operation mode 131 will be illustrated and described in detail with reference to FIG.

  For illustrative purposes only, the height 121 and / or the difference 122 are shown and described as being used as a basis for selecting or switching between operating modes, however, the selection of operating modes or the differences between those operating modes may be used. Other suitable condition data may be used as a switching criterion.

  FIG. 2 shows a mobile platform 200 having devices 251 to 254 for detecting states to implement the method 100 by switching the operation mode 131. In FIG. 2, the devices 251 to 254 may include at least a barometer 251, one or more ultrasonic detectors 252, a GPS 253, and a binocular imaging device 254. Of the devices 251 to 254, the barometer 251, the ultrasonic detector 252, and / or the GPS 253 can detect the height (or altitude) of the mobile platform 200, and the binocular imaging device 254 It can be a source of information for the differences 122. In FIG. 2, the ultrasonic detector 252 can detect a distance to an object 288 which may be a ground. Accordingly, the distance between the ultrasonic detector 252 and the object 288 can represent the ground level and the vertical height 121 of the mobile platform 200.

  In FIG. 2, the barometer 251 can be installed on the main body 260 of the mobile platform 200. The ultrasonic detector 252 can be located around a lower portion of the body 260. The GPS 253 can be installed in the main body 260 of the mobile platform 200. The binocular imaging device 254 can be located below the main body 260. However, in the present disclosure, the barometer 251 and the GPS 253 may be disposed on any part of the mobile platform 200, such as inside the main body 260, below the main body 260, or on any side of the main body 260. it can. The ultrasonic detector 252 can be disposed at any position around the main body 260. The binocular imaging device 254 can be located at any suitable location on the lower portion of the body 260.

  Although illustrated and described for purposes of illustration only as using devices 251-254, any other suitable device may be used to detect conditions for determining switching between operating modes 131. Can also be used. The mobile platform 200 can include any conventional mobile platform that can have an elevation, and FIG. 2 illustrates the unmanned aerial vehicle UAV 250 by way of example only and not by way of limitation. Shown as having.

  FIG. 3 illustrates another alternative embodiment of the method 100. In FIG. 3, the method 100 may include, at 310, categorizing the operating modes. The operating modes can be categorized, for example, based on different height grades, ie, based on height ranges and / or difference values. As mentioned above, currently available single mode operation cannot meet varying height requirements, for example, binocular systems using stereo vision technology require high altitude / height and very low altitude / Inability to meet height requirements. On the other hand, the other available modes of operation of the mobile platform 200 may be more suitable for high altitude / height or low altitude / height.

  At different altitudes or altitudes, the mobile platform 200 can be operated by using various modes of operation. In order to operate the mobile platform 200 at all heights, the operating modes can be classified into several height grades. In the following, further details of the classification will be shown and described with reference to FIG.

  It should be noted that the operation modes are described as being classified according to different height grades for the purpose of illustration only. Other appropriate information.

  FIG. 4 illustrates one embodiment of classifying the modes of operation into four modes for the method 100. In FIG. 4, a first height mode 411, a second height mode 412, a third height mode 413, and a fourth height mode 414 can be provided according to different height grades. The fourth height mode 414 is designed to be used in a fourth height range of, for example, 20 meters (20 m) or more. In the fourth height mode 414, the mobile platform 200 (shown in FIG. 2) can operate with the high altitude monocular mode. In the high altitude monocular mode, the depth of the object can be determined by, for example, a combination of the barometer 251, the GPS 253, and / or the visual detection in order to determine the height 121. The operation mode of the mobile platform 200 can be determined based on the height information.

  The third height mode 413 is designed to be used in a third height range of, for example, 3.5 meters (3.5 m) to 20 meters (20 m). Within the third height range, a binocular device with a normal resolution of 320 × 240 cannot meet the requirements of depth detection and selection of the operating mode of the mobile platform 200. To address this issue, in the third height mode 413, the height 121 of the mobile platform 200 can be determined and the operation mode can be selected by using the improved resolution binocular mode. In the binocular mode with improved resolution, the resolution may be at least 640 × 480.

  The second height mode 412 is designed to be used in a second height range of, for example, 50 centimeters (50 cm) to 3.5 meters (3.5 m). Within the second height range, the second height mode 412 may use a normal resolution binocular mode, which may use a resolution of 320 × 240.

  The first height mode 411 is designed to be used in a first height range of, for example, 10 centimeters (10 cm) to 50 centimeters (50 cm). Within the first height range, due to the short distance between the lens and the target object, there may not be enough overlap between the images acquired by the two lenses for the binocular system. Accordingly, in the first height mode 411, it is possible to use the ultra-low altitude monocular mode, in which case the other distance sensors select the operation mode of the mobile platform 200, so that the optical center of the lens may be used. The distance between the object and the ground level, ie the depth of the object, can be detected.

  Although illustrated and described as classifying the operating modes into four categories for illustrative purposes only, any suitable number of categories may be utilized in the present disclosure. In addition to the height to ground 121, the present disclosure may use other conditions when classifying operating modes and / or switching between these modes. Such a condition may have a difference 122.

  FIG. 5 illustrates another embodiment of the method 100, where the height 121 and / or the difference 122 can be used to determine the mode of operation 131. Height 121 represents the vertical distance between mobile platform 200 (shown in FIG. 2) and the ground level. Height 121 can be obtained via barometer 251, ultrasonic detector 252, and / or GPS 253 in the manner shown and described below with reference to FIG. As shown and described with reference to FIG. 1, the difference 122 represents a difference in image position of the object within the two images or frames. The difference 122 can be determined in the manner shown and described below with reference to FIGS.

  At 230, the height 121 and difference 122 information can be combined. The combined information can be used at 240 to determine the operating mode 131. Although shown and described as using the combined information to determine the operating mode 131, the height 121 and / or the difference 122 may be used separately in determining the operating mode 131. it can.

  FIG. 6 illustrates an embodiment of a mobile platform 200 that illustrates, at 120, obtaining a height 121 with the method 100. In FIG. 6, the barometer 251 (collectively shown in FIG. 2), the ultrasonic detector 252, and / or the GPS 253 can be used to obtain the height 121 of the mobile platform 200. Barometer 251 may be any type of commercially available barometer or pressure altimeter that determines height 121 based on atmospheric pressure measurements. Such a barometer 251 may comprise a water-based barometer, mercury barometer, vacuum pump oil barometer, aneroid barometer, barograph, or MEMS barometer. Barometer 251 can also include other types of barometers, such as a storm barometer.

  Additionally and / or alternatively, ultrasonic detector 252 emits ultrasonic waves and receives ultrasonic waves reflected from object 288, thereby detecting objects in the environment (shown in FIG. 2). 288 distance 121 can also be detected. Distance 121 may be a distance to a ground level, in which case object 288 is ground. The ground level 880 may be an actual ground, a water level, or a ground having any structure. Such an ultrasonic detector 252 can include any commercially available ultrasonic sensor. For illustrative purposes only, and shown and described as using a single ultrasonic detector 252 to detect object 288 in one direction, multiple objects may be detected in multiple directions to detect object 288 in multiple directions. Of the ultrasonic detector 252 can be provided.

  GPS 253 is a space satellite navigation system that can provide four or more GPS satellites with position, height, and / or time information on or near the earth in the visible region. The GPS 252 can have any commercially available GPS device. The location can be provided by GPS 253 as latitude and longitude. The height may be a height in meters or feet to ground level.

  The height 121 applicable in the present disclosure may be any vertical distance within a range of 25 centimeters (25 cm) to 100 meters (100 m) to ground level. As shown and described with reference to FIG. 4, the height 121 may be categorized in the present disclosure into a first height, a second height, a third height, and a fourth height. it can. For purposes of illustration only, and shown and described as using barometer 251, ultrasonic detector 252, or GPS 253 to detect height 121, but to detect height 121 of mobile platform 200 , Other suitable detection devices may be used.

  FIG. 7 illustrates another embodiment of the mobile platform 200 of FIG. In FIG. 7, a processor 910 can be connected with the barometer 251, the ultrasonic detector 252, the GPS 253, and the binocular imaging device 254 to collect state data, and to implement the method 100. In addition, the operation mode is controlled. Processor 910 may be a processor associated with a mobile platform to control mobile platform 300. Binocular imaging device 254 may be an imaging device having a binocular lens that captures two images of an object simultaneously.

  In a preferred embodiment of the mobile platform 200, a processor 910 may be provided to obtain and / or process information obtained from the barometer 251, the ultrasonic detector 252, the GPS 253, and / or the binocular device 254. Such information includes a height 121 (shown collectively in FIG. 1) and a difference 122. The processor 910 can determine the selection of the operation mode 131 based on the information. Further details regarding the determination of the operation mode 131 will be shown and described below with reference to FIG.

  Processor 910 may include any commercially available processing chip, or may be any custom designed processing chip for device 900 to select a mode of operation of mobile platform 200. Additionally and / or alternatively, processor 910 may include one or more general-purpose microprocessors (eg, single or multi-core processors), application-specific integrated circuits, application-specific instruction set processors, data processing units, It may include an arithmetic processing unit, a digital signal processing unit, a coprocessor, a network processing unit, an audio processing unit, an encryption processing unit, and the like. Processor 910 may be configured to perform any of the methods described herein, including, but not limited to, various operations associated with selecting an operating mode. In some embodiments, processor 910 may include dedicated hardware for handling certain operations related to selecting an operation mode.

  FIG. 8 illustrates another embodiment for the method 100, in which four operating modes can be selected (or switched) based on the height 121 and / or the difference 122. Upon takeoff, the mobile platform 200 (shown in FIG. 2) can operate at a first height. The device that selects the operation mode of the mobile platform 200 can operate according to the corresponding operation mode, that is, the first height mode 411. As shown and described with reference to FIG. 4, the first height mode 411 is an ultra-low altitude monocular in combination with a distance sensor (not shown) to select an operating mode of the mobile platform 200. Mode.

  When operating in the first height mode 411 and, at 930, when two conditions are met, the mobile platform 200 can switch to the second height mode 412. The two conditions may include that the difference 122 of the binocular imaging device 254 is less than or equal to the first difference threshold Td1, and that the height 121 of the mobile platform 200 is greater than the first height threshold Th1. In an alternative embodiment, the operation mode can be switched from the first height mode 411 to the second height mode 412 only when the height 121 of the mobile platform 200 exceeds the first height threshold Th1.

  The first difference threshold Td1 can be selected from a first difference range between 62 centimeters (62 cm) and 82 centimeters (82 cm), and is preferably selected to be 72 centimeters (72 cm). can do. The first height threshold Th1 can be selected from values within a first height range of 20 centimeters (20 cm) to 80 centimeters (80 cm), and preferably is 50 centimeters (50 cm). Can be selected to be

  When the difference 122 of the binocular imaging devices of the stereo vision system is less than or equal to the third difference threshold Td3, the mobile platform 200 can switch to the third height mode 413. As shown and described with reference to FIG. 4, the third height mode 413 may include a high resolution binocular mode.

  The third difference threshold value Td3 can be selected from a third difference range of 5 cm (5 cm) to 15 cm (15 cm), and is preferably selected to be 10 cm (10 cm). can do.

  At 934, the operating mode can be switched to the fourth height mode 414 when the two conditions are met. The two conditions may include that the difference 122 of the binocular imaging device of the stereo vision system is less than or equal to the fifth difference threshold Td5 and that the height of the mobile platform 200 is greater than the third height threshold Th3. . The fourth height mode 414 may include a high altitude monocular mode of operation, which may include a barometer, GPS, and / or vision detector, as shown and described above with reference to FIG. Can be used. In an alternative embodiment, the operation mode can be switched from the third height mode 413 to the fourth height mode 414 only when the height 121 of the mobile platform 200 exceeds the third height threshold Th3.

  The fifth difference threshold Td5 can be selected from a value within a fifth difference range of 1 cm (1 cm) to 3 cm (3 cm), and is preferably 2 cm (2 cm). Can be selected as The third height threshold Th3 can be selected from values within a third height range of 15 meters (15 m) to 25 meters (25 m), and preferably is 20 meters (20 m). Can be selected.

  When operating in the fourth height mode 414, the mobile platform 200 can switch to another operating mode when any of the specific conditions 931 and 933 are met. In 931, for example, when the difference 122 is equal to or larger than the sixth difference threshold Td6 and the height 121 of the mobile platform 200 is equal to or smaller than the fourth threshold Th4, the mobile platform 200 enters the third height mode 413. Can be switched. In an alternative embodiment, the mobile platform 200 can be switched to the third height mode 413 only when the height 121 of the mobile platform 200 falls below the fourth threshold Th4.

  At 933, when the difference 122 is greater than or equal to the fourth difference threshold Td4, the mobile platform 200 can switch to the second height mode 412.

  The sixth difference threshold Td6 can be selected from a value within a fifth difference range of 1.5 centimeters (1.5 cm) to 4 centimeters (4 cm), and preferably is 2.5 centimeters. Meters (2.5 cm). The fourth height threshold Th4 can be selected from values within a fourth height range of 15 meters (15 m) to 22 meters (22 m), and preferably is 18 meters (18 m). Can be selected. The fourth difference threshold Td4 can be selected from a value within a fourth difference range of 9 centimeters (9 cm) to 15 centimeters (15 cm), and is preferably 12 centimeters (12 cm). Can be selected as

  When operating in the third height mode 413, if the difference 122 is greater than or equal to the fourth difference threshold Td4, the mobile platform 200 can switch to the second height mode 412.

  When operating in the second height mode 412, the mobile platform 200 can switch to the first height mode 411 when the condition is satisfied at 935. At 935, when the difference 122 is greater than or equal to the second difference threshold Td2 and the height 121 of the mobile platform 200 is less than or equal to the second height threshold Th2, the mobile platform 200 enters the first height mode 411. Can be switched. In an alternative embodiment, the mobile platform 200 can switch to the first height mode 411 only when the height 121 of the mobile platform 200 falls below the second threshold Th2.

  The second difference threshold Td2 can be selected from values within a second difference range of 60 centimeters (60 cm) to 80 centimeters (80 cm), and is preferably 70 centimeters (70 cm). Can be chosen to be The second height threshold Th2 can be selected from a value within a second height range of 25 centimeters (25 cm) to 65 centimeters (65 cm), and preferably is 45 centimeters (45 cm). Can be selected to be

  The second difference threshold Td2 may be higher than the first difference threshold Td1. One or both of the first and second difference thresholds Td1, Td2 may be greater than one or both of the third and fourth difference thresholds Td3, Td4. The first height threshold Th1 may be higher than the second height threshold Th2. One or both of the first and second height thresholds Th1, Th2 may be greater than one or both of the third and fourth height thresholds Th3, Th4. The third difference threshold Td3 may be higher than the fourth difference threshold Td4. One or both of the third and fourth difference thresholds Td3, Td4 may be greater than one or both of the fifth and sixth difference thresholds Td5, Td6. The sixth difference threshold Td6 may be higher than the fifth difference threshold Td5.

式(1)

式(2)

式(3)
ここで、ｃ_ｘ及びｃ_ｙは、レンズ５１０ａ及び５１０ｂの個々の中心Ｏ_ｌ、Ｏ_ｒ座標を表しており、ｘ_ｉ及びｙ_ｉは、それぞれ、画像５２０ａ及び５２０ｂのそれぞれの内部における対象の物体５９８の座標を表しており、Ｔは、ベースラインであり（換言すれば、レンズ５１０ａ及び５１０ｂの中心座標の間の距離であり）、ｆは、レンズ５１０ａ及び５１０ｂの修正された焦点距離であり、ｉは、複数の対象物体５９８上における且つ／又は対象物体５９８の（図１０に示されている）複数の特徴点３５５におけるインデックスであり、且つ、ｄは、ここでは、次式として表される画像５２０ａ及び５２０ｂの間の双眼差異である。

式(4) FIG. 9 shows an exemplary method 700 for identifying binocular differences between two stereoscopic images 520a, 520b acquired by two lenses 510a, 510b from an object of interest 598. Using the method of triangulation, the difference d between the images 520a and 520b can be determined. Specifically, the position of the target object 598 having the index i represented by the coordinates (X _i , Y _i , Z _i ) can be given by the following equation.

Equation (1)

Equation (2)

Equation (3)
Here, _{c x} and _{c y} are lenses 510a and 510b of each of the center _O l, represents a _{O r} coordinates, _{x i} and _{y i,} respectively, the object of interest in each of the internal image 520a and 520b 598, where T is the baseline (in other words, the distance between the center coordinates of lenses 510a and 510b) and f is the modified focal length of lenses 510a and 510b. , I are indices on the plurality of target objects 598 and / or at a plurality of feature points 355 (shown in FIG. 10) of the target object 598, and d is represented by the following equation: Binocular difference between the images 520a and 520b.

Equation (4)

Based on the description of FIG. 9, the difference d, by a point _{X r} on the second image 520b is _{X l} matching of the first image 520a, it is possible to calculate, in this case, _{X l} is , Known elements. In Figure 9, the detection of the matching point X _r on the second image 520b is an embodiment for illustration is only for illustrative purposes, are shown. In Figure 9, ^{I L} represents the first image 520a of the same target object 598, and, ^{I R} represents the second image 520b. Point _x ^{i l} on the first image 520a may be a known, and, matching point _x ^{i r} on the second image 520b is "the most similar" to the point ^x _{l i} on the first image 520a It can be defined as a point, which can be represented by the following equation:
^{_{d = argmin d | I L (}} x l) -I R (x l + d) | Equation (5)
Here, d is two lenses 510a, represents the difference 510b, ^{I L} represents the first image 520a of the same target object 598, ^{I R} represents a second image 520b and, _{x l} is the point _x ^{i l} of the first image 520a.

  When specifying the matching accuracy and the visual range, there are possible matching errors, and the difference d cannot be less than or greater than a certain predetermined value. In a preferred embodiment, the difference d is greater than 5 pixels and less than one-fifth of the width of the second image 520b, which may be of the same size as the first image 520a. As an illustrative example, assuming that f = 480, T = 0.15 m, and the image resolution is 320 × 240 pixels, subtract the effective visual range of 1.5 m to 15.4 m. be able to.

  FIG. 10 illustrates an embodiment of a method 700 for matching points in the second image 520b with corresponding points 355 in the first image 520a. In FIG. 10, a 3 × 3 pixel block having a comparison point at the center is obtained from each of the images 520a and 520b. When the first and second images 520a and 520b are color images, the color component values can be compared for each pixel of the 3 × 3 pixel block. Conversely, when the images 520a and 520b are black and white images, the gray scale values of each pixel can be compared. Based on Equation 5, the point with the minimum sum of the differences of all nine pixel values can be selected as the matching point. This process may be performed iteratively for all selected feature points on the first image 520a.

  Alternatively, a method using a binary stable independent basic feature ("BRIEF") descriptor to match points in the second image 520b with corresponding points 355 in the first image 520a may be used. . In an embodiment, a first binary string representing the first region around the selected feature point of the first image 520a may be constructed by comparing the intensity of each point pair in the region. The first binary string may be a first BRIFF descriptor of a selected feature point of the first image 520a.

  Similarly, by comparing the intensity of each point pair in the second region, a second binary string representing the second region around point 355 of the second image 520b can be constructed. The second binary string may be a second BRIFF descriptor.

  By comparing the Hamming distance between the first and second BRIEF descriptors, the similarity between the selected feature point of the first image 520a and the point 355 of the second image 520b can be calculated. . When the Hamming distance between the first and second BRIFF descriptors is less than the first Hamming threshold, determine the point 355 of the second image 520b as matching the selected feature point of the first image 520a. can do.

  Referring now to FIG. 11, an embodiment of a method 700 for obtaining a difference d via a feature point 355 of an object of interest 598 is shown. At 922, a plurality of feature points 355 on the object of interest 598 can be selected. Feature point 355 may be selected using one or more of a variety of different methods. In one exemplary embodiment, the feature points 355 may be identified as a predefined shape of the object 598 of interest. In another embodiment, feature point 355 may be recognized as one or more portions of target object 598 having a particular color or intensity. In another embodiment, the feature points 355 may be selected as random parts of the object 598 of interest. In another embodiment, the feature points 355 are regular on the object of interest 598, for example, every pixel, every skipped pixel, every two skipped pixels, every three skipped pixels, etc. Can be selected at intervals spaced apart from each other. The feature points 355 can have varying shapes and sizes as needed. In some embodiments, a feature point 355 may be selected by using a combination of the above methods.

  At 924, the selected feature points 355 can be matched from the first image 520a to the second image 520b. In the preferred embodiment, matching of the feature points 355 consists of two procedures, as shown in FIG. At 924A in FIG. 12, the feature point 355 of the first image can be selected. Starting from the calculated point and along a line parallel to the centered line of the lenses 510a, 510b, the matching points can be scanned. The matching start point can be calculated based on the coordinates of the point on the first image 520a, the direction of the baseline, and / or the length thereof. Although preferably limited to only one direction along the selected line, scanning can be performed in any of one or more predefined directions.

  At 924B, while scanning for each point, the similarity between the two points is calculated according to the scheme previously illustrated and described in detail herein with reference to FIG. The point 355 of the second image 520b having the minimum sum of the difference between the feature point 355 of the first image 520a and the feature point 355 can be selected as the matching point corresponding to the feature point 355.

  Referring again to FIG. 11, at 926, a feature difference d between the respective feature points 355 of the two images 520a and 520b can be found. The difference d can be determined by using any of a variety of methods. In one embodiment, the difference d can be found based on the average value of the difference d for each of the feature points 355. Exemplary types of average values may include, but are not limited to, numerical average, geometric average, median, and / or mode. In another embodiment, the difference d may be found by selecting one or more of the feature points 355 and obtaining the difference d based on the selected feature point 355.

  FIG. 13 illustrates one embodiment of a method 300 for determining an operating mode of the mobile platform 200 (shown in FIG. 15), in which case based on the height grade (shown in FIG. 4). A sensor 360 (shown in FIG. 15) can be selected for each of the four height modes 411-414. At 320 in FIG. 13, the height grade of the mobile platform 200 can be detected. The height grade of the mobile platform 200 can be detected, for example, in the manner shown and described above with reference to FIGS. 1, 5, and 6. At 330, one or more sensors 360 can be selected according to the detected height grade, and these sensors 360 can be predefined. The selection of sensor 360 will be illustrated and described in further detail below with reference to FIG. At 340, certain information can be obtained to operate the mobile platform. For example, the information can include the height of the mobile platform, distance to object 288 (shown in FIG. 2), displacement, velocity, and the like.

  FIG. 14 illustrates an alternative embodiment of the method 300 and illustrates the selection of sensors for each of the four height modes of FIG. In FIG. 14, at 321 the first mobile platform 200 (shown in FIG. 4) detects that the mobile platform 200 (shown in FIG. 15) is within a first height grade. The height mode 411 can be selected. When the mobile platform 200 selects the first height mode 411 at 321, one image sensor 363 can be selected at 322. In an alternative embodiment, at 321, when the first height mode 411 is selected, one image sensor 363 and one distance sensor 361 can be selected. For example, a distance sensor 361 can be utilized to determine the distance between the mobile platform 200 (shown in FIG. 2) and the object of interest 288. For illustrative purposes only, one distance sensor 361 and one image sensor 363 are shown and described as being selected, but instead, for the first height mode 411, any other suitable type of sensor. 360 can also be selected.

  The distance sensor 361 described herein can include, but is not limited to, an ultrasonic detector and / or a laser detection device that detects distance.

  In FIG. 14, the mobile platform 200 can detect the height of the second height grade. Upon detecting the second height grade, the mobile platform 200 may select a second height mode 412 (shown in FIG. 4) at 323. At 323, when the mobile 200 has selected the second height mode 412, at 324, at least two image sensors 363 (shown in FIG. 15) may be selected. Each of the image sensors 363 selected for the second height mode 412 may have a first resolution.

  The mobile platform 200 can detect the operating height of the third height grade. Upon detecting the second height grade, mobile platform 200 may select 325 a third height mode 413 (shown in FIG. 4). At 325, when the mobile platform 200 selects the third height mode 413, at 326, at least two image sensors 363 can be selected. Each of the image sensors 363 selected for the third height mode 413 can have a second resolution. The second resolution may be higher than the first resolution used in the second height mode 412.

  The mobile platform 200 can detect the operating height of the fourth height grade. Upon detecting the fourth height grade, the mobile platform 200 may select a third height mode 414 (shown in FIG. 4) at 327. When the mobile platform 200 selects the fourth height mode 414 at 327, one image sensor 363 and one height sensor 362 can be selected at 328. Height sensor 362 may include, but is not limited to, a barometer 251 (shown collectively in FIG. 6) and / or a global positioning system ("GPS") 253.

  FIG. 15 illustrates another alternative embodiment of the mobile platform flight motion system 400 of FIG. 2, in which the processor 910 (shown in FIG. 7) (see FIGS. 13 and 14). One or more sensors 360 may be selected according to the method 300 (shown). Referring to FIG. 15, a distance sensor 361, a height sensor 362, and one or more image sensors 363 can also be associated with and communicate with the processor 910. As described above with reference to FIG. 14, the distance sensor 361 can include, but is not limited to, an ultrasonic detector 252 (shown in FIG. 7) and / or a laser detection device. Height sensor 362 can include, but is not limited to, a barometer 251 (collectively shown in FIG. 7) and / or a global positioning system (“GPS”) 253. Image sensor 363 may include, but is not limited to, binocular imaging device 254 (shown in FIG. 2).

  In a preferred embodiment of the flight motion system 400, the processor 910 can obtain and process measurements obtained from the sensor 360. Such measurements may include a distance to the object 288 (shown in FIG. 2), a height 121 (collectively shown in FIG. 1), and / or a difference 122, It is not limited to these. Based on the measurements, the processor 910 can determine a selectable mode of operation and one or more predefined sensors 360 that can be included for the selected mode of operation.

  Various modifications and alternatives to the described embodiments are possible, and specific examples thereof are shown in the drawings for purposes of illustration and described in detail herein. I have. However, it is to be understood that the described embodiments are not limited to the particular forms or methods disclosed, but, rather, that the present disclosure includes all modifications, equivalents, and alternatives. I want to.

In another exemplary embodiment of the disclosed device, the first height threshold is greater than the second height threshold,
Any one of the third and fourth height thresholds is greater than the first and second height thresholds, and
The third height threshold is higher than the fourth height threshold.

It is a top-level flow chart of an exemplary shaped condition of a method for selecting the operating mode within a wide range of heights. Ru GENERAL schematic der shows the mobile platform, which was charged Bei. Ru flowchart der for illustrative alternative embodiment of the method of FIG. 4 is an exemplary block diagram of an alternative embodiment of the method of FIG. 3, where the method classifies the four modes of operation based on height. FIG. 2 is an exemplary block diagram of another alternative embodiment of the method of FIG. 1, wherein height and difference information is used to determine an operating mode of operation. FIG. 3 is an exemplary block diagram of an alternative embodiment of the mobile platform of FIG. 2, where the mobile platform uses a barometer, an ultrasonic detector, and / or GPS to obtain height information. . FIG. 3 is an exemplary top-level diagram illustrating another alternative embodiment of the mobile platform of FIG. 2, where a processor is collecting state data and controlling an operating mode. FIG. 4 is an exemplary flowchart of another alternative embodiment of the method of FIG. 1, illustrating switching conditions between four different modes of operation. FIG. 4 is an exemplary detail drawing illustrating one embodiment of a stereoscopic imaging method, in which the differences of the method of FIG. 1 have been determined. FIG. 10 is an exemplary diagram of an alternative embodiment of the method of FIG. 9, illustrating an exemplary method of matching two images with an overlap area. FIG. 10 is an exemplary diagram illustrating the alternative embodiment of FIG. 9, wherein the first image is matched with a second image having a plurality of feature points. FIG. 12 is an exemplary diagram illustrating an alternative embodiment of the method of FIG. 11, wherein each feature point is matched by calculating similarity. 5 is an exemplary top-level flowchart of an alternative embodiment of a method of determining an operating mode, wherein a sensor of a mobile platform has been selected for each of the four operating modes of FIG. FIG. 14 is an exemplary flow chart of an alternative embodiment of the method of FIG. 13, illustrating a method of selecting a sensor based on each of the operating modes. FIG. 4 is an exemplary top-level diagram illustrating yet another alternative embodiment of the mobile platform flight motion system of FIG. 2, wherein the processor selects a mobile platform sensor for each of the four modes of operation. I have.

FIG. 7 illustrates another embodiment of the mobile platform 200 of FIG. In FIG. 7, a processor 910 can be connected with the barometer 251, the ultrasonic detector 252, the GPS 253, and the binocular imaging device 254 to collect state data, and to implement the method 100. In addition, the operation mode is controlled. Processor 910 may be a processor associated with a mobile platform to control mobile platform 200 . The binocular imaging device 254 may be an imaging device having a binocular lens that captures two images of an object simultaneously.

When operating in the fourth height mode 414, the mobile platform 200 can switch to another operating mode when any of the specific conditions 931 and 933 are met. In 931, for example, when the difference 122 is equal to or greater than the sixth difference threshold Td6 and the height 121 of the mobile platform 200 is equal to or less than the fourth height threshold Th4, the mobile platform 200 sets the third height mode. 413 can be switched. In an alternative embodiment, the mobile platform 200 can be switched to the third height mode 413 only when the height 121 of the mobile platform 200 falls below the fourth height threshold Th4.

Claims (90)

A method for selecting an operation mode of a mobile platform,
Detecting a height grade of the mobile platform;
Selecting an operating mode of the mobile platform according to the result of the detection.   Detecting the height grade comprises determining a height of the mobile platform and a difference between first and second images of the remote object from at least one of the mobile platforms. Item 1. The method according to Item 1.   The method of claim 2, wherein the determining comprises obtaining the height via at least one of a barometer, an ultrasonic detector, and a global positioning system (“GPS”).   4. The method of claim 2, wherein the determining comprises obtaining the difference between the first and second images of the object captured by a binocular imaging system associated with the mobile platform. 5. Method.   The method of claim 4, further comprising classifying the operating mode based on the height and / or difference values.   The method of claim 5, further comprising activating the mobile platform to operate in a first height mode.   The method of claim 6, wherein the first height mode comprises a very low altitude monocular mode.   The method of claim 7, further comprising providing a distance sensor for the moving platform to support the ultra-low altitude monocular mode.   The step of selecting the operation mode includes the step of switching the operation mode based on the height grade, and the height grade is one of the determined height and the determined difference. The method according to claim 7, wherein the determination is based on at least one.   7. The method of claim 6, wherein selecting the operating mode comprises switching the mobile platform to a second height mode when the height exceeds a first height threshold.   The method of claim 6, wherein selecting the operating mode comprises switching the mobile platform to a second height mode when the difference is less than a first difference threshold.   Selecting the operating mode comprises switching the mobile platform to a second height mode when the height is above a first height threshold and the difference is below the first difference threshold. The method of claim 6, wherein:   13. The method according to any one of claims 9 to 12, wherein switching the mobile platform to a second height mode comprises selecting a stereo vision mode having a first resolution.   14. The method according to any one of claims 6 to 13, wherein selecting the operating mode comprises switching the mobile platform to a third height mode when the difference is less than or equal to a third difference threshold. the method of.   15. The method of claim 14, wherein switching the mobile platform to a third height mode comprises switching from a binocular imaging device to a stereo vision mode with improved resolution.   16. The method of claim 14 or 15, wherein selecting the operating mode comprises switching the mobile platform to a fourth height mode when the height exceeds a third height threshold.   Selecting the operating mode comprises switching the mobile platform to a fourth height mode when the height is above a third height threshold and the difference is below a fifth difference threshold. A method according to claim 14 or claim 15.   The step of switching the mobile platform to a fourth height mode comprises: moving the binocular imaging device to a height in a combination of a barometer, a GPS, and a visual measurement of a vertical distance between at least one of the mobile platforms and a ground level. 18. The method of claim 17, comprising switching to an advanced monocular mode.   19. The method of claim 16, wherein selecting the operating mode comprises switching the mobile platform to the third height mode when the height is less than a fourth height threshold. The method described in the section.   Selecting the operating mode comprises switching the mobile platform to the third height mode when the height is less than a fourth height threshold and the difference is above a sixth difference threshold. The method according to any one of claims 16 to 18.   21. The method according to any one of claims 16 to 20, wherein selecting the operating mode comprises switching the mobile platform to the second height mode when the difference exceeds a fourth difference threshold. the method of.   22. The method of claim 21, wherein selecting the operating mode comprises switching the mobile platform to the first height mode when the height is less than a second height threshold.   Selecting the operating mode comprises switching the mobile platform to the first height mode when the height is less than a second height threshold and the difference exceeds a second difference threshold. 22. The method of claim 21. The second difference threshold is greater than the first difference threshold,
At least one of the first and second difference thresholds is greater than at least one of the third and fourth difference thresholds;
The third difference threshold is higher than the fourth difference threshold,
At least one of the third and fourth difference thresholds is greater than at least one of the fifth and sixth difference thresholds, and
24. The method of claim 23, wherein the sixth difference threshold is above the fifth threshold. The first height threshold is greater than the second height threshold,
At least one of the third and fourth height thresholds is greater than at least one of the first and second height thresholds, and
The method according to claim 23 or 24, wherein the third height threshold is above the fourth threshold. The step of determining the difference between the first and second images includes:
Selecting a plurality of feature points from the first image;
Matching the feature points of the first image with points of the second image.   The method of claim 26, wherein the feature points comprise pixels of the first image or the second image. The step of matching the plurality of feature points includes:
Scanning the second image to identify points in the second image that match selected feature points of the first image;
Calculating a similarity between the selected feature point of the first image and the point of the second image. The step of calculating the similarity includes:
Constructing a first binary string representing a first region around the selected feature point by comparing the intensities of each point pair in the region to form a first binary stable independent basic feature ("BRIEF"). Generating a descriptor;
Constructing a second binary string representing a second region around the point of the second image by comparing the intensity of each point pair of the second region, and generating a second BRIFF descriptor;
Determining that the point of the second image matches the selected feature point when a Hamming distance between the first and second BRIEF descriptors is less than a first Hamming threshold. 29. The method of claim 28, comprising:   The step of calculating the similarity comprises comparing the selected feature point of the first image with a 3 × 3 pixel area centered about the point on the second image. 28. The method according to 28.   31. The method of claim 30, wherein the comparing step comprises comparing a sum of differences for each color component of each pixel of the color image or a sum of differences in grayscale values of each pixel of the black and white image. .   The method of claim 31, wherein determining the difference comprises obtaining the difference based on an average of the differences of the feature points.   33. The method of claim 30, wherein determining the difference comprises selecting one or more feature points and acquiring the difference based on the difference between the selected feature points. The method described in the section.   A system for selecting a mode of operation of a mobile platform configured to perform the detection process according to any one of claims 1 to 33.   A computer program product having instructions for selecting a mode of operation of a mobile platform configured to perform the detection process according to any one of claims 1 to 33. A device for selecting an operation mode of a mobile platform,
A binocular imaging device associated with the mobile platform;
A processor,
Detecting the height grade of the mobile platform; and
A processor configured to select an operation mode of the mobile platform according to a result of the detection.   39. The apparatus of claim 36, wherein the processor is configured to determine a difference between the height of the mobile platform and first and second images of a remote object from the perspective of the mobile platform.   The apparatus of claim 37, further comprising a barometer associated with the mobile platform to obtain the height of the mobile platform.   39. The apparatus according to claim 37 or claim 38, further comprising an ultrasound detector associated with the mobile platform to obtain the height of the mobile platform.   40. The apparatus of claim 39, further comprising a GPS associated with the mobile platform to obtain the height and / or location of the mobile platform.   41. The processor of claim 39 or 40, wherein the processor is configured to obtain the difference between the first and second images of the object captured by a binocular imaging system associated with the mobile platform. Equipment.   42. The apparatus of claim 41, wherein the mode of operation of the mobile platform is categorized based on the height value and / or the difference value.   43. The apparatus of claim 42, wherein the processor is configured to initialize the mobile platform to operate in a first height mode.   44. The apparatus of claim 43, wherein the first height mode comprises a very low altitude monocular mode.   The apparatus of claim 44, wherein a distance sensor of the moving platform is provided to support the ultra-low altitude monocular mode.   The processor is configured to switch the operating mode based on the height grade, and wherein the height grade is at least one of the determined height and the determined difference. The apparatus of claim 44 or claim 45, wherein the apparatus is determined based on:   47. The apparatus of claim 46, wherein the processor is configured to switch to a second height mode when the height exceeds a first height threshold.   48. The apparatus of claim 47, wherein the processor is configured to switch to a second height mode when the difference is less than a first difference threshold.   48. The processor of claim 47, wherein the processor is configured to switch to a second height mode when the height is above a first height threshold and the difference is below the first difference threshold. apparatus.   50. The apparatus of claim 49, wherein the second height mode comprises a stereo vision mode having a first resolution.   51. The apparatus of any one of claims 46 to 50, wherein the processor is configured to switch to a third height mode when the difference is less than or equal to a third difference threshold.   52. The apparatus according to claim 50 or 51, wherein the third height mode is a stereo vision mode with improved resolution.   53. The apparatus of any one of claims 46 to 52, wherein the processor is configured to switch to a fourth height mode when the height exceeds a third height threshold.   53. The processor of claim 46, wherein the processor is configured to switch to a fourth height mode when the height is above a third height threshold and the difference is below the fifth difference threshold. An apparatus according to any one of the preceding claims.   55. The apparatus of claim 54, wherein the fourth height mode is a high altitude monocular mode in a combination of a barometer, GPS, and / or a visual measurement of a vertical distance between the mobile platform and a ground level.   56. The apparatus of any of claims 53-55, wherein the processor is configured to switch to the third height mode when the height is less than a fourth height threshold.   56. The processor of claims 53-55, wherein the processor is configured to switch to the third height mode when the height is below a fourth height threshold and the difference is above a sixth difference threshold. An apparatus according to any one of the preceding claims.   58. The apparatus of claim 56 or 57, wherein the processor is configured to switch to the second height mode when the difference exceeds a fourth difference threshold.   The apparatus of claim 58, wherein the processor is configured to switch to the first height mode when the height is less than a second height threshold.   60. The processor of claim 59, wherein the processor is configured to switch to the first height mode when the height is less than a second height threshold and the difference exceeds a second difference threshold. apparatus. The second difference threshold is greater than the first difference threshold,
One or both of the first and second difference thresholds are greater than one or both of the third and fourth difference thresholds;
The third difference threshold is higher than the fourth difference threshold,
One or both of the third and fourth difference thresholds is greater than one or both of the fifth and sixth difference thresholds, and
61. The device of claim 60, wherein the sixth difference threshold is above the fifth threshold. The first height threshold is greater than the second height threshold,
Any one of the third and fourth height thresholds is greater than the first and second height thresholds, and
62. The apparatus according to claim 60 or 61, wherein the third height threshold is above the fourth threshold. The processor is configured to determine the height of the unmanned aerial vehicle (“UAV”) and the difference in terms of the UAV, and
63. The apparatus of any one of claims 36 to 62, wherein the processor is configured to select an operating mode of the UAV according to the determination.   64. The processor of claim 63, wherein the processor is configured to select a plurality of feature points on the first image and match the plurality of feature points of the first image with points of the second image. Equipment.   The apparatus of claim 64, wherein the feature points comprise pixels of the first image or the second image.   66. The apparatus according to claim 64 or 65, wherein the difference is at least 5 pixels and does not exceed one fifth of the width of the first image or the second image.   The processor scans the second image to identify a point in the second image that matches a selected feature point in the first image, and the selected feature point and the point in the first image. 67. The apparatus of any one of claims 64-66, wherein the apparatus is configured to calculate a similarity between. The processor comprises:
By comparing the intensity of each point pair in the region, a first binary string representing a first region around the selected feature point is constructed to form a first binary stable independent basic feature ("BRIEF"). ) Generate a descriptor,
Constructing a second binary string representing a second region around the point of the second image by comparing the intensity of each point pair of the second region to generate a second BRIFF descriptor; ,
When the Hamming distance between the first BRIFF descriptor and the second BRIFF descriptor is less than a first Hamming threshold, it is determined that the point in the second image matches the selected feature point. 68. The apparatus of claim 67, wherein the apparatus is configured for:   69. The similarity is calculated by comparing the selected feature point of the first image with a 3x3 pixel area centered about the point on the second image. The described device.   70. The 3x3 pixel area of claim 69, wherein the 3x3 pixel areas are compared by a sum of differences for each color component of each pixel of the color image or a sum of differences in grayscale values of each pixel of the black and white image. apparatus.   71. The apparatus of claim 70, wherein the differences are determined based on an average of the feature differences.   72. The apparatus of claim 70 or 71, wherein the processor is configured to select one or more feature points and obtain the differences based on the feature differences of the selected feature points. A method for determining an operation mode of a mobile platform,
Detecting a height grade of the mobile platform;
Selecting one or more sensors based on the result of the detection;
Obtaining a measurement value from the selected sensor,
The method, wherein the height grade is selected from a group of four height grades.   74. The method of claim 73, wherein selecting the one or more sensors comprises selecting one image sensor at a first height grade.   74. The method of claim 73, wherein selecting the one or more sensors comprises selecting one image sensor and one distance sensor at a first height grade.   77. The method according to claim 75, wherein the distance sensor comprises an ultrasonic detector and / or a laser detection device. Selecting the one or more sensors further comprises selecting at least two image sensors at a second height grade;
77. The method of claim 75 or 76, wherein the at least two image sensors have a first resolution.   The step of selecting the one or more sensors further comprises selecting at least two image sensors having a second resolution at a third height grade, wherein the second resolution is at an improved resolution. 78. The method of claim 77, wherein:   79. The method of claim 78, wherein the second resolution is greater than the first resolution.   80. The method of claim 79, wherein selecting the one or more sensors further comprises selecting one image sensor and one height sensor in a fourth height grade.   81. The method of claim 80, wherein the height sensor comprises a barometer and / or a global positioning system ("GPS"). A mobile platform flight operation system,
A height sensor for detecting a height grade of the mobile platform;
A processor configured to select one or more sensors based on the detected height grade and obtain measurements from the selected sensors,
The system wherein the height grade is selected from a group of four height grades.   83. The method of claim 82, wherein the processor is configured to select one image sensor at a first height grade.   83. The method of claim 82, wherein the processor is configured to select one image sensor and one distance sensor at a first height grade.   85. The method of claim 83 or 84, wherein the height sensor comprises a distance sensor, and wherein the distance sensor comprises an ultrasonic detector and / or a laser detection device. The processor is configured to select at least two image sensors at a second height grade;
86. The method of claim 84 or 85, wherein the at least two image sensors have a first resolution.   87. The method of claim 86, wherein the processor is configured to select at least two image sensors having a second resolution at a third height grade, wherein the second resolution is an improved resolution. .   The method of claim 87, wherein the second resolution is greater than the first resolution.   89. The method of claim 88, wherein the processor is configured to select one image sensor and one height sensor in a fourth height grade.   90. The method of claim 89, wherein the height sensor comprises a barometer and / or a global positioning system ("GPS").

Images