WO2020244649A1 - 一种避障方法、装置和电子设备 - Google Patents

一种避障方法、装置和电子设备 Download PDF

Info

Publication number
WO2020244649A1
WO2020244649A1 PCT/CN2020/094764 CN2020094764W WO2020244649A1 WO 2020244649 A1 WO2020244649 A1 WO 2020244649A1 CN 2020094764 W CN2020094764 W CN 2020094764W WO 2020244649 A1 WO2020244649 A1 WO 2020244649A1
Authority
WO
WIPO (PCT)
Prior art keywords
obstacle
image
drone
category
electronic device
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2020/094764
Other languages
English (en)
French (fr)
Inventor
冯银华
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Autel Robotics Co Ltd
Original Assignee
Autel Robotics Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Autel Robotics Co Ltd filed Critical Autel Robotics Co Ltd
Publication of WO2020244649A1 publication Critical patent/WO2020244649A1/zh
Priority to US17/457,306 priority Critical patent/US20220091608A1/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/0011—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots associated with a remote control arrangement
    • G05D1/0038—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots associated with a remote control arrangement by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/0011—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots associated with a remote control arrangement
    • G05D1/0044—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots associated with a remote control arrangement by providing the operator with a computer generated representation of the environment of the vehicle, e.g. virtual reality, maps
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C11/00—Photogrammetry or videogrammetry, e.g. stereogrammetry; Photographic surveying
    • G01C11/02—Picture taking arrangements specially adapted for photogrammetry or photographic surveying, e.g. controlling overlapping of pictures
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C11/00—Photogrammetry or videogrammetry, e.g. stereogrammetry; Photographic surveying
    • G01C11/04—Interpretation of pictures
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
    • G01C21/20—Instruments for performing navigational calculations
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/0011—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots associated with a remote control arrangement
    • G05D1/005—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots associated with a remote control arrangement by providing the operator with signals other than visual, e.g. acoustic, haptic
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/10—Simultaneous control of position or course in three dimensions
    • G05D1/101—Simultaneous control of position or course in three dimensions specially adapted for aircraft
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/10—Simultaneous control of position or course in three dimensions
    • G05D1/101—Simultaneous control of position or course in three dimensions specially adapted for aircraft
    • G05D1/106—Change initiated in response to external conditions, e.g. avoidance of elevated terrain or of no-fly zones
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/10—Terrestrial scenes
    • G06V20/17—Terrestrial scenes taken from planes or by drones
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/20—Scenes; Scene-specific elements in augmented reality scenes
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
    • G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B64—AIRCRAFT; AVIATION; COSMONAUTICS
    • B64U—UNMANNED AERIAL VEHICLES [UAV]; EQUIPMENT THEREFOR
    • B64U10/00—Type of UAV
    • B64U10/10—Rotorcrafts
    • B64U10/13—Flying platforms
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B64—AIRCRAFT; AVIATION; COSMONAUTICS
    • B64U—UNMANNED AERIAL VEHICLES [UAV]; EQUIPMENT THEREFOR
    • B64U2101/00—UAVs specially adapted for particular uses or applications
    • B64U2101/30—UAVs specially adapted for particular uses or applications for imaging, photography or videography
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B64—AIRCRAFT; AVIATION; COSMONAUTICS
    • B64U—UNMANNED AERIAL VEHICLES [UAV]; EQUIPMENT THEREFOR
    • B64U2201/00—UAVs characterised by their flight controls
    • B64U2201/10—UAVs characterised by their flight controls autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS]
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B64—AIRCRAFT; AVIATION; COSMONAUTICS
    • B64U—UNMANNED AERIAL VEHICLES [UAV]; EQUIPMENT THEREFOR
    • B64U2201/00—UAVs characterised by their flight controls
    • B64U2201/20—Remote controls

Definitions

  • This application relates to the technical field of unmanned aerial vehicles, in particular to an obstacle avoidance method, device and electronic equipment.
  • UAVs mainly have three flight modes: autonomous flight, remote control of UAV flight and a combination of the first two flight modes.
  • the current obstacle avoidance method is mostly that the UAV autonomously detects the position of the obstacle and takes obstacle avoidance measures according to the position of the obstacle.
  • the drone transmits the video images of the surrounding environment to the remote control operator, and the operator judges the location of the obstacle with his naked eyes by watching the image and video, and controls the drone to avoid the obstacle.
  • the inventor found that the related technology has at least the following problems: the operator can visually judge the location of the obstacle by watching the image and video returned by the drone, and cannot intuitively display the obstacle and user experience to the operator. Poor.
  • the purpose of the embodiments of the present invention is to provide an obstacle avoidance method, device and electronic equipment, which can intuitively display obstacles to the drone operator.
  • an embodiment of the present invention provides an obstacle avoidance method, the method is used in an electronic device, and the method includes:
  • obstacle data in the flying environment of the drone and the image taken by the drone, wherein the obstacle data is obtained by the drone based on the image, and the obstacle data includes the physical information of the obstacle A position and an image position corresponding to the obstacle, where the physical position of the obstacle includes the distance and orientation of the obstacle;
  • the superimposed image is displayed on the display screen of the electronic device, so that the user controls the drone to avoid obstacles according to the superimposed image.
  • the superimposing the virtual image on the image at the image position corresponding to the obstacle on the image to obtain the superimposed image includes:
  • the virtual image is superimposed on the image using augmented reality technology to obtain the superimposed image.
  • the method further includes:
  • the method further includes:
  • the obstacle distance is less than the preset safety distance threshold, acquiring the obstacle avoidance switch status and flight speed of the drone;
  • an obstacle avoidance instruction is sent to the drone and/or a danger warning is given.
  • the method further includes:
  • the obtaining a virtual image of the obstacle based on the category of the obstacle includes:
  • an embodiment of the present invention provides an obstacle avoidance device, the device is used in an electronic device, and the device includes:
  • the image and obstacle data acquisition module is used to acquire obstacle data in the flying environment of the drone and the image taken by the drone, wherein the obstacle data is obtained by the drone based on the image,
  • the obstacle data includes the physical position of the obstacle and the image position corresponding to the obstacle, and the physical position of the obstacle includes the distance and orientation of the obstacle;
  • the recognition module is used to recognize the image to obtain the category of the obstacle
  • a virtual image acquisition module configured to acquire a virtual image of the obstacle based on the category of the obstacle
  • An overlay module configured to overlay the virtual image on the image position corresponding to the obstacle on the image to obtain an overlay image
  • the display module is configured to display the superimposed image on the display screen of the electronic device, so that the user controls the drone to avoid obstacles according to the superimposed image.
  • the superposition module is specifically used for:
  • the virtual image is superimposed on the image using augmented reality technology to obtain the superimposed image.
  • the device further includes:
  • the voice prompt module is used to perform voice prompts according to the category of the obstacle and the physical location of the obstacle, so that the user knows the category and physical location of the obstacle.
  • the device further includes:
  • Obstacle avoidance assistance module used to determine whether the obstacle distance is less than a preset safety distance threshold; if the obstacle distance is less than the preset safety distance threshold, obtain the obstacle avoidance switch status and flight speed of the drone ; If the obstacle avoidance switch is turned off and the flight speed is greater than a preset flight speed threshold, an obstacle avoidance instruction is sent to the drone and/or a danger reminder.
  • the device further includes:
  • the nearest obstacle determination module is used to determine the preset number of obstacles in each obstacle with the largest obstacle distance
  • the virtual image acquisition module is specifically used for:
  • an embodiment of the present invention provides an electronic device, and the electronic device includes:
  • At least one processor and,
  • a memory communicatively connected with the at least one processor; wherein,
  • the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor, so that the at least one processor can execute the foregoing method.
  • an embodiment of the present invention provides a non-volatile computer-readable storage medium, the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by an electronic device, The electronic device executes the above-mentioned method.
  • the embodiments of the present application also provide a computer program product.
  • the computer program product includes a computer program stored on a non-volatile computer-readable storage medium.
  • the computer program includes program instructions. When the program instructions are executed by the electronic device, the electronic device is caused to execute the above-mentioned method.
  • the obstacle avoidance method, device and electronic equipment of the embodiments of the present invention acquire images and obstacle data taken by a drone through the electronic equipment, wherein the obstacle data is obtained by the drone based on the image.
  • the electronic device recognizes the image, obtains the category of the obstacle, and obtains the virtual image of the obstacle according to the category of the obstacle.
  • the virtual image is superimposed on the corresponding image position of the obstacle on the image to obtain the superimposed image.
  • the superimposed image is displayed on the display screen.
  • Figure 1a is a schematic diagram of one of the application scenarios of the obstacle avoidance method and device according to the embodiments of the present invention.
  • FIG. 1b is a schematic diagram of another application scenario of the obstacle avoidance method and device according to the embodiment of the present invention.
  • FIG. 2 is a schematic flowchart of an embodiment of the obstacle avoidance method of the present invention.
  • FIG. 3 is a schematic flowchart of an embodiment of the obstacle avoidance method of the present invention.
  • FIG. 4 is a schematic structural diagram of an embodiment of the obstacle avoidance device of the present invention.
  • Figure 5 is a schematic structural diagram of an embodiment of the obstacle avoidance device of the present invention.
  • Fig. 6 is a schematic diagram of the hardware structure of the controller of the path planning system in an embodiment of the unmanned aerial vehicle of the present invention.
  • the obstacle avoidance method, device and electronic equipment provided by the embodiments of the present invention may be applicable to the application scenario shown in FIG. 1a.
  • the application scenario includes a drone 100, an electronic device 200, and an obstacle 400.
  • the UAV 100 may be a suitable unmanned aerial vehicle, including a fixed-wing unmanned aerial vehicle and a rotary-wing unmanned aerial vehicle, such as a helicopter, a quadrotor, and an aircraft with other numbers of rotors and/or rotor configurations.
  • the UAV 100 may also be other movable objects, such as a manned aircraft, a model airplane, an unmanned airship, and an unmanned hot air balloon.
  • Obstacles 400 such as people, animals, buildings, mountains, trees, forests, signal towers or other movable or non-movable objects ( Figure 1a shows only one obstacle, there may be more obstacles in practical applications or No obstacles).
  • the electronic device 200 is, for example, a smart phone, a tablet computer, a computer, a remote control, etc.
  • a communication connection can be established through wireless communication modules (such as a signal receiver, a signal transmitter, etc.) respectively provided in each of them, and data/commands can be uploaded or issued.
  • the electronic device 200 is a remote control
  • the remote control needs to have a display screen to display the images or data returned by the drone.
  • the application scenario may also include a remote control 300, and the electronic device 200 establishes a communication connection with the drone 100 through the remote control 300 ( The following takes the application scenario shown in Figure 1b as an example for description).
  • the drone 100 includes a fuselage, an arm connected to the fuselage, a power device provided on the arm, and a control system provided on the fuselage.
  • the power device is used to provide thrust and lift for the flight of the UAV 100.
  • the control system is the central nerve of the UAV 100 and may include multiple functional units, such as a flight control system, a vision system, and other systems with specific functions.
  • the vision system includes image acquisition devices and vision chips, and the flight control system includes various sensors (such as gyroscopes, accelerometers) and flight controllers.
  • the image acquisition device may include at least one monocular camera or at least one binocular camera.
  • the UAV 100 needs to recognize and avoid obstacles 400 in front of the flight.
  • the drone 100 can obtain images around the drone through an image acquisition device, and the vision chip performs monocular or binocular recognition based on the images to obtain obstacle data around the drone 100.
  • obstacle data such as the distance between each obstacle and the UAV (hereinafter referred to as obstacle distance), the position of each obstacle relative to the UAV, and the coordinate position of each obstacle in the image (hereinafter referred to as Image location) and so on.
  • the image, obstacle data or other data obtained by the drone can be transmitted to the electronic device 200 through the remote control 300.
  • the electronic device 200 after the electronic device 200 obtains the image taken by the drone and the obstacle data through the remote control, it recognizes the obstacle in the image, recognizes the type of the obstacle, and then obtains the obstacle according to the type of the obstacle.
  • Virtual images of objects may be a typical image image representing an obstacle category. For example, if the recognized obstacle is a person, the virtual image may be a human-shaped image.
  • the electronic device 200 superimposes the virtual image on the corresponding image position of each obstacle on the image to obtain the superimposed image, and displays the superimposed image on the display screen of the electronic device 200.
  • the obstacle can be displayed intuitively to the drone operator, and the user experience is good.
  • FIG. 2 is a schematic flowchart of an obstacle avoidance method provided by an embodiment of the present invention. The method may be executed by the electronic device 200 in FIG. 1a or FIG. 1b. As shown in FIG. 2, the method includes:
  • the obstacle data is obtained by the drone based on the image, and the obstacle data includes the physical information of the obstacle.
  • the location and the image location corresponding to the obstacle, and the physical location of the obstacle includes the distance and orientation of the obstacle.
  • the electronic device 200 establishes a communication connection with the drone through a remote control, and receives image data and obstacle data sent by the drone.
  • the electronic device can also directly establish a communication connection with the drone, and directly receive image data and obstacle data sent by the drone.
  • the UAV uses its image acquisition device to collect images of the surrounding environment, it performs monocular or binocular recognition based on the image to obtain the physical position of each obstacle in the image, where the physical position includes the distance and orientation of the obstacle Wait. Take binocular recognition as an example to illustrate the process of UAV obtaining obstacle data.
  • UAV extracts feature points and matches feature points on binocular images, uses matching algorithms to obtain the parallax of feature points on binocular images, and then uses all
  • the parallax obtains the depth value of the feature point, that is, the distance between the drone and the feature point (the obstacle distance of the feature point), and then the physical coordinates of the obstacle and the position relative to the drone can be obtained.
  • the location of each feature point on the image is determined by its image location (for example, pixel coordinates), the physical location of each obstacle corresponds to an image location.
  • the image recognition of the image may be based on a neural network model of deep learning.
  • the neural network model based on deep learning can be trained in advance by other devices, and then the neural network model is loaded on the electronic device.
  • the neural network model based on deep learning can also be trained by the electronic device itself.
  • the neural network model can be obtained through training on a large amount of sample data and labels (ie categories) corresponding to the sample data, for example, based on data training on the PASCAL VOC data set.
  • the neural network model is a network model based on SSD (Single Shot MultiBox Detector) algorithm. In other embodiments, it can also be replaced by other deep learning networks, for example, YOLO (You Only Look Once), Fast-RCNN (Regions with CNN), etc.
  • the minimum circumscribed frame for example, frames the minimum circumscribed rectangular area of the obstacle in the image.
  • the virtual image may be a typical image image representing an obstacle category.
  • the virtual image may be a human-shaped image.
  • the virtual image may be a tree image.
  • the virtual image is an image of a dog.
  • Each virtual image can be stored in the electronic device in advance, and after the obstacle category is obtained, the corresponding virtual image can be called in the electronic device according to the obstacle category.
  • each virtual image after obtaining the virtual image of each obstacle in the image, each virtual image can be superimposed with the image sent by the drone to obtain the superimposed image.
  • the method includes:
  • the obstacle data is obtained by the drone based on the image, and the obstacle data includes the physical information of the obstacle.
  • the location and the image location corresponding to the obstacle, and the physical location of the obstacle includes the distance and orientation of the obstacle.
  • the virtual image of the obstacle is superimposed on the image position corresponding to the obstacle in the image, and the superimposed image of the two is obtained.
  • augmented reality technology may be used to superimpose a virtual image of an obstacle on the image position corresponding to the obstacle in the image to obtain the superimposed image.
  • the UAV's obstacle avoidance is a continuous process.
  • the image acquisition device will continuously obtain images of the surrounding environment of the UAV, and the UAV obtains the obstacles in the image based on the images.
  • the physical location of the image and the image location corresponding to the obstacle, and then the image and obstacle data are transmitted to the electronic device.
  • the electronic device recognizes the image, obtains the category of the obstacle in the image, and obtains the virtual image corresponding to the obstacle according to the category of the obstacle, and then superimposes the virtual image with the image sent by the corresponding drone to obtain Overlay the image.
  • the electronic device Since the drone continuously obtains images of the surrounding environment, the electronic device will also continuously obtain the superimposed image, and display the superimposed image on the electronic device, which can dynamically display the obstacles around the drone.
  • the obstacle By superimposing the virtual image of the obstacle and the real image obtained by the drone, the obstacle can be displayed intuitively to the drone operator, and the user experience is good.
  • the drone operator can visually observe obstacles by watching the display screen of the electronic device.
  • the user in order to further enhance the user experience, the user can be prompted by voice according to the type of obstacle and the physical location of the obstacle. For example, if the obstacle category is a tree and the obstacle distance is 5m, the user can be voiced. Prompt "there is a big tree five meters ahead” and so on.
  • the drone operator can "see” and “hear” the state of the obstacle at the same time, making it easier to understand the state of the obstacle.
  • the UAV After obtaining the physical location of the obstacle, the UAV can use its own obstacle avoidance system to perform obstacle avoidance operations. However, in some applications, the drone will turn off the obstacle avoidance switch, so that its own obstacle avoidance system does not work. In this state, there is a certain degree of risk in the flight of the drone. Therefore, in some embodiments, in order to reduce the risk of drone flight, electronic equipment may be used to assist the drone in avoiding obstacles. In this embodiment, the method further includes:
  • the obstacle distance is less than the preset safety distance threshold, acquiring the obstacle avoidance switch status and flight speed of the drone;
  • an obstacle avoidance instruction is sent to the drone and/or a danger warning is given.
  • the obstacle avoidance switch status and flight speed of the drone will be obtained. If the obstacle avoidance switch is off and the flight speed is greater than the preset flight At the speed threshold, the electronic device takes obstacle avoidance measures to assist the drone in avoiding obstacles. Specifically, for example, sending a control instruction to make the drone pause, or sending a control instruction to make the drone turn on the obstacle avoidance switch, or directly issuing a voice prompt to remind the drone operator.
  • the preset safety distance threshold and the preset flight speed threshold can be set in combination with application conditions, such as the performance of the drone.
  • an embodiment of the present invention also provides an obstacle avoidance device, which can be used in the electronic equipment shown in FIG. 1a or FIG. 1b.
  • the obstacle avoidance device 400 includes:
  • the image and obstacle data acquisition module 401 is used to acquire obstacle data in the flying environment of the drone and the image taken by the drone.
  • the obstacle data is obtained by the drone based on the image.
  • the obstacle data includes the physical position of the obstacle and the image position corresponding to the obstacle, and the physical position of the obstacle includes the distance and orientation of the obstacle;
  • the recognition module 402 is configured to recognize the image and obtain the category of the obstacle
  • the virtual image acquisition module 403 is configured to acquire a virtual image of the obstacle based on the category of the obstacle;
  • the superimposing module 404 is configured to superimpose the virtual image on the image position corresponding to the obstacle on the image to obtain a superimposed image
  • the display module 405 is configured to display the superimposed image on the display screen of the electronic device, so that the user controls the drone to avoid obstacles according to the superimposed image.
  • an image and obstacle data taken by a drone are acquired through an electronic device, where the obstacle data is obtained by the drone based on the image.
  • the electronic device recognizes the image, obtains the category of the obstacle, and obtains the virtual image of the obstacle according to the category of the obstacle.
  • the virtual image is superimposed on the corresponding image position of the obstacle on the image to obtain the superimposed image.
  • the superimposed image is displayed on the display screen.
  • the superposition module 404 is specifically used for:
  • the virtual image is superimposed on the image using augmented reality technology to obtain the superimposed image.
  • the obstacle avoidance device 400 further includes:
  • the voice prompt module 406 is used to perform voice prompts according to the category of the obstacle and the physical location of the obstacle, so that the user knows the category and physical location of the obstacle.
  • the obstacle avoidance device 400 further includes:
  • the obstacle avoidance assistance module 407 is used to determine whether the obstacle distance is less than a preset safety distance threshold; if the obstacle distance is less than the preset safety distance threshold, obtain the obstacle avoidance switch status and flight speed of the UAV; If the obstacle avoidance switch is turned off and the flight speed is greater than a preset flight speed threshold, an obstacle avoidance instruction is sent to the drone and/or a danger warning is given.
  • the obstacle avoidance device 400 further includes:
  • the nearest obstacle determination module 408 is used to determine a preset number of obstacles in each obstacle with the smallest obstacle distance
  • the virtual image acquisition module 403 is specifically used for:
  • the above-mentioned device can execute the method provided in the embodiment of the present application, and has corresponding functional modules and beneficial effects for executing the method.
  • the above-mentioned device can execute the method provided in the embodiment of the present application, and has corresponding functional modules and beneficial effects for executing the method.
  • the methods provided in the embodiments of this application please refer to the methods provided in the embodiments of this application.
  • FIG. 6 is a schematic diagram of the hardware structure of an embodiment of the electronic device 200 of the present invention. As shown in FIG. 6, the electronic device 200 includes:
  • One or more processors 201 and a memory 202 are taken as an example in FIG. 6.
  • the processor 201 and the memory 202 may be connected by a bus or in other ways.
  • the connection by a bus is taken as an example.
  • the memory 202 can be used to store non-volatile software programs, non-volatile computer-executable programs and modules, such as program instructions corresponding to the obstacle avoidance method in the embodiments of the present application /Module (for example, the image and obstacle data acquisition module 401, the recognition module 402, the virtual image acquisition module 403, the overlay module 404, and the display module 405 shown in FIG. 4).
  • the processor 201 executes various functional applications and data processing of the electronic device by running non-volatile software programs, instructions, and modules stored in the memory 202, that is, implements the obstacle avoidance method of the foregoing method embodiment.
  • the memory 202 may include a program storage area and a data storage area.
  • the program storage area may store an operating system and an application program required by at least one function; the data storage area may store data created according to the use of the controller.
  • the memory 202 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state storage devices.
  • the memory 202 may optionally include memories remotely provided with respect to the processor 201, and these remote memories may be connected to the electronic device through a network. Examples of the aforementioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
  • the one or more modules are stored in the memory 202, and when executed by the one or more processors 201, the obstacle avoidance method in any of the foregoing method embodiments is executed, for example, the above-described
  • the method steps 101 to 105, the method steps 101 to 105 in FIG. 3; the functions of the modules 401-405 in FIG. 4 and the modules 401-408 in FIG. 5 are realized.
  • the embodiment of the present application provides a non-volatile computer-readable storage medium, the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are executed by one or more processors to execute the above-described The method steps 101 to 105 in Fig. 2 and the method steps 101 to 105 in Fig. 3; realize the functions of the modules 401-405 in Fig. 4 and the modules 401-408 in Fig. 5.
  • the device embodiments described above are merely illustrative.
  • the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in One place, or it can be distributed to multiple network units. Some or all of the modules may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.
  • each embodiment can be implemented by software plus a general hardware platform, and of course, it can also be implemented by hardware.
  • Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be implemented by computer programs instructing relevant hardware.
  • the programs can be stored in a computer readable storage medium. When executed, it may include the procedures of the above-mentioned method embodiments.
  • the storage medium can be a magnetic disk, an optical disc, a read-only memory (Read-Only Memory, ROM), or a random access memory (Random Access Memory, RAM), etc.

Landscapes

  • Engineering & Computer Science (AREA)
  • Remote Sensing (AREA)
  • Physics & Mathematics (AREA)
  • Radar, Positioning & Navigation (AREA)
  • General Physics & Mathematics (AREA)
  • Automation & Control Theory (AREA)
  • Aviation & Aerospace Engineering (AREA)
  • Multimedia (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Acoustics & Sound (AREA)
  • Human Computer Interaction (AREA)
  • Control Of Position, Course, Altitude, Or Attitude Of Moving Bodies (AREA)
  • User Interface Of Digital Computer (AREA)
  • Processing Or Creating Images (AREA)

Abstract

一种避障方法、装置和电子设备，方法包括：获取无人机飞行环境中的障碍物数据和无人机拍摄的图像，障碍物数据由无人机基于图像获得，障碍物数据包括障碍物的物理位置以及障碍物对应的图像位置，障碍物的物理位置包括障碍物的距离和方位（101）；对图像进行识别，获得障碍物的类别（102）；基于障碍物的类别获得障碍物的虚拟图像（103）；将虚拟图像叠加在障碍物在图像上的相应图像位置，以获得叠加图像（104）；在电子设备的显示屏幕上显示叠加图像（105）。通过将障碍物的虚拟图像和无人机获得的真实图像相叠加呈现的方式，能直观的向无人机操纵者显示障碍物，用户体验好。

Description

一种避障方法、装置和电子设备
本申请要求于2019年6月6日提交中国专利局、申请号为201910490281.6、申请名称为“一种避障方法、装置和电子设备”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及无人飞行器技术领域，特别涉及一种避障方法、装置和电子设备。
背景技术
目前，无人机主要有三种飞行方式：自主飞行、通过遥控器控制无人机飞行和前两种方式相结合飞行。无人机在飞行过程中，需要躲避障碍物飞行，目前的避障方法多是无人机自主对障碍物的位置进行检测，并根据障碍物的位置采取避障措施。或者，无人机将周围环境的视频图像传递给遥控器操纵者，操纵者通过观看图像视频肉眼判断障碍物的位置，并控制无人机躲避障碍物。
在实现本发明过程中，发明人发现相关技术中至少存在如下问题：操纵者通过观看无人机传回的图像视频肉眼判断障碍物位置的方式，不能直观的向操纵者显示障碍物、用户体验较差。
发明内容
本发明实施例的目的是提供一种避障方法、装置和电子设备，能直观的向无人机操纵者显示障碍物。
第一方面，本发明实施例提供了一种避障方法，所述方法用于电子设备，所述方法包括：
获取无人机飞行环境中的障碍物数据和所述无人机拍摄的图像，其中，所述障碍物数据由所述无人机基于所述图像获得，所述障碍物数据包括障碍物的物理位置以及所述障碍物对应的图像位置，所述障碍物的物理位置包括所述障碍物的距离和方位；
对所述图像进行识别，以获得所述障碍物的类别；
基于所述障碍物的类别，获得所述障碍物的虚拟图像；
将所述虚拟图像叠加在所述图像上的所述障碍物对应的所述图像位置，以获得叠加图像；
在所述电子设备的显示屏幕上显示所述叠加图像，以使用户根据所述叠加图像控制所述无人机避障。
在一些实施例中，所述将所述虚拟图像叠加在所述图像上的所述障碍物对应的所述图像位置，以获得叠加图像，包括：
利用增强现实技术将所述虚拟图像叠加在所述图像上，以获得所述叠加图 像。
在一些实施例中，所述方法还包括：
根据所述障碍物的类别和所述障碍物的物理位置进行语音提示，以使所述用户知晓所述障碍物的类别和物理位置。
在一些实施例中，所述方法还包括：
判断所述障碍物距离是否小于预设安全距离阈值；
如果所述障碍物距离小于所述预设安全距离阈值，获取所述无人机的避障开关状态以及飞行速度；
如果所述避障开关关闭且所述飞行速度大于预设飞行速度阈值，则发送避障指令给所述无人机和/或进行危险提示。
在一些实施例中，所述方法还包括：
确定各障碍物中障碍物距离最大的预设数量的障碍物；则，
所述基于所述障碍物的类别，获得所述障碍物的虚拟图像，包括：
基于所述障碍物的类别，获得所述预设数量的障碍物对应的虚拟图像。
第二方面，本发明实施例提供了一种避障装置，所述装置用于电子设备，所述装置包括：
图像以及障碍物数据获取模块，用于获取无人机飞行环境中的障碍物数据和所述无人机拍摄的图像，其中，所述障碍物数据由所述无人机基于所述图像获得，所述障碍物数据包括障碍物的物理位置以及所述障碍物对应的图像位置，所述障碍物的物理位置包括所述障碍物的距离和方位；
识别模块，用于对所述图像进行识别，以获得所述障碍物的类别；
虚拟图像获取模块，用于基于所述障碍物的类别，获得所述障碍物的虚拟图像；
叠加模块，用于将所述虚拟图像叠加在所述图像上的所述障碍物对应的所述图像位置，以获得叠加图像；
显示模块，用于在所述电子设备的显示屏幕上显示所述叠加图像，以使用户根据所述叠加图像控制所述无人机避障。
在一些实施例中，所述叠加模块具体用于：
利用增强现实技术将所述虚拟图像叠加在所述图像上，以获得所述叠加图像。
在一些实施例中，所述装置还包括：
语音提示模块，用于根据所述障碍物的类别和所述障碍物的物理位置进行语音提示，以使所述用户知晓所述障碍物的类别和物理位置。
在一些实施例中，所述装置还包括：
避障辅助模块，用于判断所述障碍物距离是否小于预设安全距离阈值；如果所述障碍物距离小于所述预设安全距离阈值，获取所述无人机的避障开关状态以及飞行速度；如果所述避障开关关闭且所述飞行速度大于预设飞行速度阈值，则发送避障指令给所述无人机和/或进行危险提示。
在一些实施例中，所述装置还包括：
最近障碍物确定模块，用于确定各障碍物中障碍物距离最大的预设数量的障碍物；
所述虚拟图像获取模块具体用于：
基于所述障碍物的类别，获得所述预设数量的障碍物对应的虚拟图像。
第三方面，本发明实施例提供了一种电子设备，所述电子设备包括：
至少一个处理器；以及，
与所述至少一个处理器通信连接的存储器；其中，
所述存储器存储有可被所述至少一个处理器执行的指令，所述指令被所述至少一个处理器执行，以使所述至少一个处理器能够执行上述的方法。
第四方面，本发明实施例提供了一种非易失性计算机可读存储介质，所述计算机可读存储介质存储有计算机可执行指令，当所述计算机可执行指令被电子设备执行时，使所述电子设备执行上述的方法。
第五方面，本申请实施例还提供了一种计算机程序产品，所述计算机程序产品包括存储在非易失性计算机可读存储介质上的计算机程序，所述计算机程序包括程序指令，当所述程序指令被电子设备执行时，使所述电子设备执行上述的方法。
本发明实施例的避障方法、装置和电子设备，通过电子设备获取无人机拍摄的图像和障碍物数据，其中障碍物数据为无人机基于所述图像获得。电子设备对图像进行识别，获得障碍物的类别，并根据障碍物的类别获得障碍物的虚拟图像，将虚拟图像叠加在障碍物在图像上的相应图像位置，获得叠加图像，然后在电子设备的显示屏幕上显示所述叠加图像。通过将障碍物的虚拟图像和无人机获得的真实图像相叠加呈现的方式，能直观的向无人机操纵者显示障碍物，用户体验好。
附图说明
一个或多个实施例通过与之对应的附图中的图片进行示例性说明，这些示例性说明并不构成对实施例的限定，附图中具有相同参考数字标号的元件表示为类似的元件，除非有特别申明，附图中的图不构成比例限制。
图1a是本发明实施例避障方法和装置的其中一个应用场景示意图；
图1b是本发明实施例避障方法和装置的另一个应用场景示意图；
图2是本发明避障方法的一个实施例的流程示意图；
图3是本发明避障方法的一个实施例的流程示意图；
图4是本发明避障装置的一个实施例的结构示意图；
图5是本发明避障装置的一个实施例的结构示意图；
图6是本发明无人机的一个实施例中路径规划系统的控制器的硬件结构示意图。
具体实施方式
为使本发明实施例的目的、技术方案和优点更加清楚，下面将结合本发明实施例中的附图，对本发明实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例是本发明一部分实施例，而不是全部的实施例。基于本发明中的实施例，本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其他实施例，都属于本发明保护的范围。
本发明实施例提供的避障方法、装置和电子设备可以适用于如图1a所示的应用场景。所述应用场景包括无人机100、电子设备200和障碍物400。其中，无人机100可以为合适的无人飞行器，包括固定翼无人飞行器和旋转翼无人飞行器，例如直升机、四旋翼机和具有其它数量的旋翼和/或旋翼配置的飞行器。无人机100还可以是其他可移动物体，例如载人飞行器、航模、无人飞艇和无人热气球等。障碍物400例如人、动物、建筑物、山体、树木、森林、信号塔或其他可移动或不可移动物体(图1a中只示出了一个障碍物，实际应用中可能会有更多障碍物或者没有障碍物)。电子设备200例如智能手机、平板电脑、电脑、遥控器等。
无人机100和电子设备200之间，可以通过分别设置在各自内部的无线通信模块(例如信号接收器、信号发送器等)建立通信连接，上传或者下发数据/指令。在电子设备200为遥控器的场合，遥控器需带有显示屏，以显示无人机传回的图像或数据等。在电子设备200为智能手机、平板电脑、电脑等的场合，如图1b所示，所述应用场景中还可以包括遥控器300，电子设备200通过遥控器300与无人机100建立通信连接(以下以图1b所示的应用场景为例进行说明)。
其中，在一些实施例中，无人机100包括机身、与机身相连的机臂、设于机臂的动力装置和设于机身的控制系统。动力装置用于提供无人机100飞行的推力、升力，控制系统是无人机100的中枢神经，可以包括多个功能性单元，例如飞控系统、视觉系统以及其他具有特定功能的系统。视觉系统包括图像采集装置和视觉芯片等，飞控系统包括各类传感器(例如陀螺仪、加速计)和飞行控制器。其中，图像采集装置可以包括至少一个单目相机或者至少一个双目相机。
无人机100在飞行过程中，需要识别并躲避飞行前方的障碍物400。无人机100可以通过图像采集装置获得无人机周围的图像，视觉芯片基于所述图像进行单目或双目识别获得无人机100周围的障碍物数据。其中，障碍物数据例如各障碍物与所述无人机的距离(下称障碍物距离)、各障碍物相对于无人机的方位以及各障碍物在所述图像中的坐标位置(下称图像位置)等。无人机获得的图像、障碍物数据或其他数据可以通过遥控器300传输至电子设备200上。
在本发明实施例中，电子设备200通过遥控器获得无人机拍摄的图像以及 障碍物数据后，对图像中的障碍物进行识别，识别出障碍物的类别，然后根据障碍物的类别获得障碍物的虚拟图像。其中，所述虚拟图像可以是代表障碍物类别的典型图像形象，例如如果识别出的障碍物为人，则所述虚拟图像可以为人形图像。然后，电子设备200将该虚拟图像叠加在各障碍物在图像上的相应图像位置，以获得叠加图像，并在电子设备200的显示屏幕上显示该叠加图像。通过将障碍物的虚拟图像和无人机获得的真实图像相叠加呈现的方式，能直观的向无人机操纵者显示障碍物，用户体验好。
图2为本发明实施例提供的一种避障方法的流程示意图，所述方法可以由图1a或图1b中电子设备200执行，如图2所示，所述方法包括：
101：获取无人机飞行环境中的障碍物数据和所述无人机拍摄的图像，所述障碍物数据由所述无人机基于所述图像获得，所述障碍物数据包括障碍物的物理位置以及所述障碍物对应的图像位置，所述障碍物的物理位置包括所述障碍物的距离和方位。
在其中一些实施例中，电子设备200通过遥控器与无人机建立通信连接，接收无人机发送的图像数据和障碍物数据等。在另一些实施例中，电子设备也可以直接与无人机建立通信连接，直接接收无人机发送的图像数据和障碍物数据等。无人机利用其图像采集装置采集周围环境的图像后，基于该图像进行单目或者双目识别，获得所述图像中各障碍物的物理位置，其中所述物理位置包括障碍物的距离和方位等。以双目识别为例说明无人机获取障碍物数据的过程，无人机对双目图像进行特征点提取、特征点匹配、利用匹配算法获得特征点在双目图像上的视差，然后利用所述视差获得该特征点的深度值，即无人机距离该特征点的距离(该特征点的障碍物距离)，进而可以获得该障碍物的物理坐标和相对于所述无人机的方位等。由于所述图像上每个特征点的位置是由其图像位置(例如像素坐标)确定的，那么，每个障碍物的物理位置都对应一图像位置。
102：对所述图像进行识别，获得所述障碍物的类别。
其中，对所述图像进行图像识别可以基于深度学习的神经网络模型进行识别。在其中一些实施例中，可以预先通过其他装置训练基于深度学习的神经网络模型，再将所述神经网络模型加载在电子设备上。在另一些实施例中，也可以由电子设备自身训练基于深度学习的神经网络模型。所述神经网络模型可以通过大量样本数据以及样本数据对应的标签(即类别)训练获得，例如基于数据集PASCAL VOC上的数据训练获得。在其中一些实施例中，所述神经网络模型为基于SSD(Single Shot MultiBox Detector)算法的网络模型。在另一些实施例中，也可以被其它深度学习网络替换，例如，YOLO(You Only Look Once)、Fast-RCNN(Regions with CNN)等。
将所述图像输入所述神经网络模型，将获得所述图像中各个障碍物的最小外接框和最小外接框对应的类别。其中，所述最小外接框例如在所述图像中框 住障碍物的最小外接矩形区域。通过判断图像位置位于的最小外接框，可以获得该图像位置处的障碍物的类别。
103：基于所述障碍物的类别获得所述障碍物的虚拟图像。
其中，所述虚拟图像可以是代表障碍物类别的典型图像形象，例如如果识别出的障碍物为人，则所述虚拟图像可以为人形图像。如果识别出的障碍物为树，则所述虚拟图像可以为树形图像。如果识别出的障碍物为狗，则所述虚拟图像为狗的形象。各个虚拟图像可以事先存储在电子设备中，获得障碍物的类别后，可以根据障碍物的类别在电子设备中调用对应的虚拟图像。
在本实施例中，获得图像中各障碍物的虚拟图像后可以将各个虚拟图像与无人机发送的图像叠加，获得叠加图像。在另一些实施例中，也可以仅将距离最小的数个障碍物的虚拟图像与无人机发送的图像叠加。如此可以避免叠加图像中虚拟图像过多，区分效果差，影响客户体验。
具体的，在103之前先确定各障碍物中障碍物距离最小的预设数量的障碍物，例如先获得各障碍物中障碍物距离最小的3个障碍物，再获得该预设数量的障碍物对应的虚拟图像。其中，预设数量例如3或4等。其中，距离最小的3个障碍物是指距离从小到大的排列中，排在前三位的障碍物。该实施例请参照图3，在图3所示的实施例中，所述方法包括：
101：获取无人机飞行环境中的障碍物数据和所述无人机拍摄的图像，所述障碍物数据由所述无人机基于所述图像获得，所述障碍物数据包括障碍物的物理位置以及所述障碍物对应的图像位置，所述障碍物的物理位置包括所述障碍物的距离和方位。
102：对所述图像进行识别，获得所述障碍物的类别。
102b：确定各障碍物中障碍物距离最小的预设数量的障碍物。
103b：基于所述障碍物的类别获得所述预设数量的障碍物对应的虚拟图像。
104：将所述虚拟图像叠加在所述图像上的所述障碍物对应的所述图像位置，以获得叠加图像。
105：在所述电子设备的显示屏幕上显示所述叠加图像，以使用户根据所述叠加图像控制所述无人机避障。
104：将所述虚拟图像叠加在所述图像上的所述障碍物对应的所述图像位置，以获得叠加图像。
将障碍物的虚拟图像叠加在所述图像中该障碍物对应的图像位置，获得二者的叠加图像。
其中，在一些实施例中，可以利用增强现实技术(Augmented Reality，AR)将障碍物的虚拟图像叠加在所述图像中该障碍物对应的图像位置，以获得该叠加图像。
需要说明的是，无人机进行避障是一段持续的过程，在此过程中，图像获 取装置将不断的获取无人机周围环境的图像，无人机基于所述图像获得图像中各障碍物的物理位置以及障碍物对应的图像位置，然后将所述图像以及障碍物数据传递至电子设备。电子设备针对所述图像进行识别，获得该图像中障碍物的类别，并根据障碍物的类别获得障碍物对应的虚拟图像，然后将该虚拟图像与对应的无人机发送的图像进行叠加，获得叠加图像。由于无人机不断的获取周围环境的图像，电子设备端也将不断的获得叠加图像，将该叠加图像在电子设备端进行显示，可以动态的显示无人机周围障碍物的情况。通过将障碍物的虚拟图像和无人机获得的真实图像相叠加呈现的方式，能直观的向无人机操纵者显示障碍物，用户体验好。
105：在所述电子设备的显示屏幕上显示所述叠加图像，以使用户根据所述叠加图像控制所述无人机避障。
无人机操纵者通过观看电子设备的显示屏幕，能直观的观测到障碍物。在另一些实施例中，为进一步增强用户体验，可以通过语音的方式根据障碍物的类别和障碍物的物理位置向用户进行提示，例如障碍物类别为树，障碍物距离为5m，则可以语音提示“前方五米有大树”等。通过显示叠加图像和语音提示相结合的方式，可以使无人机操纵者能同时“看”和“听”障碍物的状态，从而更易了解障碍物的状态。
无人机获得障碍物的物理位置后可以利用其自身的避障系统进行避障操作。但在某些应用场合，无人机会关闭避障开关，从而使其自身的避障系统不工作。在这种状态下，无人机的飞行存在一定的风险性。因此，在一些实施例中，为降低无人机飞行的风险性，可以利用电子设备辅助无人机进行避障。在该实施例中，所述方法还包括：
判断所述障碍物距离是否小于预设安全距离阈值；
如果所述障碍物距离小于预设安全距离阈值，获取所述无人机的避障开关状态以及飞行速度；
如果所述避障开关关闭且所述飞行速度大于预设飞行速度阈值，则发送避障指令给所述无人机和/或进行危险提示。
当各障碍物的距离中，有障碍物距离小于预设安全距离阈值时，则获得无人机的避障开关状态和飞行速度，如果避障开关处于关闭状态且飞行速度较大大于预设飞行速度阈值，则电子设备采取避障措施以辅助无人机躲避障碍物。具体的，例如发送控制指令令无人机暂停，或者发送控制指令令无人机打开避障开关，或者直接发出语音提示以提醒无人机操纵者。其中，预设安全距离阈值和预设飞行速度阈值等可以结合应用情况例如结合无人机的性能等设定。
相应的，如图4所示，本发明实施例还提供了一种避障装置，所述装置可以用于图1a或图1b所示的电子设备，避障装置400包括：
图像以及障碍物数据获取模块401，用于获取无人机飞行环境中的障碍物数据和所述无人机拍摄的图像，所述障碍物数据由所述无人机基于所述图像获 得，所述障碍物数据包括障碍物的物理位置以及所述障碍物对应的图像位置，所述障碍物的物理位置包括所述障碍物的距离和方位；
识别模块402，用于对所述图像进行识别，获得所述障碍物的类别；
虚拟图像获取模块403，用于基于所述障碍物的类别获得所述障碍物的虚拟图像；
叠加模块404，用于将所述虚拟图像叠加在所述图像上的所述障碍物对应的所述图像位置，以获得叠加图像；
显示模块405，用于在所述电子设备的显示屏幕上显示所述叠加图像，以使用户根据所述叠加图像控制所述无人机避障。
本发明实施例通过电子设备获取无人机拍摄的图像和障碍物数据，其中障碍物数据为无人机基于所述图像获得。电子设备对图像进行识别，获得障碍物的类别，并根据障碍物的类别获得障碍物的虚拟图像，将虚拟图像叠加在障碍物在图像上的相应图像位置，获得叠加图像，然后在电子设备的显示屏幕上显示所述叠加图像。通过将障碍物的虚拟图像和无人机获得的真实图像相叠加呈现的方式，能直观的向无人机操纵者显示障碍物，用户体验好。
在其中一些实施例中，叠加模块404具体用于：
利用增强现实技术将所述虚拟图像叠加在所述图像上，以获得所述叠加图像。
在另一些实施例中，请参照图5，避障装置400还包括：
语音提示模406，用于根据所述障碍物的类别和所述障碍物的物理位置进行语音提示，以使所述用户知晓所述障碍物的类别和物理位置。
在另一些实施例中，请参照图5，避障装置400还包括：
避障辅助模块407，用于判断所述障碍物距离是否小于预设安全距离阈值；如果所述障碍物距离小于预设安全距离阈值，获取所述无人机的避障开关状态以及飞行速度；如果所述避障开关关闭且所述飞行速度大于预设飞行速度阈值，则发送避障指令给所述无人机和/或进行危险提示。
在另一些实施例中，请参照图5，避障装置400还包括：
最近障碍物确定模块408，用于确定各障碍物中障碍物距离最小的预设数量的障碍物；
虚拟图像获取模块403具体用于：
基于所述障碍物的类别获得所述预设数量的障碍物对应的虚拟图像。
需要说明的是，上述装置可执行本申请实施例所提供的方法，具备执行方法相应的功能模块和有益效果。未在装置实施例中详尽描述的技术细节，可参见本申请实施例所提供的方法。
图6是本发明电子设备200的一个实施例的硬件结构示意图，如图6所示，电子设备200包括：
一个或多个处理器201以及存储器202，图6中以一个处理器201为例。
处理器201和存储器202可以通过总线或者其他方式连接，图6中以通过总线连接为例。
存储器202作为一种非易失性计算机可读存储介质，可用于存储非易失性软件程序、非易失性计算机可执行程序以及模块，如本申请实施例中的避障方法对应的程序指令/模块(例如，附图4所示的图像以及障碍物数据获取模块401、识别模块402、虚拟图像获取模块403、叠加模块404和显示模块405)。处理器201通过运行存储在存储器202中的非易失性软件程序、指令以及模块，从而执行电子设备的各种功能应用以及数据处理，即实现上述方法实施例的避障方法。
存储器202可以包括存储程序区和存储数据区，其中，存储程序区可存储操作系统、至少一个功能所需要的应用程序；存储数据区可存储根据控制器的使用所创建的数据等。此外，存储器202可以包括高速随机存取存储器，还可以包括非易失性存储器，例如至少一个磁盘存储器件、闪存器件、或其他非易失性固态存储器件。在一些实施例中，存储器202可选包括相对于处理器201远程设置的存储器，这些远程存储器可以通过网络连接至电子设备。上述网络的实例包括但不限于互联网、企业内部网、局域网、移动通信网及其组合。
所述一个或者多个模块存储在所述存储器202中，当被所述一个或者多个处理器201执行时，执行上述任意方法实施例中的避障方法，例如，执行以上描述的图2中的方法步骤101至步骤105、图3中的方法步骤101至步骤105；实现图4中的模块401-405、图5中的模块401-408的功能。
上述产品可执行本申请实施例所提供的方法，具备执行方法相应的功能模块和有益效果。未在本实施例中详尽描述的技术细节，可参见本申请实施例所提供的方法。
本申请实施例提供了一种非易失性计算机可读存储介质，所述计算机可读存储介质存储有计算机可执行指令，该计算机可执行指令被一个或多个处理器执行，执行以上描述的图2中的方法步骤101至步骤105、图3中的方法步骤101至步骤105；实现图4中的模块401-405、图5中的模块401-408的功能。
以上所描述的装置实施例仅仅是示意性的，其中所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部模块来实现本实施例方案的目的。
通过以上的实施例的描述，本领域普通技术人员可以清楚地了解到各实施例可借助软件加通用硬件平台的方式来实现，当然也可以通过硬件。本领域普通技术人员可以理解实现上述实施例方法中的全部或部分流程是可以通过计算机程序来指令相关的硬件来完成，所述的程序可存储于一计算机可读取存储 介质中，该程序在执行时，可包括如上述各方法的实施例的流程。其中，所述的存储介质可为磁碟、光盘、只读存储记忆体(Read-Only Memory,ROM)或随机存储记忆体(RandomAccessMemory,RAM)等。
最后应说明的是：以上实施例仅用以说明本发明的技术方案，而非对其限制；在本发明的思路下，以上实施例或者不同实施例中的技术特征之间也可以进行组合，步骤可以以任意顺序实现，并存在如上所述的本发明的不同方面的许多其它变化，为了简明，它们没有在细节中提供；尽管参照前述实施例对本发明进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述各实施例所记载的技术方案进行修改，或者对其中部分技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本发明各实施例技术方案的范围。

Claims (12)

  1. 一种避障方法，所述方法用于电子设备，其特征在于，所述方法包括：
    获取无人机飞行环境中的障碍物数据和所述无人机拍摄的图像，其中，所述障碍物数据由所述无人机基于所述图像获得，所述障碍物数据包括障碍物的物理位置以及所述障碍物对应的图像位置，所述障碍物的物理位置包括所述障碍物的距离和方位；
    对所述图像进行识别，以获得所述障碍物的类别；
    基于所述障碍物的类别，获得所述障碍物的虚拟图像；
    将所述虚拟图像叠加在所述图像上的所述障碍物对应的所述图像位置，以获得叠加图像；
    在所述电子设备的显示屏幕上显示所述叠加图像，以使用户根据所述叠加图像控制所述无人机避障。
  2. 根据权利要求1所述的方法，其特征在于，所述将所述虚拟图像叠加在所述图像上的所述障碍物对应的所述图像位置，以获得叠加图像，包括：
    利用增强现实技术将所述虚拟图像叠加在所述图像上，以获得所述叠加图像。
  3. 根据权利要求1或2所述的方法，其特征在于，所述方法还包括：
    根据所述障碍物的类别和所述障碍物的物理位置进行语音提示，以使所述用户知晓所述障碍物的类别和物理位置。
  4. 根据权利要求1-3任意一项所述的方法，其特征在于，所述方法还包括：
    判断所述障碍物距离是否小于预设安全距离阈值；
    如果所述障碍物距离小于所述预设安全距离阈值，获取所述无人机的避障开关状态以及飞行速度；
    如果所述避障开关关闭且所述飞行速度大于预设飞行速度阈值，则发送避障指令给所述无人机和/或进行危险提示。
  5. 根据权利要求1-4任意一项所述的方法，其特征在于，所述方法还包括：
    确定各障碍物中障碍物距离最小的预设数量的障碍物；则，
    所述基于所述障碍物的类别，获得所述障碍物的虚拟图像，包括：
    基于所述障碍物的类别，获得所述预设数量的障碍物对应的虚拟图像。
  6. 一种避障装置，所述装置用于电子设备，其特征在于，所述装置包括：
    图像以及障碍物数据获取模块，用于获取无人机飞行环境中的障碍物数据和所述无人机拍摄的图像，其中，所述障碍物数据由所述无人机基于所述图像获得，所述障碍物数据包括障碍物的物理位置以及所述障碍物对应的图像位置，所述障碍物的物理位置包括所述障碍物的距离和方位；
    识别模块，用于对所述图像进行识别，以获得所述障碍物的类别；
    虚拟图像获取模块，用于基于所述障碍物的类别，获得所述障碍物的虚拟图像；
    叠加模块，用于将所述虚拟图像叠加在所述图像上的所述障碍物对应的所述图像位置，以获得叠加图像；
    显示模块，用于在所述电子设备的显示屏幕上显示所述叠加图像，以使用户根据所述叠加图像控制所述无人机避障。
  7. 根据权利要求6所述的装置，其特征在于，所述叠加模块具体用于：
    利用增强现实技术将所述虚拟图像叠加在所述图像上，以获得所述叠加图像。
  8. 根据权利要求6或7所述的装置，其特征在于，所述装置还包括：
    语音提示模块，用于根据所述障碍物的类别和所述障碍物的物理位置进行语音提示，以使所述用户知晓所述障碍物的类别和物理位置。
  9. 根据权利要求6-8任意一项所述的装置，其特征在于，所述装置还包括：
    避障辅助模块，用于判断所述障碍物距离是否小于预设安全距离阈值；如果所述障碍物距离小于所述预设安全距离阈值，获取所述无人机的避障开关状态以及飞行速度；如果所述避障开关关闭且所述飞行速度大于预设飞行速度阈值，则发送避障指令给所述无人机和/或进行危险提示。
  10. 根据权利要求6-9任意一项所述的装置，其特征在于，所述装置还包括：
    最近障碍物确定模块，用于确定各障碍物中障碍物距离最小的预设数量的障碍物；
    所述虚拟图像获取模块具体用于：
    基于所述障碍物的类别，获得所述预设数量的障碍物对应的虚拟图像。
  11. 一种电子设备，其特征在于，所述电子设备包括：
    至少一个处理器；以及，
    与所述至少一个处理器通信连接的存储器；其中，
    所述存储器存储有可被所述至少一个处理器执行的指令，所述指令被所述至少一个处理器执行，以使所述至少一个处理器能够执行权利要求1-5任一项所述的方法。
  12. 一种非易失性计算机可读存储介质，其特征在于，所述计算机可读存储介质存储有计算机可执行指令，当所述计算机可执行指令被电子设备执行时，使所述电子设备执行如权利要求1-5任一项所述的方法。
PCT/CN2020/094764 2019-06-06 2020-06-05 一种避障方法、装置和电子设备 Ceased WO2020244649A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US17/457,306 US20220091608A1 (en) 2019-06-06 2021-12-02 Method, apparatus, and electronic device for obstacle avoidance

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910490281.6A CN110244760A (zh) 2019-06-06 2019-06-06 一种避障方法、装置和电子设备
CN201910490281.6 2019-06-06

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/457,306 Continuation US20220091608A1 (en) 2019-06-06 2021-12-02 Method, apparatus, and electronic device for obstacle avoidance

Publications (1)

Publication Number Publication Date
WO2020244649A1 true WO2020244649A1 (zh) 2020-12-10

Family

ID=67886344

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/094764 Ceased WO2020244649A1 (zh) 2019-06-06 2020-06-05 一种避障方法、装置和电子设备

Country Status (3)

Country Link
US (1) US20220091608A1 (zh)
CN (1) CN110244760A (zh)
WO (1) WO2020244649A1 (zh)

Families Citing this family (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110244760A (zh) * 2019-06-06 2019-09-17 深圳市道通智能航空技术有限公司 一种避障方法、装置和电子设备
CN111142150A (zh) * 2020-01-06 2020-05-12 中国石油化工股份有限公司 地震勘探自动智能避障设计方法
FR3107359B1 (fr) * 2020-02-18 2022-02-25 Thales Sa Procede et dispositif de determination d'obstacles d'altitude
CN111611869B (zh) * 2020-04-25 2021-06-01 哈尔滨理工大学 一种基于串行深度神经网络的端到端单目视觉避障方法
CN111650953B (zh) * 2020-06-09 2024-04-16 浙江商汤科技开发有限公司 飞行器避障处理方法、装置、电子设备及存储介质
CN111814720B (zh) * 2020-07-17 2022-06-17 电子科技大学 一种基于无人机视觉的机场跑道异物检测与分类方法
CN111814721B (zh) * 2020-07-17 2022-05-24 电子科技大学 基于无人机高低空联合扫描的机场跑道异物检测分类方法
CN113795803B (zh) * 2020-08-17 2024-05-14 深圳市大疆创新科技有限公司 无人飞行器的飞行辅助方法、设备、芯片、系统及介质
CN112085960A (zh) * 2020-09-21 2020-12-15 北京百度网讯科技有限公司 车路协同信息处理方法、装置、设备及自动驾驶车辆
CN112486208A (zh) * 2020-12-22 2021-03-12 安徽配隆天环保科技有限公司 一种无人机超声红外避障系统
EP4273656B1 (en) * 2021-02-19 2025-01-29 Anarky Labs Oy Apparatus, method and software for assisting an operator in flying a drone using a remote controller and ar glasses
CN113311857A (zh) * 2021-04-29 2021-08-27 重庆交通大学 一种基于无人机的环境感知与避障系统及方法
WO2023184487A1 (zh) * 2022-04-01 2023-10-05 深圳市大疆创新科技有限公司 无人机避障方法、装置、无人机、遥控设备和存储介质
CN115752468A (zh) * 2022-11-15 2023-03-07 沈阳工程学院 一种基于手眼协调的无人机避障方法
CN116061810A (zh) * 2023-01-03 2023-05-05 深圳松鼠机器人科技有限公司 一种基于载具的交互显示系统
CN116039512A (zh) * 2023-01-03 2023-05-02 深圳松鼠机器人科技有限公司 虚拟内容的显示方法、终端设备及存储介质

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070093945A1 (en) * 2005-10-20 2007-04-26 Grzywna Jason W System and method for onboard vision processing
CN106126042A (zh) * 2016-07-01 2016-11-16 京东方科技集团股份有限公司 一种环境提醒方法和系统
CN107077145A (zh) * 2016-09-09 2017-08-18 深圳市大疆创新科技有限公司 显示无人飞行器的障碍检测的方法和系统
CN107526443A (zh) * 2017-09-29 2017-12-29 北京金山安全软件有限公司 一种增强现实方法、装置、系统、电子设备及存储介质
CN108521808A (zh) * 2017-10-31 2018-09-11 深圳市大疆创新科技有限公司 一种障碍信息显示方法、显示装置、无人机及系统
CN108521807A (zh) * 2017-04-27 2018-09-11 深圳市大疆创新科技有限公司 无人机的控制方法、设备及障碍物的提示方法、设备
CN108629842A (zh) * 2017-03-16 2018-10-09 亮风台（上海）信息科技有限公司 一种无人驾驶设备运动信息提供及运动控制方法与设备
CN110244760A (zh) * 2019-06-06 2019-09-17 深圳市道通智能航空技术有限公司 一种避障方法、装置和电子设备

Family Cites Families (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105025272A (zh) * 2015-07-28 2015-11-04 深圳乐行天下科技有限公司 一种机器人及其混合视频流生成方法
EP3485465A1 (en) * 2016-09-23 2019-05-22 Apple Inc. Augmented reality display
IL250253B (en) * 2017-01-24 2021-10-31 Arbe Robotics Ltd A method for separating targets and echoes from noise, in radar signals
US11074827B2 (en) * 2017-08-25 2021-07-27 Aurora Flight Sciences Corporation Virtual reality system for aerial vehicle
WO2019044536A1 (ja) * 2017-08-31 2019-03-07 ソニー株式会社 情報処理装置、情報処理方法、プログラム、および移動体
CN107818333B (zh) * 2017-09-29 2020-04-07 爱极智（苏州）机器人科技有限公司 基于深度信念网络的机器人避障行为学习和目标搜索方法
CN108873931A (zh) * 2018-06-05 2018-11-23 北京理工雷科电子信息技术有限公司 一种基于主观与客观结合的无人机视觉防撞方法
CN109733283B (zh) * 2019-01-09 2020-07-31 吉林大学 基于ar的被遮挡障碍物识别预警系统及识别预警方法
JP7222285B2 (ja) * 2019-03-20 2023-02-15 株式会社リコー 表示制御装置、表示装置、表示システム、移動体、プログラム、画像生成方法

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070093945A1 (en) * 2005-10-20 2007-04-26 Grzywna Jason W System and method for onboard vision processing
CN106126042A (zh) * 2016-07-01 2016-11-16 京东方科技集团股份有限公司 一种环境提醒方法和系统
CN107077145A (zh) * 2016-09-09 2017-08-18 深圳市大疆创新科技有限公司 显示无人飞行器的障碍检测的方法和系统
CN108629842A (zh) * 2017-03-16 2018-10-09 亮风台（上海）信息科技有限公司 一种无人驾驶设备运动信息提供及运动控制方法与设备
CN108521807A (zh) * 2017-04-27 2018-09-11 深圳市大疆创新科技有限公司 无人机的控制方法、设备及障碍物的提示方法、设备
CN107526443A (zh) * 2017-09-29 2017-12-29 北京金山安全软件有限公司 一种增强现实方法、装置、系统、电子设备及存储介质
CN108521808A (zh) * 2017-10-31 2018-09-11 深圳市大疆创新科技有限公司 一种障碍信息显示方法、显示装置、无人机及系统
CN110244760A (zh) * 2019-06-06 2019-09-17 深圳市道通智能航空技术有限公司 一种避障方法、装置和电子设备

Also Published As

Publication number Publication date
US20220091608A1 (en) 2022-03-24
CN110244760A (zh) 2019-09-17

Similar Documents

Publication Publication Date Title
WO2020244649A1 (zh) 一种避障方法、装置和电子设备
US11797028B2 (en) Unmanned aerial vehicle control method and device and obstacle notification method and device
US11295458B2 (en) Object tracking by an unmanned aerial vehicle using visual sensors
US20190004543A1 (en) Detecting optical discrepancies in captured images
CN105912980A (zh) 无人机以及无人机系统
US20220137647A1 (en) System and method for operating a movable object based on human body indications
CN108122553A (zh) 一种无人机控制方法、装置、遥控设备和无人机系统
CN113566825B (zh) 基于视觉的无人机导航方法、系统及存储介质
US11961407B2 (en) Methods and associated systems for managing 3D flight paths
CN104881039A (zh) 一种无人机返航的方法及系统
CN106444843A (zh) 无人机相对方位控制方法及装置
CN106292719B (zh) 地面站融合系统及地面站视频数据融合方法
US12130641B2 (en) Method, apparatus and unmanned aerial vehicle for processing depth map
CN108885469A (zh) 用于在跟踪系统中初始化目标物体的系统和方法
CN105589466A (zh) 无人飞行器的飞行控制装置及其飞行控制方法
US20240192705A1 (en) Automated Unmanned Aerial Vehicle Dock Verification And Landing
US20220342428A1 (en) Unmanned aerial vehicles
WO2018045976A1 (zh) 一种飞行器的飞行控制方法和飞行控制装置
CN110187720A (zh) 无人机导引方法、装置、系统、介质及电子设备
CN114384925A (zh) 一种车载无人机升降方法及配对方法
CN107077143A (zh) 一种无人机的线缆避障方法和系统及无人机
CN108700885A (zh) 一种飞行控制方法、遥控装置、遥控系统
US12222735B2 (en) Systems, methods and programs for continuously directing an unmanned vehicle to an environment agnostic destination marked by a user
CN117693722A (zh) 无人机的控制方法、控制装置、无人机及存储介质
CN118567383A (zh) 车辆脱困方法、电子设备及存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20819017

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20819017

Country of ref document: EP

Kind code of ref document: A1