WO2019061466A1

WO2019061466A1 - Flight control method, remote control device, and remote control system

Info

Publication number: WO2019061466A1
Application number: PCT/CN2017/104911
Authority: WO
Inventors: 封旭阳; 赵丛; 李思晋
Original assignee: 深圳市大疆创新科技有限公司
Priority date: 2017-09-30
Filing date: 2017-09-30
Publication date: 2019-04-04
Also published as: CN108700885A; CN108700885B

Abstract

Provided in an embodiment of the present invention are a flight control method, a remote control device, and a remote control system. The method comprises: detecting a movement track of a feature position region of a target object in a space, and determining a non-touch control movement on the basis of the movement track; generating a control command according to the non-touch control movement; and transmitting the control command to an aerial vehicle by means of a wireless link so as to control flight of the aerial vehicle. The invention provides a new manner of controlling an aerial vehicle, and enables a remote control device to have a higher level of intelligence.

Description

Flight control method, remote control device and remote control system

Technical field

The present invention relates to the field of electronic technologies, and in particular, to a flight control method, a remote control device, and a remote control system.

Background technique

With the continuous development of electronic technology, aircraft (such as drones, remote-controlled aircraft, aerial vehicles, etc.) have also become popular. People can carry cameras, spray devices, etc. through drones to achieve aerial photography, spraying and other tasks.

When maneuvering an aircraft, it is often necessary for the operator to manipulate the remote control device with both hands to achieve control of the aircraft. Taking the drone as an example, the operator can control the rocker on the remote control device by both hands, and shake the rocker up and down as needed to control the drone, and generate control according to the rocking direction and the rocking amplitude of the rocker. Command to control the flight of the aircraft.

How to conveniently control the aircraft has become a research hotspot.

Summary of the invention

The embodiment of the invention discloses a flight control method, a remote control device and a remote control system, which can enrich the control of the aircraft.

In a first aspect, an embodiment of the present invention discloses a flight control method, which is applied to a remote control device, where the remote control device is used to remotely control aircraft flight, and the method includes:

Detecting a movement trajectory of the feature position area on the target object in the space, and determining a levitation control action based on the movement trajectory;

Generating a control instruction according to the suspension control action;

The control command is transmitted to the aircraft over a wireless link to control the aircraft to fly.

In a second aspect, an embodiment of the present invention discloses a remote control device for remotely controlling aircraft flight, including: a memory and a processor;

The memory is configured to store program instructions;

The processor is configured to execute program instructions stored in the memory, when program instructions are executed, The processor is used to:

Generating a control instruction according to the suspension control action;

In a third aspect, an embodiment of the present invention discloses a remote control system, including:

At least one camera device and/or at least one sensor, the camera device comprising a red, green and blue camera device;

Aircraft

A remote control device as in the second aspect.

In the embodiment of the present invention, the remote control device can detect the movement trajectory of the feature position area on the target object in the space, determine the levitation control action, generate a control instruction according to the levitation control action, and finally use the wireless link to control the command. Sending to the aircraft and controlling the flight of the aircraft can accurately control the flight of the aircraft without the operator touching the remote control device, enriching the control mode of the aircraft and improving the intelligence of the remote control device.

DRAWINGS

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings to be used in the embodiments will be briefly described below. It is obvious that the drawings in the following description are only some embodiments of the present invention. Those skilled in the art can also obtain other drawings based on these drawings without paying for creative labor.

FIG. 1 is a schematic diagram of a scenario for flight control according to an embodiment of the present invention; FIG.

2 is a schematic diagram of another scenario for flight control according to an embodiment of the present invention;

3 is a schematic flow chart of a flight control method according to an embodiment of the present invention;

4 is a schematic flow chart of another flight control method according to an embodiment of the present invention;

FIG. 5 is a schematic structural diagram of a remote control device according to an embodiment of the present invention; FIG.

FIG. 6 is a schematic structural diagram of a remote control system according to an embodiment of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described in the following with reference to the accompanying drawings.

UAVs have a very wide range of uses. They are currently well used in aerial photography, agriculture, express transportation, disaster relief, surveying and mapping, news reporting, power inspection, disaster relief, etc., making drone technology a Popular emerging technologies. In the UAV technology, achieving precise control of the drone is an important research direction.

There are two ways to control a drone. The first way is to require the operator to manipulate the remote control device such as the joystick with both hands to achieve more precise control of the drone.

However, the above method requires an artificial contact with the remote control device, and the remote control device can recognize the operation of the operator, and the control method of the aircraft is limited.

The second way is that the operator performs a wave motion against the aircraft, and then the aircraft uses the camera device (such as a camera, a close-up binocular stereo module, etc.) provided on the aircraft to collect the depth information of the palm, and the depth information is used to identify the operation. The player's wave action then controls the aircraft to fly in accordance with the flight instructions represented by the wave action.

The second method described above can control the flight of the aircraft without a remote control device. However, the control can only be controlled by the action of the operator waving, the control method is relatively simple, and the operator's hand needs to be very close to the aircraft and in the same horizontal plane. The aircraft can take in the waved action and cannot control the flight of the aircraft remotely.

Therefore, how to remotely control the remote control device to fly, and to break the limitations of artificial contact with the remote control device, and improve the intelligence of the remote control device has become an urgent problem to be solved.

In order to solve the above technical problems, the present invention provides a flight control method, a remote control device, and a remote control system.

For a clearer description, please refer to FIG. 1 and FIG. 2 below, which are schematic diagrams of scenarios for flight control according to an embodiment of the present invention. In FIG. 1 and FIG. 2, the hand is the target object, and the knuckle point is the feature position area of the target object, but it should be understood that in the embodiment of the present invention, the hand and the knuckle joint point are only for the target object and the target. An example of the feature location area of the object, in other embodiments, the target object and the feature location area of the target object may also be other objects, which is not limited in this embodiment of the present invention.

It should also be noted that the execution body of 101-106 is a remote control device, and the remote control device can be used for Remotely control the flight of the aircraft. Specifically, the remote control device may be a wearable device, an augmented reality device, or the like. The wearable device may be, for example, a smart watch, a smart bracelet, smart glasses, etc., and the augmented reality device may be, for example, a head mounted display or the like. In FIG. 2, the remote control device is exemplified by a smart watch, but it should be understood that in other embodiments, the remote control device may be any of the above-mentioned remote control devices.

The embodiments of the present invention are specifically described below with reference to FIGS. 1 and 2.

In 101, the remote control device can acquire the depth of the detection area through a depth information collection device (for example, a binocular stereo vision module, a 3D time of flight (ToF) module, a depth sensor, etc.) disposed on the remote control device. information.

In one embodiment, as shown in FIG. 2, above the dial of the remote control device, an area between two broken lines may be the detection area, and the remote control device may acquire depth information in the detection area.

In one embodiment, the depth information may be determined from a depth map of the current frame.

In 102, the remote control device can detect whether there is a hand (ie, a target object) in the detection area by using the depth information, and if present, the opponent can perform rough positioning, and specifically, the palm can perform rough positioning.

In an embodiment, as shown in FIG. 2, the remote control device may use the surface of the current dial as a reference coordinate system, project the depth information of the current frame onto the reference coordinate system to generate a point cloud image, and then the point cloud image. The presence or absence of the palm is detected, and if present, the palm is roughly positioned and steps 103-106 are performed.

In one embodiment, if the palm is not detected, the remote control can enter a sleep mode. The remote control device can adjust the frequency of detecting the palm from the first frequency to the second frequency (the second frequency is less than the first frequency). If the remote control device receives a wake-up command (for example, shaking the remote control device, etc.), the sleep mode can be released and the second frequency can be adjusted back to the first frequency.

In 103, if the remote control device detects the palm, the knuckle point can be accurately positioned after the palm is roughly positioned. As shown in FIG. 2, a black dot on the tip of the finger can be used to indicate a feature position area (ie, a knuckle point) of the target object, and the remote control device can accurately position the knuckle point.

In one embodiment, the remote control device may extract a region of interest (ROI) in the point cloud image, and perform palm fitting according to the point cloud data provided by the point cloud image within the ROI range. Thereby precise positioning of each knuckle point.

At 104, the remote control device can perform a movement trajectory fit based on the positional movement of the knuckle joint point.

In an embodiment, the remote control device may further remove the positional deviation of the movement trajectory caused by the shaking of the remote control device according to the preset filter function and the posture information of the remote control device, to implement smoothing filtering processing, and obtain the knuckle point. The movement track.

In 105, the remote control device can determine a hovering control action according to the movement trajectory and generate a corresponding control command. For example, if the remote control device fits a horizontal to right movement trajectory, then the levitation control action is to control the aircraft to fly horizontally to the right. The control command is an instruction to control the aircraft to fly horizontally to the right.

At 106, the remote control device can transmit the control command to the drone via a wireless link (e.g., cellular mobile data network, Bluetooth, infrared, etc.) to cause the drone to fly in accordance with the indication of the control command.

It can be seen that the operator can control the flight of the aircraft without touching the remote control device, which breaks the limitation of the artificial contact remote control device and realizes the precise control of the aircraft, and the form of the remote control device is not limited to the traditional rocker. It can also be a wearable device, an augmented reality device, etc., which enriches the control mode of the aircraft and improves the intelligence of the remote control device.

For a clearer description, the flight control method described in the embodiment of the present invention will be described below. FIG. 3 is a schematic flowchart of a flight control method according to an embodiment of the present invention. The method shown in FIG. 3 may include:

S301. Detect a movement trajectory of a feature position area on the target object in the space, and determine a levitation control action based on the movement trajectory.

It should be noted that the execution body of the embodiment of the present invention may be a remote control device for remotely controlling aircraft flight.

In one embodiment, the remote control device is a wearable device or an augmented reality device.

In one embodiment, the wearable device is any one or more of a smart watch, smart glasses, and a smart bracelet; the augmented reality device is a head mounted display.

In an embodiment, the remote control device detects a movement trajectory of the feature location area on the target object in the space, and specifically includes: acquiring image feature information of the detection area; and determining the target object from the image feature information of the detection area. The movement track of the feature position area on the space.

It should be noted that the detection area may be an area that can be detected by the depth information collection device on the remote control device.

It should also be noted that the image feature information can be used to represent feature information in the scene image within the range of the detection region.

In an embodiment, as shown in FIG. 2, the remote control device may be a smart watch, and the detection area may be an area above the dial of the smart watch and between two broken lines, and the remote control device may collect the detection area. Image feature information.

In some feasible implementation manners, the remote control device may acquire an image in real time, acquire image feature information in the image, or acquire image feature information in the image according to a preset time, where each frame image is provided. The image feature information may correspond to image feature information provided by an image captured at a time of, for example, 12:30:10, and the corresponding time may be 12:30:10, and the current time may correspond to the current frame image. Image feature information.

The image feature information may be the image feature information in the current frame image, or may be the image feature information in the continuous multi-frame image, and the like, which is not limited in this embodiment of the present invention.

In an embodiment, the remote control device may acquire a current frame image of the detection area, extract image feature information in the current frame image, and detect a movement trajectory of the feature position area on the target object in the space according to the image feature information. .

In an embodiment, the remote control device, before determining the movement trajectory of the feature location area on the target object in the image from the image feature information of the detection area, further includes: detecting whether the detection area exists a target object; if there is a target object, the acquisition frequency when acquiring the image feature information of the detection area is adjusted to the first frequency.

In one embodiment, the target object is a hand, and the feature location area on the target object is an knuckle point on the hand.

In one embodiment, the target object may also be other biometric parts, such as eyes, mouth, head, etc., and the feature area on the target object may also be the pupil of the eye, the lip of the mouth, the hair of the head. Etc., the present invention does not impose any limitation on this.

In some possible implementations, the remote control device may determine, according to the image feature information, whether the target object exists after extracting the image feature information in the current frame image, and if present, acquire the image feature of the detection region. The acquisition frequency at the time of information is adjusted to the first frequency.

It should be noted that the first frequency may be, for example, a high acquisition frequency such as 50 Hz, 100 Hz, or the like. That is to say, if the remote control device detects the target object, the acquisition frequency of the remote control device at this time may be the first frequency, which may facilitate fitting the movement trajectory of the target object.

In one embodiment, if the target object is not present, the acquisition frequency when acquiring the image feature information of the detection area is adjusted to the second frequency.

It should be noted that the second frequency may be, for example, a lower acquisition frequency such as 5 Hz, 10 Hz, or the like. That is to say, if the remote control device detects that the target object does not exist, the acquisition frequency of the remote control device may be the second frequency, which may reduce the power consumption of the remote control device and increase the usage time of the remote control device.

In one embodiment, the remote control device is in a sleep mode at the second frequency, the method further comprising: if receiving the wake-up command, acquiring frequency of the image feature information of the detection region from The second frequency is adjusted to the first frequency; wherein the first frequency is higher than the second frequency.

It should also be noted that the wake-up command may be, for example, an operation command for recognizing a wrist by an inertial measurement unit. Alternatively, when the remote control device is a smart wristband or a smart watch, it is determined that the wake-up command is received by recognizing the action of raising the hand and the face of the dial facing up.

In some feasible implementation manners, if the remote control device receives the wake-up command, it may determine that the working mode (ie, the mode of detecting the target object) needs to be entered, and the acquiring frequency corresponding to the working mode may be the first frequency, and the remote control device may The acquisition frequency can be adjusted from the second frequency to the first frequency.

In an embodiment, determining the movement trajectory of the feature location area on the target object in the image from the image feature information of the detection area comprises: determining features on the target object according to image feature information of the detection area a location area; determining a movement trajectory of the feature location area in the space according to at least two frames of image feature information of the detection area.

It should be noted that the image feature information provided by the at least two frames of the detection area may be image feature information provided by two consecutive frames of images. In an embodiment, the remote control device determines the feature location area on the target object by using the image feature information provided by the current frame image, and then the root device can And according to the image feature information provided by the current frame image and the image feature information provided by the next frame image of the current frame image feature information, the movement trajectory of the feature location region in the space is determined.

It should also be noted that the moving track may refer to a position change track of the feature position area in space.

In an embodiment, the image feature information of the detection area is depth information of the detection area.

In some feasible implementation manners, the remote control device may first acquire a depth image in the detection area, and then determine depth information from the depth image, and according to the depth change indicated by the depth information, the contour movement of the target object The direction, size and other parameters determine the movement trajectory of the feature location area on the target object in space.

The remote control device may acquire image feature information in the detection area by using a binocular stereo vision module, a 3D TOF module, a depth sensor, or the like, and further detect whether the target object exists in the detection area.

In an embodiment, the determining the feature location area on the target object according to the depth information of the detection area comprises: performing projection processing on the reference coordinate system according to the depth information of the detection area, and generating on the reference coordinate system a point cloud image, the reference coordinate system is determined according to the remote control device; determining contour information of the target object according to the point cloud image; determining position information of the feature position region according to the contour information of the target object, A feature location area on the target object is determined.

It should be noted that the reference coordinate system may be a coordinate system centered on the remote control device. For example, the center point or other points of the surface of the remote control device may be taken as the origin of the reference coordinate system, and then the horizontal axis and the vertical axis of the reference coordinate system are established with the surface of the remote control device (for example, the dial of the smart watch). .

It should also be noted that the depth information may be determined from an image in which the distance (depth) value from each point in the detection area to the detection area is taken as a pixel value.

It should also be noted that when the light beam is irradiated onto the surface of the target object, the reflected light carries information such as the orientation and distance of the target object. If the light beam is scanned according to the preset trajectory, the reflection is recorded while scanning. Information, when the scan is very fine, you can get a lot of light spots, so A point cloud image can be formed. That is to say, the point cloud image can indicate information such as the orientation and distance of the target object.

In some feasible implementation manners, the remote control device may first determine depth information represented by the depth map of the detection area, then project the depth information into a reference coordinate system, and generate a point cloud image in the reference coordinate system, The point cloud image can represent information such as the orientation and distance of the target object, and the contour information of the target object can be determined according to the information such as the azimuth and the distance. The position of the point cloud data in the contour is dense, and can be regarded as the feature position on the target object. region.

In an embodiment, the determining the location information of the feature location area according to the profile information of the target object comprises: determining initial location information of the feature location area according to the profile information of the target object; filtering according to a preset The function and the posture information of the remote control device perform smoothing processing on the initial position information to obtain position information of the feature position area.

It should be noted that the location information of the feature location area may be used to indicate the location change of the feature location area itself.

It should also be noted that the initial location information of the feature location area may include information of the location movement of the feature location area due to the movement of the remote control device itself.

In some feasible implementation manners, the remote control device can remove the position of the feature location area caused by the movement of the remote control device by using a preset filter function and the attitude information of the remote control device measured by the inertial measurement unit. Move to obtain the location information of the feature location area.

In one embodiment, the detecting a movement trajectory of the feature location area on the target object in the space comprises: acquiring motion sensing data of the feature location area on the target object by using a sensor disposed on the feature location area; Obtaining a movement trajectory of the feature location area on the target object in the space according to the motion sensing data.

It should be noted that the sensor may be pre-set in the area of the feature, and the sensor may be, for example, a temperature sensor, an infrared sensor, or the like, which is not limited in the embodiment of the present invention.

It should also be noted that the sensor can be wirelessly connected to the remote control device, and the sensor transmits the mobile sensing data to the remote control device by wireless transmission.

In some possible embodiments, the operator can wear gloves, the sensor is provided at the knuckle point of the glove, and the operator can issue some control actions, and the sensor converts these control actions The motion sensing data is sent to the remote control device, and the remote control device can perform data fitting according to the motion sensing data to obtain a movement track of the operator's knuckle point in the space.

In an embodiment, the image feature information of the detection area is color information or infrared information of the detection area.

It should be noted that the color information may be, for example, color information in an image captured in the detection area, for example, the image is an RGB (Red Green Blue) image captured by the imaging device. The infrared information may be, for example, information represented by an image taken by the infrared camera.

In some possible implementations, the feature location area on the target object may be determined from image feature information of the detection area based on depth learning. After the feature position area on the target object is located, the movement track of the feature position area on the target object in the space may be obtained according to the position change of the feature position area on the target object.

It should also be noted that after the remote control device determines the movement trajectory of the feature position area on the target object in space, the levitation control action can be determined according to the movement trajectory.

The suspension control action may refer to a control action triggered by the operator when the remote control device is not touched. For example, the palm (target object) shown in FIG. 2 performs a control action in a space area above the smart watch (remote control device).

S302. Generate a control instruction according to the suspension control action.

It should be noted that the control command can be used to control the flight of the aircraft.

In some possible implementations, the suspension control action and the control command may have a corresponding relationship. Specifically, the correspondence may be a correspondence between a direction and an angle. For example, the hovering control action is a horizontal left motion, then the control command may be an instruction to control the aircraft horizontally to the left.

In one embodiment, the generating the control instruction according to the suspension control action comprises: obtaining a type of the suspension control action and a motion vector of the feature location area according to the determined contour information; according to the type of the suspension control action The motion vector of the feature location area generates a control instruction.

In one embodiment, the motion vector of the feature location area includes a motion direction and a motion amplitude of the feature location area determined according to at least two frame depth information.

It should be noted that the moving direction of the feature location area may be any direction, such as upper, lower, left, right, upper left, lower right, lower left, upper right, and the like, which is not limited in this embodiment of the present invention.

It should also be noted that the type of the suspension control action can be used to indicate the suspension control action. The direction may be, for example, a type that controls the upward direction, a type that controls the downward direction, a type that controls the leftward, a type that controls the rightward, and the like.

In one embodiment, the target object may be a palm, the feature location area of the target object is a joint point, and the remote control device determines the suspension control action after positioning the joint finger point, and then the remote control device The type of the levitation control action may be determined by the classifier, for example, to control the downward type, and the motion vector of each knuckle point may be calculated according to the depth information provided by the at least two frames of depth information, and the motion vector may include a motion direction. , the magnitude of the movement and other parameters. The remote control device can determine a control command that can be used to control the aircraft to fly downward in accordance with parameters in the motion vector.

S303. Send the control command to the aircraft through a wireless link to control the aircraft to fly.

It should be noted that the wireless link may be, for example, a cellular mobile data network, a wireless fidelity (WiFi), an infrared, a Bluetooth, etc., and the present invention does not impose any limitation.

In some possible implementations, the remote control device can transmit a control command to the aircraft via a wireless link, and after receiving the control command, the aircraft can fly according to the instruction of the control command. For example, the control command indicates that the flight is horizontal to the right, and the aircraft can fly to the right according to the level of the control command.

It can be seen that, in the embodiment of the present invention, the remote control device can detect the movement trajectory of the feature location area on the target object in the space, and determine the corresponding levitation control action, generate a control instruction according to the levitation control action, and finally pass the wireless link. Sending the control command to the aircraft can control the flight of the aircraft without the operator touching the remote control device, which breaks the limitation of the artificial contact with the remote control device, realizes the precise control of the aircraft, and enriches the control mode of the aircraft. Improves the intelligence of the remote control device.

Please refer to FIG. 4 , which is a schematic flowchart diagram of another flight control method according to an embodiment of the present invention. The method as shown in FIG. 4 may include:

S401. Detect a movement track of a feature position area on the target object in the space.

In an embodiment, the remote control device may acquire depth information of the detection area before detecting a movement trajectory of the feature position area on the target object in the space; according to the depth information of the detection area It is detected whether there is a target object; if there is a target object, a movement trajectory of the feature position area on the detection target object in space is performed.

In one embodiment, a virtual image is superimposed on the detection area.

It should be noted that the virtual image may be a non-realistic image generated by a virtual reality device, an augmented reality device, or the like.

S402. Generate and display the virtual image in the detection area according to location information of the target object.

In one embodiment, the virtual image may be superimposed and displayed in the detection area by the remote control device.

It should be noted that the remote control device can display the virtual image in any orientation directly below, directly above the target object, and the present invention does not impose any limitation.

In one embodiment, the remote control device may be an augmented reality device a, and the detection area may be an area directly in front of the line of sight of the augmented reality device a, and the augmented reality device a may detect the target object (eg, a palm) within the detection area. The virtual image is displayed directly below the palm, and the virtual image can be, for example, a virtual airplane.

In one embodiment, the virtual image may also be superimposed and displayed in the detection area by other devices.

In one embodiment, the remote control device is a smart watch, the operator can wear an augmented reality device b, the augmented reality device b can establish a connection with the smart watch, and then the remote device detects the target object, the enhancement The reality device b can project a virtual image on the detection area of the smart watch, such as a virtual aircraft.

S403. Control the virtual image movement according to the movement trajectory.

It should be noted that the movement trajectory of the feature position area of the target object can be used as the movement trajectory of the virtual image.

In one embodiment, the target object may be a palm, the feature location area of the target object may be a joint point, and the movement track fitted by the remote control device may mean that the joint point moves horizontally to the right, then the virtual image is also It can be moved to the right horizontally according to the movement trajectory.

In one embodiment, the virtual image may be a virtual aircraft, and the operator may “pinch” the virtual aircraft in the detection area, and the operator's knuckles point moves horizontally to the right, and the virtual aircraft can also follow the operation. The author's knuckle points move horizontally to the right. At this time, the remote control device can also obtain a suspension control action according to the movement trajectory of the knuckle joint point, and determine a corresponding control command, send the control command to the aircraft, and control the aircraft to fly according to the control instruction.

It can be seen that, in the embodiment of the present invention, the remote control device can display the virtual image in the detection area, and control the movement of the virtual image according to the movement trajectory of the feature position area on the target object in the space, thereby enhancing the interest in remote control of the aircraft. Sex, but also improve the intelligence of the remote control device.

Embodiments of the present invention provide a remote control device. FIG. 5 is a schematic structural diagram of a remote control device according to an embodiment of the present invention. The remote control device described in this embodiment includes:

Memory 501 and processor 502;

The memory 501 is configured to store program instructions;

The processor 502 is configured to execute the program instructions stored in the memory, when the program instructions are executed, to:

Generating a control instruction according to the suspension control action;

In an embodiment, the processor 502 is configured to: acquire depth information of the detection area before detecting a movement trajectory of the feature location area on the target object in the space; and detect whether the target exists according to the depth information of the detection area. An object; if there is a target object, performing a movement trajectory of the feature position area on the detection target object in the space.

In an embodiment, when the processor 502 detects a movement trajectory of a feature location area on the target object in the space, the processor 502 is specifically configured to:

Obtaining image feature information of the detection area;

A movement trajectory of the feature position area on the target object in the space is determined from the image feature information of the detection area.

In an embodiment, the processor 502 is further configured to: detect whether the target object exists in the detection area; if the target object exists, acquire frequency when acquiring image feature information of the detection area Adjust to the first frequency.

In an embodiment, the processor 502 is further configured to: if the target object does not exist, adjust an acquisition frequency when acquiring image feature information of the detection area to a second frequency.

In an embodiment, the remote control device is in a sleep mode at the second frequency, and the processor 502 is further configured to: when the wake-up instruction is received, the image feature information of the detection area is acquired The acquisition frequency is adjusted from the second frequency to the first frequency; wherein the first frequency is higher than the second frequency.

In an embodiment, when the processor 502 determines the movement trajectory of the feature location area on the target object in the space from the image feature information of the detection area, the processor 502 is specifically configured to: according to the image feature information of the detection area Determining a feature location area on the target object; determining a movement trajectory of the feature location area in the space according to the image feature information provided by the at least two frames of the detection area.

In an embodiment, when the processor 502 determines the feature location area on the target object according to the image feature information of the detection area, the processor 502 is specifically configured to: perform projection processing on the reference coordinate system according to the depth information of the detection area, and generate a point cloud image on the reference coordinate system, the reference coordinate system is determined according to the remote control device; determining contour information of the target object according to the point cloud image; determining a feature according to the contour information of the target object Position information of the location area to determine a feature location area on the target object.

In an embodiment, when the processor 502 determines the location information of the feature location area according to the profile information of the target object, the method is specifically configured to: determine initial location information of the feature location area according to the profile information of the target object. And performing smoothing processing on the initial position information according to the preset filter function and the posture information of the remote control device to obtain position information of the feature position area.

In an embodiment, when the processor 502 generates a control instruction according to the levitation control action, the method is specifically configured to: obtain a type of the levitation control action and the feature position according to the determined contour information. a motion vector of the region; generating a control instruction according to the type of the suspension control action and the motion vector of the feature location region.

In an embodiment, when the processor 502 detects a movement trajectory of the feature location area on the target object in the space, the processor 502 is configured to: acquire a feature location area on the target object by using a sensor disposed on the feature location area Moving sensing data; obtaining a moving trajectory of the feature location area on the target object in the space according to the motion sensing data.

In one embodiment, the image feature information of the detection area is color information or infrared information of the detection area; the processor 502 determines a feature position on the target object from image feature information of the detection area. When the trajectory of the region is in the space, the method is specifically: determining, according to the depth learning, the feature location region on the target object from the image feature information of the detection region; acquiring the feature location region of the target object in the space Move the track.

In one embodiment, a virtual image is superimposed and displayed in the detection area.

In an embodiment, the processor 502 is further configured to: generate and display the virtual image in the detection area according to location information of the target object.

In an embodiment, the processor 502 is further configured to: control the virtual image movement according to the movement trajectory.

Embodiments of the present invention provide a remote control system. FIG. 6 is a schematic structural diagram of a remote control system according to an embodiment of the present invention. As shown in FIG. 6, the remote control system includes: at least one camera device and/or at least one sensor, wherein the camera device includes a red, green and blue camera device; an aircraft; a remote control device.

The remote control device 601 is the remote control device disclosed in the foregoing embodiment of the present invention, and the principles and implementations are similar to the foregoing embodiments, and details are not described herein again.

The camera 603 can be disposed on the remote control device for capturing depth information of the detection area.

In an embodiment, the camera device 603 may include a red, green and blue camera device, and the red, green and blue camera device may acquire a color map of a feature location area of the target object, and the remote control device 601 may use the color map to target the target object. The location area is positioned and tracked to obtain a moving track.

In one embodiment, the sensor 604 can be disposed at a feature location area of the target object, such as a temperature sensor, a distance sensor, an infrared sensor, and the like.

In one embodiment, the sensor 604 can also be disposed on the remote control device 601 for acquiring depth information of the detection area. The sensor 604 can be, for example, a depth sensor or the like.

In particular, the remote control system can be applied to remotely controlled aircraft devices. The remote control device 601 can be configured to detect a movement trajectory of the feature position area on the target object in the space, and determine a levitation control action based on the movement trajectory, generate a control instruction according to the levitation control action, and use the remote link to control the command. Send to the aircraft to control the flight of the aircraft.

It should be noted that the remote control device 601 can be used to perform the flight control method shown in the foregoing method embodiment, and the specific implementation process can refer to the method embodiment, and details are not described herein.

It should be noted that, for the foregoing various method embodiments, for the sake of simple description, they are all expressed as a series of action combinations, but those skilled in the art should understand that the present invention is not limited by the described action sequence. Because certain steps may be performed in other sequences or concurrently in accordance with the present invention. In the following, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

A person skilled in the art can understand that all or part of the steps of the foregoing embodiments can be completed by a program to instruct related hardware. The program can be stored in a computer readable storage medium, and the storage medium can include: Flash disk, Read-Only Memory (ROM), Random Access Memory (RAM), disk or optical disk.

A flight control method, a remote controller and a remote control system provided by the embodiments of the present invention are described in detail. The principles and embodiments of the present invention are described in the following. The description of the above embodiments is only used for To help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in specific embodiments and application scopes. It should not be construed as limiting the invention.

Claims

A flight control method is characterized in that it is applied to a remote control device for remotely controlling aircraft flight, and the method includes:

Detecting a movement trajectory of the feature position area on the target object in the space, and determining a levitation control action based on the movement trajectory;

Generating a control instruction according to the suspension control action;

The control command is transmitted to the aircraft over a wireless link to control the aircraft to fly.
The method according to claim 1, wherein the detecting a movement trajectory of the feature position area on the target object in the space comprises:

Obtaining image feature information of the detection area;

A movement trajectory of the feature position area on the target object in the space is determined from the image feature information of the detection area.
The method of claim 2, wherein the determining, before determining the movement trajectory of the feature location area on the target object in the image from the image feature information of the detection area, further comprises:

Detecting whether the target object exists in the detection area;

If the target object exists, the acquisition frequency when acquiring the image feature information of the detection area is adjusted to the first frequency.
The method of claim 3, wherein the method further comprises:

If the target object does not exist, the acquisition frequency when acquiring the image feature information of the detection area is adjusted to the second frequency.
The method of claim 4, wherein the remote control device is in a sleep mode at the second frequency, the method further comprising:

If the wake-up instruction is received, the acquisition frequency when acquiring the image feature information of the detection area is adjusted from the second frequency to the first frequency;

Wherein the first frequency is higher than the second frequency.
The method according to claim 2, wherein the determining a movement trajectory of the feature location area on the target object in the space from the image feature information of the detection area comprises:

Determining a feature location area on the target object according to image feature information of the detection area;

And determining, according to the image feature information provided by the at least two frames of the detection area, a movement trajectory of the feature location area in the space.
The method according to claim 2, wherein the image feature information of the detection area is depth information of the detection area.
The method according to claim 7, wherein the determining the feature location area on the target object according to the image feature information of the detection area comprises:

Performing a projection process on the reference coordinate system according to the depth information of the detection area, and generating a point cloud image on the reference coordinate system, the reference coordinate system being determined according to the remote control device;

Determining contour information of the target object according to the point cloud image;

Determining location information of the feature location area according to the contour information of the target object to determine a feature location area on the target object.
The method according to claim 8, wherein the determining the location information of the feature location area according to the profile information of the target object comprises:

Determining initial position information of the feature location area according to the contour information of the target object;

The initial position information is smoothed and filtered according to the preset filter function and the posture information of the remote control device, and the position information of the feature position area is obtained.
The method according to claim 8 or 9, wherein the generating the control instruction according to the suspension control action comprises:

Obtaining a type of the suspension control action and a motion vector of the feature location area according to the determined contour information;

A control instruction is generated according to the type of the suspension control action and the motion vector of the feature position area.
The method according to claim 10, wherein the motion vector of the feature location area comprises: a motion direction and a motion amplitude of the feature location area determined according to at least two frame depth information.
The method according to claim 1, wherein the detecting a movement trajectory of the feature position area on the target object in the space comprises:

Acquiring movement sensing data of the feature location area on the target object by a sensor disposed on the feature location area;

Obtaining a movement trajectory of the feature location area on the target object in the space according to the motion sensing data.
The method according to claim 2, wherein the image feature information of the detection area is color information or infrared information of the detection area;

Determining a movement trajectory of the feature location area on the target object in the space from the image feature information of the detection area, including:

Determining a feature location area on the target object from image feature information of the detection area based on depth learning;

Obtaining a movement trajectory of the feature position area of the target object in the space.
The method of claim 1 wherein a virtual image is superimposed on said detection area.
The method of claim 14 further comprising:

The virtual image is generated and displayed in the detection area according to the location information of the target object.
The method of claim 14 or 15, further comprising:

The virtual image movement is controlled according to the movement trajectory.
The method of claim 1 wherein said remote control device is a wearable device or an augmented reality device.
The method according to claim 17, wherein the wearable device is any one or more of a smart watch, smart glasses, and a smart bracelet;

The augmented reality device is a head mounted display.
The method of claim 1 wherein said target object is a hand and said feature location area on said target object is an knuckle point on the hand.
A remote control device for remotely controlling aircraft flight, comprising: a memory and a processor;

The memory is configured to store program instructions;

The processor is configured to execute the program instructions stored by the memory, when the program instructions are executed, for:

Detecting a movement trajectory of the feature position area on the target object in the space, and determining a levitation control action based on the movement trajectory;

Generating a control instruction according to the suspension control action;

The control command is transmitted to the aircraft over a wireless link to control the aircraft to fly.
The remote control device according to claim 20, wherein when the processor detects a movement trajectory of the feature location area on the target object in the space, the method is specifically configured to:

Obtaining image feature information of the detection area;

A movement trajectory of the feature position area on the target object in the space is determined from the image feature information of the detection area.
The remote control device according to claim 21, wherein the processor is further configured to:

Detecting whether the target object exists in the detection area;

If the target object exists, the acquisition frequency when acquiring the image feature information of the detection area is adjusted to the first frequency.
The remote control device of claim 22, wherein the processor is further configured to:

If the target object does not exist, the acquisition frequency when acquiring the image feature information of the detection area is adjusted to the second frequency.
The remote control device of claim 23, wherein the remote control device is in a sleep mode at the second frequency, the processor is further configured to:

If the wake-up instruction is received, the acquisition frequency when acquiring the image feature information of the detection area is adjusted from the second frequency to the first frequency;

Wherein the first frequency is higher than the second frequency.
The remote control device according to claim 21, wherein the processor is configured to: when determining a movement trajectory of the feature location area on the target object in the space from the image feature information of the detection area, specifically:

Determining a feature location area on the target object according to image feature information of the detection area;

And determining, according to the image feature information provided by the at least two frames of the detection area, a movement trajectory of the feature location area in the space.
The remote control device according to claim 21, wherein the image feature information of the detection area is depth information of the detection area.
The remote control device according to claim 26, wherein when the processor determines the feature location area on the target object according to the image feature information of the detection area, the processor is specifically configured to:

Performing a projection process on the reference coordinate system according to the depth information of the detection area, and generating a point cloud image on the reference coordinate system, the reference coordinate system being determined according to the remote control device;

Determining contour information of the target object according to the point cloud image;

Determining location information of the feature location area according to the contour information of the target object to determine a feature location area on the target object.
The remote control device according to claim 27, wherein when the processor determines the location information of the feature location area according to the profile information of the target object, the method is specifically configured to:

Determining initial position information of the feature location area according to the contour information of the target object;

The initial position information is smoothed and filtered according to the preset filter function and the posture information of the remote control device, and the position information of the feature position area is obtained.
The remote control device according to claim 27 or 28, wherein when the processor generates a control command according to the levitation control action, the processor is specifically configured to:

Obtaining a type of the suspension control action and a motion vector of the feature location area according to the determined contour information;

A control instruction is generated according to the type of the suspension control action and the motion vector of the feature position area.
The remote control device according to claim 29, wherein the motion vector of the feature position region comprises: a motion direction and a motion amplitude of the feature position region determined according to at least two frame depth information.
The remote control device according to claim 20, wherein when the processor detects a movement trajectory of the feature location area on the target object in the space, the method is specifically configured to:

Acquiring movement sensing data of the feature location area on the target object by a sensor disposed on the feature location area;

Obtaining a movement trajectory of the feature location area on the target object in the space according to the motion sensing data.
The remote control device according to claim 19, wherein the image feature information of the detection area is color information or infrared information of the detection area;

When the processor determines the movement trajectory of the feature location area on the target object in the space from the image feature information of the detection area, the processor is specifically configured to:

Determining a feature location area on the target object from image feature information of the detection area based on depth learning;

Obtaining a movement trajectory of the feature position area of the target object in the space.
A remote control device according to claim 20, wherein a virtual image is superimposed and displayed in said detection area.
The remote control device of claim 33, wherein the processor is further configured to:

The virtual image is generated and displayed in the detection area according to the location information of the target object.
The remote control device according to claim 33 or 34, wherein the processor is further configured to:

The virtual image movement is controlled according to the movement trajectory.
A remote control device according to claim 20, wherein said remote control device is a wearable device or an augmented reality device.
The remote control device according to claim 36, wherein the wearable device is any one or more of a smart watch, smart glasses, and a smart bracelet;

The augmented reality device is a head mounted display.
The remote control device according to claim 20, wherein the target object is a hand, and the feature position area on the target object is an knuckle point of the hand.
A remote control system, comprising:

At least one camera device and/or at least one sensor, the camera device comprising a red, green and blue camera device;

Aircraft

A remote control device according to any of claims 1-19.