WO2018133589A1 - Aerial photography method, device, and unmanned aerial vehicle - Google Patents

Aerial photography method, device, and unmanned aerial vehicle Download PDF

Info

Publication number
WO2018133589A1
WO2018133589A1 PCT/CN2017/115877 CN2017115877W WO2018133589A1 WO 2018133589 A1 WO2018133589 A1 WO 2018133589A1 CN 2017115877 W CN2017115877 W CN 2017115877W WO 2018133589 A1 WO2018133589 A1 WO 2018133589A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
cameras
images
aerial
splicing
Prior art date
Application number
PCT/CN2017/115877
Other languages
French (fr)
Chinese (zh)
Inventor
胡华智
Original Assignee
亿航智能设备(广州)有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 亿航智能设备(广州)有限公司 filed Critical 亿航智能设备(广州)有限公司
Publication of WO2018133589A1 publication Critical patent/WO2018133589A1/en

Links

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20Image signal generators
    • H04N13/271Image signal generators wherein the generated image signals comprise depth maps or disparity maps
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00Television systems
    • H04N7/18Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
    • H04N7/183Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a single remote source
    • H04N7/185Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a single remote source from a mobile camera, e.g. for remote control

Definitions

  • the invention relates to the technical field of drones, in particular to an aerial photography method, a device and a drone.
  • the current aerial photography system is mainly composed of a drone and a remote control device.
  • the user controls the drone through a remote control device, and the drone transmits the captured image to the remote control device for viewing by the user through an aerial camera.
  • a virtual reality (VR) device such as VR glasses is started as a remote control device.
  • the drone transmits the captured image to the VR glasses in real time, and the user watches the captured image in real time through the VR glasses, and the user also controls the posture and shooting angle of the drone through the VR glasses.
  • VR virtual reality
  • VR glasses completely isolate the human eye from the outside, it provides an immersive experience for the user and improves the user experience to a certain extent.
  • the image transmitted by the drone to the VR glasses is a 2D image, the advantage of the VR glasses cannot be fully utilized, and the method gives the user an immersive experience.
  • the main object of the embodiments of the present invention is to provide an aerial photography method, device and drone, which aim to realize an immersive experience for the user when performing aerial photography through the drone. .
  • an aerial photography method is proposed on the one hand, and the method comprises the following steps:
  • the 3D image is sent out.
  • the image is a photo or video stream.
  • the splicing the two images collected by the two cameras into one 3D image includes:
  • the two video streams collected by the two cameras are respectively sampled into two preset resolution video streams, and the preset resolution is lower than the original resolution;
  • the two preset resolution video streams are spliced into one 3D video stream.
  • the two cameras are arranged side by side, and the splicing the two images collected by the two cameras into one 3D image includes:
  • the two images collected by the two cameras are spliced side by side to form a 3D image in a left and right format.
  • the two images collected by the two cameras are side-by-side spliced together: the image captured by the left camera is spliced to the left, and the image captured by the right camera is spliced to the right.
  • the method further includes: performing depth detection of the captured scene by using the 3D image to obtain depth information.
  • the sending the 3D image outward comprises: transmitting the 3D image to a head mounted virtual reality device.
  • an aerial camera device comprising:
  • An image acquisition module for collecting images through two cameras
  • An image processing module configured to splicing two images collected by the two cameras into one 3D image
  • An image sending module configured to send the 3D image outward.
  • the image is a photo or video stream.
  • the image processing module is configured to: separately sample two video streams collected by the two cameras into two preset resolution video streams, and the two The preset resolution video streams are spliced into a 3D video stream, wherein the preset resolution is lower than the original resolution.
  • the two cameras are arranged side by side, and the image processing module is configured to: splicing the two images collected by the two cameras side by side to obtain a 3D image in a left and right format.
  • the image processing module is configured to: splicing the image collected by the left camera to the left side, and splicing the image collected by the right camera to the right side.
  • the device further includes a depth detecting module, configured to: perform depth detection of the shooting scene by using the 3D image to obtain depth information.
  • a depth detecting module configured to: perform depth detection of the shooting scene by using the 3D image to obtain depth information.
  • the image sending module is configured to: send the 3D image to the head mounted virtual reality device.
  • the invention also proposes a drone comprising:
  • One or more processors are One or more processors;
  • One or more applications wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more applications being configured to use Perform the aforementioned aerial photography method.
  • an aerial photography method is proposed on the one hand, and the method comprises the following steps:
  • the 3D image is sent out.
  • the image is a photo or video stream.
  • the splicing the two images collected by the two cameras into one 3D image includes:
  • the two video streams collected by the two cameras are respectively sampled into two preset resolution video streams, and the preset resolution is lower than the original resolution;
  • the two preset resolution video streams are spliced into one 3D video stream.
  • the two cameras are arranged side by side, and the splicing the two images collected by the two cameras into one 3D image includes:
  • the two images collected by the two cameras are spliced side by side to form a 3D image in a left and right format.
  • the two images collected by the two cameras are side-by-side spliced together: the image captured by the left camera is spliced to the left, and the image captured by the right camera is spliced to the right.
  • the method further includes: performing depth detection of the captured scene by using the 3D image to obtain depth information.
  • the sending the 3D image outward comprises: transmitting the 3D image to a head mounted virtual reality device.
  • an aerial camera device comprising:
  • An image acquisition module for collecting images through two cameras
  • An image processing module configured to splicing two images collected by the two cameras into one 3D image
  • An image sending module configured to send the 3D image outward.
  • the image is a photo or video stream.
  • the image processing module is configured to: separately sample two video streams collected by the two cameras into two preset resolution video streams, and the two The preset resolution video streams are spliced into a 3D video stream, wherein the preset resolution is lower than the original resolution.
  • the two cameras are arranged side by side, and the image processing module is configured to: splicing the two images collected by the two cameras side by side to obtain a 3D image in a left and right format.
  • the image processing module is configured to: splicing the image collected by the left camera to the left side, and splicing the image collected by the right camera to the right side.
  • the device further includes a depth detecting module, configured to: perform depth detection of the shooting scene by using the 3D image to obtain depth information.
  • a depth detecting module configured to: perform depth detection of the shooting scene by using the 3D image to obtain depth information.
  • the image sending module is configured to: send the 3D image to the head mounted virtual reality device.
  • the invention also proposes a drone comprising:
  • One or more processors are One or more processors;
  • One or more applications wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more applications being configured to use Perform the aforementioned aerial photography method.
  • An aerial photography method provided by an embodiment of the present invention collects two images through two cameras, and splicing the collected two images into a 3D image and transmitting the same, so that the drone can provide a 3D image during the aerial photography process, and The user can view the realistic 3D images in real time, allowing the user to have an immersive feeling, realizing an immersive experience, and greatly improving the user's aerial photography experience.
  • the depth detection is also performed through the 3D image, so that the drone can simultaneously realize 3D aerial photography, obstacle avoidance, and tracking by using a set of binocular cameras (ie, two cameras).
  • a set of binocular cameras ie, two cameras.
  • FIG. 1 is a flow chart of an aerial photography method according to a first embodiment of the present invention
  • FIG. 2 is a flow chart of an aerial photography method according to a second embodiment of the present invention.
  • Figure 3 is a block diagram showing an aerial photographing apparatus of a third embodiment of the present invention.
  • Fig. 4 is a block diagram showing the aerial device of the fourth embodiment of the present invention.
  • first, second, and the like in the present invention are used for the purpose of description only, and are not to be construed as indicating or implying their relative importance or implicitly indicating the number of technical features indicated.
  • features defining “first” or “second” may include at least one of the features, either explicitly or implicitly.
  • the technical solutions between the various embodiments may be combined with each other, but must be based on the realization of those skilled in the art, and when the combination of the technical solutions is contradictory or impossible to implement, it should be considered that the combination of the technical solutions does not exist. It is also within the scope of protection required by the present invention.
  • the aerial photographing method and the aerial photographing device of the embodiment of the present invention are mainly applied to the drone, and can of course be applied to other aircrafts, which is not limited by the present invention.
  • the embodiment of the present invention is described in detail by applying to an unmanned aerial vehicle as an example.
  • an aerial photography method according to a first embodiment of the present invention is proposed.
  • the method includes the following steps:
  • the drone is provided with two cameras to form a set of binocular cameras.
  • the two cameras are preferably arranged side by side. Of course, it can also be staggered, that is, the two cameras are not on the same horizontal line.
  • the two cameras are separated by a certain distance. In theory, the larger the separation distance, the better.
  • the drone acquires images simultaneously (synchronously) through two cameras, and the acquired images may be photos or video streams.
  • the two images collected by the two cameras are spliced side by side, preferably, the image collected by the left camera is spliced to the left, and the image captured by the right camera is spliced to the right, and finally a 3D (3D) format is obtained. )image.
  • the two images collected by the two cameras may be spliced together side by side to obtain a 3D image of the top and bottom format.
  • the 3D image is a 3D photo or a 3D video stream.
  • the drone before performing image splicing, the drone first performs resolution reduction processing on the original image, and then splicing the reduced resolution image to reduce the final 3D image size, thereby avoiding excessive consumption during subsequent transmission.
  • the bandwidth resources thereby increasing the transmission speed and improving the real-time performance of image transmission.
  • the drone first samples the two video streams collected by the two cameras into two preset resolution video streams, and then splices the two preset resolution video streams into A 3D video stream in which the preset resolution is lower than the original resolution.
  • the two cameras of the drone respectively capture a video stream of 4K resolution, and the drone samples two video streams of 4K resolution into two video streams of 720P format, and the sampling method can adopt a general downsampling algorithm.
  • the four pixels are merged into one pixel; the sampled two 720P video streams are placed on the left side of the left camera and the right camera is placed on the right side while the frame synchronization is maintained.
  • the drone can save the original images captured by the two cameras in the local storage space. Further, before the saving, the original image is also compressed to save storage space, such as compressing the video stream into a H.265 format video file.
  • the drone transmits the obtained 3D image, for example, to a remote control device or a terminal device that establishes a wireless communication connection with the drone, such as a mobile phone, a tablet computer, or a head mounted virtual reality (VR) device. (such as VR glasses, VR helmets, etc.), etc., or uploaded to a server via a wireless communication network.
  • a remote control device or a terminal device that establishes a wireless communication connection with the drone, such as a mobile phone, a tablet computer, or a head mounted virtual reality (VR) device. (such as VR glasses, VR helmets, etc.), etc., or uploaded to a server via a wireless communication network.
  • VR virtual reality
  • the drone before transmitting the 3D image, the drone further performs compression processing on the 3D image to reduce the size of the 3D image, improve transmission efficiency, and realize real-time transmission. For example, a 3D video stream is compressed into a H.264 format video stream and then sent out.
  • the drone uses 3D images to perform depth detection of the captured scene, acquires depth information, and uses depth information to implement functions such as target ranging, face recognition, gesture recognition, target tracking, and the like, and can be combined with the drone.
  • Attitude information and depth information can be used to avoid obstacles (such as forward obstacle avoidance), so that a set of binocular cameras can simultaneously achieve various functions such as aerial photography, obstacle avoidance, tracking, and ranging.
  • depth detection is performed by using a 3D image, that is, depth detection is performed by using a difference (parallax) between left and right or upper and lower images in a 3D image (such as a 3D video stream).
  • Parallax is the difference in direction produced by observing the same target from two points with a certain distance, so there is parallax in the image obtained by binocular cameras (such as left camera and right camera) of the same target at different positions. The closer the target is to the camera, the larger the parallax in the image of the binocular camera. Therefore, the distance from the target to the camera, that is, the depth of the target, can be calculated according to the parallax size of the target in the two images obtained by the binocular camera.
  • Depth detection The image is divided into several effective areas, the target distance of each area is calculated in turn, and the distance and the area orientation are fed back to the flight control. The flight control can realize obstacle avoidance according to the distance and orientation of the front target.
  • a remote control device such as VR glasses
  • the pitch angle is greater than the preset pitch angle (can be set according to actual needs)
  • the drone prompts the user to disable the obstacle function and/or maintain the hover state.
  • the drone when the user selects a target that needs to be tracked, the drone will adjust the posture of itself and the pan/tilt to align with the selected target. Because the target tracking accuracy based on depth information is more accurate than the previous method of planar vision, it can realize the tracking function with practical application value for the drone.
  • the drone when the user triggers the photographing instruction, such as waving a wave, a two-hand frame, and the like in front of the remote control device (such as VR glasses), the drone takes two full-resolution photos through two cameras. Store locally and use the two photos you took to stitch into a 3D photo.
  • the photographing instruction such as waving a wave, a two-hand frame, and the like in front of the remote control device (such as VR glasses)
  • the drone takes two full-resolution photos through two cameras. Store locally and use the two photos you took to stitch into a 3D photo.
  • the drone can also perform image quality improvement through the respective screens of the two cameras, such as performing denoising processing, background blurring processing, and the like.
  • the two photos taken by the binocular camera can be matched by the feature points to find a completely overlapping area, and the picture in the area is equivalent to two shots.
  • Overlay processing (such as the simplest weighted average) after multiple shots of the same picture can effectively reduce noise.
  • the foreground and the back view of the picture can be distinguished, and the back scene can be blurred by a fuzzy algorithm (such as the simplest Gaussian blur filter), thereby forming a background blur effect.
  • an aerial photography method according to a second embodiment of the present invention is proposed.
  • the method includes the following steps:
  • the binocular cameras of the drone are arranged side by side, and the drone simultaneously collects the video stream through the binocular camera.
  • the two video streams collected by the two cameras are respectively sampled into two preset resolution video streams.
  • the two preset resolution video streams are spliced side by side to obtain a left and right format 3D video stream.
  • the drone firstly samples the two video streams collected by the binocular camera into two preset resolution video streams, and then splicing the two preset resolution video streams side by side, preferably, The video stream collected by the left camera is spliced to the left, and the video stream collected by the right camera is spliced to the right, and finally a 3D video stream of the left and right format is obtained, wherein the preset resolution is lower than the original resolution.
  • the two cameras of the drone each shoot a 4K resolution video stream, and the drone first samples two 4K resolution video streams into two 720P format video streams, and then two 720P format videos.
  • the stream is spliced left and right (or up and down) into a 3D video stream with a resolution of 2560*720 in the left and right format (or top and bottom format).
  • the drone also saves the 4K resolution video stream captured by the two cameras in the local storage space. Further, before the saving, the original video stream is also compressed to save storage space, such as compressing the 4K resolution video stream into a H.265 format video file.
  • the 3D video stream is compressed and sent to the VR glasses.
  • the drone compresses the 3D video stream into a video stream of the H.264 format, and then transmits the video stream to the VR glasses for real-time transmission.
  • the VR glasses After the VR glasses receive the 3D video stream and play it immediately, the user can watch the 3D video captured by the drone in real time, making the picture more realistic, giving the user an immersive feeling and greatly improving the user experience.
  • the user can control the flight attitude and shooting angle of the drone through the VR glasses.
  • the drone When the user triggers the camera command, such as waving a wave or a two-hand frame in front of the VR glasses, the drone shoots two through two cameras. Full-resolution photos are stored locally, and the two photos taken are stitched into a 3D photo and sent back to the VR glasses, allowing users to view the captured 3D photos in real time. Further, the drone can also perform image quality improvement through the respective screens of the two cameras, such as performing denoising processing, background blurring processing, and the like.
  • S25 Perform depth detection of the shooting scene by using the 3D video stream to obtain depth information.
  • the drone also uses the 3D video stream to perform depth detection of the shooting scene to acquire depth information.
  • the depth detection is performed by using a 3D video stream, that is, using the difference (parallax) between the left and right video streams in the 3D video stream to implement depth detection.
  • the depth information is used to implement obstacle avoidance (such as forward obstacle avoidance), face recognition, gesture recognition, target tracking, etc., and the target distance measurement can be realized by combining the attitude information and the depth information of the drone, and the specific implementation is realized.
  • obstacle avoidance such as forward obstacle avoidance
  • face recognition such as face recognition
  • gesture recognition such as target tracking
  • target distance measurement can be realized by combining the attitude information and the depth information of the drone, and the specific implementation is realized.
  • the process is the same as the prior art and will not be described here.
  • the two video streams captured by the binocular camera are spliced into one 3D video stream, and the 3D video stream is transmitted to the VR glasses in real time for the user to view in real time, and the depth detection is performed through the 3D video stream, so that the drone utilizes one.
  • the group binocular camera can realize 3D aerial photography, obstacle avoidance, tracking, ranging and other functions at the same time, without using two sets of binocular cameras (ie four cameras) to achieve 3D shooting and depth detection respectively, thus lowering Costs enable multiple functions.
  • the device includes an image acquisition module, an image processing module, and an image transmission module, wherein:
  • Image acquisition module used to capture images through two cameras.
  • the drone is provided with two cameras to form a set of binocular cameras.
  • the two cameras are preferably arranged side by side. Of course, it can also be staggered, that is, the two cameras are not on the same horizontal line.
  • the two cameras are separated by a certain distance. In theory, the larger the separation distance, the better.
  • the image acquisition module acquires images simultaneously (synchronously) through two cameras, and the acquired images may be photos or video streams.
  • Image processing module used to splicing two images acquired by two cameras into one 3D image.
  • the image processing module splices the two images collected by the two cameras side by side, preferably, the image collected by the left camera is spliced to the left, and the image captured by the right camera is spliced to the right, and finally a left and right format is obtained.
  • 3D (stereo) image the image processing module can also splicing the two images collected by the two cameras side by side, and finally obtaining a 3D image of the top and bottom format.
  • the 3D image is a 3D photo or a 3D video stream.
  • the image processing module before performing image splicing, the image processing module first performs resolution reduction processing on the original image, and then splicing the reduced resolution image to reduce the final 3D image size, thereby avoiding excessive consumption during subsequent transmission.
  • the bandwidth resources thereby increasing the transmission speed and improving the real-time performance of image transmission.
  • the image processing module first samples the two video streams collected by the two cameras into two preset resolution video streams, and then splices the two preset resolution video streams into A 3D video stream in which the preset resolution is lower than the original resolution.
  • the two cameras of the drone respectively capture a video stream of 4K resolution
  • the image processing module samples two video streams of 4K resolution into two video streams of 720P format
  • the sampling mode can adopt a general downsampling algorithm. For example, combine 4 pixels into one pixel; in the case of keeping the video stream frames of two 720P formats synchronized, the picture taken by the left camera is placed on the left side, and the picture taken by the right camera is placed on the right side, and two 720P are placed.
  • the formatted video stream is spliced left and right into a 3D video stream with a resolution of 2560*720.
  • the image processing module can also save the original images captured by the two cameras in a local storage space. Further, before saving, the image processing module further compresses the original image to save storage space, such as compressing the video stream into a video file of the H.265 format.
  • Image Send Mode Used to send 3D images outward.
  • the image sending module sends the obtained 3D image in real time (or timing), for example, to a remote control device or a terminal device that establishes a wireless communication connection with the drone, such as a mobile phone, a tablet computer, or a headset virtual reality device.
  • Equipment such as VR glasses, VR helmets, etc., etc., or uploaded to a server via a wireless communication network.
  • the image processing module before transmitting the 3D image, the image processing module further performs compression processing on the 3D image to reduce the size of the 3D image, improve transmission efficiency, and realize real-time transmission. For example, a 3D video stream is compressed into a H.264 format video stream and then sent out.
  • the image acquisition module takes two full-resolution photos through two cameras.
  • the image processing module uses the two photos taken to form a 3D photo.
  • the image processing module can also perform image quality improvement through the respective screens of the two cameras, such as performing denoising processing, background blurring processing, and the like.
  • the two photos taken by the binocular camera can be completely overlapped by matching the feature points, and the image in the area is equivalent to two shots, and the image processing module performs multiple shots.
  • the same picture is superimposed (such as the simplest weighted average) to effectively reduce noise.
  • the image processing module can blur the back scene through a fuzzy algorithm (such as the simplest Gaussian blur filtering), thereby forming a background blur effect.
  • the aerial camera of the embodiment of the invention collects two images through two cameras, and splicing the collected two images into a 3D image and transmitting them, so that the drone can provide 3D images during the aerial photography process, and enables the user to real-time Viewing the realistic 3D image of the screen brings the user an immersive experience, which greatly enhances the user's aerial photography experience.
  • an aerial photographing apparatus according to a fourth embodiment of the present invention is proposed.
  • a depth detecting module is added to the third embodiment, and the depth detecting module is configured to: perform depth detection of a shooting scene by using a 3D image.
  • obstacle avoidance such as forward obstacle avoidance
  • a set of binocular cameras can simultaneously achieve various functions such as aerial photography, obstacle avoidance, tracking, and ranging.
  • depth detection is performed by using a 3D image, that is, depth detection is performed by using a difference (parallax) between left and right or upper and lower images in a 3D image (such as a 3D video stream).
  • Parallax is the difference in direction produced by observing the same target from two points with a certain distance, so there is parallax in the image obtained by binocular cameras (such as left camera and right camera) of the same target at different positions. The closer the target is to the camera, the larger the parallax in the image of the binocular camera, so the depth detection module can calculate the distance from the target to the camera according to the parallax size of the target in the two images obtained by the binocular camera, ie the depth of the target. To achieve depth detection.
  • the depth detection module prompts The user's obstacle avoidance function is disabled and/or the drone is kept in a hovering state.
  • the drone when the user selects a target that needs to be tracked, the drone will adjust the posture of itself and the pan/tilt to align with the selected target. Because the target tracking accuracy based on depth information is more accurate than the previous method of planar vision, it can realize the tracking function with practical application value for the drone.
  • the 3D image is transmitted to the user in real time, and the depth detection is performed through the 3D image, so that the drone uses a set of binocular cameras ( That is to say, two cameras can realize 3D aerial photography, obstacle avoidance, tracking, ranging and other functions at the same time, without using two sets of binocular cameras (ie four cameras) to realize 3D shooting and depth detection respectively, thereby lowering The cost of implementing a variety of functions.
  • the invention also proposes a drone, the drone comprising: one or more processors; a memory; one or more applications, wherein the one or more applications are stored in the memory and Configured to be performed by the one or more processors, the one or more applications configured to perform an aerial photography method.
  • the aerial photography method includes the steps of: acquiring images by two cameras; splicing two images acquired by the two cameras into one 3D image; and transmitting the 3D images outward.
  • the aerial photography method described in this embodiment is the aerial photography method according to the above embodiment of the present invention, and details are not described herein again.
  • the foregoing embodiment method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be through hardware, but in many cases, the former is better.
  • Implementation Based on such understanding, the technical solution of the present invention, which is essential or contributes to the prior art, may be embodied in the form of a software product stored in a storage medium (such as ROM/RAM, disk,
  • the optical disc includes a number of instructions for causing a terminal device (which may be a cell phone, a computer, a server, an air conditioner, or a network device, etc.) to perform the methods described in various embodiments of the present invention.
  • An aerial photography method provided by an embodiment of the present invention collects two images through two cameras, and splicing the collected two images into a 3D image and transmitting the same, so that the drone can provide a 3D image during the aerial photography process, and The user can view the realistic 3D images in real time, allowing the user to have an immersive feeling, realizing an immersive experience, and greatly improving the user's aerial photography experience.
  • the depth detection is also performed through the 3D image, so that the drone can simultaneously realize 3D aerial photography, obstacle avoidance, and tracking by using a set of binocular cameras (ie, two cameras).
  • a variety of functions, such as ranging, do not need to separately use two sets of binocular cameras (ie, four cameras) to achieve 3D shooting and depth detection, respectively, thereby achieving a variety of functions at a lower cost, and therefore, industrial applicability.

Abstract

An aerial photography method, device, and unmanned aerial vehicle. The method comprises the following steps: acquiring images using two cameras; stitching the two images acquired by the two cameras into one 3D image; and transmitting the 3D image. As a result, the unmanned aerial vehicle can provide the 3D image during an aerial photography process, creating real-time lively 3D images for a user to view and bringing immersive experience for the user, significantly enhancing user experience in aerial photography. Moreover, the embodiment can further comprise: detecting a depth using the 3D image while transmitting the 3D image in real time for a user to view. By employs only a set of binocular vision cameras (the two cameras), the unmanned aerial vehicle can implement a plurality of functions such as 3D aerial photography, obstacle avoidance, tracking, and distance detection, without employing two sets of binocular vision cameras (four cameras) for separately implementing 3D aerial photography and depth detection, thereby implementing the plurality of functions at lower cost.

Description

航拍方法、装置和无人机Aerial photography method, device and drone 技术领域Technical field
本发明涉及无人机技术领域,尤其是涉及一种航拍方法、装置和无人机。The invention relates to the technical field of drones, in particular to an aerial photography method, a device and a drone.
背景技术Background technique
随着无人机技术的迅速发展,无人机的应用越来越广泛,其中,航拍是比较重要的应用领域。目前的航拍系统,主要由无人机和遥控设备组成,用户通过遥控设备对无人机进行控制,无人机通过一个航拍摄像头将拍摄的图像实时传送给遥控设备供用户观看。With the rapid development of drone technology, the application of drones has become more and more extensive. Among them, aerial photography is an important application field. The current aerial photography system is mainly composed of a drone and a remote control device. The user controls the drone through a remote control device, and the drone transmits the captured image to the remote control device for viewing by the user through an aerial camera.
为了向用户提供一种更加真实的体验,现有技术中开始将头戴式虚拟现实(Virtual Reality,VR)设备如VR眼镜作为遥控设备。无人机将拍摄的图像实时传输给VR眼镜,用户则通过VR眼镜实时观看拍摄图像,同时用户还通过VR眼镜来控制无人机的姿态和拍摄角度。In order to provide a more realistic experience for users, in the prior art, a virtual reality (VR) device such as VR glasses is started as a remote control device. The drone transmits the captured image to the VR glasses in real time, and the user watches the captured image in real time through the VR glasses, and the user also controls the posture and shooting angle of the drone through the VR glasses.
由于VR眼镜将人眼与外部完全隔绝,因此为用户提供了一种沉浸式体验,在一定程度上提升了用户体验。然而,由于无人机传送给VR眼镜的图像为2D图像,因此并不能充分发挥VR眼镜的优势,法给用户带来身临其境的体验。Since VR glasses completely isolate the human eye from the outside, it provides an immersive experience for the user and improves the user experience to a certain extent. However, since the image transmitted by the drone to the VR glasses is a 2D image, the advantage of the VR glasses cannot be fully utilized, and the method gives the user an immersive experience.
技术问题technical problem
本发明实施例的主要目的在于提供一种航拍方法、装置和无人机,旨在实现通过无人机进行航拍时给用户带来身临其境的体验。。The main object of the embodiments of the present invention is to provide an aerial photography method, device and drone, which aim to realize an immersive experience for the user when performing aerial photography through the drone. .
技术解决方案Technical solution
为达以上目的,一方面提出一种航拍方法,所述方法包括以下步骤:In order to achieve the above object, an aerial photography method is proposed on the one hand, and the method comprises the following steps:
通过两个摄像头采集图像;Acquiring images through two cameras;
将所述两个摄像头采集的两个图像拼接为一个3D图像;Splicing two images acquired by the two cameras into one 3D image;
向外发送所述3D图像。The 3D image is sent out.
可选地,所述图像为照片或视频流。Optionally, the image is a photo or video stream.
可选地,当所述图像为视频流时,所述将所述两个摄像头采集的两个图像拼接为一个3D图像包括:Optionally, when the image is a video stream, the splicing the two images collected by the two cameras into one 3D image includes:
将所述两个摄像头采集的两个视频流分别采样为两个预设分辨率的视频流,所述预设分辨率低于原始分辨率;The two video streams collected by the two cameras are respectively sampled into two preset resolution video streams, and the preset resolution is lower than the original resolution;
将所述两个预设分辨率的视频流拼接为一个3D视频流。The two preset resolution video streams are spliced into one 3D video stream.
可选地,所述两个摄像头左右并排设置,所述将所述两个摄像头采集的两个图像拼接为一个3D图像包括:Optionally, the two cameras are arranged side by side, and the splicing the two images collected by the two cameras into one 3D image includes:
将所述两个摄像头采集的两个图像左右并排拼接在一起,获得一个左右格式的3D图像。The two images collected by the two cameras are spliced side by side to form a 3D image in a left and right format.
可选地,所述将所述两个摄像头采集的两个图像左右并排拼接在一起包括:将左边摄像头采集的图像拼接在左边,将右边摄像头采集的图像拼接在右边。Optionally, the two images collected by the two cameras are side-by-side spliced together: the image captured by the left camera is spliced to the left, and the image captured by the right camera is spliced to the right.
可选地,将所述两个摄像头采集的两个图像拼接为一个3D图像的步骤之后还包括:利用所述3D图像进行拍摄场景的深度探测,获取深度信息。Optionally, after the step of splicing the two images collected by the two cameras into one 3D image, the method further includes: performing depth detection of the captured scene by using the 3D image to obtain depth information.
可选地,所述向外发送所述3D图像包括:向头戴式虚拟现实设备发送所述3D图像。Optionally, the sending the 3D image outward comprises: transmitting the 3D image to a head mounted virtual reality device.
另一方面,提出一种航拍装置,所述装置包括:In another aspect, an aerial camera device is provided, the device comprising:
图像采集模块,用于通过两个摄像头采集图像;An image acquisition module for collecting images through two cameras;
图像处理模块,用于将所述两个摄像头采集的两个图像拼接为一个3D图像;An image processing module, configured to splicing two images collected by the two cameras into one 3D image;
图像发送模块,用于向外发送所述3D图像。An image sending module, configured to send the 3D image outward.
可选地,所述图像为照片或视频流。Optionally, the image is a photo or video stream.
可选地,当所述图像为视频流时,所述图像处理模块用于:将所述两个摄像头采集的两个视频流分别采样为两个预设分辨率的视频流,将所述两个预设分辨率的视频流拼接为一个3D视频流,其中,所述预设分辨率低于原始分辨率。Optionally, when the image is a video stream, the image processing module is configured to: separately sample two video streams collected by the two cameras into two preset resolution video streams, and the two The preset resolution video streams are spliced into a 3D video stream, wherein the preset resolution is lower than the original resolution.
可选地,所述两个摄像头左右并排设置,所述图像处理模块用于:将所述两个摄像头采集的两个图像左右并排拼接在一起,获得一个左右格式的3D图像。Optionally, the two cameras are arranged side by side, and the image processing module is configured to: splicing the two images collected by the two cameras side by side to obtain a 3D image in a left and right format.
可选地,所述图像处理模块用于:将左边摄像头采集的图像拼接在左边,将右边摄像头采集的图像拼接在右边。Optionally, the image processing module is configured to: splicing the image collected by the left camera to the left side, and splicing the image collected by the right camera to the right side.
可选地,所述装置还包括深度探测模块,所述深度探测模块用于:利用所述3D图像进行拍摄场景的深度探测,获取深度信息。Optionally, the device further includes a depth detecting module, configured to: perform depth detection of the shooting scene by using the 3D image to obtain depth information.
可选地,所述图像发送模块用于:向头戴式虚拟现实设备发送所述3D图像。Optionally, the image sending module is configured to: send the 3D image to the head mounted virtual reality device.
本发明还提出一种无人机,包括:The invention also proposes a drone comprising:
一个或多个处理器;One or more processors;
存储器;Memory
一个或多个应用程序,其中所述一个或多个应用程序被存储在所述存储器中并被配置为由所述一个或多个处理器执行,所述一个或多个应用程序被配置为用于执行前述航拍方法。One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more applications being configured to use Perform the aforementioned aerial photography method.
为达以上目的,一方面提出一种航拍方法,所述方法包括以下步骤:In order to achieve the above object, an aerial photography method is proposed on the one hand, and the method comprises the following steps:
通过两个摄像头采集图像;Acquiring images through two cameras;
将所述两个摄像头采集的两个图像拼接为一个3D图像;Splicing two images acquired by the two cameras into one 3D image;
向外发送所述3D图像。The 3D image is sent out.
可选地,所述图像为照片或视频流。Optionally, the image is a photo or video stream.
可选地,当所述图像为视频流时,所述将所述两个摄像头采集的两个图像拼接为一个3D图像包括:Optionally, when the image is a video stream, the splicing the two images collected by the two cameras into one 3D image includes:
将所述两个摄像头采集的两个视频流分别采样为两个预设分辨率的视频流,所述预设分辨率低于原始分辨率;The two video streams collected by the two cameras are respectively sampled into two preset resolution video streams, and the preset resolution is lower than the original resolution;
将所述两个预设分辨率的视频流拼接为一个3D视频流。The two preset resolution video streams are spliced into one 3D video stream.
可选地,所述两个摄像头左右并排设置,所述将所述两个摄像头采集的两个图像拼接为一个3D图像包括:Optionally, the two cameras are arranged side by side, and the splicing the two images collected by the two cameras into one 3D image includes:
将所述两个摄像头采集的两个图像左右并排拼接在一起,获得一个左右格式的3D图像。The two images collected by the two cameras are spliced side by side to form a 3D image in a left and right format.
可选地,所述将所述两个摄像头采集的两个图像左右并排拼接在一起包括:将左边摄像头采集的图像拼接在左边,将右边摄像头采集的图像拼接在右边。Optionally, the two images collected by the two cameras are side-by-side spliced together: the image captured by the left camera is spliced to the left, and the image captured by the right camera is spliced to the right.
可选地,将所述两个摄像头采集的两个图像拼接为一个3D图像的步骤之后还包括:利用所述3D图像进行拍摄场景的深度探测,获取深度信息。Optionally, after the step of splicing the two images collected by the two cameras into one 3D image, the method further includes: performing depth detection of the captured scene by using the 3D image to obtain depth information.
可选地,所述向外发送所述3D图像包括:向头戴式虚拟现实设备发送所述3D图像。Optionally, the sending the 3D image outward comprises: transmitting the 3D image to a head mounted virtual reality device.
另一方面,提出一种航拍装置,所述装置包括:In another aspect, an aerial camera device is provided, the device comprising:
图像采集模块,用于通过两个摄像头采集图像;An image acquisition module for collecting images through two cameras;
图像处理模块,用于将所述两个摄像头采集的两个图像拼接为一个3D图像;An image processing module, configured to splicing two images collected by the two cameras into one 3D image;
图像发送模块,用于向外发送所述3D图像。An image sending module, configured to send the 3D image outward.
可选地,所述图像为照片或视频流。Optionally, the image is a photo or video stream.
可选地,当所述图像为视频流时,所述图像处理模块用于:将所述两个摄像头采集的两个视频流分别采样为两个预设分辨率的视频流,将所述两个预设分辨率的视频流拼接为一个3D视频流,其中,所述预设分辨率低于原始分辨率。Optionally, when the image is a video stream, the image processing module is configured to: separately sample two video streams collected by the two cameras into two preset resolution video streams, and the two The preset resolution video streams are spliced into a 3D video stream, wherein the preset resolution is lower than the original resolution.
可选地,所述两个摄像头左右并排设置,所述图像处理模块用于:将所述两个摄像头采集的两个图像左右并排拼接在一起,获得一个左右格式的3D图像。Optionally, the two cameras are arranged side by side, and the image processing module is configured to: splicing the two images collected by the two cameras side by side to obtain a 3D image in a left and right format.
可选地,所述图像处理模块用于:将左边摄像头采集的图像拼接在左边,将右边摄像头采集的图像拼接在右边。Optionally, the image processing module is configured to: splicing the image collected by the left camera to the left side, and splicing the image collected by the right camera to the right side.
可选地,所述装置还包括深度探测模块,所述深度探测模块用于:利用所述3D图像进行拍摄场景的深度探测,获取深度信息。Optionally, the device further includes a depth detecting module, configured to: perform depth detection of the shooting scene by using the 3D image to obtain depth information.
可选地,所述图像发送模块用于:向头戴式虚拟现实设备发送所述3D图像。Optionally, the image sending module is configured to: send the 3D image to the head mounted virtual reality device.
本发明还提出一种无人机,包括:The invention also proposes a drone comprising:
一个或多个处理器;One or more processors;
存储器;Memory
一个或多个应用程序,其中所述一个或多个应用程序被存储在所述存储器中并被配置为由所述一个或多个处理器执行,所述一个或多个应用程序被配置为用于执行前述航拍方法。One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more applications being configured to use Perform the aforementioned aerial photography method.
 
有益效果Beneficial effect
本发明实施例所提供的一种航拍方法,通过两个摄像头采集两个图像,并将采集的两个图像拼接为3D图像后发送出去,使得无人机能够在航拍过程中提供3D图像,并使得用户可以实时查看到画面逼真的3D图像,让用户产生身临其境的感觉,实现真正的沉浸式体验,极大的提升了用户的航拍体验。An aerial photography method provided by an embodiment of the present invention collects two images through two cameras, and splicing the collected two images into a 3D image and transmitting the same, so that the drone can provide a 3D image during the aerial photography process, and The user can view the realistic 3D images in real time, allowing the user to have an immersive feeling, realizing an immersive experience, and greatly improving the user's aerial photography experience.
进一步地,在将3D图像实时传送给用户观看的同时,还通过3D图像进行深度探测,使得无人机利用一组双目摄像头(即两个摄像头)就能同时实现3D航拍、避障、跟踪、测距等多种功能,无需分别利用两组双目摄像头(即四个摄像头)分别实现3D拍摄和深度探测,从而以较低的成本实现了多种功能。Further, while the 3D image is transmitted to the user for viewing in real time, the depth detection is also performed through the 3D image, so that the drone can simultaneously realize 3D aerial photography, obstacle avoidance, and tracking by using a set of binocular cameras (ie, two cameras). A variety of functions, such as ranging, do not need to use two sets of binocular cameras (ie, four cameras) to achieve 3D shooting and depth detection respectively, thereby achieving multiple functions at a lower cost.
附图说明DRAWINGS
图1是本发明第一实施例的航拍方法的流程图;1 is a flow chart of an aerial photography method according to a first embodiment of the present invention;
图2是本发明第二实施例的航拍方法的流程图;2 is a flow chart of an aerial photography method according to a second embodiment of the present invention;
图3是本发明第三实施例的航拍装置的模块示意图;Figure 3 is a block diagram showing an aerial photographing apparatus of a third embodiment of the present invention;
图4是本发明第四实施例的航拍装置的模块示意图。Fig. 4 is a block diagram showing the aerial device of the fourth embodiment of the present invention.
本发明目的的实现、功能特点及优点将结合实施例,参照附图做进一步说明。The implementation, functional features, and advantages of the present invention will be further described in conjunction with the embodiments.
本发明的实施方式Embodiments of the invention
应当理解,此处所描述的具体实施例仅仅用以解释本发明,并不用于限定本发明。下面将结合本发明实施例中的附图,对本发明实施例中的技术方案进行清楚、完整地描述,显然,所描述的实施例仅仅是本发明的一部分实施例,而不是全部的实施例。基于本发明中的实施例,本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其他实施例,都属于本发明保护的范围。It is understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. The technical solutions in the embodiments of the present invention are clearly and completely described in the following with reference to the accompanying drawings in the embodiments of the present invention. It is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative efforts are within the scope of the present invention.
需要说明,本发明实施例中所有方向性指示(诸如上、下、左、右、前、后……)仅用于解释在某一特定姿态(如附图所示)下各部件之间的相对位置关系、运动情况等,如果该特定姿态发生改变时,则该方向性指示也相应地随之改变。It should be noted that all directional indications (such as up, down, left, right, front, back, ...) in the embodiments of the present invention are only used to explain between components in a certain posture (as shown in the drawing). Relative positional relationship, motion situation, etc., if the specific posture changes, the directional indication also changes accordingly.
另外,在本发明中涉及“第一”、“第二”等的描述仅用于描述目的,而不能理解为指示或暗示其相对重要性或者隐含指明所指示的技术特征的数量。由此,限定有“第一”、“第二”的特征可以明示或者隐含地包括至少一个该特征。另外,各个实施例之间的技术方案可以相互结合,但是必须是以本领域普通技术人员能够实现为基础,当技术方案的结合出现相互矛盾或无法实现时应当认为这种技术方案的结合不存在,也不在本发明要求的保护范围之内。In addition, the descriptions of "first", "second", and the like in the present invention are used for the purpose of description only, and are not to be construed as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Thus, features defining "first" or "second" may include at least one of the features, either explicitly or implicitly. In addition, the technical solutions between the various embodiments may be combined with each other, but must be based on the realization of those skilled in the art, and when the combination of the technical solutions is contradictory or impossible to implement, it should be considered that the combination of the technical solutions does not exist. It is also within the scope of protection required by the present invention.
本发明实施例的航拍方法和航拍装置,主要应用于无人机,当然也可以应用于其它的飞行器,本发明对此不作限定。本发明实施例以应用于无人机为例进行详细说明。The aerial photographing method and the aerial photographing device of the embodiment of the present invention are mainly applied to the drone, and can of course be applied to other aircrafts, which is not limited by the present invention. The embodiment of the present invention is described in detail by applying to an unmanned aerial vehicle as an example.
实施例一Embodiment 1
参见图1,提出本发明第一实施例的航拍方法,所述方法包括以下步骤:Referring to FIG. 1, an aerial photography method according to a first embodiment of the present invention is proposed. The method includes the following steps:
S11、通过两个摄像头采集图像。S11. Acquire an image through two cameras.
本发明实施例中,无人机设置有两个摄像头,组成一组双目摄像头。两个摄像头优选左右并排设置。当然,也可以交错设置,即两个摄像头不在同一条水平线上。两个摄像头之间间隔一定距离,理论上间隔距离越大越好。In the embodiment of the invention, the drone is provided with two cameras to form a set of binocular cameras. The two cameras are preferably arranged side by side. Of course, it can also be staggered, that is, the two cameras are not on the same horizontal line. The two cameras are separated by a certain distance. In theory, the larger the separation distance, the better.
本步骤S11中,无人机通过两个摄像头同时(同步)采集图像,采集的图像可以是照片或者视频流。In this step S11, the drone acquires images simultaneously (synchronously) through two cameras, and the acquired images may be photos or video streams.
S12、将两个摄像头采集的两个图像拼接为一个3D图像。S12. Splicing two images acquired by the two cameras into one 3D image.
具体的,将两个摄像头采集的两个图像左右并排拼接在一起,优选地,将左边摄像头采集的图像拼接在左边,将右边摄像头采集的图像拼接在右边,最终获得一个左右格式的3D(立体)图像。可选地,也可以将两个摄像头采集的两个图像上下并列拼接在一起,获得一个上下格式的3D图像。所述3D图像即3D照片或者3D视频流。Specifically, the two images collected by the two cameras are spliced side by side, preferably, the image collected by the left camera is spliced to the left, and the image captured by the right camera is spliced to the right, and finally a 3D (3D) format is obtained. )image. Optionally, the two images collected by the two cameras may be spliced together side by side to obtain a 3D image of the top and bottom format. The 3D image is a 3D photo or a 3D video stream.
进一步地,在进行图像拼接之前,无人机先对原始图像进行分辨率降低处理,然后对降低了分辨率的图像进行拼接处理,以减小最终的3D图像尺寸,避免后续传输时消耗太多的带宽资源,从而提高传输速度,提高图像传输的实时性。Further, before performing image splicing, the drone first performs resolution reduction processing on the original image, and then splicing the reduced resolution image to reduce the final 3D image size, thereby avoiding excessive consumption during subsequent transmission. The bandwidth resources, thereby increasing the transmission speed and improving the real-time performance of image transmission.
以采集的图像为视频流为例,无人机首先将两个摄像头采集的两个视频流分别采样为两个预设分辨率的视频流,然后将两个预设分辨率的视频流拼接为一个3D视频流,其中,预设分辨率低于原始分辨率。Taking the captured image as a video stream as an example, the drone first samples the two video streams collected by the two cameras into two preset resolution video streams, and then splices the two preset resolution video streams into A 3D video stream in which the preset resolution is lower than the original resolution.
例如,无人机的两个摄像头各自拍摄4K分辨率的视频流,无人机将两个4K分辨率的视频流采样成为两个720P格式的的视频流,采样方式可以采用通用的降采样算法进行,例如将4个像素合并成为一个像素;采样后的两个720P的视频流在保持帧同步的情况下,将左摄像头拍摄的画面置于左边,右摄像头拍摄的画面置于右边,拼接成为一个分辨率为2560*720的左右格式的3D视频流。For example, the two cameras of the drone respectively capture a video stream of 4K resolution, and the drone samples two video streams of 4K resolution into two video streams of 720P format, and the sampling method can adopt a general downsampling algorithm. For example, the four pixels are merged into one pixel; the sampled two 720P video streams are placed on the left side of the left camera and the right camera is placed on the right side while the frame synchronization is maintained. A 3D video stream with a resolution of 2560*720.
此外,无人机还可以将两个摄像头拍摄的原始图像保存于本地的存储空间内。进一步地,在保存前,还对原始图像进行压缩处理,以节省存储空间,如将视频流压缩为H.265格式的视频文件。In addition, the drone can save the original images captured by the two cameras in the local storage space. Further, before the saving, the original image is also compressed to save storage space, such as compressing the video stream into a H.265 format video file.
S13、向外发送3D图像。S13. Send a 3D image outward.
本步骤S13中,无人机将获得的3D图像发送出去,例如发送给与无人机建立了无线通信连接的遥控设备或终端设备,如手机、平板电脑、头戴式虚拟现实(VR)设备(如VR眼镜、VR头盔等)等,或者通过无线通信网络上传至服务器等。In this step S13, the drone transmits the obtained 3D image, for example, to a remote control device or a terminal device that establishes a wireless communication connection with the drone, such as a mobile phone, a tablet computer, or a head mounted virtual reality (VR) device. (such as VR glasses, VR helmets, etc.), etc., or uploaded to a server via a wireless communication network.
进一步地,在发送3D图像之前,无人机还对3D图像进行压缩处理,以减小3D图像的尺寸,提高传输效率,实现实时传输。例如,将3D视频流压缩为H.264格式的视频流,然后向外发送。Further, before transmitting the 3D image, the drone further performs compression processing on the 3D image to reduce the size of the 3D image, improve transmission efficiency, and realize real-time transmission. For example, a 3D video stream is compressed into a H.264 format video stream and then sent out.
更进一步地,无人机还利用3D图像进行拍摄场景的深度探测,获取深度信息,并利用深度信息实现目标测距、人脸识别、手势识别、目标跟踪等功能,并可以结合无人机的姿态信息与深度信息实现避障(如前向避障),从而利用一组双目摄像头就能同时实现航拍、避障、跟踪、测距等多种功能。 Further, the drone uses 3D images to perform depth detection of the captured scene, acquires depth information, and uses depth information to implement functions such as target ranging, face recognition, gesture recognition, target tracking, and the like, and can be combined with the drone. Attitude information and depth information can be used to avoid obstacles (such as forward obstacle avoidance), so that a set of binocular cameras can simultaneously achieve various functions such as aerial photography, obstacle avoidance, tracking, and ranging.
其中,利用3D图像进行深度探测,即利用3D图像(如3D视频流)中左右或上下两个图像的差异(视差)来实现深度探测。视差就是从有一定距离的两个点上观察同一个目标所产生的方向差异,因此同一个目标在不同位置的双目摄像头(如左摄像头和右摄像头)获得的图像中存在视差。距离摄像头越近的目标,在双目摄像头的图像中的视差越大,因此可以根据双目摄像头获得的两幅图像中目标的视差大小来计算目标到摄像头的距离,即目标的深度,从而实现深度探测。将图像分成几个有效区域,依次计算每个区域的目标距离,并将距离与区域方位反馈给飞控,飞控根据前方目标的距离与方位,就可以实现避障。Wherein, depth detection is performed by using a 3D image, that is, depth detection is performed by using a difference (parallax) between left and right or upper and lower images in a 3D image (such as a 3D video stream). Parallax is the difference in direction produced by observing the same target from two points with a certain distance, so there is parallax in the image obtained by binocular cameras (such as left camera and right camera) of the same target at different positions. The closer the target is to the camera, the larger the parallax in the image of the binocular camera. Therefore, the distance from the target to the camera, that is, the depth of the target, can be calculated according to the parallax size of the target in the two images obtained by the binocular camera. Depth detection. The image is divided into several effective areas, the target distance of each area is calculated in turn, and the distance and the area orientation are fed back to the flight control. The flight control can realize obstacle avoidance according to the distance and orientation of the front target.
可选地,当用户通过遥控设备(如VR眼镜)控制摄像头俯仰(通常通过控制安置摄像头的云台俯仰来实现摄像头俯仰)时,当俯仰角度大于预设的俯仰角度(可以根据实际需要设定)时,无人机则提示用户避障功能失效和/或保持悬停状态。Optionally, when the user controls the camera pitch by a remote control device (such as VR glasses) (usually implementing the camera pitch by controlling the pan tilt of the camera), when the pitch angle is greater than the preset pitch angle (can be set according to actual needs) When the drone prompts the user to disable the obstacle function and/or maintain the hover state.
本发明实施例中,当用户选择好需要跟踪拍摄的目标后,无人机将调整自身和云台的姿态以对准选中的目标。因为基于深度信息的目标跟踪准确率比以往的平面视觉的方法更加精确,从而能够为无人机实现具有实际应用价值的跟踪拍摄功能。In the embodiment of the present invention, when the user selects a target that needs to be tracked, the drone will adjust the posture of itself and the pan/tilt to align with the selected target. Because the target tracking accuracy based on depth information is more accurate than the previous method of planar vision, it can realize the tracking function with practical application value for the drone.
本发明实施例中,当用户触发拍照指令时,如在遥控设备(如VR眼镜)前做出挥手、双手画框等动作时,无人机则通过两个摄像头拍摄两张全分辨率的照片存储于本地,并利用拍摄的两张照片拼接成一张3D照片。In the embodiment of the present invention, when the user triggers the photographing instruction, such as waving a wave, a two-hand frame, and the like in front of the remote control device (such as VR glasses), the drone takes two full-resolution photos through two cameras. Store locally and use the two photos you took to stitch into a 3D photo.
进一步地,无人机还可以通过两个摄像头各自的画面进行画质改善,如进行去噪处理、背景虚化处理等。具体的,双目摄像头拍摄的两张照片,通过特征点匹配后,可以找出完全重合的区域,在该区域内的画面相当于进行了两次拍摄。对同一个画面进行多次拍摄后进行叠加处理(如最简单的加权平均)可以有效的降低噪声。同时,根据双目的深度信息,可以辨别画面的前景和后景,通过模糊算法(如最简单的高斯模糊滤波)可以对后景进行模糊处理,从而形成背景虚化的效果。Further, the drone can also perform image quality improvement through the respective screens of the two cameras, such as performing denoising processing, background blurring processing, and the like. Specifically, the two photos taken by the binocular camera can be matched by the feature points to find a completely overlapping area, and the picture in the area is equivalent to two shots. Overlay processing (such as the simplest weighted average) after multiple shots of the same picture can effectively reduce noise. At the same time, according to the depth information of the dual purpose, the foreground and the back view of the picture can be distinguished, and the back scene can be blurred by a fuzzy algorithm (such as the simplest Gaussian blur filter), thereby forming a background blur effect.
本发明实施例的航拍方法,相对于现有技术,具有以下技术效果:The aerial photography method of the embodiment of the present invention has the following technical effects as compared with the prior art:
1)通过图像拼接的方式制作3D图像,使得无人机可以通过双目摄像头拍摄3D画面,并实时传送到VR眼镜等终端设备,用户可以通过VR眼镜进行3D视频的实时观看,实时提供了航拍的3D视频源,大大提升了用户体验;1) Make 3D images by means of image stitching, so that the drone can shoot 3D images through binocular cameras and transmit them to terminal devices such as VR glasses in real time. Users can watch 3D videos in real time through VR glasses, and provide aerial photography in real time. The 3D video source greatly enhances the user experience;
2)通过3D图像进行深度探测,使得无人机利用同一组双目摄像头,还可以实现拍摄场景的深度探测,获取深度信息,实现前向避障、目标跟踪等功能,无需分别利用两组双目摄像头(即四个摄像头)分别实现3D拍摄和深度探测,从而以较低的成本实现了多种功能;2) Deep detection by 3D image, so that the drone can use the same set of binocular cameras to realize depth detection of the shooting scene, obtain depth information, realize forward obstacle avoidance, target tracking and other functions, without using two sets of double The camera (ie, four cameras) realizes 3D shooting and depth detection respectively, thereby realizing various functions at a low cost;
3)还可以将手机等终端设备上基于双目摄拍摄的特色应用应用在无人机上,进一步拓展无人机的功能;3) It is also possible to apply the special application based on binocular shooting on the terminal device such as mobile phone to the drone, and further expand the function of the drone;
4)结合深度信息进行目标跟踪,提高了对拍摄对象(如人、车)追踪的准确度,使得无人机进行目标跟踪时更加可靠和有效;4) Combining depth information for target tracking improves the accuracy of tracking the subject (such as people and cars), making the drone more reliable and effective for target tracking;
5)利用深度信息进行目标测距,可以精确测量目标的距离,有助于安全的近距离拍摄。5) Using depth information for target ranging, you can accurately measure the distance of the target, which is safe for close-up shooting.
实施例二Embodiment 2
参见图2,提出本发明第二实施例的航拍方法,所述方法包括以下步骤:Referring to FIG. 2, an aerial photography method according to a second embodiment of the present invention is proposed. The method includes the following steps:
S21、通过两个左右并排设置的摄像头采集视频流。S21. Collecting a video stream through two cameras arranged side by side.
本实施例中,无人机的双目摄像头左右并排设置,无人机通过双目摄像头同时采集视频流。In this embodiment, the binocular cameras of the drone are arranged side by side, and the drone simultaneously collects the video stream through the binocular camera.
S22、将两个摄像头采集的两个视频流分别采样为两个预设分辨率的视频流。S22. The two video streams collected by the two cameras are respectively sampled into two preset resolution video streams.
S23、将两个预设分辨率的视频流左右并排拼接在一起,获得一个左右格式的3D视频流。S23. The two preset resolution video streams are spliced side by side to obtain a left and right format 3D video stream.
具体的,无人机首先将双目摄像头采集的两个视频流分别采样为两个预设分辨率的视频流,然后将两个预设分辨率的视频流左右并排拼接在一起,优选地,将左边摄像头采集的视频流拼接在左边,将右边摄像头采集的视频流拼接在右边,最终获得一个左右格式的3D视频流,其中,预设分辨率低于原始分辨率。例如,无人机的两个摄像头各自拍摄4K分辨率的视频流,无人机首先将两个4K分辨率的视频流采样成为两个720P格式的的视频流,然后将两个720P格式的视频流左右(或上下)拼接成为一个分辨率为2560*720的左右格式(或上下格式)的3D视频流。Specifically, the drone firstly samples the two video streams collected by the binocular camera into two preset resolution video streams, and then splicing the two preset resolution video streams side by side, preferably, The video stream collected by the left camera is spliced to the left, and the video stream collected by the right camera is spliced to the right, and finally a 3D video stream of the left and right format is obtained, wherein the preset resolution is lower than the original resolution. For example, the two cameras of the drone each shoot a 4K resolution video stream, and the drone first samples two 4K resolution video streams into two 720P format video streams, and then two 720P format videos. The stream is spliced left and right (or up and down) into a 3D video stream with a resolution of 2560*720 in the left and right format (or top and bottom format).
同时,无人机还将两个摄像头拍摄的4K分辨率的视频流保存于本地的存储空间内。进一步地,在保存前,还对原始视频流进行压缩处理,以节省存储空间,如将4K分辨率的视频流压缩为H.265格式的视频文件。At the same time, the drone also saves the 4K resolution video stream captured by the two cameras in the local storage space. Further, before the saving, the original video stream is also compressed to save storage space, such as compressing the 4K resolution video stream into a H.265 format video file.
S24、将3D视频流压缩后发送给VR眼镜。S24. The 3D video stream is compressed and sent to the VR glasses.
本实施例中,无人机将3D视频流压缩为H.264格式的视频流,然后发送给VR眼镜,实现实时传输。In this embodiment, the drone compresses the 3D video stream into a video stream of the H.264 format, and then transmits the video stream to the VR glasses for real-time transmission.
VR眼镜接收到3D视频流后立即播放,用户则可以实时观看无人机拍摄到的3D视频,使得画面更加逼真,让用户有身临其境的感觉,极大的提升了用户体验。After the VR glasses receive the 3D video stream and play it immediately, the user can watch the 3D video captured by the drone in real time, making the picture more realistic, giving the user an immersive feeling and greatly improving the user experience.
用户可以通过VR眼镜控制无人机的飞行姿态和拍摄角度,当用户触发拍照指令时,如在VR眼镜前做出挥手、双手画框等动作时,无人机则通过两个摄像头拍摄两张全分辨率的照片存储于本地,并利用拍摄的两张照片拼接成一张3D照片传送回VR眼镜,用户则可以实时观看拍摄到的3D照片。进一步地,无人机还可以通过两个摄像头各自的画面进行画质改善,如进行去噪处理、背景虚化处理等。The user can control the flight attitude and shooting angle of the drone through the VR glasses. When the user triggers the camera command, such as waving a wave or a two-hand frame in front of the VR glasses, the drone shoots two through two cameras. Full-resolution photos are stored locally, and the two photos taken are stitched into a 3D photo and sent back to the VR glasses, allowing users to view the captured 3D photos in real time. Further, the drone can also perform image quality improvement through the respective screens of the two cameras, such as performing denoising processing, background blurring processing, and the like.
S25、利用3D视频流进行拍摄场景的深度探测,获取深度信息。S25. Perform depth detection of the shooting scene by using the 3D video stream to obtain depth information.
本实施例中,无人机还同时利用3D视频流进行拍摄场景的深度探测,获取深度信息。其中,利用3D视频流进行深度探测,即利用3D视频流中左右两个视频流的差异(视差)来实现深度探测。In this embodiment, the drone also uses the 3D video stream to perform depth detection of the shooting scene to acquire depth information. The depth detection is performed by using a 3D video stream, that is, using the difference (parallax) between the left and right video streams in the 3D video stream to implement depth detection.
S26、根据深度信息实现前向避障、目标跟踪、目标测距等功能。S26. Perform forward obstacle avoidance, target tracking, and target ranging according to the depth information.
本实施例中,利用深度信息实现避障(如前向避障)、人脸识别、手势识别、目标跟踪等功能,并可以结合无人机的姿态信息与深度信息实现目标测距,具体实现过程与现有技术相同,在此不赘述。In this embodiment, the depth information is used to implement obstacle avoidance (such as forward obstacle avoidance), face recognition, gesture recognition, target tracking, etc., and the target distance measurement can be realized by combining the attitude information and the depth information of the drone, and the specific implementation is realized. The process is the same as the prior art and will not be described here.
本实施例将双目摄像头拍摄的两个视频流拼接为一个3D视频流,一边将3D视频流实时传送给VR眼镜供用户实时查看,一边通过3D视频流进行深度探测,使得无人机利用一组双目摄像头就能同时实现3D航拍、避障、跟踪、测距等多种功能,无需分别利用两组双目摄像头(即四个摄像头)分别实现3D拍摄和深度探测,从而以较低的成本实现了多种功能。In this embodiment, the two video streams captured by the binocular camera are spliced into one 3D video stream, and the 3D video stream is transmitted to the VR glasses in real time for the user to view in real time, and the depth detection is performed through the 3D video stream, so that the drone utilizes one. The group binocular camera can realize 3D aerial photography, obstacle avoidance, tracking, ranging and other functions at the same time, without using two sets of binocular cameras (ie four cameras) to achieve 3D shooting and depth detection respectively, thus lowering Costs enable multiple functions.
实施例三Embodiment 3
参见图3,提出本发明第三实施例的航拍装置,所述装置包括图像采集模块、图像处理模块和图像发送模块,其中:Referring to FIG. 3, an aerial device according to a third embodiment of the present invention is proposed. The device includes an image acquisition module, an image processing module, and an image transmission module, wherein:
图像采集模块:用于通过两个摄像头采集图像。Image acquisition module: used to capture images through two cameras.
本发明实施例中,无人机设置有两个摄像头,组成一组双目摄像头。两个摄像头优选左右并排设置。当然,也可以交错设置,即两个摄像头不在同一条水平线上。两个摄像头之间间隔一定距离,理论上间隔距离越大越好。In the embodiment of the invention, the drone is provided with two cameras to form a set of binocular cameras. The two cameras are preferably arranged side by side. Of course, it can also be staggered, that is, the two cameras are not on the same horizontal line. The two cameras are separated by a certain distance. In theory, the larger the separation distance, the better.
图像采集模块通过两个摄像头同时(同步)采集图像,采集的图像可以是照片或者视频流。The image acquisition module acquires images simultaneously (synchronously) through two cameras, and the acquired images may be photos or video streams.
图像处理模块:用于将两个摄像头采集的两个图像拼接为一个3D图像。Image processing module: used to splicing two images acquired by two cameras into one 3D image.
具体的,图像处理模块将两个摄像头采集的两个图像左右并排拼接在一起,优选地,将左边摄像头采集的图像拼接在左边,将右边摄像头采集的图像拼接在右边,最终获得一个左右格式的3D(立体)图像。可选地,图像处理模块也可以将两个摄像头采集的两个图像上下并列拼接在一起,最终获得一个上下格式的3D图像。所述3D图像即3D照片或者3D视频流。Specifically, the image processing module splices the two images collected by the two cameras side by side, preferably, the image collected by the left camera is spliced to the left, and the image captured by the right camera is spliced to the right, and finally a left and right format is obtained. 3D (stereo) image. Optionally, the image processing module can also splicing the two images collected by the two cameras side by side, and finally obtaining a 3D image of the top and bottom format. The 3D image is a 3D photo or a 3D video stream.
进一步地,在进行图像拼接之前,图像处理模块先对原始图像进行分辨率降低处理,然后对降低了分辨率的图像进行拼接处理,以减小最终的3D图像尺寸,避免后续传输时消耗太多的带宽资源,从而提高传输速度,提高图像传输的实时性。Further, before performing image splicing, the image processing module first performs resolution reduction processing on the original image, and then splicing the reduced resolution image to reduce the final 3D image size, thereby avoiding excessive consumption during subsequent transmission. The bandwidth resources, thereby increasing the transmission speed and improving the real-time performance of image transmission.
以采集的图像为视频流为例,图像处理模块首先将两个摄像头采集的两个视频流分别采样为两个预设分辨率的视频流,然后将两个预设分辨率的视频流拼接为一个3D视频流,其中,预设分辨率低于原始分辨率。Taking the captured image as a video stream as an example, the image processing module first samples the two video streams collected by the two cameras into two preset resolution video streams, and then splices the two preset resolution video streams into A 3D video stream in which the preset resolution is lower than the original resolution.
例如,无人机的两个摄像头各自拍摄4K分辨率的视频流,图像处理模块将两个4K分辨率的视频流采样成为两个720P格式的的视频流,采样方式可以采用通用的降采样算法进行,例如将4个像素合并成为一个像素;在保持两个720P格式的视频流帧同步的情况下,将左摄像头拍摄的画面置于左边,右摄像头拍摄的画面置于右边,将两个720P格式的视频流左右拼接成为一个分辨率为2560*720的左右格式的3D视频流。For example, the two cameras of the drone respectively capture a video stream of 4K resolution, and the image processing module samples two video streams of 4K resolution into two video streams of 720P format, and the sampling mode can adopt a general downsampling algorithm. For example, combine 4 pixels into one pixel; in the case of keeping the video stream frames of two 720P formats synchronized, the picture taken by the left camera is placed on the left side, and the picture taken by the right camera is placed on the right side, and two 720P are placed. The formatted video stream is spliced left and right into a 3D video stream with a resolution of 2560*720.
此外,图像处理模块还可以将两个摄像头拍摄的原始图像保存于本地的存储空间内。进一步地,在保存前,图像处理模块还对原始图像进行压缩处理,以节省存储空间,如将视频流压缩为H.265格式的视频文件。In addition, the image processing module can also save the original images captured by the two cameras in a local storage space. Further, before saving, the image processing module further compresses the original image to save storage space, such as compressing the video stream into a video file of the H.265 format.
图像发送模:用于向外发送3D图像。Image Send Mode: Used to send 3D images outward.
具体的,图像发送模将获得的3D图像实时(或定时的)发送出去,例如发送给与无人机建立了无线通信连接的遥控设备或终端设备,如手机、平板电脑、头戴式虚拟现实设备(如VR眼镜、VR头盔等)等,或者通过无线通信网络上传至服务器等。Specifically, the image sending module sends the obtained 3D image in real time (or timing), for example, to a remote control device or a terminal device that establishes a wireless communication connection with the drone, such as a mobile phone, a tablet computer, or a headset virtual reality device. Equipment (such as VR glasses, VR helmets, etc.), etc., or uploaded to a server via a wireless communication network.
进一步地,在发送3D图像之前,图像处理模块还对3D图像进行压缩处理,以减小3D图像的尺寸,提高传输效率,实现实时传输。例如,将3D视频流压缩为H.264格式的视频流,然后向外发送。Further, before transmitting the 3D image, the image processing module further performs compression processing on the 3D image to reduce the size of the 3D image, improve transmission efficiency, and realize real-time transmission. For example, a 3D video stream is compressed into a H.264 format video stream and then sent out.
本发明实施例中,当用户触发拍照指令时,如在遥控设备(如VR眼镜)前做出挥手、双手画框等动作时,图像采集模块则通过两个摄像头拍摄两张全分辨率的照片存储于本地,图像处理模块则利用拍摄的两张照片拼接成一张3D照片。In the embodiment of the present invention, when the user triggers a photographing instruction, such as waving a wave, a two-hand frame, or the like in front of a remote control device (such as VR glasses), the image acquisition module takes two full-resolution photos through two cameras. Stored locally, the image processing module uses the two photos taken to form a 3D photo.
进一步地,图像处理模块还可以通过两个摄像头各自的画面进行画质改善,如进行去噪处理、背景虚化处理等。具体的,双目摄像头拍摄的两张照片,通过特征点匹配后,可以找出完全重合的区域,在该区域内的画面相当于进行了两次拍摄,图像处理模块对进行多次拍摄后的同一个画面进行叠加处理(如最简单的加权平均)就可以有效的降低噪声。同时,根据双目的深度信息,可以辨别画面的前景和后景,图像处理模块通过模糊算法(如最简单的高斯模糊滤波)可以对后景进行模糊处理,从而形成背景虚化的效果。Further, the image processing module can also perform image quality improvement through the respective screens of the two cameras, such as performing denoising processing, background blurring processing, and the like. Specifically, the two photos taken by the binocular camera can be completely overlapped by matching the feature points, and the image in the area is equivalent to two shots, and the image processing module performs multiple shots. The same picture is superimposed (such as the simplest weighted average) to effectively reduce noise. At the same time, according to the depth information of the dual purpose, the foreground and the back scene of the picture can be distinguished, and the image processing module can blur the back scene through a fuzzy algorithm (such as the simplest Gaussian blur filtering), thereby forming a background blur effect.
本发明实施例的航拍装置,通过两个摄像头采集两个图像,并将采集的两个图像拼接为3D图像后发送出去,使得无人机能够在航拍过程中提供3D图像,并使得用户可以实时查看到画面逼真的3D图像,给用户带来了身临其境的体验,极大的提升了用户的航拍体验。The aerial camera of the embodiment of the invention collects two images through two cameras, and splicing the collected two images into a 3D image and transmitting them, so that the drone can provide 3D images during the aerial photography process, and enables the user to real-time Viewing the realistic 3D image of the screen brings the user an immersive experience, which greatly enhances the user's aerial photography experience.
实施例四Embodiment 4
参照图4,提出本发明第四实施例的航拍装置,本实施例在第三实施例的基础上增加了深度探测模块,所述深度探测模块用于:利用3D图像进行拍摄场景的深度探测,获取深度信息,然后就可以利用深度信息实现目标测距、人脸识别、手势识别、目标跟踪等功能,并可以结合无人机的姿态信息与深度信息实现避障(如前向避障),从而利用一组双目摄像头就能同时实现航拍、避障、跟踪、测距等多种功能。Referring to FIG. 4, an aerial photographing apparatus according to a fourth embodiment of the present invention is proposed. In this embodiment, a depth detecting module is added to the third embodiment, and the depth detecting module is configured to: perform depth detection of a shooting scene by using a 3D image. Obtain depth information, and then use depth information to achieve target ranging, face recognition, gesture recognition, target tracking and other functions, and can be combined with the attitude information and depth information of the drone to achieve obstacle avoidance (such as forward obstacle avoidance). Thus, a set of binocular cameras can simultaneously achieve various functions such as aerial photography, obstacle avoidance, tracking, and ranging.
其中,利用3D图像进行深度探测,即利用3D图像(如3D视频流)中的左右或上下两个图像的差异(视差)来实现深度探测。视差就是从有一定距离的两个点上观察同一个目标所产生的方向差异,因此同一个目标在不同位置的双目摄像头(如左摄像头和右摄像头)获得的图像中存在视差。距离摄像头越近的目标,在双目摄像头的图像中的视差越大,因此深度探测模块可以根据双目摄像头获得的两幅图像中目标的视差大小来计算目标到摄像头的距离,即目标的深度,从而实现深度探测。Wherein, depth detection is performed by using a 3D image, that is, depth detection is performed by using a difference (parallax) between left and right or upper and lower images in a 3D image (such as a 3D video stream). Parallax is the difference in direction produced by observing the same target from two points with a certain distance, so there is parallax in the image obtained by binocular cameras (such as left camera and right camera) of the same target at different positions. The closer the target is to the camera, the larger the parallax in the image of the binocular camera, so the depth detection module can calculate the distance from the target to the camera according to the parallax size of the target in the two images obtained by the binocular camera, ie the depth of the target. To achieve depth detection.
可选地,当用户通过遥控设备(如VR眼镜)控制摄像头俯仰(通常通过控制安置摄像头的云台俯仰来实现摄像头俯仰)时,当俯仰角度大于预设的俯仰角度时,深度探测模块则提示用户避障功能失效和/或保持无人机处于悬停状态。Optionally, when the user controls the camera pitch by a remote control device (such as VR glasses) (usually implementing camera tilt by controlling the pan tilt of the camera), when the pitch angle is greater than the preset pitch angle, the depth detection module prompts The user's obstacle avoidance function is disabled and/or the drone is kept in a hovering state.
本发明实施例中,当用户选择好需要跟踪拍摄的目标后,无人机将调整自身和云台的姿态以对准选中的目标。因为基于深度信息的目标跟踪准确率比以往的平面视觉的方法更加精确,从而能够为无人机实现具有实际应用价值的跟踪拍摄功能。In the embodiment of the present invention, when the user selects a target that needs to be tracked, the drone will adjust the posture of itself and the pan/tilt to align with the selected target. Because the target tracking accuracy based on depth information is more accurate than the previous method of planar vision, it can realize the tracking function with practical application value for the drone.
本实施例中,将双目摄像头拍摄的两个图像拼接为一个3D图像后,一边将3D图像实时传送给用户观看,一边通过3D图像进行深度探测,使得无人机利用一组双目摄像头(即两个摄像头)就能同时实现3D航拍、避障、跟踪、测距等多种功能,无需分别利用两组双目摄像头(即四个摄像头)分别实现3D拍摄和深度探测,从而以较低的成本实现了多种功能。In this embodiment, after the two images captured by the binocular camera are spliced into one 3D image, the 3D image is transmitted to the user in real time, and the depth detection is performed through the 3D image, so that the drone uses a set of binocular cameras ( That is to say, two cameras can realize 3D aerial photography, obstacle avoidance, tracking, ranging and other functions at the same time, without using two sets of binocular cameras (ie four cameras) to realize 3D shooting and depth detection respectively, thereby lowering The cost of implementing a variety of functions.
本发明同时提出一种无人机,所述无人机包括:一个或多个处理器;存储器;一个或多个应用程序,其中所述一个或多个应用程序被存储在所述存储器中并被配置为由所述一个或多个处理器执行,所述一个或多个应用程序被配置为用于执行航拍方法。所述航拍方法包括以下步骤:通过两个摄像头采集图像;将所述两个摄像头采集的两个图像拼接为一个3D图像;向外发送所述3D图像。本实施例中所描述的航拍方法为本发明中上述实施例所涉及的航拍方法,在此不再赘述。The invention also proposes a drone, the drone comprising: one or more processors; a memory; one or more applications, wherein the one or more applications are stored in the memory and Configured to be performed by the one or more processors, the one or more applications configured to perform an aerial photography method. The aerial photography method includes the steps of: acquiring images by two cameras; splicing two images acquired by the two cameras into one 3D image; and transmitting the 3D images outward. The aerial photography method described in this embodiment is the aerial photography method according to the above embodiment of the present invention, and details are not described herein again.
通过以上的实施方式的描述,本领域的技术人员可以清楚地了解到上述实施例方法可借助软件加必需的通用硬件平台的方式来实现,当然也可以通过硬件,但很多情况下前者是更佳的实施方式。基于这样的理解,本发明的技术方案本质上或者说对现有技术做出贡献的部分可以以软件产品的形式体现出来,该计算机软件产品存储在一个存储介质(如ROM/RAM、磁碟、光盘)中,包括若干指令用以使得一台终端设备(可以是手机,计算机,服务器,空调器,或者网络设备等)执行本发明各个实施例所述的方法。Through the description of the above embodiments, those skilled in the art can clearly understand that the foregoing embodiment method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be through hardware, but in many cases, the former is better. Implementation. Based on such understanding, the technical solution of the present invention, which is essential or contributes to the prior art, may be embodied in the form of a software product stored in a storage medium (such as ROM/RAM, disk, The optical disc includes a number of instructions for causing a terminal device (which may be a cell phone, a computer, a server, an air conditioner, or a network device, etc.) to perform the methods described in various embodiments of the present invention.
以上参照附图说明了本发明的优选实施例,并非因此局限本发明的权利范围。本领域技术人员不脱离本发明的范围和实质,可以有多种变型方案实现本发明,比如作为一个实施例的特征可用于另一实施例而得到又一实施例。凡在运用本发明的技术构思之内所作的任何修改、等同替换和改进,均应在本。The preferred embodiments of the present invention have been described above with reference to the drawings, and are not intended to limit the scope of the invention. A person skilled in the art can implement the invention in various variants without departing from the scope and spirit of the invention. For example, the features of one embodiment can be used in another embodiment to obtain a further embodiment. Any modifications, equivalent substitutions and improvements made within the technical concept of the present invention should be used herein.
工业实用性Industrial applicability
本发明实施例所提供的一种航拍方法,通过两个摄像头采集两个图像,并将采集的两个图像拼接为3D图像后发送出去,使得无人机能够在航拍过程中提供3D图像,并使得用户可以实时查看到画面逼真的3D图像,让用户产生身临其境的感觉,实现真正的沉浸式体验,极大的提升了用户的航拍体验。An aerial photography method provided by an embodiment of the present invention collects two images through two cameras, and splicing the collected two images into a 3D image and transmitting the same, so that the drone can provide a 3D image during the aerial photography process, and The user can view the realistic 3D images in real time, allowing the user to have an immersive feeling, realizing an immersive experience, and greatly improving the user's aerial photography experience.
进一步地,在将3D图像实时传送给用户观看的同时,还通过3D图像进行深度探测,使得无人机利用一组双目摄像头(即两个摄像头)就能同时实现3D航拍、避障、跟踪、测距等多种功能,无需分别利用两组双目摄像头(即四个摄像头)分别实现3D拍摄和深度探测,从而以较低的成本实现了多种功能,因此,具有工业实用性。Further, while the 3D image is transmitted to the user for viewing in real time, the depth detection is also performed through the 3D image, so that the drone can simultaneously realize 3D aerial photography, obstacle avoidance, and tracking by using a set of binocular cameras (ie, two cameras). A variety of functions, such as ranging, do not need to separately use two sets of binocular cameras (ie, four cameras) to achieve 3D shooting and depth detection, respectively, thereby achieving a variety of functions at a lower cost, and therefore, industrial applicability.

Claims (15)

  1. 一种航拍方法,包括以下步骤:An aerial photography method comprising the following steps:
    通过两个摄像头采集图像;Acquiring images through two cameras;
    将所述两个摄像头采集的两个图像拼接为一个3D图像;Splicing two images acquired by the two cameras into one 3D image;
    向外发送所述3D图像。The 3D image is sent out.
  2. 根据权利要求1所述的航拍方法,其中,所述图像为照片或视频流。The aerial photography method of claim 1 wherein the image is a photo or video stream.
  3. 根据权利要求2所述的航拍方法,其特征在于,当所述图像为视频流时,所述将所述两个摄像头采集的两个图像拼接为一个3D图像包括:The aerial photography method according to claim 2, wherein when the image is a video stream, the splicing the two images collected by the two cameras into one 3D image comprises:
    将所述两个摄像头采集的两个视频流分别采样为两个预设分辨率的视频流,所述预设分辨率低于原始分辨率;The two video streams collected by the two cameras are respectively sampled into two preset resolution video streams, and the preset resolution is lower than the original resolution;
    将所述两个预设分辨率的视频流拼接为一个3D视频流。The two preset resolution video streams are spliced into one 3D video stream.
  4. 根据权利要求1所述的航拍方法,其中,所述两个摄像头左右并排设置,所述将所述两个摄像头采集的两个图像拼接为一个3D图像包括:The aerial photography method according to claim 1, wherein the two cameras are arranged side by side, and the splicing the two images collected by the two cameras into one 3D image comprises:
    将所述两个摄像头采集的两个图像左右并排拼接在一起,获得一个左右格式的3D图像。The two images collected by the two cameras are spliced side by side to form a 3D image in a left and right format.
  5. 根据权利要求4所述的航拍方法,其中,所述将所述两个摄像头采集的两个图像左右并排拼接在一起包括:The aerial photography method according to claim 4, wherein said splicing the two images collected by the two cameras side by side side by side comprises:
    将左边摄像头采集的图像拼接在左边,将右边摄像头采集的图像拼接在右边。The image captured by the left camera is stitched to the left, and the image captured by the right camera is stitched to the right.
  6. 根据权利要求1-5任一项所述的航拍方法,其中,将所述两个摄像头采集的两个图像拼接为一个3D图像的步骤之后还包括:The aerial photography method according to any one of claims 1-5, wherein the step of splicing the two images acquired by the two cameras into one 3D image further comprises:
    利用所述3D图像进行拍摄场景的深度探测,获取深度信息。The depth detection of the captured scene is performed by using the 3D image to obtain depth information.
  7. 根据权利要求1-5任一项所述的航拍方法,其中,所述向外发送所述3D图像包括:向头戴式虚拟现实设备发送所述3D图像。The aerial photography method according to any one of claims 1 to 5, wherein the outward transmission of the 3D image comprises transmitting the 3D image to a head mounted virtual reality device.
  8. 一种航拍装置,包括:An aerial camera device comprising:
    图像采集模块,用于通过两个摄像头采集图像;An image acquisition module for collecting images through two cameras;
    图像处理模块,用于将所述两个摄像头采集的两个图像拼接为一个3D图像;An image processing module, configured to splicing two images collected by the two cameras into one 3D image;
    图像发送模块,用于向外发送所述3D图像。An image sending module, configured to send the 3D image outward.
  9. 根据权利要求8所述的航拍装置,其中,所述图像为照片或视频流。The aerial camera of claim 8 wherein the image is a photo or video stream.
  10. 根据权利要求9所述的航拍装置,其中,当所述图像为视频流时,所述图像处理模块用于:将所述两个摄像头采集的两个视频流分别采样为两个预设分辨率的视频流,将所述两个预设分辨率的视频流拼接为一个3D视频流,其中,所述预设分辨率低于原始分辨率。The aerial image device according to claim 9, wherein when the image is a video stream, the image processing module is configured to separately sample two video streams collected by the two cameras into two preset resolutions. The video stream is spliced into two 3D video streams, wherein the preset resolution is lower than the original resolution.
  11. 根据权利要求8所述的航拍装置,其中,所述两个摄像头左右并排设置,所述图像处理模块用于:将所述两个摄像头采集的两个图像左右并排拼接在一起,获得一个左右格式的3D图像。The aerial photographing device according to claim 8, wherein the two cameras are arranged side by side, the image processing module is configured to: splicing the two images collected by the two cameras side by side to obtain a left and right format. 3D image.
  12. 根据权利要求11所述的航拍装置,其中,所述图像处理模块用于:将左边摄像头采集的图像拼接在左边,将右边摄像头采集的图像拼接在右边。The aerial image capturing device according to claim 11, wherein the image processing module is configured to: splicing the image captured by the left camera to the left and stitching the image captured by the right camera to the right.
  13. 根据权利要求8-12任项所述的航拍装置,其中,所述装置还包括深度探测模块,所述深度探测模块用于:利用所述3D图像进行拍摄场景的深度探测,获取深度信息。The aerial photographing apparatus according to any one of claims 8 to 12, wherein the apparatus further comprises a depth detecting module, wherein the depth detecting module is configured to: perform depth detection of the shooting scene by using the 3D image to acquire depth information.
  14. 根据权利要求8-12任一项所述的航拍装置,其中,所述图像发送模块用于:向头戴式虚拟现实设备发送所述3D图像。The aerial photographing apparatus according to any one of claims 8 to 12, wherein the image transmitting module is configured to: transmit the 3D image to a head mounted virtual reality device.
  15. 一种无人机,包括:A drone that includes:
    一个或多个处理器;One or more processors;
    存储器;Memory
    一个或多个应用程序,其中所述一个或多个应用程序被存储在所述存储器中并被配置为由所述一个或多个处理器执行,所述一个或多个应用程序被配置为用于执行权利要求1至7任一项所述的方法。One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more applications being configured to use The method of any one of claims 1 to 7 is performed.
PCT/CN2017/115877 2017-01-17 2017-12-13 Aerial photography method, device, and unmanned aerial vehicle WO2018133589A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201710031741.XA CN107071389A (en) 2017-01-17 2017-01-17 Take photo by plane method, device and unmanned plane
CN201710031741.X 2017-01-17

Publications (1)

Publication Number Publication Date
WO2018133589A1 true WO2018133589A1 (en) 2018-07-26

Family

ID=59597930

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2017/115877 WO2018133589A1 (en) 2017-01-17 2017-12-13 Aerial photography method, device, and unmanned aerial vehicle

Country Status (2)

Country Link
CN (1) CN107071389A (en)
WO (1) WO2018133589A1 (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109729337A (en) * 2018-11-15 2019-05-07 华南师范大学 A kind of vision synthesizer and its control method applied to dual camera
CN112506228A (en) * 2020-12-28 2021-03-16 广东电网有限责任公司中山供电局 Substation unmanned aerial vehicle optimal emergency hedge path selection method

Families Citing this family (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107071389A (en) * 2017-01-17 2017-08-18 亿航智能设备(广州)有限公司 Take photo by plane method, device and unmanned plane
CN107741781A (en) * 2017-09-01 2018-02-27 中国科学院深圳先进技术研究院 Flight control method, device, unmanned plane and the storage medium of unmanned plane
WO2019051649A1 (en) * 2017-09-12 2019-03-21 深圳市大疆创新科技有限公司 Method and device for image transmission, movable platform, monitoring device, and system
CN108304000B (en) * 2017-10-25 2024-01-23 河北工业大学 Real-time VR system of cloud platform
CN108616679A (en) * 2018-04-09 2018-10-02 沈阳上博智像科技有限公司 The method of binocular camera and control binocular camera
CN108521558A (en) * 2018-04-10 2018-09-11 深圳慧源创新科技有限公司 Unmanned plane figure transmission method, system, unmanned plane and unmanned plane client
CN108985193A (en) * 2018-06-28 2018-12-11 电子科技大学 A kind of unmanned plane portrait alignment methods based on image detection
CN110460677A (en) * 2019-08-23 2019-11-15 临工集团济南重机有限公司 A kind of excavator and excavator tele-control system
CN111006586B (en) * 2019-12-12 2020-07-24 天目爱视(北京)科技有限公司 Intelligent control method for 3D information acquisition
WO2022088072A1 (en) * 2020-10-30 2022-05-05 深圳市大疆创新科技有限公司 Visual tracking method and apparatus, movable platform, and computer-readable storage medium
CN112714281A (en) * 2020-12-19 2021-04-27 西南交通大学 Unmanned aerial vehicle carries VR video acquisition transmission device based on 5G network
CN113784051A (en) * 2021-09-23 2021-12-10 深圳市道通智能航空技术股份有限公司 Method, device, equipment and medium for controlling shooting of aircraft based on portrait mode

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8860780B1 (en) * 2004-09-27 2014-10-14 Grandeye, Ltd. Automatic pivoting in a wide-angle video camera
CN106162145A (en) * 2016-07-26 2016-11-23 北京奇虎科技有限公司 Stereoscopic image generation method based on unmanned plane, device
CN106184787A (en) * 2016-07-14 2016-12-07 科盾科技股份有限公司北京分公司 There is the aircraft of DAS (Driver Assistant System) and landing thereof and the method for collision free
CN107071389A (en) * 2017-01-17 2017-08-18 亿航智能设备(广州)有限公司 Take photo by plane method, device and unmanned plane

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2016086379A1 (en) * 2014-12-04 2016-06-09 SZ DJI Technology Co., Ltd. Imaging system and method
CN105872523A (en) * 2015-10-30 2016-08-17 乐视体育文化产业发展(北京)有限公司 Three-dimensional video data obtaining method, device and system
CN105828062A (en) * 2016-03-23 2016-08-03 常州视线电子科技有限公司 Unmanned aerial vehicle 3D virtual reality shooting system
CN105791810A (en) * 2016-04-27 2016-07-20 深圳市高巨创新科技开发有限公司 Virtual stereo display method and device
CN205847443U (en) * 2016-07-13 2016-12-28 杭州翼飞电子科技有限公司 The more than enough people of a kind of energy shares the device that unmanned plane 3D schemes to pass in real time

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8860780B1 (en) * 2004-09-27 2014-10-14 Grandeye, Ltd. Automatic pivoting in a wide-angle video camera
CN106184787A (en) * 2016-07-14 2016-12-07 科盾科技股份有限公司北京分公司 There is the aircraft of DAS (Driver Assistant System) and landing thereof and the method for collision free
CN106162145A (en) * 2016-07-26 2016-11-23 北京奇虎科技有限公司 Stereoscopic image generation method based on unmanned plane, device
CN107071389A (en) * 2017-01-17 2017-08-18 亿航智能设备(广州)有限公司 Take photo by plane method, device and unmanned plane

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109729337A (en) * 2018-11-15 2019-05-07 华南师范大学 A kind of vision synthesizer and its control method applied to dual camera
CN112506228A (en) * 2020-12-28 2021-03-16 广东电网有限责任公司中山供电局 Substation unmanned aerial vehicle optimal emergency hedge path selection method
CN112506228B (en) * 2020-12-28 2023-11-07 广东电网有限责任公司中山供电局 Optimal emergency risk avoiding path selection method for unmanned aerial vehicle of transformer substation

Also Published As

Publication number Publication date
CN107071389A (en) 2017-08-18

Similar Documents

Publication Publication Date Title
WO2018133589A1 (en) Aerial photography method, device, and unmanned aerial vehicle
US10484621B2 (en) Systems and methods for compressing video content
US20190246104A1 (en) Panoramic video processing method, device and system
US10171792B2 (en) Device and method for three-dimensional video communication
US9940697B2 (en) Systems and methods for combined pipeline processing of panoramic images
US10116922B2 (en) Method and system for automatic 3-D image creation
US10395338B2 (en) Virtual lens simulation for video and photo cropping
US20150358539A1 (en) Mobile Virtual Reality Camera, Method, And System
WO2016187924A1 (en) System and method for showing image information
WO2017124868A1 (en) Method and apparatus for acquiring panorama image
WO2014162324A1 (en) Spherical omnidirectional video-shooting system
JP6057570B2 (en) Apparatus and method for generating stereoscopic panoramic video
WO2017166360A1 (en) Video calling method and device
WO2020093850A1 (en) Dual-light image integration method and apparatus, and unmanned aerial vehicle
WO2017166714A1 (en) Method, device, and system for capturing panoramic image
WO2012039306A1 (en) Image processing device, image capture device, image processing method, and program
WO2012029299A1 (en) Image capture device, playback device, and image-processing method
US11258949B1 (en) Electronic image stabilization to improve video analytics accuracy
US20150244756A1 (en) Method, Apparatus and System for Determining Terminal That is to Share Real-Time Video
WO2021237616A1 (en) Image transmission method, mobile platform, and computer readable storage medium
WO2022047701A1 (en) Image processing method and apparatus
WO2018121401A1 (en) Splicing method for panoramic video images, and panoramic camera
US20150326847A1 (en) Method and system for capturing a 3d image using single camera
WO2023036218A1 (en) Method and apparatus for determining width of viewpoint
CN108419052B (en) Panoramic imaging method for multiple unmanned aerial vehicles

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17893175

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 04/11/2019)

122 Ep: pct application non-entry in european phase

Ref document number: 17893175

Country of ref document: EP

Kind code of ref document: A1