WO2020037615A1 - 云台系统及其图像处理方法、无人机 - Google Patents

云台系统及其图像处理方法、无人机 Download PDF

Info

Publication number
WO2020037615A1
WO2020037615A1 PCT/CN2018/102000 CN2018102000W WO2020037615A1 WO 2020037615 A1 WO2020037615 A1 WO 2020037615A1 CN 2018102000 W CN2018102000 W CN 2018102000W WO 2020037615 A1 WO2020037615 A1 WO 2020037615A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
focus
far
focal length
stitched
Prior art date
Application number
PCT/CN2018/102000
Other languages
English (en)
French (fr)
Inventor
卢庆博
陆真国
Original Assignee
深圳市大疆创新科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 深圳市大疆创新科技有限公司 filed Critical 深圳市大疆创新科技有限公司
Priority to CN201880039151.4A priority Critical patent/CN110771140A/zh
Priority to PCT/CN2018/102000 priority patent/WO2020037615A1/zh
Publication of WO2020037615A1 publication Critical patent/WO2020037615A1/zh
Priority to US17/180,572 priority patent/US20210176395A1/en

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/67—Focus control based on electronic image sensor signals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/80—Camera processing pipelines; Components thereof
    • G—PHYSICS
    • G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
    • G03B—APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
    • G03B17/00—Details of cameras or camera bodies; Accessories therefor
    • G03B17/56—Accessories
    • G03B17/561—Support related camera accessories
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4007—Scaling of whole images or parts thereof, e.g. expanding or contracting based on interpolation, e.g. bilinear interpolation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4038—Image mosaicing, e.g. composing plane images from plane sub-images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/69—Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/695—Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/95—Computational photography systems, e.g. light-field imaging systems
    • H04N23/951—Computational photography systems, e.g. light-field imaging systems by using two or more images to influence resolution, frame rate or aspect ratio
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/48—Increasing resolution by shifting the sensor relative to the scene
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B64—AIRCRAFT; AVIATION; COSMONAUTICS
    • B64U—UNMANNED AERIAL VEHICLES [UAV]; EQUIPMENT THEREFOR
    • B64U10/00—Type of UAV
    • B64U10/10—Rotorcrafts
    • B64U10/13—Flying platforms
    • B64U10/14—Flying platforms with four distinct rotor axes, e.g. quadcopters
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B64—AIRCRAFT; AVIATION; COSMONAUTICS
    • B64U—UNMANNED AERIAL VEHICLES [UAV]; EQUIPMENT THEREFOR
    • B64U20/00—Constructional aspects of UAVs
    • B64U20/80—Arrangement of on-board electronics, e.g. avionics systems or wiring
    • B64U20/87—Mounting of imaging devices, e.g. mounting of gimbals
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B64—AIRCRAFT; AVIATION; COSMONAUTICS
    • B64U—UNMANNED AERIAL VEHICLES [UAV]; EQUIPMENT THEREFOR
    • B64U2101/00—UAVs specially adapted for particular uses or applications
    • B64U2101/30—UAVs specially adapted for particular uses or applications for imaging, photography or videography
    • G—PHYSICS
    • G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
    • G03B—APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
    • G03B15/00—Special procedures for taking photographs; Apparatus therefor
    • G03B15/006—Apparatus mounted on flying objects

Definitions

  • the present invention relates to the field of image processing, and in particular, to an image processing method of a PTZ system, a PTZ system and an unmanned aerial vehicle.
  • Super-resolution reconstruction image technology is a type of technology that acquires high-resolution images from low-resolution images. It can be roughly divided into two categories: reconstructing high-resolution images from multiple low-resolution images, and reconstructing high-resolution images from a single low-resolution image. The super-resolution reconstructed image will provide more image details than the original image, so it has full applications in the fields of security and medical treatment.
  • the super-resolution reconstruction technology is mainly based on two types of algorithms: single frame or multi-frame. These methods basically use the method of modeling training first, then estimating prediction to reconstruct high-resolution images. Especially thanks to the rapid development of deep learning, the super-resolution reconstruction effect has been greatly improved. However, the reconstruction of high-resolution images using these methods still has the problem that the reconstructed details may be different from the real scene, and the reconstructed results have poor visual effects in some scenes.
  • An embodiment of the present invention provides an image processing method of a PTZ system, a PTZ system, and an unmanned aerial vehicle.
  • the image processing method is used in a PTZ system.
  • the PTZ system includes a PTZ and a zoom lens mounted on the PTZ.
  • the image processing method includes: controlling the zoom lens to take a frame of reference image at a first focal length, where the first focal length is a shooting focal length desired by a user; adjusting the zoom lens to a second focal length; controlling the zoom lens at The second focal segment captures multiple frames of far-focus images with different shooting ranges, and the focal length of the lens corresponding to the second focal segment is greater than the focal length of the lens corresponding to the first focal segment; stitching the multiple far-focus images into one frame stitching image; And, processing the reference image and the stitched image to obtain a target reconstructed image.
  • a reference image at a first focal length is captured by controlling a zoom lens, and then a plurality of frames of far-focus images with different shooting ranges are captured at a second focal length, and then the multiple frames are far-focused.
  • the images are stitched into a frame stitching image, and the reference image and the stitched image are processed to obtain a target reconstructed image. Since the resolution of the far-focus image captured in the second focal segment is high, the resolution of the target reconstructed image is high.
  • a gimbal system includes a gimbal, a zoom lens and a processor mounted on the gimbal.
  • the processor is configured to control the zoom lens to take a frame of reference image at a first focal length, where the first focal length is a desired focal length of the user; adjust the zoom lens to a second focal length; and control the zoom lens to
  • the second focal segment captures multiple frames of far-focus images with different shooting ranges, and the focal length of the lens corresponding to the second focal segment is greater than the focal length of the lens corresponding to the first focal segment; stitching the multiple telephoto images into a frame stitching image; and , Processing the reference image and the stitched image to obtain a target reconstructed image.
  • the processor is configured to control the zoom lens to take a reference image at a first focal length, and then take a plurality of frames of far focus images with different shooting ranges under the second focal length, and then stitch the multiple frames of far focus images together. Create a frame of stitched images, and process the reference and stitched images to obtain the target reconstructed image. Since the resolution of the far-focus image captured in the second focal segment is high, the resolution of the target reconstructed image is high.
  • the obtained reconstructed image has a small difference from the real scene, that is, it can truly reflect the target scene, and on the other hand, the visual effect in the target scene is better.
  • An unmanned aerial vehicle includes the gimbal system and a fuselage described in the above embodiment.
  • the gimbal system is disposed on the fuselage.
  • the drone according to the embodiment of the present invention controls a zoom lens to capture a reference image at a first focal length, and then captures a plurality of frames of far focus images with different shooting ranges under the second focal length, and then stitches the multiple frames of far focus images into a frame of stitched images. And process the reference image and the stitched image to obtain the target reconstructed image. Since the resolution of the far-focus image captured in the second focal segment is high, the resolution of the target reconstructed image is high. With the drone of the present invention, the target reconstructed image obtained on the one hand has a small difference from the real scene, that is, it can truly reflect the target scene, and on the other hand, the visual effect in the target scene is better.
  • FIG. 1 is a schematic structural diagram of a pan / tilt system according to an embodiment of the present invention
  • FIG. 2 is a schematic flowchart of an image processing method of a pan / tilt system according to an embodiment of the present invention
  • FIG. 3 is a schematic diagram of a reference image taken by a zoom lens according to an embodiment of the present invention at a first focal length
  • FIG. 4 is a schematic diagram of a far-focus image of multiple frames with different shooting ranges taken by a zoom lens in a second focal segment according to an embodiment of the present invention
  • FIG. 5 is a schematic diagram of stitching a multi-frame far-focus image of a partial region into a framed mosaic image of a partial region according to an embodiment of the present invention
  • FIG. 6 to 11 are schematic flowcharts of an image processing method of a pan / tilt system according to an embodiment of the present invention.
  • FIG. 12 is a schematic structural diagram of an unmanned aerial vehicle and a gimbal system according to an embodiment of the present invention.
  • first and second are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as “first” and “second” may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present invention, the meaning of "a plurality" is two or more, unless it is specifically and specifically defined otherwise.
  • an embodiment of the present invention provides an image processing method of the PTZ system 100.
  • the gimbal system 100 includes a gimbal 10 and a zoom lens 20 mounted on the gimbal 10.
  • the image processing method includes the following steps:
  • S11 Control the zoom lens 20 to take a frame of reference image at a first focal length, and the first focal length is a focal length desired by the user;
  • S15 Process the reference image and the stitched image to obtain a target reconstructed image.
  • the attitude of the pan / tilt head 10 may be a gesture that follows the movement of the imaging object.
  • the gimbal 10 includes one or more axes. When the head 10 includes only one axis, the head 10 is a single-axis head. When the head 10 includes a plurality of axes, the head 10 is a multi-axis head 10.
  • the multi-axis head 10 may be a two-axis head, such as a head including a yaw axis and a roll axis, or a head including a yaw axis and a pitch axis;
  • the multi-axis head 10 can also be a three-axis head, and includes a yaw axis, a roll axis, and a pitch axis.
  • the pan / tilt head 10 in this embodiment is schematically described by taking a three-axis pan / tilt as an example.
  • the zoom lens 20 can change the focal length within a certain shooting range. Under different focal lengths, the zoom lens 20 has different angles of view, different fields of view, and different sizes of framing shooting ranges, thereby shooting at different focal lengths. The shooting range of the images covering the scene will also be different (ie, the shooting range will be different).
  • the zoom lens 20 may be an optical zoom lens, a digital zoom lens, or a combination of multiple zoom lenses.
  • the zoom lens 20 is mounted on the gimbal 10 so that the zoom lens 20 follows the imaging object in real time.
  • the zoom lens 20 can be rotated relative to the Y (Yaw) axis, R (Roll) axis, and P (Pitch) axis of the pan / tilt head 10, or it can follow the other one when one or both axes of the pan / tilt head 10 are locked. Or the two axes are rotated, for example, when the R (Roll) axis and the P (Pitch) axis are locked, the zoom lens 20 is rotated only relative to the Y (Yaw) axis.
  • the zoom lens 20 in this embodiment can capture an image at a first focal length, or can transform from a first focal length to a second focal length, and then capture an image at a second focal length.
  • the first focal length is a focal length desired by the user.
  • the corresponding focal lengths of the lenses are small, the angle of view taken by the zoom lens 20 is larger, the subject is smaller, and the depth of field is longer.
  • the focal length of the lens corresponding to the second focal length is greater than the focal length of the lens corresponding to the first focal length.
  • the focal length of the lens corresponding to the second focal length is relatively large, the angle of view taken by the zoom lens 20 is small, the subject is large, and the depth of field is short.
  • an object to be imaged in the target scene 200 is determined, such as a person, an animal, a scene, and the like.
  • the zoom lens 20 is controlled to take a reference image at a first focal length.
  • the reference image has a larger viewing angle, a smaller subject, and a lower resolution.
  • the zoom lens 20 is controlled to capture a plurality of frames of far-focus images in different shooting ranges at the second focal length.
  • the multi-frame means two frames or more.
  • different shooting ranges indicate that the zoom lens 20 focuses on a plurality of different areas in the target scene 200 so as to obtain clear images of the plurality of different areas.
  • nine areas in the target scene 200 are focused at the second focal length.
  • the focal length of the zoom lens 20 focused in each area may be the same or slightly different, but all fall within the shooting range of the second focal range.
  • step S14 after obtaining multiple frames of the far-focus image, the multiple frames of the far-focus image are stitched to form a frame of the stitched image.
  • the far-focus image of the I area and the far-focus image of the VI area are stitched to form a stitched image of the I + IV area.
  • the far-focus images of the other seven areas are also stitched in order to form a frame of stitched images.
  • step S15 based on the reference image, the stitched image is further processed to obtain a target reconstructed image. Because the target reconstructed image is a mosaic of far-focus images with clear details, the target reconstructed image has high resolution and clear details, which can truly reflect the captured target scene 200, and in some scenes, no Good visual effects.
  • a reference image at a first focal length is captured by controlling the zoom lens 20, and then a plurality of frames of far-focus images at different shooting ranges are captured at a second focal length.
  • the telefocus images are stitched into a frame stitched image, and the reference image and stitched image are processed to obtain the target reconstructed image. Since the resolution of the far-focus image captured in the second focal segment is high, the resolution of the target reconstructed image is high.
  • At least two frames of the far-focus images in different frames with different shooting ranges are adjacent in the horizontal direction, and at least two frames of the far-focus images in multiple frames with different shooting ranges are in the same vertical range. adjacent.
  • At least two frames of the far-focus images are connected in the horizontal direction or there are overlaps, at least two frames of the far-focus images with different shooting ranges are adjacent in the horizontal direction.
  • the shooting ranges of at least two frames of the far-focus images in the vertical direction are contiguous or overlapped, the shooting ranges of the two frames of at least two different far-focus images in the vertical direction are adjacent.
  • FIG. 4 and continue to take nine frames of the far-focus image as an example.
  • the two frames of the far-focus image adjacent to each other in the horizontal direction or the vertical shooting range are adjacent.
  • the shooting range of the far-focus image in the I region and the far-focus image of the II region in the horizontal direction are adjacent, and the shooting range of the far-focus image in the I region and the tele-focus image in the IV region are in the vertical direction.
  • the feature point pairs are matched, and the far-focus image of the I area and the far-focus image of the IV area are stitched by an algorithm to obtain a stitched image of the I + IV area.
  • the far-focus images of the other seven regions are also stitched in order to form a frame of stitched images. Since at least two frames of the far-focus images with different shooting ranges are adjacent in the horizontal direction and the vertical direction, when the far-focus images with different shooting ranges are stitched into one frame stitching image, the two adjacent frames are far away.
  • the focus image does not appear tomographic in the horizontal and vertical directions (the target scene 200 is discontinuous).
  • the far-focus images in different shooting ranges of multiple frames form a matrix of m * n, where m ⁇ 2, n ⁇ 2, and m and n are integers.
  • m can be an integer such as 2, 3, 4, 10, 20, 100
  • n can be an integer. It is an integer such as 2, 6, 8, 15, 25, 80, and so on.
  • the larger the values of m and n the greater the number of details of the target scene 200, that is, the target scene 200 can be restored more.
  • the zoom lens 20 captures the far focus images of the four areas at the second focal length, so as to obtain four frames with 2 * 2 Far focus image of matrix distribution.
  • the zoom lens 20 sequentially focuses the nine regions at the second focal length, and shoots the far-focus images of the nine regions. , So as to obtain nine frames to form a 3 * 3 matrix distributed far-focus image.
  • the image processing method further includes the following steps:
  • the size of the target reconstructed image is determined by one parameter or multiple parameters such as size, resolution, and the like.
  • the size of the target reconstructed image may be the same as the size of the reference pattern, or may be larger than the size of the reference image.
  • the resolution of the target reconstructed image needs to be further determined.
  • the resolution of the target reconstructed image can be set according to the requirements of the displayed image, or it can be determined according to the resolution of the reference image.
  • the second focal length is calculated on the basis of the first focal length, so as to obtain the second focal length corresponding to the larger focal length of the lens.
  • the pan / tilt head 10 has one or more axes, and the image processing method further includes the following steps:
  • the attitude angle of the gimbal 10 may be a yaw axis attitude angle, a roll axis attitude angle, and a pitch axis attitude angle.
  • Means. Obtain the close focus attitude angle of the yaw axis, the close focus state angle of the roll axis, and the close focus attitude angle of the pitch axis in order, and determine the far focus attitude of the gimbal 10 when the zoom lens 20 shoots the far focus image according to the second focal length.
  • Angles calculate the direction and angle of the rotation required to adjust the near-focus attitude angle to the distant attitude angle.
  • the angle and direction to obtain the required rotation can be calculated by formulas, such as converting the yaw axis attitude angle, roll axis attitude angle, and pitch axis attitude angle from the near focus attitude angle and the far focus attitude angle to Represented by a quaternion to quickly calculate the angle and direction of the required rotation.
  • the angle and direction of the required rotation can also be obtained by looking up the table.
  • step S15 the step of processing the reference image and the stitched image to obtain the target reconstructed image includes the following sub-steps:
  • S153 Perform crop processing on the stitched image according to the mapping matrix to obtain a target reconstructed image.
  • the reference image is upsampled, that is, based on the reference image, a new pixel is inserted between each pixel using a suitable interpolation algorithm to obtain a sampled image.
  • a suitable interpolation algorithm include nearest-neighbor interpolation, bilinear interpolation, mean interpolation, and median interpolation. Taking the nearest neighbor difference algorithm as an example, among the four neighboring pixels of the pixel to be determined, the gray value of the neighboring pixel closest to the pixel to be determined is assigned to the pixel to be determined, thereby achieving fast interpolation.
  • the one-to-one mapping relationship between the corresponding points of the sampled image and the stitched image that is, calculate the mapping matrix of the sampled image and the stitched image. Since the stitched image is composed of multiple frames of far-focus images, the size of the stitched image is larger than the size of the sampled image. Therefore, the redundant part of the stitched image and the sampled image need to be trimmed. According to the mapping matrix of the sampled image and the stitched image, the difference pixel area of the sampled image and the stitched image can be determined, and the stitched image is trimmed according to the mapping matrix to remove the redundant part of the stitched image and the sampled image.
  • a far-focus image that is consistent with the size of the target reconstructed image need only be stitched, and the target reconstructed image can be obtained without trimming the stitched image.
  • the target reconstructed image can reconstruct the target reconstructed image.
  • the mapping matrix includes a homography matrix (Homography matrix) or an affine transformation matrix (Affine transformation matrix).
  • Homography matrix homography matrix
  • Adjae transformation matrix affine transformation matrix
  • step S151 the step of up-sampling the reference image to obtain an up-sampled image includes the following sub-steps:
  • S1512 The reference image is up-sampled by a cubic spline interpolation algorithm to obtain an up-sampled image.
  • the upsampling interpolation algorithm may be a bilinear interpolation (Culinear) and a cubic spline interpolation (Cubic) interpolation algorithm.
  • the bilinear interpolation algorithm performs linear interpolation in the horizontal and vertical directions of the reference image
  • the cubic spline interpolation algorithm uses the three bending moment equation and the first boundary condition to segment the reference image into cubic interpolations.
  • the bi-linear interpolation algorithm or the cubic spline interpolation algorithm is used to up-sample the reference image, and the sampled image is less prone to distortion.
  • the image processing method further includes the following steps:
  • S20 Perform a truncation operation on at least one frame of the far-focus image according to the size of the target reconstructed image;
  • step S14 the step of stitching multiple frames of far-focus images into one stitched image includes the following sub-steps:
  • S141 Merge the far-focus image and other far-focus images after the interception operation into a frame stitching image.
  • an embodiment of the present invention provides a PTZ system 100.
  • the image processing method of the above embodiment can be applied to the PTZ system 100.
  • the gimbal system 100 includes a gimbal 10, a zoom lens 20 mounted on the gimbal 10, and a processor 30.
  • the processor 30 is configured to: control the zoom lens 20 to take a frame of reference image at the first focal length, the first focal length is the user's desired focal length; adjust the zoom lens 20 to the second focal length; control the zoom lens 20 to take multiple frames at the second focal length For far-focus images of different shooting ranges, the focal length of the lens corresponding to the second focal length is greater than the focal length of the lens corresponding to the first focal length; stitching multiple frames of far-focus images into one stitching image; and, processing the reference image and the stitching image to obtain the target reconstructed image.
  • the processor 30 may be used to implement steps S11, S12, S13, S14, and S15.
  • the attitude of the pan / tilt head 10 may be a gesture that follows the movement of the imaging object.
  • the gimbal 10 includes one or more axes. When the head 10 includes only one axis, the head 10 is a single-axis head. When the head 10 includes a plurality of axes, the head 10 is a multi-axis head 10.
  • the multi-axis head 10 may be a two-axis head, such as a head including a yaw axis and a roll axis, or a head including a yaw axis and a pitch axis;
  • the multi-axis head 10 can also be a three-axis head, and includes a yaw axis, a roll axis, and a pitch axis.
  • the pan / tilt head 10 in this embodiment is schematically described by taking a three-axis pan / tilt as an example.
  • the zoom lens 20 can change the focal length within a certain shooting range. Under different focal lengths, the zoom lens 20 has a different angle of view, a field of view shooting range of different sizes, and a viewfinder shooting range of different sizes, thereby shooting at different focal lengths. The shooting range of the image covering the scene will also be different (ie, the shooting range will be different).
  • the zoom lens 20 may be an optical zoom lens, a digital zoom lens, or a combination of multiple zoom lenses.
  • the zoom lens 20 is mounted on the gimbal 10 so that the zoom lens 20 follows the imaging object in real time.
  • the zoom lens 20 can be rotated relative to the Y (Yaw) axis, R (Roll) axis, and P (Pitch) axis of the pan / tilt head 10, or it can follow the other one when one or both axes of the pan / tilt head 10 are locked. Or the two axes are rotated, for example, when the R (Roll) axis and the P (Pitch) axis are locked, the zoom lens 20 is rotated only relative to the Y (Yaw) axis.
  • the zoom lens 20 in this embodiment can capture an image at a first focal length, or can transform from a first focal length to a second focal length, and then capture an image at a second focal length.
  • the first focal length is a focal length desired by the user.
  • the corresponding focal lengths of the lenses are small, the angle of view taken by the zoom lens 20 is larger, the subject is smaller, and the depth of field is longer.
  • the focal length of the lens corresponding to the second focal length is greater than the focal length of the lens corresponding to the first focal length.
  • the focal length of the lens corresponding to the second focal length is larger, and the zoom lens 20 has a smaller angle of view, a larger subject, and a shorter depth of field.
  • an object to be imaged in the target scene 200 is determined, such as a person, an animal, a scene, and the like.
  • the processor 30 controls the zoom lens 20 to take a reference image at a first focal length.
  • the reference image has a larger viewing angle, a smaller subject, and a lower resolution.
  • the processor 30 controls the zoom lens 20 to take a plurality of frames of far-focus images in different shooting ranges at the second focal length.
  • the multi-frame means two frames or more.
  • different shooting ranges indicate that the zoom lens 20 focuses on a plurality of different areas in the target scene 200 so as to obtain clear images of the plurality of different areas.
  • nine areas in the target scene 200 are focused at the second focal length.
  • the focal length of the zoom lens 20 focused in each area may be the same or slightly different, but all fall within the shooting range of the second focal range.
  • nine frames of far-focus images of nine different shooting ranges are obtained, and each frame of the far-focus images can clearly display details of at least one region in the target scene 200.
  • the processor 30 obtains the multi-frame far-focus images, the multi-frame far-focus images are stitched to form one frame of stitched images. As shown in FIG.
  • the far-focus image of the I area and the far-focus image of the VI area are stitched to form a stitched image of the I + IV area.
  • the far-focus images of the other seven areas are also stitched in order to form a frame of stitched images.
  • the processor 30 further processes the stitched image based on the reference image to obtain a target reconstructed image. Because the target reconstructed image is a mosaic of far-focus images with clear details, the target reconstructed image has high resolution and clear details, which can truly reflect the captured target scene 200, and in some scenes, no Good visual effects.
  • the processor 30 is configured to control the zoom lens 20 to take a reference image at a first focal length, and then take a plurality of frames of far-focus images at different shooting ranges under the second focal length, and then The telefocus images are stitched into a frame stitched image, and the reference image and stitched image are processed to obtain the target reconstructed image. Since the resolution of the far-focus image captured in the second focal segment is high, the resolution of the target reconstructed image is high.
  • the obtained reconstructed image has a small difference from the real scene, that is, it can truly reflect the target scene 200, and on the other hand, the visual effect in the target scene is better.
  • At least two frames of the far-focus images in different frames with different shooting ranges are adjacent in the horizontal direction, and at least two frames of the far-focus images in multiple frames with different shooting ranges are in the same vertical range. adjacent.
  • At least two frames of the far-focus images are connected in the horizontal direction or there are overlaps, at least two frames of the far-focus images with different shooting ranges are adjacent in the horizontal direction.
  • the shooting ranges of at least two frames of the far-focus images in the vertical direction are contiguous or overlapped, the shooting ranges of the two frames of at least two different far-focus images in the vertical direction are adjacent.
  • FIG. 4 and continue to take nine frames of the far-focus image as an example.
  • the two frames of the far-focus image adjacent to each other in the horizontal direction or the vertical shooting range are adjacent.
  • the shooting range of the far-focus image in the I region is adjacent to the shooting range of the far-focus image in the II region in the horizontal direction
  • the shooting range of the far-focus image in the I region is adjacent to the shooting range in the vertical direction of the far-focus image in the IV region.
  • the feature point pairs are matched, and the far-focus image of the I area and the far-focus image of the IV area are stitched by an algorithm to obtain a stitched image of the I + IV area.
  • the far-focus images of the other seven regions are also stitched in order to form a frame of stitched images. Since at least two frames of the far-focus images with different shooting ranges are adjacent in the horizontal direction and the vertical direction, when the far-focus images with different shooting ranges are stitched into one frame stitching image, the two adjacent frames are far away.
  • the focus image does not appear tomographic in the horizontal and vertical directions (the target scene 200 is discontinuous).
  • the far-focus images in different shooting ranges of multiple frames form a matrix of m * n, where m ⁇ 2, n ⁇ 2, and m and n are integers.
  • m can be an integer such as 2, 3, 4, 10, 20, 100
  • n can be an integer. It is an integer such as 2, 6, 8, 15, 25, 80, and so on.
  • the larger the values of m and n the greater the number of details of the target scene 200, that is, the target scene 200 can be restored more.
  • the zoom lens 20 captures the far focus images of the four areas at the second focal length, so as to obtain four frames with 2 * 2 Far focus image of matrix distribution.
  • the zoom lens 20 sequentially focuses the nine regions at the second focal length, and shoots the far-focus images of the nine regions. , So as to obtain nine frames to form a 3 * 3 matrix distributed far-focus image.
  • the processor 30 is further configured to calculate the second focal length according to the first focal length and the size of the target reconstructed image. With reference to FIG. 6, that is, the processor 30 may also be used to implement step S16.
  • the size of the target reconstructed image is determined by one parameter or multiple parameters such as size, resolution, and the like.
  • the size of the target reconstructed image may be the same as the size of the reference pattern, or may be larger than the size of the reference image.
  • the resolution of the target reconstructed image needs to be further determined.
  • the resolution of the target reconstructed image can be set according to the requirements of the displayed image, or it can be determined according to the resolution of the reference image.
  • the second focal length is calculated on the basis of the first focal length, so as to obtain the second focal length corresponding to the larger focal length of the lens.
  • the gimbal 10 has one or more axes
  • the processor 30 is further configured to: obtain a close-focus attitude angle of each axis of the gimbal 10 when the zoom lens 20 takes a reference image; according to the second focal length and The close focus attitude angle calculates the telephoto attitude angle of the gimbal 10 when the zoom lens 20 shoots a telefocus image; and, each axis of the gimbal 10 is controlled to be adjusted from the close focus attitude angle to the near focus attitude angle and the far focus attitude angle. Telephoto attitude angle.
  • the processor 30 can also be used to implement steps S17, S18, and S19.
  • the attitude angle of the gimbal 10 may be a yaw axis attitude angle, a roll axis attitude angle, and a pitch axis attitude angle.
  • Means. Obtain the close focus attitude angle of the yaw axis, the close focus state angle of the roll axis, and the close focus attitude angle of the pitch axis in order, and determine the far focus attitude of the gimbal 10 when the zoom lens 20 shoots the far focus image according to the second focal length.
  • Angles calculate the direction and angle of the rotation required to adjust the near-focus attitude angle to the distant attitude angle.
  • the angle and direction to obtain the required rotation can be calculated by formulas, such as converting the yaw axis attitude angle, roll axis attitude angle, and pitch axis attitude angle from the near focus attitude angle and the far focus attitude angle to Represented by a quaternion to quickly calculate the angle and direction of the required rotation.
  • the angle and direction of the required rotation can also be obtained by looking up the table.
  • the processor 30 is further configured to: up-sample the reference image to obtain an up-sampled image; calculate a mapping matrix between the up-sampled image and the stitched image; and perform crop processing on the stitched image according to the mapping matrix to: A target reconstructed image is obtained.
  • the processor 30 can also be used to implement steps S151, S152, and S153.
  • the processor 30 performs upsampling on the reference image, that is, based on the reference image, a new pixel is inserted between each pixel using an appropriate interpolation algorithm to obtain a sampled image.
  • interpolation algorithms include nearest-neighbor interpolation, bilinear interpolation, mean interpolation, and median interpolation. Taking the nearest neighbor difference algorithm as an example, among the four neighboring pixels of the pixel to be determined, the gray value of the neighboring pixel closest to the pixel to be determined is assigned to the pixel to be determined, thereby achieving fast interpolation.
  • the one-to-one mapping relationship between the corresponding points of the sampled image and the stitched image that is, calculate the mapping matrix of the sampled image and the stitched image. Since the stitched image is composed of multiple frames of far-focus images, the size of the stitched image is larger than the size of the sampled image. Therefore, the redundant part of the stitched image and the sampled image need to be trimmed. According to the mapping matrix of the sampled image and the stitched image, the difference pixel area of the sampled image and the stitched image can be determined, and the stitched image is trimmed according to the mapping matrix to remove the redundant part of the stitched image and the sampled image.
  • the target reconstructed image according to the size of the target reconstructed image, only the far-focus images that are consistent with the size of the target reconstructed image need to be stitched, and the target reconstructed image can be obtained without trimming the stitched image.
  • the far-focus images of the central area are stitched, and the obtained stitched image can reconstruct the target reconstructed image.
  • the mapping matrix includes a homography matrix (Homography matrix) or an affine transformation matrix (Affine transformation matrix).
  • Homography matrix homography matrix
  • Adjae transformation matrix affine transformation matrix
  • the processor 30 is further configured to: up-sample the reference image by using a bilinear interpolation algorithm to obtain an up-sampled image; or up-sample the reference image by using a cubic spline interpolation algorithm to obtain an up-sampled image .
  • the processor 30 may also be used to implement step S1511 or S1512.
  • the upsampling interpolation algorithm may be a bilinear interpolation (Culinear) and a cubic spline interpolation (Cubic) interpolation algorithm.
  • the bilinear interpolation algorithm performs linear interpolation in the horizontal and vertical directions of the reference image
  • the cubic spline interpolation algorithm uses the three bending moment equation and the first boundary condition to segment the reference image into cubic interpolations.
  • the bi-linear interpolation algorithm or the cubic spline interpolation algorithm is used to up-sample the reference image, and the sampled image is less prone to distortion.
  • the processor 30 is further configured to: perform an interception operation on at least one frame of the far-focus image according to the size of the target reconstructed image; and stitch the far-focus image and other far-focus images after the interception operation into a frame stitching image.
  • the processor 30 may also be used to implement steps S20 and S141.
  • an embodiment of the present invention provides a drone 1000.
  • the drone 1000 includes the gimbal system 100 and the airframe 300 in the above embodiment.
  • the gimbal system 100 is provided on the body 300.
  • the drone 1000 may be a four-rotor aircraft, a six-rotor aircraft, an eight-rotor aircraft, etc., which is not limited herein.
  • the unmanned aerial vehicle 1000 in this embodiment is exemplified by a quadrotor aircraft, which can be equipped with a gimbal system 100.
  • the gimbal 10 is fixedly connected to the body 300, and the zoom lens 20 is mounted on the gimbal 10.
  • the zoom lens 20 is used for taking a picture or photography when the drone 1000 is flying or hovering.
  • the drone 1000 controls a zoom lens 20 to capture a reference image at a first focal length, and then captures a plurality of frames of far-focus images with different shooting ranges under a second focal length, and then stitches the multiple-frame far-focus images into one frame. Stitch the images and process the reference and stitched images to get the target reconstructed image. Since the resolution of the far-focus image captured in the second focal segment is high, the resolution of the target reconstructed image is high.
  • the unmanned aerial vehicle 1000 of the present invention on the one hand, the target reconstructed image obtained has a small difference from the real scene, that is, it can truly reflect the target scene 200, and on the other hand, the visual effect in the target scene is better.
  • Any process or method description in a flowchart or otherwise described herein can be understood as a module, fragment, or portion of code that includes one or more executable instructions for implementing a particular logical function or step of a process
  • the scope of the preferred embodiments of the present invention includes additional implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order according to the functions involved, which should It is understood by those skilled in the art to which the embodiments of the present invention pertain.
  • Logic and / or steps represented in a flowchart or otherwise described herein, for example, a sequenced list of executable instructions that may be considered to implement a logical function, may be embodied in any computer-readable medium, For use by instruction execution systems, devices, or devices (such as computer-based systems, systems that include processing modules, or other systems that can take instructions from and execute instructions) Or equipment.
  • a "computer-readable medium” may be any device that can contain, store, communicate, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device.
  • computer-readable media include the following: electrical connections (control methods) with one or more wirings, portable computer disk cartridges (magnetic devices), random access memory (RAM), Read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disk read-only memory (CDROM).
  • the computer-readable medium may even be paper or other suitable medium on which the program can be printed, because, for example, by optically scanning the paper or other medium, followed by editing, interpretation, or other suitable Processing to obtain the program electronically and then store it in computer memory.
  • each part of the embodiments of the present invention may be implemented by hardware, software, firmware, or a combination thereof.
  • multiple steps or methods may be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system.
  • a suitable instruction execution system For example, if implemented in hardware, as in another embodiment, it may be implemented using any one or a combination of the following techniques known in the art: Discrete logic circuits, application-specific integrated circuits with suitable combinational logic gate circuits, programmable gate arrays (PGA), field programmable gate arrays (FPGA), etc.
  • a person of ordinary skill in the art can understand that all or part of the steps carried by the methods in the foregoing embodiments can be implemented by a program instructing related hardware.
  • the program can be stored in a computer-readable storage medium.
  • the program is When executed, one or a combination of the steps of the method embodiment is included.
  • each functional unit in each embodiment of the present invention may be integrated into one processing module, or each unit may exist separately physically, or two or more units may be integrated into one module.
  • the above integrated modules may be implemented in the form of hardware or software functional modules. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.
  • the aforementioned storage medium may be a read-only memory, a magnetic disk, or an optical disk.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Computing Systems (AREA)
  • Studio Devices (AREA)
  • Aviation & Aerospace Engineering (AREA)
  • Image Processing (AREA)

Abstract

一种云台系统（100）及其图像处理方法。云台系统（100）包括云台（10）和搭载在云台（10）上的变焦镜头（20）。图像处理方法包括：控制变焦镜头（20）在第一焦段拍摄一帧参考图像，第一焦段为用户期望的拍摄焦段；调整变焦镜头（20）至第二焦段；控制变焦镜头（20）在第二焦段拍摄多帧不同拍摄范围的远焦图像，第二焦段对应的镜头焦距大于第一焦段对应的镜头焦距；将多帧远焦图像拼接成一帧拼接图像；和处理参考图像及拼接图像以得到目标重建图像。

Description

云台系统及其图像处理方法、无人机 技术领域
本发明涉及图像处理领域，特别涉及一种云台系统的图像处理方法、云台系统及无人机。
背景技术
超分辨重建图像技术是从低分辨率图像获取高分辨率图像的一类技术。大体可以分为两类：从多张低分辨图像重建出高分辨率图像，以及从单张低分辨率图像重建高分辨率图像。经过超分辨率重建的图像会比原图像提供更多的图像细节，因此在安防、医疗等领域都有充分的应用。
目前，超分辨重建技术主要基于单帧或者基于多帧两类算法。这些方法基本采用先建模训练，然后估计预测的方式重建出高分辨率图像。特别得益于深度学习的快速发展，超分辨率重建效果有了很大的提升。然而，采用这些方法重建高分辨率图像仍存在重建的细节可能与真实场景存在差异、重建的结果在某些场景下的视觉效果较差的问题。
发明内容
本发明实施方式提供一种云台系统的图像处理方法、云台系统及无人机。
本发明实施方式的图像处理方法用于云台系统。所述云台系统包括云台和搭载在所述云台上的变焦镜头。所述图像处理方法包括：控制所述变焦镜头在第一焦段拍摄一帧参考图像，所述第一焦段为用户期望的拍摄焦段；调整所述变焦镜头至第二焦段；控制所述变焦镜头在所述第二焦段拍摄多帧不同拍摄范围的远焦图像，所述第二焦段对应的镜头焦距大于所述第一焦段对应的镜头焦距；将多帧所述远焦图像拼接成一帧拼接图像；和，处理所述参考图像及所述拼接图像以得到目标重建图像。
本发明实施方式的云台系统的图像处理方法中，通过控制变焦镜头拍摄第一焦段下的参考图像，然后再拍摄第二焦段下多帧不同拍摄范围的远焦图像，接着将多帧远焦图像拼接成一帧拼接图像，并处理参考图像及拼接图像以得到目标重建图像。由于第二焦段拍摄的远焦图像的分辨率高，因此目标重建图像的分辨率较高。采用本发明的图像处理方法，获得的目标重建图像一方面与真实场景的差异较小，即能够真实反映目标场景，另一方面，在目标场景下的视觉效果较好。
本发明实施方式的云台系统包括云台、搭载在所述云台上的变焦镜头以及处理器。 所述处理器用于：控制所述变焦镜头在第一焦段拍摄一帧参考图像，所述第一焦段为用户期望的拍摄焦段；调整所述变焦镜头至第二焦段；控制所述变焦镜头在所述第二焦段拍摄多帧不同拍摄范围的远焦图像，所述第二焦段对应的镜头焦距大于所述第一焦段对应的镜头焦距；将多帧所述远焦图像拼接成一帧拼接图像；和，处理所述参考图像及所述拼接图像以得到目标重建图像。
本发明实施方式的云台系统中，处理器用于控制变焦镜头拍摄第一焦段下的参考图像，然后再拍摄第二焦段下多帧不同拍摄范围的远焦图像，接着将多帧远焦图像拼接成一帧拼接图像，并处理参考图像及拼接图像以得到目标重建图像。由于第二焦段拍摄的远焦图像的分辨率高，因此目标重建图像的分辨率较高。采用本发明的云台系统，获得的目标重建图像一方面与真实场景的差异较小，即能够真实反映目标场景，另一方面，在目标场景下的视觉效果较好。
本发明实施方式的无人机包括上述实施方式所述的云台系统和机身。所述云台系统设置在所述机身上。
本发明实施方式的无人机通过控制变焦镜头拍摄第一焦段下的参考图像，然后再拍摄第二焦段下多帧不同拍摄范围的远焦图像，接着将多帧远焦图像拼接成一帧拼接图像，并处理参考图像及拼接图像以得到目标重建图像。由于第二焦段拍摄的远焦图像的分辨率高，因此目标重建图像的分辨率较高。采用本发明的无人机，获得的目标重建图像一方面与真实场景的差异较小，即能够真实反映目标场景，另一方面，在目标场景下的视觉效果较好。
本发明实施方式的附加方面和优点将在下面的描述中部分给出，部分将从下面的描述中变得明显，或通过本发明的实践了解到。
附图说明
本发明的上述和/或附加的方面和优点可以从结合下面附图对实施方式的描述中将变得明显和容易理解，其中：
图1是本发明实施方式的云台系统的结构示意图；
图2是本发明实施方式的云台系统的图像处理方法的流程示意图；
图3为本发明实施方式的变焦镜头在第一焦段拍摄的参考图像的示意图；
图4为本发明实施方式的变焦镜头在第二焦段拍摄的多帧不同拍摄范围的远焦图像的示意图；
图5是本发明实施方式的部分区域的多帧远焦图像拼接成一帧部分区域的拼接图像的示意图；
图6至图11是本发明实施方式的云台系统的图像处理方法的流程示意图；及
图12是本发明实施方式的无人机与云台系统的结构示意图。
具体实施方式
下面详细描述本发明的实施方式，实施方式的示例在附图中示出，其中，相同或类似的标号自始至终表示相同或类似的元件或具有相同或类似功能的元件。下面通过参考附图描述的实施方式是示例性的，仅用于解释本发明，而不能理解为对本发明的限制。
在本发明的实施方式的描述中，术语“第一”、“第二”仅用于描述目的，而不能理解为指示或暗示相对重要性或者隐含指明所指示的技术特征的数量。由此，限定有“第一”、“第二”的特征可以明示或者隐含地包括一个或者更多个所述特征。在本发明的实施方式的描述中，“多个”的含义是两个或两个以上，除非另有明确具体的限定。
请参阅图1和图2，本发明实施方式提供一种云台系统100的图像处理方法。其中，云台系统100包括云台10和搭载在云台10上的变焦镜头20。图像处理方法包括以下步骤：
S11：控制变焦镜头20在第一焦段拍摄一帧参考图像，第一焦段为用户期望的拍摄焦段；
S12：调整变焦镜头20至第二焦段；
S13：控制变焦镜头20在第二焦段拍摄多帧不同拍摄范围的远焦图像，第二焦段对应的镜头焦距大于第一焦段对应的镜头焦距；
S14：将多帧远焦图像拼接成一帧拼接图像；和
S15：处理参考图像及拼接图像以得到目标重建图像。
具体地，云台10的姿态可以呈跟随成像物体运动的姿态。其中，云台10包括一个轴或多个轴。当云台10仅包括一个轴时，云台10为单轴云台。当云台10包括多个轴时，云台10为多轴云台10。多轴云台10可以为两轴云台，例如包括偏航(Yaw)轴与横滚(Roll)轴的云台，或者是包括偏航(Yaw)轴与俯仰(Pitch)轴的云台；多轴云台10也可以为三轴云台，同时包括偏航(Yaw)轴、横滚(Roll)轴和俯仰(Pitch)轴。本实施例的云台10以三轴云台为例进行示意性说明。
变焦镜头20能够在一定拍摄范围内变换焦距，不同焦距下，变焦镜头20具有不同大小的视场角，具有不同大小的视野范围，以及不同大小的取景拍摄范围，由此在不同焦距下拍摄出的图像涵盖场景的拍摄范围也会不同(即，拍摄范围不同)。在本实施例中，变焦镜头20可以为光学变焦镜头、数字变焦镜头、或者多种变焦镜头的组 合。变焦镜头20搭载在云台10上，以使变焦镜头20实时跟随成像物体。具体地，变焦镜头20可以相对云台10的Y(Yaw)轴、R(Roll)轴和P(Pitch)轴转动，也可以在云台10的一个或者两个轴锁定时，跟随其他的一个或者两个轴转动，例如在R(Roll)轴和P(Pitch)轴锁定时，变焦镜头20仅相对Y(Yaw)轴转动。本实施例中的变焦镜头20可以在第一焦段下拍摄图像，也可以从第一焦段变换为第二焦段，然后在第二焦段下拍摄图像。其中，第一焦段为用户期望的拍摄焦段。在第一焦段内，对应的镜头焦距均较小，变焦镜头20拍摄的视角较大、主体较小、景深较长。第二焦段对应的镜头焦距大于第一焦段对应的镜头焦距。在第二焦段内，第二焦段对应的镜头焦距均较大，变焦镜头20拍摄的视角较小、主体较大、景深较短。
请结合图3，本发明实施方式的图像处理方法中，首先确定目标场景200中待成像的物体，例如人物、动物、景物等。在步骤S11中，控制变焦镜头20在第一焦段拍摄一帧参考图像。此时，参考图像的视角较大、主体较小且分辨率较低。在步骤S13中，控制变焦镜头20在第二焦段拍摄多帧不同拍摄范围的远焦图像。在本实施例中，多帧表示两帧或者两帧以上。此外，不同拍摄范围表示变焦镜头20对目标场景200中多个不同区域进行对焦，如此以得到多个不同区域的清晰图像。例如图4中，对目标场景200中的九个区域以第二焦段进行对焦，例如，在I区域中，对太阳进行对焦，在VI区域中，对左起第三座山的山峰进行对焦。其中，每个区域对焦的变焦镜头20的镜头焦距可以相同，也可以略有差异，但均落入第二焦段的拍摄范围内。在变焦镜头20以第二焦段拍摄后，获得九个不同拍摄范围的九帧远焦图像，每帧远焦图像均能清晰地显示目标场景200中的至少一个区域的细节。在步骤S14中，在获得多帧远焦图像后，对多帧远焦图像进行拼接，以形成一帧拼接图像。如图5，将I区域的远焦图像以及VI区域的远焦图像进行拼接，以形成I+IV区域的拼接图像。其他七个区域的远焦图像也依次进行拼接，从而形成一帧拼接图像。在步骤S15中，以参考图像为参照基础，进一步地处理拼接图像，以得到目标重建图像。由于目标重建图像是由多帧细节清晰的远焦图像拼接而成，因此，目标重建图像的分辨率高而且细节清晰，能够真实反映拍摄的目标场景200，而且在某些场景下不会引入不好的视觉效果问题。
本发明实施方式的云台系统100的图像处理方法中，通过控制变焦镜头20拍摄第一焦段下的参考图像，然后再拍摄第二焦段下多帧不同拍摄范围的远焦图像，接着将多帧远焦图像拼接成一帧拼接图像，并处理参考图像及拼接图像以得到目标重建图像。由于第二焦段拍摄的远焦图像的分辨率高，因此目标重建图像的分辨率较高。采用本发明的图像处理方法，获得的目标重建图像一方面与真实场景的差异较小，即能够真实反映目标场景200，另一方面，在目标场景200下的视觉效果较好。
在本实施例中，多帧不同拍摄范围的远焦图像中的至少两帧在水平方向上拍摄范围相邻，多帧不同拍摄范围的远焦图像中的至少两帧在垂直方向上拍摄范围相邻。
当至少有两帧远焦图像在水平方向上拍摄范围相接或者存在重叠时，至少有两帧不同拍摄范围的远焦图像在水平方向上的拍摄范围相邻。此外，当至少有两帧远焦图像在垂直方向上拍摄范围相接或者存在重叠时，至少有不同拍摄范围的两帧远焦图像在垂直方向上的拍摄范围相邻。请参阅图4，继续以九帧远焦图像为例，区域相邻的两帧远焦图像在水平方向上或者垂直方向上的拍摄范围相邻。例如I区域的远焦图像与II区域的远焦图像在水平方向上的拍摄范围相邻，并且I区域的远焦图像与IV区域的远焦图像在垂直方向上的拍摄范围相邻。当至少有两帧远焦图像在水平方向或者垂直方向上的拍摄范围相接或者存在重叠时，可以先提取相邻的两帧远焦图像相接的边界上或者重叠区域的特征点，然后根据匹配特征点的方式进行拼接。如图5，以拼接I区域的远焦图像与IV区域的远焦图像为例，I区域的远焦图像的X区域、Y区域上均存在特征点(分别用小黑点表示)，IV区域的远焦图像的X’区域、Y’区域上均存在特征点(分别用小黑点表示)。然后通过匹配X区域与X’区域的特征点，Y区域与Y’区域的特征点，从而得到I区域的远焦图像与IV区域的远焦图像之间的特征点对。然后匹配特征点对，并采用算法拼接I区域的远焦图像与IV区域的远焦图像，得到I+IV区域的拼接图像。依此类推，其他七个区域的远焦图像也依次进行拼接，从而形成一帧拼接图像。由于多帧不同拍摄范围的远焦图像的至少两帧在水平方向以及垂直方向上拍摄范围相邻，在将多帧不同拍摄范围的远焦图像拼接成一帧拼接图像时，相邻的两帧远焦图像在水平方向以及垂直方向不会出现断层(目标场景200不连续)。
在本实施例中，多帧不同拍摄范围的远焦图像形成m*n的矩阵，其中m≥2，n≥2，且m、n均为整数。
在参考图像的尺寸一定的情况，可以根据目标场景200的细节数量选取不同拍摄范围的对焦区域，例如，m可以取值为2，3，4，10，20，100等整数，n可以取值为2，6，8，15，25，80等整数。m和n的取值越大，目标场景200的细节数量更多，即更能还原目标场景200。例如，若需要得到左上、右上、左下以及右下的四个区域均清晰的目标重建图像，变焦镜头20在第二焦段拍摄该四个区域的远焦图像，如此以得到四帧以2*2矩阵分布的远焦图像。再以图4的实施例为例，若需要得到九个区域均清晰的目标重建图像，则变焦镜头20在第二焦段依次对这九个区域进行对焦，并拍摄该九个区域的远焦图像，从而得到九帧以形成3*3矩阵分布的远焦图像。
请参阅图6，在本实施例中，在步骤S12之前，图像处理方法还包括以下步骤：
S16：根据第一焦段及目标重建图像的大小计算第二焦段。
具体地，目标重建图像的大小由尺寸、分辨率等的一个参数或者多个参数决定。目标重建图像的尺寸可以与参考图案的尺寸一致，也可以大于参考图像的尺寸。在目标重建图像的尺寸确定的情况下，需要进一步确定目标重建图像的分辨率。目标重建图像的分辨率可以根据显示图像的要求设定，也可以根据参考图像的分辨率确定。最后，根据目标重建图像的分辨率以及尺寸，在第一焦段的基础上，计算第二焦段，如此，以得到对应镜头焦距较大的第二焦段。
请参阅图7，在本实施例中，云台10具有一个轴或多个轴，图像处理方法还包括以下步骤：
S17：获取云台10在变焦镜头20拍摄参考图像时每一个轴的近焦姿态角；
S18：根据第二焦段及近焦姿态角计算云台10在变焦镜头20拍摄远焦图像时的远焦姿态角；和
S19：根据近焦姿态角和远焦姿态角控制云台10的每一个轴由近焦姿态角调整至远焦姿态角。
具体地，请结合图1，以三轴云台10为例，在变焦镜头20拍摄参考图像时，云台10的姿态角可以用偏航轴姿态角、横滚轴姿态角以及俯仰轴姿态角表示。依次获取偏航轴的近焦姿态角、横滚轴的近焦态角以及俯仰轴的近焦姿态角，并根据第二焦段确定云台10在变焦镜头20拍摄远焦图像时的远焦姿态角(偏航轴的远焦姿态角、横滚轴的远焦姿态角以及俯仰轴的远焦姿态角)计算由近焦姿态角调整至远焦姿态角的所需旋转的方向及角度。在一个例子中，获得所需旋转的角度及方向可以通过公式计算，例如将偏航轴姿态角、横滚轴姿态角以及俯仰轴姿态角表示的近焦姿态角以及远焦姿态角均转换为用四元数表示，从而快速计算所需旋转的角度及方向。在另一个例子中，所需旋转的角度及方向还可以通过查阅表格的方式获得。重复测试将近焦姿态角调节至任意远焦姿态角时所需旋转的角度及方向，以获得近焦姿态角与远焦姿态角一一对应的关系表格。如此，在获取到变焦镜头20的近焦姿态角后，可以查阅近焦姿态角与远焦姿态角对应的关系表格，以直接获得所需旋转的角度及方向，从而将云台10的每一个轴由近焦姿态角调整至远焦姿态角。
请参阅图8，在本实施例中，在步骤S15中，处理参考图像及拼接图像以得到目标重建图像的步骤包括以下子步骤：
S151：对参考图像进行上采样以获得上采样图像；
S152：计算上采样图像与拼接图像之间的映射矩阵；和
S153：根据映射矩阵对拼接图像进行裁剪处理以得到目标重建图像。
具体地，由于最终获得的目标重建图像的分辨率比参考图像的分辨率高，因此， 需要对参考图像进行放大处理。在步骤S151中，对参考图像进行上采样(Upsampling)，即在参考图像的基础上，在各个像素点之间采用合适的插值算法插入新的像素，从而得到采样图像。常用的插值算法包括最近邻插值算法、双线性插值算法、均值插值算法、中值插值算法等。以最邻近差值算法为例，在待求像素的四邻像素中，将距离待求像素最近的邻像素灰度赋给待求像素，从而实现快速插值。此外，在得到插值好的采样图像后，还需进一步计算采样图像与拼接图像对应点之间一对一的映射关系，即计算采样图像与拼接图像的映射矩阵。由于拼接图像是由多帧远焦图像拼接而成，拼接图像的尺寸比采样图像的尺寸大，因此，需要裁去拼接图像中与采样图像多余的部分。根据采样图像与拼接图像的映射矩阵，可以确定采样图像与拼接图像的差异像素区域，并根据映射矩阵对拼接图像进行剪裁处理，以去除拼接图像中与采样图像多余的部分。当然，在其他实施例中，根据目标重建图像的大小，只需拼接与目标重建图像的大小一致的远焦图像，而无需对拼接图像进行剪裁就能得到目标重建图像。继续以图4为例，只拼接中心区域(对焦区域分别为Ⅱ区域、Ⅳ区域、Ⅴ区域、Ⅵ区域以及Ⅷ区域)的远焦图像，得到的拼接图像就能重建出目标重建图像。
在本实施例中，映射矩阵包括单应矩阵(Homography Matrix)或仿射变换矩阵(Affine Transformation Matrix)。利用FindHomography函数可以获得采样图像与参考图像的单应矩阵，从而将采样图像上的像素点的坐标映射成拼接图像上的像素点的坐标。根据采样图像与参考图像的仿射变换矩阵，对采样图像进行仿射变换，以实现采样图像与拼接图像的线性变换。
请参阅图9和图10，在本实施例中，在步骤S151中，对参考图像进行上采样以获得上采样图像的步骤包括以下子步骤：
S1511：通过双线性插值算法对参考图像进行上采样以获得上采样图像；或者
S1512：通过三次样条插值算法对参考图像进行上采样以获得上采样图像。
在本实施例中，上采样的插值算法可以为双线性插值(Bilinear Interpolation)算法和三次样条插值(Cubic Spline Interpolation)算法。其中，双线性插值算法是在参考图像的水平方向以及垂直方向分别进行一次线性插值，三次样条插值算法是利用三弯矩方程和第一边界条件，将参考图像进行分段三次插值。相较于最近邻插值算法，采用双线性插值算法或者三次样条插值算法对参考图像进行上采样，获得采样图像较不易失真。
请参阅图11，在本实施例中，图像处理方法还包括以下步骤：
S20：根据目标重建图像的大小对至少一帧远焦图像进行截取操作；
在步骤S14中，将多帧远焦图像拼接成一帧拼接图像的步骤包括以下子步骤：
S141：将进行截取操作后的远焦图像及其他远焦图像拼接成一帧拼接图像。
当多帧远焦图像拼接而成的一帧拼接图像的大小大于目标重建图像的大小时，需要对多帧远焦图像进行截取操作。具体地，在将多帧不同拍摄范围的远焦图像拼接成一帧图像之前，计算多帧远焦图像总的大小与目标重建图像的大小差异，并计算出多于目标重建图像的像素区域，即计算所需截取的远焦图像的区域并对需要截取的远焦图像进行截取操作。在截取多余的像素区域后，再将截取后的远焦图像与未截取的远焦图像拼接成一帧拼接图像。如此，无需对拼接图像进行剪裁就能获取目标重建图像。
请继续参阅图1，本发明实施方式提供一种云台系统100。上述实施方式的图像处理方法可以应用在云台系统100中。云台系统100包括云台10、搭载在云台10上的变焦镜头20以及处理器30。处理器30用于：控制变焦镜头20在第一焦段拍摄一帧参考图像，第一焦段为用户期望的拍摄焦段；调整变焦镜头20至第二焦段；控制变焦镜头20在第二焦段拍摄多帧不同拍摄范围的远焦图像，第二焦段对应的镜头焦距大于第一焦段对应的镜头焦距；将多帧远焦图像拼接成一帧拼接图像；和，处理参考图像及拼接图像以得到目标重建图像。请结合图2，也即是说，处理器30可以用于实施步骤S11、S12、S13、S14及S15。
具体地，云台10的姿态可以呈跟随成像物体运动的姿态。其中，云台10包括一个轴或多个轴。当云台10仅包括一个轴时，云台10为单轴云台。当云台10包括多个轴时，云台10为多轴云台10。多轴云台10可以为两轴云台，例如包括偏航(Yaw)轴与横滚(Roll)轴的云台，或者是包括偏航(Yaw)轴与俯仰(Pitch)轴的云台；多轴云台10也可以为三轴云台，同时包括偏航(Yaw)轴、横滚(Roll)轴和俯仰(Pitch)轴。本实施例的云台10以三轴云台为例进行示意性说明。
变焦镜头20能够在一定拍摄范围内变换焦距，不同焦距下，变焦镜头20具有不同大小的视场角，具有不同大小的视野拍摄范围，以及不同大小的取景拍摄范围，由此在不同焦距下拍摄出的图像涵盖场景的拍摄范围也会不同(即，拍摄范围不同)。在本实施例中，变焦镜头20可以为光学变焦镜头、数字变焦镜头、或者多种变焦镜头的组合。变焦镜头20搭载在云台10上，以使变焦镜头20实时跟随成像物体。具体地，变焦镜头20可以相对云台10的Y(Yaw)轴、R(Roll)轴和P(Pitch)轴转动，也可以在云台10的一个或者两个轴锁定时，跟随其他的一个或者两个轴转动，例如在R(Roll)轴和P(Pitch)轴锁定时，变焦镜头20仅相对Y(Yaw)轴转动。本实施例中的变焦镜头20可以在第一焦段下拍摄图像，也可以从第一焦段变换为第二焦段，然后在第二焦段下拍摄图像。其中，第一焦段为用户期望的拍摄焦段。在第一焦段内，对应的镜头焦距均较小，变焦镜头20拍摄的视角较大、主体较小、景深较长。第二焦段对应的镜头焦距大于第一焦段对应的镜头焦距。在第二焦段内，第二焦段对应的镜头焦距均较大，变焦 镜头20拍摄的视角较小、主体较大、景深较短。
请结合图3，本发明实施方式的图像处理方法中，首先确定目标场景200中待成像的物体，例如人物、动物、景物等。然后处理器30控制变焦镜头20在第一焦段拍摄一帧参考图像。此时，参考图像的视角较大、主体较小且分辨率较低。接着处理器30控制变焦镜头20在第二焦段拍摄多帧不同拍摄范围的远焦图像。在本实施例中，多帧表示两帧或者两帧以上。此外，不同拍摄范围表示变焦镜头20对目标场景200中多个不同区域进行对焦，如此以得到多个不同区域的清晰图像。例如图4中，对目标场景200中的九个区域以第二焦段进行对焦，例如，在I区域中，对太阳进行对焦，在VI区域中，对左起第三座山的山峰进行对焦。其中，每个区域对焦的变焦镜头20的镜头焦距可以相同，也可以略有差异，但均落入第二焦段的拍摄范围内。在变焦镜头20以第二焦段拍摄后，获得九个不同拍摄范围的九帧远焦图像，每帧远焦图像均能清晰地显示目标场景200中的至少一个区域的细节。处理器30在获得多帧远焦图像后，对多帧远焦图像进行拼接，以形成一帧拼接图像。如图5，将I区域的远焦图像以及VI区域的远焦图像进行拼接，以形成I+IV区域的拼接图像。其他七个区域的远焦图像也依次进行拼接，从而形成一帧拼接图像。处理器30以参考图像为参照基础，进一步地处理拼接图像，以得到目标重建图像。由于目标重建图像是由多帧细节清晰的远焦图像拼接而成，因此，目标重建图像的分辨率高而且细节清晰，能够真实反映拍摄的目标场景200，而且在某些场景下不会引入不好的视觉效果问题。
本发明实施方式的云台系统100中，处理器30用于控制变焦镜头20拍摄第一焦段下的参考图像，然后再拍摄第二焦段下多帧不同拍摄范围的远焦图像，接着将多帧远焦图像拼接成一帧拼接图像，并处理参考图像及拼接图像以得到目标重建图像。由于第二焦段拍摄的远焦图像的分辨率高，因此目标重建图像的分辨率较高。采用本发明的云台系统100，获得的目标重建图像一方面与真实场景的差异较小，即能够真实反映目标场景200，另一方面，在目标场景下的视觉效果较好。
在本实施例中，多帧不同拍摄范围的远焦图像中的至少两帧在水平方向上拍摄范围相邻，多帧不同拍摄范围的远焦图像中的至少两帧在垂直方向上拍摄范围相邻。
当至少有两帧远焦图像在水平方向上拍摄范围相接或者存在重叠时，至少有两帧不同拍摄范围的远焦图像在水平方向上的拍摄范围相邻。此外，当至少有两帧远焦图像在垂直方向上拍摄范围相接或者存在重叠时，至少有不同拍摄范围的两帧远焦图像在垂直方向上的拍摄范围相邻。请参阅图4，继续以九帧远焦图像为例，区域相邻的两帧远焦图像在水平方向上或者垂直方向上的拍摄范围相邻。例如I区域的远焦图像与II区域的远焦图像在水平方向上的拍摄范围相邻，并且I区域的远焦图像与IV区域的 远焦图像在垂直方向上的拍摄范围相邻。当至少有两帧远焦图像在水平方向或者垂直方向上的拍摄范围相接或者存在重叠时，可以先提取相邻的两帧远焦图像相接的边界上或者重叠区域的特征点，然后根据匹配特征点的方式进行拼接。如图5，以拼接I区域的远焦图像与IV区域的远焦图像为例，I区域的远焦图像的X区域、Y区域上均存在特征点(分别用小黑点表示)，IV区域的远焦图像的X’区域、Y’区域上均存在特征点(分别用小黑点表示)。然后通过匹配X区域与X’区域的特征点，Y区域与Y’区域的特征点，从而得到I区域的远焦图像与IV区域的远焦图像之间的特征点对。然后匹配特征点对，并采用算法拼接I区域的远焦图像与IV区域的远焦图像，得到I+IV区域的拼接图像。依此类推，其他七个区域的远焦图像也依次进行拼接，从而形成一帧拼接图像。由于多帧不同拍摄范围的远焦图像的至少两帧在水平方向以及垂直方向上拍摄范围相邻，在将多帧不同拍摄范围的远焦图像拼接成一帧拼接图像时，相邻的两帧远焦图像在水平方向以及垂直方向不会出现断层(目标场景200不连续)。
在本实施例中，多帧不同拍摄范围的远焦图像形成m*n的矩阵，其中m≥2，n≥2，且m、n均为整数。
在参考图像的尺寸一定的情况，可以根据目标场景200的细节数量选取不同拍摄范围的对焦区域，例如，m可以取值为2，3，4，10，20，100等整数，n可以取值为2，6，8，15，25，80等整数。m和n的取值越大，目标场景200的细节数量更多，即更能还原目标场景200。例如，若需要得到左上、右上、左下以及右下的四个区域均清晰的目标重建图像，变焦镜头20在第二焦段拍摄该四个区域的远焦图像，如此以得到四帧以2*2矩阵分布的远焦图像。再以图4的实施例为例，若需要得到九个区域均清晰的目标重建图像，则变焦镜头20在第二焦段依次对这九个区域进行对焦，并拍摄该九个区域的远焦图像，从而得到九帧以形成3*3矩阵分布的远焦图像。
在本实施例中，处理器30还用于根据第一焦段及目标重建图像的大小计算第二焦段。请结合图6，也即是说，处理器30还可以用于实施步骤S16。
具体地，目标重建图像的大小由尺寸、分辨率等的一个参数或者多个参数决定。目标重建图像的尺寸可以与参考图案的尺寸一致，也可以大于参考图像的尺寸。在目标重建图像的尺寸确定的情况下，需要进一步确定目标重建图像的分辨率。目标重建图像的分辨率可以根据显示图像的要求设定，也可以根据参考图像的分辨率确定。最后，根据目标重建图像的分辨率以及尺寸，在第一焦段的基础上，计算第二焦段，如此，以得到对应镜头焦距较大的第二焦段。
在本实施例中，云台10具有一个轴或多个轴，处理器30还用于：获取云台10在变焦镜头20拍摄参考图像时每一个轴的近焦姿态角；根据第二焦段及近焦姿态角计算 云台10在变焦镜头20拍摄远焦图像时的远焦姿态角；和，根据近焦姿态角和远焦姿态角控制云台10的每一个轴由近焦姿态角调整至远焦姿态角。请结合图7，也即是说，处理器30还可以用于实施步骤S17、S18及S19。
具体地，请结合图1，以三轴云台10为例，在变焦镜头20拍摄参考图像时，云台10的姿态角可以用偏航轴姿态角、横滚轴姿态角以及俯仰轴姿态角表示。依次获取偏航轴的近焦姿态角、横滚轴的近焦态角以及俯仰轴的近焦姿态角，并根据第二焦段确定云台10在变焦镜头20拍摄远焦图像时的远焦姿态角(偏航轴的远焦姿态角、横滚轴的远焦姿态角以及俯仰轴的远焦姿态角)计算由近焦姿态角调整至远焦姿态角的所需旋转的方向及角度。在一个例子中，获得所需旋转的角度及方向可以通过公式计算，例如将偏航轴姿态角、横滚轴姿态角以及俯仰轴姿态角表示的近焦姿态角以及远焦姿态角均转换为用四元数表示，从而快速计算所需旋转的角度及方向。在另一个例子中，所需旋转的角度及方向还可以通过查阅表格的方式获得。重复测试将近焦姿态角调节至任意远焦姿态角时所需旋转的角度及方向，以获得近焦姿态角与远焦姿态角一一对应的关系表格。如此，在获取到变焦镜头20的近焦姿态角后，可以查阅近焦姿态角与远焦姿态角对应的关系表格，以直接获得所需旋转的角度及方向，从而将云台10的每一个轴由近焦姿态角调整至远焦姿态角。
在本实施例中，处理器30还用于：对参考图像进行上采样以获得上采样图像；计算上采样图像与拼接图像之间的映射矩阵；和，根据映射矩阵对拼接图像进行裁剪处理以得到目标重建图像。请结合图8，也即是说，处理器30还可以用于实施步骤S151、S152及S153。
具体地，由于最终获得的目标重建图像的分辨率比参考图像的分辨率高，因此，需要对参考图像进行放大处理。处理器30对参考图像进行上采样(Upsampling)，即在参考图像的基础上，在各个像素点之间采用合适的插值算法插入新的像素，从而得到采样图像。常用的插值算法包括最近邻插值算法、双线性插值算法、均值插值算法、中值插值算法等。以最邻近差值算法为例，在待求像素的四邻像素中，将距离待求像素最近的邻像素灰度赋给待求像素，从而实现快速插值。此外，在得到插值好的采样图像后，还需进一步计算采样图像与拼接图像对应点之间一对一的映射关系，即计算采样图像与拼接图像的映射矩阵。由于拼接图像是由多帧远焦图像拼接而成，拼接图像的尺寸比采样图像的尺寸大，因此，需要裁去拼接图像中与采样图像多余的部分。根据采样图像与拼接图像的映射矩阵，可以确定采样图像与拼接图像的差异像素区域，并根据映射矩阵对拼接图像进行剪裁处理，以去除拼接图像中与采样图像多余的部分。当然，在其他实施例中，根据目标重建图像的大小，只需拼接与目标重建图像的大小 一致的远焦图像，而无需对拼接图像进行剪裁就能得到目标重建图像。继续以图4为例，只拼接中心区域(对焦区域分别为Ⅱ区域、Ⅳ区域、Ⅴ区域、Ⅵ区域以及Ⅷ区域)的远焦图像，得到的拼接图像就能重建出目标重建图像。
在本实施例中，映射矩阵包括单应矩阵(Homography Matrix)或仿射变换矩阵(Affine Transformation Matrix)。利用FindHomography函数可以获得采样图像与参考图像的单应矩阵，从而将采样图像上的像素点的坐标映射成拼接图像上的像素点的坐标。根据采样图像与参考图像的仿射变换矩阵，对采样图像进行仿射变换，以实现采样图像与拼接图像的线性变换。
在本实施例中，处理器30还用于：通过双线性插值算法对参考图像进行上采样以获得上采样图像；或者，通过三次样条插值算法对参考图像进行上采样以获得上采样图像。请结合图9和图10，也即是说，处理器30还可以用于实施步骤S1511或者S1512。
在本实施例中，上采样的插值算法可以为双线性插值(Bilinear Interpolation)算法和三次样条插值(Cubic Spline Interpolation)算法。其中，双线性插值算法是在参考图像的水平方向以及垂直方向分别进行一次线性插值，三次样条插值算法是利用三弯矩方程和第一边界条件，将参考图像进行分段三次插值。相较于最近邻插值算法，采用双线性插值算法或者三次样条插值算法对参考图像进行上采样，获得采样图像较不易失真。
在本实施例中，处理器30还用于：根据目标重建图像的大小对至少一帧远焦图像进行截取操作；和，将进行截取操作后的远焦图像及其他远焦图像拼接成一帧拼接图像。请结合图11，也即是说，处理器30还可以用于实施步骤S20及S141。
当多帧远焦图像拼接而成的一帧拼接图像的大小大于目标重建图像的大小时，需要对多帧远焦图像进行截取操作。具体地，在将多帧不同拍摄范围的远焦图像拼接成一帧图像之前，计算多帧远焦图像总的大小与目标重建图像的大小差异，并计算出多于目标重建图像的像素区域，即计算所需截取的远焦图像的区域并对需要截取的远焦图像进行截取操作。在截取多余的像素区域后，再将截取后的远焦图像与未截取的远焦图像拼接成一帧拼接图像。如此，无需对拼接图像进行剪裁就能获取目标重建图像。
请参阅图12，本发明实施方式提供一种无人机1000。无人机1000包括上述实施例中的云台系统100和机身300。云台系统100设置在机身300上。
本发明实施方式的无人机1000可以为四旋翼飞行器、六旋翼飞行器、八旋翼飞行器等，在此不做限定。本实施例的无人机1000以四旋翼飞行器为例，其可以搭载有云台系统100。云台10与机身300固定连接，变焦镜头20安装在云台10上。变焦镜头20用于在无人机1000飞行或悬停时进行拍照或者摄影。
本发明实施方式的无人机1000通过控制变焦镜头20拍摄第一焦段下的参考图像， 然后再拍摄第二焦段下多帧不同拍摄范围的远焦图像，接着将多帧远焦图像拼接成一帧拼接图像，并处理参考图像及拼接图像以得到目标重建图像。由于第二焦段拍摄的远焦图像的分辨率高，因此目标重建图像的分辨率较高。采用本发明的无人机1000，获得的目标重建图像一方面与真实场景的差异较小，即能够真实反映目标场景200，另一方面，在目标场景下的视觉效果较好。
在本说明书的描述中，参考术语“一个实施方式”、“一些实施方式”、“示意性实施方式”、“示例”、“具体示例”或“一些示例”等的描述意指结合所述实施方式或示例描述的具体特征、结构、材料或者特点包含于本发明的至少一个实施方式或示例中。在本说明书中，对上述术语的示意性表述不一定指的是相同的实施方式或示例。而且，描述的具体特征、结构、材料或者特点可以在任何的一个或多个实施方式或示例中以合适的方式结合。
流程图中或在此以其他方式描述的任何过程或方法描述可以被理解为，表示包括一个或更多个用于实现特定逻辑功能或过程的步骤的可执行指令的代码的模块、片段或部分，并且本发明的优选实施方式的范围包括另外的实现，其中可以不按所示出或讨论的顺序，包括根据所涉及的功能按基本同时的方式或按相反的顺序，来执行功能，这应被本发明的实施例所属技术领域的技术人员所理解。
在流程图中表示或在此以其他方式描述的逻辑和/或步骤，例如，可以被认为是用于实现逻辑功能的可执行指令的定序列表，可以具体实现在任何计算机可读介质中，以供指令执行系统、装置或设备(如基于计算机的系统、包括处理模块的系统或其他可以从指令执行系统、装置或设备取指令并执行指令的系统)使用，或结合这些指令执行系统、装置或设备而使用。就本说明书而言，"计算机可读介质"可以是任何可以包含、存储、通信、传播或传输程序以供指令执行系统、装置或设备或结合这些指令执行系统、装置或设备而使用的装置。计算机可读介质的更具体的示例(非穷尽性列表)包括以下：具有一个或多个布线的电连接部(控制方法)，便携式计算机盘盒(磁装置)，随机存取存储器(RAM)，只读存储器(ROM)，可擦除可编辑只读存储器(EPROM或闪速存储器)，光纤装置，以及便携式光盘只读存储器(CDROM)。另外，计算机可读介质甚至可以是可在其上打印所述程序的纸或其他合适的介质，因为可以例如通过对纸或其他介质进行光学扫描，接着进行编辑、解译或必要时以其他合适方式进行处理来以电子方式获得所述程序，然后将其存储在计算机存储器中。
应当理解，本发明的实施方式的各部分可以用硬件、软件、固件或它们的组合来实现。在上述实施方式中，多个步骤或方法可以用存储在存储器中且由合适的指令执行系统执行的软件或固件来实现。例如，如果用硬件来实现，和在另一实施方式中一 样，可用本领域公知的下列技术中的任一项或他们的组合来实现：具有用于对数据信号实现逻辑功能的逻辑门电路的离散逻辑电路，具有合适的组合逻辑门电路的专用集成电路，可编程门阵列(PGA)，现场可编程门阵列(FPGA)等。
本技术领域的普通技术人员可以理解实现上述实施例方法携带的全部或部分步骤是可以通过程序来指令相关的硬件完成，所述的程序可以存储于一种计算机可读存储介质中，该程序在执行时，包括方法实施例的步骤之一或其组合。
此外，在本发明的各个实施例中的各功能单元可以集成在一个处理模块中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个模块中。上述集成的模块既可以采用硬件的形式实现，也可以采用软件功能模块的形式实现。所述集成的模块如果以软件功能模块的形式实现并作为独立的产品销售或使用时，也可以存储在一个计算机可读取存储介质中。上述提到的存储介质可以是只读存储器，磁盘或光盘等。
尽管上面已经示出和描述了本发明的实施方式，可以理解的是，上述实施方式是示例性的，不能理解为对本发明的限制，本领域的普通技术人员在本发明的范围内可以对上述实施实施进行变化、修改、替换和变型。

Claims (19)

  1. 一种云台系统的图像处理方法，其特征在于，所述云台系统包括云台和搭载在所述云台上的变焦镜头，所述图像处理方法包括：
    控制所述变焦镜头在第一焦段拍摄一帧参考图像，所述第一焦段为用户期望的拍摄焦段；
    调整所述变焦镜头至第二焦段；
    控制所述变焦镜头在所述第二焦段拍摄多帧不同拍摄范围的远焦图像，所述第二焦段对应的镜头焦距大于所述第一焦段对应的镜头焦距；
    将多帧所述远焦图像拼接成一帧拼接图像；和
    处理所述参考图像及所述拼接图像以得到目标重建图像。
  2. 根据权利要求1所述的图像处理方法，其特征在于，所述图像处理方法在所述调整所述变焦镜头至第二焦段的步骤前，还包括：
    根据所述第一焦段及所述目标重建图像的大小计算所述第二焦段。
  3. 根据权利要求1所述的图像处理方法，其特征在于，所述云台具有一个轴或多个轴，所述图像处理方法还包括：
    获取所述云台在所述变焦镜头拍摄所述参考图像时每一个轴的近焦姿态角；
    根据所述第二焦段及所述近焦姿态角计算所述云台在所述变焦镜头拍摄所述远焦图像时的远焦姿态角；和
    根据所述近焦姿态角和所述远焦姿态角控制所述云台的每一个轴由所述近焦姿态角调整至所述远焦姿态角。
  4. 根据权利要求1所述的图像处理方法，其特征在于，所述处理所述参考图像及所述拼接图像以得到目标重建图像，包括：
    对所述参考图像进行上采样以获得上采样图像；
    计算所述上采样图像与所述拼接图像之间的映射矩阵；和
    根据所述映射矩阵对所述拼接图像进行裁剪处理以得到所述目标重建图像。
  5. 根据权利要求4所述的图像处理方法，其特征在于，所述映射矩阵包括单应矩阵或仿射变换矩阵。
  6. 根据权利要求4所述的图像处理方法，其特征在于，所述对所述参考图像进行上采样以获得上采样图像的步骤包括：
    通过双线性插值算法对所述参考图像进行上采样以获得所述上采样图像；或者
    通过三次样条插值算法对所述参考图像进行上采样以获得所述上采样图像。
  7. 根据权利要求1所述的图像处理方法，其特征在于，多帧不同拍摄范围的所述远焦图像中的至少两帧在水平方向上拍摄范围相邻，多帧不同拍摄范围的所述远焦图像中的至少两帧在垂直方向上拍摄范围相邻。
  8. 根据权利要求1所述的图像处理方法，其特征在于，多帧不同拍摄范围的所述远焦图像形成m*n的矩阵，其中m≥2，n≥2，且m、n均为整数。
  9. 根据权利要求1所述的图像处理方法，其特征在于，所述图像处理方法还包括：
    根据所述目标重建图像的大小对至少一帧所述远焦图像进行截取操作；
    所述将多帧所述远焦图像拼接成一帧拼接图像的步骤包括：
    将进行截取操作后的所述远焦图像及其他所述远焦图像拼接成一帧所述拼接图像。
  10. 一种云台系统，其特征在于，所述云台系统包括云台、搭载在所述云台上的变焦镜头以及处理器，所述处理器用于：
    控制所述变焦镜头在第一焦段拍摄一帧参考图像，所述第一焦段为用户期望的拍摄焦段；
    调整所述变焦镜头至第二焦段；
    控制所述变焦镜头在所述第二焦段拍摄多帧不同拍摄范围的远焦图像，所述第二焦段对应的镜头焦距大于所述第一焦段对应的镜头焦距；
    将多帧所述远焦图像拼接成一帧拼接图像；和
    处理所述参考图像及所述拼接图像以得到目标重建图像。
  11. 根据权利要求10所述的云台系统，其特征在于，所述处理器还用于：
    根据所述第一焦段及所述目标重建图像的大小计算所述第二焦段。
  12. 根据权利要求10所述的云台系统，其特征在于，所述云台具有一个轴或多个轴，所述处理器还用于：
    获取所述云台在所述变焦镜头拍摄所述参考图像时每一个轴的近焦姿态角；
    根据所述第二焦段及所述近焦姿态角计算所述云台在所述变焦镜头拍摄所述远焦图像时的远焦姿态角；和
    根据所述近焦姿态角和所述远焦姿态角控制所述云台的每一个轴由所述近焦姿态角调整至所述远焦姿态角。
  13. 根据权利要求10所述的云台系统，其特征在于，所述处理器还用于：
    对所述参考图像进行上采样以获得上采样图像；
    计算所述上采样图像与所述拼接图像之间的映射矩阵；和
    根据所述映射矩阵对所述拼接图像进行裁剪处理以得到所述目标重建图像。
  14. 根据权利要求13所述的云台系统，其特征在于，所述映射矩阵包括单应矩阵或仿射变换矩阵。
  15. 根据权利要求13所述的云台系统，其特征在于，所述处理器还用于：
    通过双线性插值算法对所述参考图像进行上采样以获得所述上采样图像；或者
    通过三次样条插值算法对所述参考图像进行上采样以获得所述上采样图像。
  16. 根据权利要求10所述的云台系统，其特征在于，多帧不同拍摄范围的所述远焦图像中的至少两帧在水平方向上拍摄范围相邻，多帧不同拍摄范围的所述远焦图像中的至少两帧在垂直方向上拍摄范围相邻。
  17. 根据权利要求10所述的云台系统，其特征在于，多帧不同拍摄范围的所述远焦图像形成m*n的矩阵，其中m≥2，n≥2，且m、n均为整数。
  18. 根据权利要求10所述的云台系统，其特征在于，所述处理器还用于：
    根据所述目标重建图像的大小对至少一帧所述远焦图像进行截取操作；和
    将进行截取操作后的所述远焦图像及其他所述远焦图像拼接成一帧所述拼接图像。
  19. 一种无人机，其特征在于，所述无人机包括：
    权利要求10-18任意一项所述的云台系统；和
    机身，所述云台系统设置在所述机身上。
PCT/CN2018/102000 2018-08-23 2018-08-23 云台系统及其图像处理方法、无人机 WO2020037615A1 (zh)

Priority Applications (3)

Application Number Priority Date Filing Date Title
CN201880039151.4A CN110771140A (zh) 2018-08-23 2018-08-23 云台系统及其图像处理方法、无人机
PCT/CN2018/102000 WO2020037615A1 (zh) 2018-08-23 2018-08-23 云台系统及其图像处理方法、无人机
US17/180,572 US20210176395A1 (en) 2018-08-23 2021-02-19 Gimbal system and image processing method thereof and unmanned aerial vehicle

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2018/102000 WO2020037615A1 (zh) 2018-08-23 2018-08-23 云台系统及其图像处理方法、无人机

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/180,572 Continuation US20210176395A1 (en) 2018-08-23 2021-02-19 Gimbal system and image processing method thereof and unmanned aerial vehicle

Publications (1)

Publication Number Publication Date
WO2020037615A1 true WO2020037615A1 (zh) 2020-02-27

Family

ID=69328802

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2018/102000 WO2020037615A1 (zh) 2018-08-23 2018-08-23 云台系统及其图像处理方法、无人机

Country Status (3)

Country Link
US (1) US20210176395A1 (zh)
CN (1) CN110771140A (zh)
WO (1) WO2020037615A1 (zh)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112004025A (zh) * 2020-09-02 2020-11-27 广东电网有限责任公司 一种基于目标点云的无人机自动驾驶变焦方法、系统及设备
CN113034369A (zh) * 2021-04-07 2021-06-25 北京佰才邦技术股份有限公司 一种基于多摄像头的图像生成方法、装置和计算机设备
CN113222820A (zh) * 2021-05-20 2021-08-06 北京航空航天大学 一种位姿信息辅助的航空遥感图像拼接方法

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114071009B (zh) * 2020-07-31 2023-04-18 华为技术有限公司 一种拍摄方法及设备
CN114071004A (zh) * 2020-08-06 2022-02-18 北京小米移动软件有限公司 月亮图像获取方法和装置、电子设备、存储介质
US11615582B2 (en) * 2021-06-08 2023-03-28 Fyusion, Inc. Enclosed multi-view visual media representation
DE102022001394B3 (de) 2022-04-23 2023-06-29 Rainer Püllen Bildaufzeichnungssystem mit mehreren Zoombereichen und Flüssigkeitsinjektionslinse

Citations (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6147709A (en) * 1997-04-07 2000-11-14 Interactive Pictures Corporation Method and apparatus for inserting a high resolution image into a low resolution interactive image to produce a realistic immersive experience
US20070189747A1 (en) * 2005-12-08 2007-08-16 Sony Corporation Camera system, camera control apparatus, panorama image making method and computer program product
CN100550986C (zh) * 2004-07-05 2009-10-14 伊斯曼柯达公司 具有多个分辨率的方法和照相机
CN102013094A (zh) * 2010-11-25 2011-04-13 上海合合信息科技发展有限公司 提高文本图像清晰度的方法及系统
US20110175900A1 (en) * 2010-01-18 2011-07-21 Beeler Thabo Dominik System and method for mesoscopic geometry modulation
CN103907132A (zh) * 2011-11-03 2014-07-02 皇家飞利浦有限公司 图像数据处理
CN103986867A (zh) * 2014-04-24 2014-08-13 宇龙计算机通信科技(深圳)有限公司 一种图像拍摄终端和图像拍摄方法
CN103985085A (zh) * 2014-05-26 2014-08-13 三星电子（中国）研发中心 图像超分辨率放大的方法和装置
CN104050653A (zh) * 2014-07-07 2014-09-17 西安电子科技大学 基于非负结构稀疏的高光谱图像超分辨率算法
CN105389797A (zh) * 2015-10-16 2016-03-09 西安电子科技大学 一种基于超分辨率重建的无人机视频小目标检测方法
CN106797434A (zh) * 2014-11-04 2017-05-31 奥林巴斯株式会社 摄像装置、摄像方法、处理程序
CN107079104A (zh) * 2016-08-24 2017-08-18 深圳市大疆灵眸科技有限公司 广角照片拍摄方法、装置、云台、无人飞行器及机器人

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9736346B2 (en) * 2006-05-09 2017-08-15 Stereo Display, Inc Imaging system improving image resolution of the system with low resolution image sensor
CN107659774B (zh) * 2017-09-30 2020-10-13 北京拙河科技有限公司 一种基于多尺度相机阵列的视频成像系统及视频处理方法

Patent Citations (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6147709A (en) * 1997-04-07 2000-11-14 Interactive Pictures Corporation Method and apparatus for inserting a high resolution image into a low resolution interactive image to produce a realistic immersive experience
CN100550986C (zh) * 2004-07-05 2009-10-14 伊斯曼柯达公司 具有多个分辨率的方法和照相机
US20070189747A1 (en) * 2005-12-08 2007-08-16 Sony Corporation Camera system, camera control apparatus, panorama image making method and computer program product
CN1979335B (zh) * 2005-12-08 2010-05-19 索尼株式会社 照相机系统、照相机控制装置、全景图像形成方法
US20110175900A1 (en) * 2010-01-18 2011-07-21 Beeler Thabo Dominik System and method for mesoscopic geometry modulation
CN102013094A (zh) * 2010-11-25 2011-04-13 上海合合信息科技发展有限公司 提高文本图像清晰度的方法及系统
CN103907132A (zh) * 2011-11-03 2014-07-02 皇家飞利浦有限公司 图像数据处理
CN103986867A (zh) * 2014-04-24 2014-08-13 宇龙计算机通信科技(深圳)有限公司 一种图像拍摄终端和图像拍摄方法
CN103985085A (zh) * 2014-05-26 2014-08-13 三星电子（中国）研发中心 图像超分辨率放大的方法和装置
CN104050653A (zh) * 2014-07-07 2014-09-17 西安电子科技大学 基于非负结构稀疏的高光谱图像超分辨率算法
CN106797434A (zh) * 2014-11-04 2017-05-31 奥林巴斯株式会社 摄像装置、摄像方法、处理程序
CN105389797A (zh) * 2015-10-16 2016-03-09 西安电子科技大学 一种基于超分辨率重建的无人机视频小目标检测方法
CN107079104A (zh) * 2016-08-24 2017-08-18 深圳市大疆灵眸科技有限公司 广角照片拍摄方法、装置、云台、无人飞行器及机器人

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112004025A (zh) * 2020-09-02 2020-11-27 广东电网有限责任公司 一种基于目标点云的无人机自动驾驶变焦方法、系统及设备
CN112004025B (zh) * 2020-09-02 2021-09-14 广东电网有限责任公司 一种基于目标点云的无人机自动驾驶变焦方法、系统及设备
CN113034369A (zh) * 2021-04-07 2021-06-25 北京佰才邦技术股份有限公司 一种基于多摄像头的图像生成方法、装置和计算机设备
CN113034369B (zh) * 2021-04-07 2024-05-28 北京佰才邦技术股份有限公司 一种基于多摄像头的图像生成方法、装置和计算机设备
CN113222820A (zh) * 2021-05-20 2021-08-06 北京航空航天大学 一种位姿信息辅助的航空遥感图像拼接方法
CN113222820B (zh) * 2021-05-20 2024-05-07 北京航空航天大学 一种位姿信息辅助的航空遥感图像拼接方法

Also Published As

Publication number Publication date
US20210176395A1 (en) 2021-06-10
CN110771140A (zh) 2020-02-07

Similar Documents

Publication Publication Date Title
WO2020037615A1 (zh) 云台系统及其图像处理方法、无人机
CN113382168B (zh) 用于存储成像数据的重叠区以产生优化拼接图像的设备及方法
JP5835383B2 (ja) 情報処理方法、情報処理装置、およびプログラム
WO2017113504A1 (zh) 一种图像显示方法以及装置
US10748243B2 (en) Image distortion transformation method and apparatus
US7646404B2 (en) Foveated wide-angle imaging system and method for capturing and viewing wide-angle images in real time
JP3463612B2 (ja) 画像入力方法、画像入力装置及び記録媒体
JP5835384B2 (ja) 情報処理方法、情報処理装置、およびプログラム
JP5172863B2 (ja) バナナ・コーデック
Lai et al. Video stitching for linear camera arrays
US10012982B2 (en) System and method for focus and context views for telepresence and robotic teleoperation
WO2022032538A1 (zh) 全景拍摄方法、装置、系统及计算机可读存储介质
JP3861855B2 (ja) 画像入力装置
JP6350695B2 (ja) 装置、方法、およびプログラム
CN110047039A (zh) 一种面向虚拟现实交互的冗余视场全景影像构建方法
CN117083871A (zh) 控制相机的方法
JP6583486B2 (ja) 情報処理方法、情報処理プログラムおよび情報処理装置
CN110796690B (zh) 图像匹配方法和图像匹配装置
JP6128185B2 (ja) 装置、方法、およびプログラム
JP6777208B2 (ja) プログラム
JP6673459B2 (ja) 画像処理装置、画像処理システム及びプログラム
JP2014165866A (ja) 画像処理装置及びその制御方法、プログラム
EP3432567A1 (en) Image processing device, image processing method and image processing system
EP3595287A1 (en) Capturing video content with at least two cameras of a multi-camera rig
EP3229070B1 (en) Three-dimensional, 360-degree virtual reality camera exposure control

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18931239

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 18931239

Country of ref document: EP

Kind code of ref document: A1