WO2023115562A1 - 一种图像处理方法、装置、云台、可移动平台 - Google Patents

一种图像处理方法、装置、云台、可移动平台 Download PDF

Info

Publication number
WO2023115562A1
WO2023115562A1 PCT/CN2021/141316 CN2021141316W WO2023115562A1 WO 2023115562 A1 WO2023115562 A1 WO 2023115562A1 CN 2021141316 W CN2021141316 W CN 2021141316W WO 2023115562 A1 WO2023115562 A1 WO 2023115562A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
rotation
pan
tilt
rotation angle
Prior art date
Application number
PCT/CN2021/141316
Other languages
English (en)
French (fr)
Inventor
王晓东
Original Assignee
深圳市大疆创新科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 深圳市大疆创新科技有限公司 filed Critical 深圳市大疆创新科技有限公司
Priority to PCT/CN2021/141316 priority Critical patent/WO2023115562A1/zh
Publication of WO2023115562A1 publication Critical patent/WO2023115562A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D3/00—Control of position or direction
    • G05D3/12—Control of position or direction using feedback
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting

Definitions

  • the present application relates to the technical field of image processing, and in particular to an image processing method, device, cloud platform, and movable platform.
  • the gimbal is a supporting device for installing and fixing the shooting device.
  • the gimbal can provide a stabilization function through a control algorithm, so that the shooting device can remain stable during the photosensitive imaging process, which is conducive to generating high-quality images.
  • the resolution of the image captured by the photographing device is the number of pixels in the photosensitive pixel array of the image sensor (sensor). If you want to obtain an image whose resolution exceeds the number of pixels of the image sensor, you need to process the image using super-resolution image reconstruction technology. However, in related technologies, construction efficiency is low when super-resolution images are constructed.
  • one of the objectives of the present application is to provide an image processing method, device, pan-tilt, and movable platform, so as to improve the construction efficiency of super-resolution images.
  • an image processing method comprising:
  • the photographing device is carried on the platform, and the platform includes at least one rotating shaft;
  • the second image group collected by the shooting device includes at least one second image
  • the second image is after the shooting device captures the first image, the pan/tilt moves around
  • the at least one rotation axis is collected after two rotations in which the rotation directions are opposite and the rotation angles are different;
  • the first image and the second image group construct a third image, and the resolution of the third image is higher than that of the first image and the second image respectively resolution.
  • an image processing device including:
  • memory for storing processor-executable instructions
  • the processor implements the operations described in the first aspect when invoking the executable instructions.
  • a cloud platform which is equipped with a shooting device, including:
  • At least one motor used to drive the pan/tilt to rotate around the at least one rotation axis
  • memory for storing processor-executable instructions
  • the processor implements the operations described in the first aspect when invoking the executable instructions.
  • a mobile platform including:
  • a pan-tilt for carrying a photographing device said pan-tilt comprising at least one rotation axis;
  • a power assembly used to drive the movable platform to move in space
  • memory for storing processor-executable instructions
  • the processor implements the operations described in the first aspect when invoking the executable instructions.
  • a computer program product including a computer program, and when the computer program is executed by a processor, the operations described in the first aspect are implemented.
  • a computer-readable storage medium is provided, and several computer instructions are stored on the computer-readable storage medium, and when executed, the computer instructions perform the operations described in the first aspect.
  • Fig. 1 is a schematic structural diagram of a three-axis gimbal according to an embodiment of the present application.
  • Fig. 2 is a schematic diagram of a shooting scene shown in the present application according to an embodiment.
  • Fig. 3 is a flowchart of an image processing method according to an embodiment of the present application.
  • FIG. 4 is a schematic diagram of a photosensitive pixel array according to an embodiment of the present application.
  • Fig. 5 is a flowchart of an image processing method according to another embodiment of the present application.
  • Fig. 6 is a schematic diagram showing a spatial phase relationship between a first image and a second image according to an embodiment of the present application.
  • Fig. 7 is a flowchart of an image processing method according to another embodiment of the present application.
  • Fig. 8A is a schematic diagram of a first rotating shaft according to an embodiment of the present application.
  • Fig. 8B is a schematic diagram of a second rotating shaft according to an embodiment of the present application.
  • Fig. 9A is a schematic diagram of a third image shown according to an embodiment of the present application.
  • Fig. 9B is a schematic diagram of a third image shown according to another embodiment of the present application.
  • Fig. 9C is a schematic diagram of a third image shown according to another embodiment of the present application.
  • Fig. 10 is a flowchart of an image processing method according to another embodiment of the present application.
  • Fig. 11 is a schematic structural diagram of an image processing device according to an embodiment of the present application.
  • Fig. 12 is a schematic structural diagram of a pan/tilt according to an embodiment of the present application.
  • Fig. 13 is a schematic structural diagram of a movable platform according to an embodiment of the present application.
  • the gimbal is a supporting device for installing and fixing the shooting device.
  • the gimbal can provide a stabilization function through a control algorithm, so that the shooting device can remain stable during the photosensitive imaging process, which is conducive to generating clear images.
  • the pan-tilt can be a hand-held pan-tilt, or a pan-tilt mounted on a movable platform.
  • the gimbal can be a two-axis gimbal or a three-axis gimbal.
  • FIG. 1 shows a structural diagram of a three-axis gimbal 100 .
  • the three-axis gimbal 100 mainly includes: a pitch (pitch) axis motor 101, a roll (roll) axis motor 102, a heading (yaw) axis motor 103, a gimbal base 104, and a yaw axis arm 105 , the shooting device fixing mechanism 106 (including the inertial measurement unit IMU inside), the pitch shaft arm 107 , the roll shaft arm 108 , and the shooting device fixing mechanism 106 is used to carry the shooting device 109 .
  • the shooting device fixing mechanism 106 including the inertial measurement unit IMU inside
  • the roll axis arm 108 is used to support the pitch axis arm 107 and the pitch axis motor 101
  • the yaw axis arm 105 is used to support the yaw axis motor 103 and the roll axis motor 102
  • the pitch axis arm 107 is used to support the shooting device 109 .
  • the pitch axis motor 101 , the roll axis motor 102 and the yaw axis motor 103 may be collectively referred to as driving motors.
  • the angle sensor installed on the drive motor can measure the angle rotated by the drive motor, wherein the angle sensor can be one or more of potentiometers, hall sensors, and encoders.
  • the three-axis pan/tilt 100 can be connected to the handle or to the movable platform (not shown in FIG. 1 ) through the pan/tilt base 104 .
  • the photographing device mounted on the cloud platform is used to collect images. As shown in FIG. On the image plane, light signals are collected and sampled through the photosensitive pixel array, so as to obtain a digital sample of an image, that is, a digital image corresponding to the field of view of the optical component.
  • the photosensitive pixel array of the image sensor 220 generally adopts a Bayer (Bayer) arrangement, and the image processing algorithm restores images of red, green, and blue colors through an interpolation demosaic algorithm, and the resolution of each image is equal to the size of the photosensitive pixel array. .
  • the images of the three colors form a color image, and the resolution is also the size of the photosensitive pixel array of the image sensor.
  • the resolution of the image recovered by the corresponding method is also the size of the photosensitive pixel array of the image sensor.
  • the resolution of the image is limited by the size of the photosensitive pixel array of the image sensor 220 .
  • super-resolution (Super Resolution, SR) reconstruction technology can be used to obtain ultra-high-resolution images.
  • Super-resolution image reconstruction technology is a technology that uses multiple low-resolution (Low Resolution, LR) images containing different spatial phase information to construct a super-resolution image.
  • the so-called low resolution means that the resolution of the image is the size of the photosensitive pixel array of the image sensor.
  • Super-resolution means that the resolution of the image is greater than the size of the photosensitive pixel array of the image sensor.
  • a sub-pixel in an image corresponds to a spatial phase, and since multiple low-resolution images have sub-pixel displacements, each low-resolution image contains image information corresponding to different spatial phases. Based on deep learning or other algorithms, multiple images of specific spatial phases can be combined into images with higher resolution.
  • the photographing device acquires multiple low-resolution images containing different spatial phase information through random dithering. Due to the randomness of jitter, the displacement amount of each jitter may be different. In this way, when constructing a super-resolution image, spatial phase estimation and resampling are required, which brings a large amount of calculation, resulting in low construction efficiency of super-resolution images. At the same time, the uncertainty of dithering will also affect the quality of the final super-resolution image.
  • the drive motor can perform reverse motion compensation to offset the shake of the camera in various directions during image acquisition. That is to say, the gimbal can perform active and controllable shaking.
  • the so-called shaking mainly refers to the rotation of the gimbal around at least one rotation axis.
  • the so-called controllable jitter means that the direction and amount of jitter are controllable, rather than random. Therefore, if active and controllable shaking is performed by controlling the pan/tilt, the photographing device can acquire an image containing specified spatial phase information. Therefore, when constructing an ultra-high resolution image, the estimation of the spatial phase is omitted, the calculation amount is reduced, and the construction efficiency is improved. Moreover, the controllable dithering can also ensure the quality of the final ultra-high resolution image generation.
  • the multiple low-resolution images are images collected by the camera after performing sub-pixel displacement, that is, it is necessary to control the pan-tilt to shake at the sub-pixel level, which is a small-angle rotation for the pan-tilt.
  • the generated rotational friction force is negatively correlated with the rotation angle of the pan/tilt. That is, the smaller the rotation angle, the greater the rotational friction. Since the rotation of the pan-tilt is realized by each driving motor, when the pan-tilt rotates at a small angle, the driving motor needs to provide a relatively large rotational torque to offset the rotational friction.
  • drive motors of different sizes have different rated torques. If the size of the drive motor is small, its rated torque is correspondingly small. For a drive motor with a small rated torque, when the rotational friction is greater than the rated torque of the drive motor, the drive motor needs to provide a rotational torque exceeding the rated torque for a long time to offset the rotational friction, which will cause the drive motor to produce More heat. If there is a good assembly relationship, then the drive motor can have better heat dissipation performance, but if the assembly relationship of the drive motor is poor, then the drive motor may overheat or even burn out. Furthermore, even if the driving motor has a good assembly relationship, if the driving motor provides a rotational torque exceeding the rated torque for a long time, the life of the driving motor will be shortened.
  • Step 310 Obtain the first image captured by the photographing device
  • Step 320 Obtain a second image group captured by the photographing device, the second image group includes at least one second image, and the second image is obtained after the photographing device captures the first image. Collected after the pan/tilt performs two rotations with opposite rotation directions and different rotation angles around the at least one rotation axis;
  • Step 330 According to the rotation angle difference between the two rotations, the first image and the second image group construct a third image, and the resolution of the third image is higher than that of the first image and the second image respectively. The resolution of the second image.
  • the photographing device is carried on the platform, and the platform includes at least one rotating shaft.
  • the gimbal can be a three-axis gimbal as shown in FIG. 1 , or a two-axis gimbal or a four-axis gimbal, which is not limited in this application.
  • the first image and the second image may be images respectively captured by the shooting device when at least one type of shooting parameters are the same.
  • the shooting parameters may include parameters such as white balance, exposure parameters, magnification, and focal plane. Based on this, at least one type of shooting parameters of the first image and the second image are the same to ensure that the third image has better quality.
  • the first image and the second image are images respectively captured by the photographing device under at least the same exposure parameters.
  • the exposure parameters include one or more parameters of aperture, shutter speed, and sensitivity.
  • the second image group includes one or more second images, that is, the first image and at least one second image are required to construct the third image.
  • the second image is acquired after the pan/tilt performs two rotations with opposite rotation directions and different rotation angles around the at least one rotation axis after the first image is captured by the photographing device. For example, after the pan-tilt performs the above two rotations around one of the rotation axes, the photographing device captures the second image. For another example, after the pan/tilt performs the above two rotations around the two rotation axes, the photographing device captures the second image.
  • the pan/tilt performs two rotations with opposite rotation directions and different degrees of rotation around at least one rotation axis in sequence, so that the actual rotation direction and the actual rotation angle of the pan/tilt can be known.
  • the rotation angle of the above two rotations may be greater than a preset angle threshold.
  • the angle threshold can be determined based on the rated torque of the drive motor of the gimbal.
  • the drive motor can provide a rotational torque not greater than the rated torque to drive the pan-tilt to rotate, ensuring the normal operation of the drive motor.
  • the first rotation may be a 3° rotation of the pan/tilt in a first direction around at least one rotation axis
  • the second rotation may be a 2.9° rotation of the pan/tilt in a second direction around the same rotation axis.
  • the gimbal actually rotates by 0.1° in the first direction around the rotation axis.
  • the first direction is opposite to the second direction, for example, the first direction may be clockwise, and the second direction may be counterclockwise.
  • the rotation angle difference between the two rotations may be within a preset angle range.
  • the preset angle range may include 0.4 to 0.6 times the field angle corresponding to the unit pixel.
  • the rotation angle difference between the two rotations may be 0.5 times the field angle corresponding to the unit pixel.
  • the size of the photosensitive pixel array of the image sensor is M*N. That is, the horizontal resolution is M pixels, and the vertical resolution is N pixels.
  • the field of view of the photographing device can be divided into a horizontal field of view ⁇ and a vertical field of view ⁇ .
  • the horizontal viewing angle ⁇ may be greater than the vertical viewing angle ⁇ .
  • the above-mentioned viewing angle corresponding to the unit pixel may be the viewing angle corresponding to the unit pixel in the horizontal direction of the photosensitive pixel array of the image sensor, that is, ⁇ /M.
  • the viewing angle corresponding to the unit pixel may also be the viewing angle corresponding to the unit pixel in the vertical direction of the photosensitive pixel array of the image sensor, that is, ⁇ /N.
  • the rotation angle difference between the two rotations is 0.4 to 0.6 times the field angle corresponding to the unit pixel, and the rotation angle difference can be within the angle range of 0.4 ⁇ /M to 0.6 ⁇ /M, or it can be rotated
  • the angular difference is within an angular range of 0.4 ⁇ /N to 0.6 ⁇ /N.
  • the angle of view corresponding to a unit pixel whose rotation angle difference between two rotations is 0.5 times may be 0.5 ⁇ /M or 0.5 ⁇ /N.
  • the difference in rotation angle between two rotations may be less than 0.1 degrees.
  • the spatial phase information of the first image is different from the spatial phase information of the second image.
  • the actual rotation angle of the gimbal is smaller than the field of view angle corresponding to the unit pixel. That is to say, the gimbal performs active and controllable sub-pixel level shaking.
  • an image processing method provided by the present application controls the pan/tilt to perform two rotations with opposite rotation directions and different rotation angles in sequence around at least one rotation axis, so as to obtain the pan/tilt shake of the sub-pixel level shake amount, and when the shaking The first image and at least one second image are acquired successively.
  • this embodiment realizes the technical idea of using the shooting device mounted on the pan-tilt to obtain super-resolution images, and can control the pan-tilt to perform active and controllable shaking to obtain images containing specified spatial phase information. image. In this way, when constructing an ultra-high resolution image, no phase estimation is required, which simplifies the amount of calculation, thereby improving the efficiency of image construction.
  • the solution of this embodiment has a wider scope of application and can be applied to small-sized driving motors without causing losses to the pan-tilt driving motors, and will not generate heat due to the small rated torque of the small-sized driving motors.
  • the construction process of the third image in step 330 above may include steps as shown in FIG. 5:
  • Step 331 For each second image, according to the rotation angle difference, determine the spatial phase relationship between the first image and the second image;
  • Step 332 Based on the spatial phase relationship, interleave the pixels of the first image and the pixels of the second image to construct the third image.
  • the pan/tilt is controlled to perform two rotations in opposite directions and with different rotation angles, and then the photographing device captures a second image 620 .
  • the spatial phase relationship between the first image and the second image can be determined. It can be seen vividly from Fig. 6 that the spatial phase of the second image pixel may be the spatial phase of the sub-pixel between the first image pixel and the pixel. That is, the second image carries the image information of the sub-pixels of the first image, so the third image can be constructed by interleaving the pixels of the first image and the pixels of the second image.
  • the spatial phase relationship between the first image and the second image may include a relative offset direction and a relative offset in space, so that the determination process of the spatial phase relationship in the above step 331 includes as shown in FIG. 7 step:
  • Step 3311 Determine the actual rotation direction and actual rotation angle of the pan/tilt around the at least one rotation axis according to the rotation angle difference;
  • Step 3312 Determine the relative offset direction based on the actual rotation direction
  • Step 3313 Determine the relative offset based on the actual rotation angle.
  • the actual rotation direction and actual rotation angle of the pan/tilt can be determined.
  • the relative offset direction between the first image and the second image can be determined. As an example, if the gimbal rotates in the yaw direction, the relative offset direction between the first image and the second image is the horizontal direction; if the gimbal rotates in the pitch direction, the relative offset direction between the first image and the second image The moving direction is the vertical direction.
  • the relative offset between the first image and the second image can be determined. As an example, if the actual rotation angle of the gimbal is 0.5 times the viewing angle corresponding to the unit pixel, then the relative offset between the first image and the second image is 0.5 pixels.
  • the spatial phase information of the first image and the second image can be determined through the rotation angle difference between the two rotations, and then based on the spatial phase information, the pixels of the first image and the second image can be staggered to construct a
  • the third image has a resolution higher than that of the first image and the second image respectively. That is, the third image is a super-resolution image.
  • the gimbal may include a first axis of rotation 810 .
  • the gimbal can perform yaw rotation around the first rotation axis 810 .
  • the gimbal can perform two yaw rotations around the first rotation axis 810 with opposite rotation directions and different rotation angles, and then the photographing device captures the second image.
  • the actual rotation direction and the actual rotation angle of the pan/tilt can be determined according to the rotation angle difference between the two rotations.
  • the gimbal may first perform a yaw rotation of a first rotation angle around the first rotation axis 810 in a first direction, and then perform a yaw rotation of a second rotation angle around the first rotation axis 810 in a second direction.
  • the first direction is opposite to the second direction, for example, the first direction may be clockwise, and the second direction may be counterclockwise.
  • both the first rotation angle and the second rotation angle are greater than a preset angle threshold.
  • the angle threshold can be determined based on the rated torque of the drive motor of the gimbal.
  • first rotation angle is greater than the second rotation angle, it can be determined that the gimbal actually performs yaw rotation around the first rotation axis 810 in the first direction, and the rotation angle is the difference between the first rotation angle and the second rotation angle. On the contrary, if the first rotation angle is smaller than the second rotation angle, it can be determined that the gimbal actually performs yaw rotation in the second direction around the first rotation axis 810 .
  • the gimbal in order to obtain the second image containing the specified spatial phase information, can be rotated twice in opposite directions so that the gimbal performs a yaw rotation at a preset angle in a preset direction, The rotation angle difference between the two yaw rotations is the preset angle.
  • the preset direction may be the first direction.
  • the gimbal may first rotate around the first rotation axis 810 in the first direction at a first rotation angle.
  • the gimbal can first perform yaw rotation of the second rotation angle around the first rotation axis 810 in the second direction, and then perform yaw rotation of the first rotation angle around the first rotation axis 810 in the first direction .
  • the difference between the first rotation angle and the second rotation angle is a preset angle.
  • the photographing device can collect a second image.
  • the field of view angle corresponding to a unit pixel whose actual rotation angle of the pan/tilt is equal to 0.5 times it can be known that there is a relative offset of 0.5 pixels in the horizontal direction between the first image and the second image.
  • a third image 931 can be constructed.
  • the resolutions of the first image 910 and the second image 921 are both M*N
  • the resolution of the third image 931 is 2M*N.
  • the gimbal may include a second axis of rotation 820 .
  • the gimbal can perform pitch rotation around the second rotation axis 820 .
  • the gimbal can perform two pitching rotations around the second rotation axis 820 with opposite rotation directions and different rotation angles, and then the photographing device captures the second image.
  • the actual rotation direction and the actual rotation angle of the pan/tilt can be determined according to the rotation angle difference between the two rotations.
  • the gimbal may first perform a pitch rotation of a third rotation angle in a third direction around the second rotation axis 820 , and then perform a yaw rotation of a fourth rotation angle in a fourth direction around the second rotation axis 820 .
  • the third direction is opposite to the fourth direction, for example, the third direction may be clockwise, and the fourth direction may be counterclockwise.
  • both the third rotation angle and the fourth rotation angle are greater than a preset angle threshold.
  • the angle threshold can be determined based on the rated torque of the drive motor of the gimbal.
  • the third rotation angle is greater than the fourth rotation angle, it can be determined that the gimbal actually performs pitch rotation around the second rotation axis 820 in the third direction, and the rotation angle is the difference between the third rotation angle and the fourth rotation angle. On the contrary, if the third rotation angle is smaller than the fourth rotation angle, it can be determined that the gimbal actually performs pitch rotation in the fourth direction around the second rotation axis 820 .
  • the gimbal in order to obtain the second image containing specified spatial phase information, can be rotated twice in opposite directions, so that the gimbal can perform a preset angle of pitch rotation in a preset direction, wherein the two The rotation angle difference of the next pitch rotation is the preset angle.
  • the preset direction may be a third direction.
  • the gimbal may first perform a pitch of a third rotation angle in the third direction around the second rotation axis 820 Rotation, followed by a yaw rotation of a fourth rotation angle in a fourth direction around the second rotation axis 820 .
  • the gimbal may also perform a pitch rotation of a fourth rotation angle in a fourth direction around the second rotation axis 820 first, and then perform a yaw rotation of a third rotation angle in a third direction around the second rotation axis 820 .
  • the difference between the third rotation angle and the fourth rotation angle is a preset angle.
  • the photographing device can collect a second image. Taking the field of view corresponding to a unit pixel whose actual rotation angle of the gimbal is equal to 0.5 times as an example, it can be known that there is a relative offset of 0.5 pixels between the first image and the second image in the vertical direction. As shown in FIG. 9B , after the pixels of the first image 910 and the pixels of the second image 922 are alternately arranged, a third image 932 can be constructed. Wherein, the resolutions of the first image 910 and the second image 921 are both M*N, and the resolution of the third image 932 is M*2N.
  • the multiple second images form a second image group for constructing a third image with the first image.
  • the photographing device collects the second image; after the pan/tilt performs two pitch rotations in opposite directions around the second rotation axis, the photographing device Collecting a second image; after the pan/tilt rotates in opposite directions twice around the first rotation axis and the second rotation axis, the shooting device collects the second image.
  • the gimbal may first perform two yaw rotations in opposite directions around the first rotation axis, and then perform two pitch rotations in opposite directions around the second rotation axis.
  • the gimbal can first perform a yaw rotation of a first rotation angle in a first direction around a first rotation axis, then perform a pitch rotation of a third rotation angle in a third direction around a second rotation axis, and then rotate around a first rotation axis.
  • the rotation axis performs a yaw rotation of a second rotation angle in the second direction, and finally performs a pitch rotation of a fourth rotation angle in a fourth direction around the second rotation axis.
  • the third image can be constructed 933 , wherein the second image group 920 includes a second image 921 , a second image 922 and a second image 923 .
  • the resolutions of the first image 910 , the second image 921 , the second image 922 and the second image 923 are all M*N, while the resolution of the third image 933 is 2M*2N.
  • An image processing method provided by the present application by controlling the pan-tilt to perform two rotations with opposite rotation directions and different rotation angles sequentially around at least one rotation axis, to obtain sub-pixel-level pan-tilt shake, and obtain before and after the shake A first image and at least one second image. Since the first image and the second image have different spatial phase information, based on the super-resolution image construction technology, the super-high resolution of the third image. In addition, by controlling the gimbal to rotate in the yaw and/or pitch directions, second images including different spatial phase information can be obtained, so as to construct third images with different resolutions.
  • this embodiment realizes the technical idea of using the shooting device mounted on the pan-tilt to obtain super-resolution images, and can control the pan-tilt to perform active and controllable shaking to obtain images containing specified spatial phase information. image.
  • the solution of this embodiment has a wider scope of application and can be applied to small-sized driving motors without causing losses to the pan-tilt driving motors, and will not generate heat due to the small rated torque of the small-sized driving motors.
  • Step 1010 Obtain the first image collected by the photographing device; the photographing device is carried on the platform, and the platform includes at least one rotation axis;
  • Step 1020 Obtain a second image group captured by the photographing device, the second image group includes at least one second image, and the second image is obtained after the photographing device captures the first image. Collected after the pan/tilt performs two rotations with opposite rotation directions and different rotation angles around the at least one rotation axis;
  • Step 1030 For each second image, according to the rotation angle difference, determine the actual rotation direction and actual rotation angle of the pan/tilt around the at least one rotation axis;
  • Step 1040 Determine the relative offset direction based on the actual rotation direction
  • Step 1050 Determine the relative offset based on the actual rotation angle
  • Step 1060 Based on the relative offset direction and the relative offset amount, the pixels of the first image and the pixels of the second image are alternately arranged to construct the third image.
  • the rotation of the pan-tilt can be controlled so that the photographing device faces the target object.
  • the gimbal is in the first pose.
  • the photographing device shoots the first image with a resolution of M*N through the stabilization of the pan-tilt according to the set exposure parameters.
  • the horizontal viewing angle of the photographing device is ⁇
  • the horizontal resolution is M
  • the viewing angle corresponding to a unit pixel is ⁇ /M.
  • the gimbal performs two yaw rotations in opposite directions around the first rotation axis, so that the gimbal performs a yaw with a rotation angle of 0.5 ⁇ /M in the first preset direction turn.
  • the gimbal is in the second pose.
  • the photographing device photographs a second image with a resolution of M*N according to the same exposure parameters.
  • the gimbal performs two pitch rotations in opposite directions around the second rotation axis, so that the gimbal performs a pitch rotation with a rotation angle of 0.5 ⁇ /M in the second preset direction.
  • the gimbal is in the third position.
  • the photographing device photographs a second image with a resolution of M*N according to the same exposure parameters.
  • the gimbal performs two yaw rotations in opposite directions around the first rotation axis, so that the gimbal performs a yaw rotation with a rotation angle of 0.5 ⁇ /M in the third preset direction turn.
  • the gimbal is in the fourth position.
  • the photographing device photographs a second image with a resolution of M*N according to the same exposure parameters.
  • the first preset direction is opposite to the second preset direction.
  • the actual rotation direction and the actual rotation angle of the pan/tilt are determined according to the rotation angle difference. And based on the actual rotation direction, determine the relative offset direction between the first image and each second image. Based on the actual rotation angle, the relative offset between the first image and each second image is determined. Finally, based on the relative offset direction and relative offset, the pixels of the above-mentioned first image and the three second images are interleaved to obtain a third image with a resolution of 2M*2N.
  • An image processing method provided by the present application by controlling the pan-tilt to perform two rotations with opposite rotation directions and different rotation angles sequentially around at least one rotation axis, to obtain sub-pixel-level pan-tilt shake, and obtain before and after the shake A first image and at least one second image. Since the first image and the second image have different spatial phase information, based on the super-resolution image construction technology, the super-high resolution of the third image. Compared with related technologies, on the one hand, this embodiment realizes the technical idea of using the shooting device mounted on the pan-tilt to obtain super-resolution images, and can control the pan-tilt to perform active and controllable shaking to obtain images containing specified spatial phase information. image.
  • the solution of this embodiment has a wider scope of application and can be applied to small-sized driving motors without causing losses to the pan-tilt driving motors, and will not generate heat due to the small rated torque of the small-sized driving motors.
  • the present application also provides a schematic structural diagram of an image processing device as shown in FIG. 11 .
  • the image processing device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and of course may also include hardware required by other services.
  • the processor reads the corresponding computer program from the non-volatile memory into the memory and runs it, so as to realize the image processing method described in any of the above embodiments.
  • the present application also provides a schematic structural diagram of a pan/tilt as shown in FIG. 12 .
  • the pan/tilt includes at least one rotating shaft, at least one motor, processor, internal bus, network interface, memory and non-volatile memory, and of course may also include hardware required by other services.
  • at least one motor is used to drive the platform to rotate around at least one rotation axis.
  • the processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, so as to realize an image processing method described in any of the above embodiments.
  • the present application also provides a schematic structural diagram of a movable platform as shown in FIG. 13 .
  • the mobile platform includes a body, a gimbal, power components, a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and of course may also include hardware required by other services.
  • the pan-tilt is used to carry the photographing device, and the pan-tilt includes at least one rotating shaft.
  • the power assembly is used to drive the movable platform to move in space.
  • the processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, so as to realize an image processing method described in any of the above embodiments.
  • the present application also provides a computer program product, including a computer program, when the computer program is executed by a processor, it can be used to perform the image processing method described in any of the above embodiments method.
  • the present application also provides a computer storage medium, where a computer program is stored in the storage medium, and when the computer program is executed by a processor, it can be used to perform an image processing method described in any of the above embodiments. an image processing method.
  • the device embodiment since it basically corresponds to the method embodiment, for related parts, please refer to the part description of the method embodiment.
  • the device embodiments described above are only illustrative, and the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in One place, or it can be distributed to multiple network elements. Part or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. It can be understood and implemented by those skilled in the art without creative effort.

Landscapes

  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Automation & Control Theory (AREA)
  • Studio Devices (AREA)

Abstract

本申请提供的一种图像处理方法、装置、云台、可移动平台，云台包括至少一个转动轴，且云台上搭载有拍摄装置。拍摄装置首先采集第一图像；然后云台绕至少一个转动轴依次进行转动方向相反且转动角度不同的两次转动后，拍摄装置采集至少一张第二图像，组成第二图像组。根据两次转动的转动角度差值、第一图像以及第二图像组来构建第三图像，其中第三图像的分辨率分别高于第一图像和第二图像的分辨率，也即第三图像为超分辨率图像。通过上述方法实现了利用搭载在云台上的拍摄装置获取超分辨率图像。

Description

一种图像处理方法、装置、云台、可移动平台 技术领域
本申请涉及图像处理技术领域，尤其涉及一种图像处理方法、装置、云台、可移动平台。
背景技术
云台是一种安装和固定拍摄装置的支撑设备，云台可以通过控制算法提供增稳功能，使得拍摄装置在感光成像的过程当中能够保持稳定，有利于生成高质量的图像。
一般来说，拍摄装置采集的图像的分辨率即为图像传感器(sensor)的感光像素阵列的像素数。若想获得分辨率超过图像传感器像素数的图像，需要对图像利用超分辨率图像重建技术进行处理。然而相关技术中，在构建超分辨率图像时构建效率较低。
发明内容
有鉴于此，本申请的目的之一是提供一种图像处理方法、装置、云台、可移动平台，以提高超分辨率图像的构建效率。
为了达到上述技术效果，本发明实施例公开了如下技术方案：
第一方面，提供了一种图像处理方法，所述方法包括：
获取拍摄装置采集的第一图像；所述拍摄装置搭载在云台上，所述云台包括至少一个转动轴；
获取所述拍摄装置采集的第二图像组，所述第二图像组包括至少一张第二图像，所述第二图像是在所述拍摄装置采集所述第一图像后，所述云台绕所述至少一个转动轴依次进行转动方向相反且转动角度不同的两次转动后采集的；
根据所述两次转动的转动角度差值，所述第一图像与所述第二图像组构建第三图像，所述第三图像的分辨率分别高于所述第一图像和第二图像的分辨率。
第二方面，提供了一种图像处理装置，包括：
处理器；
用于存储处理器可执行指令的存储器；
其中，所述处理器调用所述可执行指令时实现第一方面所述的操作。
第三方面，提供了一种云台，搭载有拍摄装置，包括：
至少一个转动轴；
至少一个电机，用于驱动所述云台绕所述至少一个转动轴转动；
处理器；
用于存储处理器可执行指令的存储器；
其中，所述处理器调用所述可执行指令时实现第一方面所述的操作。
第四方面，提供了一种可移动平台，包括：
机身；
用于携带拍摄装置的云台，所述云台包括至少一个转动轴；
动力组件，用于驱动所述可移动平台在空间中运动；
处理器；
用于存储处理器可执行指令的存储器；
其中，所述处理器调用所述可执行指令时实现第一方面所述的操作。
第五方面，提供了一种计算机程序产品，包括计算机程序，所述计算机程序被处理器执行时实现第一方面所述的操作。
第六方面，提供了一种计算机可读存储介质，所述计算机可读存储介质上存储有若干计算机指令，所述计算机指令被执行时执行第一方面所述的操作。
本申请的有益效果将在以下实施例展开说明。
附图说明
为了更清楚地说明本申请实施例中的技术方案，下面将对实施例描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动性的前提下，还可以根据这些附图获得其他的附图。
图1是本申请根据一实施例示出的三轴云台的结构示意图。
图2本申请根据一实施例示出的拍摄场景的示意图。
图3本申请根据一实施例示出的一种图像处理方法的流程图。
图4本申请根据一实施例示出的感光像素阵列的示意图。
图5是本申请根据另一实施例示出的一种图像处理方法的流程图。
图6是本申请根据一实施例示出的第一图像与第二图像空间相位关系的示意图。
图7是本申请根据另一实施例示出的一种图像处理方法的流程图。
图8A是本申请根据一实施例示出的第一转动轴的示意图。
图8B是本申请根据一实施例示出的第二转动轴的示意图。
图9A是本申请根据一实施例示出的第三图像的示意图。
图9B是本申请根据另一实施例示出的第三图像的示意图。
图9C是本申请根据另一实施例示出的第三图像的示意图。
图10是本申请根据另一实施例示出的一种图像处理方法的流程图。
图11是本申请根据一实施例示出的一种图像处理装置的结构示意图。
图12是本申请根据一实施例示出的一种云台的结构示意图。
图13是本申请根据一实施例示出的一种可移动平台的结构示意图。
具体实施方式
下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本申请保护的范围。
云台是一种安装和固定拍摄装置的支撑设备，云台可以通过控制算法提供增稳功能，使得拍摄装置在感光成像的过程当中能够保持稳定，有利于生成清晰的图像。
其中，云台可以是手持云台，也可以是搭载于可移动平台上的云台。云台可以是两轴云台，也可以是三轴云台。
作为例子，图1示出了一种三轴云台100的结构图。如图1所示，三轴云台100主要包括：俯仰(pitch)轴电机101、横滚(roll)轴电机102、航向(yaw)轴电机103、云台基座104、yaw轴轴臂105、拍摄设备固定机构106(内部包含惯性测量元件IMU)、pitch轴轴臂107、roll轴轴臂108，拍摄设备固定机构106用于搭载拍摄设备109。其中，roll轴轴臂108用于支撑pitch轴轴臂107和pitch轴电机101，yaw轴轴臂105用于支撑yaw轴电机103和roll轴电机102，pitch轴轴臂107用于支撑拍摄设备109。pitch轴电机101、roll轴电机102以及yaw轴电机103可统称为驱动电机。在驱动电机转动时，通过驱动电机安装的角度传感器可以测量驱动电机所转动的角度，其中角度传感器可以为电位计、霍尔传感器、编码器中的一种或多种。三轴云台100可以通过云台基座104与手柄连接或与可移动平台连接(图1未示出)。
搭载在云台上的拍摄装置用于采集图像，如图2所示，光线经物体反射后，经过拍摄装置的镜头210等光学组件进入拍摄装置中的图像传感器(sensor)220，图像传感器220在像平面上通过感光像素阵列进行光信号的收集和采样，从而获得一幅图像的数字采样，即对应该光学组件视场角的一幅数字图像。
图像传感器220的感光像素阵列一般采用拜耳(Bayer)排列方式，图像处理算法通过插值解马赛克算法恢复出红、绿、蓝三种颜色的图像，每种图像的分辨率均为感光像素阵列的大小。三种颜色的图像组成一幅彩色的图像，分辨率也为图像传感器的感光像素阵列的大小。其他感光像素阵列排列方式的图像传感器，通过对应的方法恢复出的图像，分辨率也为图像传感器的感光像素阵列的大小。
图像的分辨率受限于图像传感器220的感光像素阵列的大小。为降低感光像素阵列对分辨率的约束，可以使用超分辨率(Super Resolution，SR)重建技术获得超高分辨率的图像。
超分辨率图像重建技术是利用多张包含不同空间相位信息的低分辨率(Low Resolution，LR)图像构建出超分辨率图像的技术。所谓低分辨 率，是指图像的分辨率为图像传感器的感光像素阵列的大小。而超分辨率，是指图像的分辨率大于图像传感器的感光像素阵列的大小。图像中一个亚像素对应于一个空间相位，由于多张低分辨率图像存在亚像素位移，因此每张低分辨率图像包含了不同空间相位对应的图像信息。基于深度学习或其他算法，可以将多张特定空间相位的图像组合成具有更高分辨率的图像。
在相关技术中，拍摄装置通过随机抖动来获取多张包含不同空间相位信息的低分辨率图像。由于抖动的随机性，每次抖动的位移量可能都不相同。如此，在构建超高分辨率图像时，就需要进行空间相位估计以及重采样，从而带来了较大的计算量，导致超分辨率图像的构建效率较低。同时抖动的不确定性也会影响最终超高分辨率图像的生成质量。
发明人发现，由于云台可以感知拍摄装置的运动信息，然后基于反馈算法来使驱动电机进行反向运动补偿，以抵消图像采集时拍摄装置在各个方向上的抖动。也即云台可以进行主动且可控的抖动。所谓抖动，主要是指云台绕至少一个转动轴进行转动。所谓可控的抖动，是指抖动方向和抖动量是可控的，而非随机的。因此，若通过控制云台进行主动可控的抖动，可以使拍摄装置获取到包含指定空间相位信息的图像。从而在构建超高分辨率图像时，省去了对空间相位的估计，减少了计算量，提高了构建效率。而且可控的抖动也能保证最终超高分辨率图像的生成质量。
然而，由于多张低分辨率图像是拍摄装置进行亚像素位移后所采集的图像，也即需要控制云台进行亚像素级抖动量的抖动，这对云台来说是一个小角度的转动。而在转动过程中，产生的转动摩擦力与云台的转动角度为负相关关系。也即转动角度越小，转动摩擦力越大。由于云台的转动是通过各驱动电机实现的，因此当云台进行小角度转动时，需要驱动电机提供一个较大的转动力矩来抵消该转动摩擦力。
对于驱动电机来说，不同尺寸的驱动电机有不同大小的额定力矩。若驱动电机的尺寸较小，相应地，其额定力矩也较小。对于额定力矩较小的驱动电机，当转动摩擦力大于驱动电机的额定力矩时，驱动电机需要在较长的时间内提供超出额定力矩的转动力矩来抵消该转动摩擦力，这会使驱动电机产生较多热量。若有较好的装配关系，那么驱动电机可以有较优的散热性能，但若驱动电机的装配关系较差，那么驱动电机可能会过热甚至会烧毁。进一步地，就算驱动电机有较好的装配关系，但长时间让驱动电机提供超出额定力矩的转动力矩，也会缩短驱动电机的寿命。
虽然可以通过增大驱动电机的尺寸以增大额定力矩来解决上述问题，但对于可移动平台或手持云台来说，驱动电机尺寸的增大无疑会增加设备的重量以及成本，这又会引发诸如缩短续航时间等问题。可见，在实现控制云台进行主动可控抖动，来获取包含指定空间相位信息的图像技术构思的过程中，仍面临着诸如额定力矩较小、发热等技术挑战。在相关技术中尚未有较好的解决方案。
为了寻求上述技术问题的解决方案，本申请提出了一种图像处理方法，包括如图3所示的步骤：
步骤310：获取拍摄装置采集的第一图像；
步骤320：获取所述拍摄装置采集的第二图像组，所述第二图像组包括至少一张第二图像，所述第二图像是在所述拍摄装置采集所述第一图像后，所述云台绕所述至少一个转动轴依次进行转动方向相反且转动角度不同的两次转动后采集的；
步骤330：根据所述两次转动的转动角度差值，所述第一图像与所述第二图像组构建第三图像，所述第三图像的分辨率分别高于所述第一图像和第二图像的分辨率。
其中，拍摄装置搭载在云台上，云台包括至少一个转动轴。云台可以是如图1所示的三轴云台，也可以是二轴云台或四轴云台等，本申请不做限制。
在一些实施例中，第一图像与第二图像可以是拍摄装置至少在一类拍摄参数相同的情况下分别采集的图像。其中，拍摄参数可以包括白平衡、曝光参数、倍率、焦平面等参数，基于此，利用第一图像与第二图像至少有一类拍摄参数相同，可以保证第三图像具有较好的质量。
在一些实施例中，第一图像与第二图像是拍摄装置至少在曝光参数相同的情况下分别采集的图像。其中，曝光参数包括光圈、快门速度、感光度中的一种或多种参数。
第二图像组包括一张或以上的第二图像，也即需要第一图像与至少一张第二图像来构建第三图像。第二图像是在拍摄装置采集第一图像后，云台绕所述至少一个转动轴依次进行转动方向相反且转动角度不同的两次转动后采集的。例如云台可以绕其中一个转动轴进行上述两次转动后，拍摄装置采集第二图像。又例如，云台可以绕两个转动轴进行上述两次转动后，拍摄装置采集第二图像。
云台绕至少一个转动轴依次进行转动方向相反且转动度不同的两次转动，可以得知云台实际转动的方向以及实际的转动角度。在一些实施例中，上述两次转动的转动角度可以大于预设的角度阈值。角度阈值可以基于云台的驱动电机的额定力矩确定。云台进行转动角度大于角度阈值的转动，可以使驱动电机提供不大于额定力矩的转动力矩来驱动云台转动，保证了驱动电机的正常工作。
例如第一次转动可以是云台绕至少一个转动轴在第一方向进行了3°的转动，第二次转动可以是云台绕同样的转动轴在第二方向进行了2.9°的转动。如此，云台实际上是绕上述转动轴在第一方向进行了0.1°的转动。其中，第一方向与第二方向相反，例如第一方向可以是顺时针方向，第二方向可以是逆时针方向。
在一些实施例中，两次转动的转动角度差值可以在预设的角度范围内。作为例子，预设的角度范围可以包括0.4至0.6倍的单位像素对应的视场角。
在一些实施例中，两次转动的转动角度差值可以是0.5倍的单位像素对应的视场角。
其中，如图4所示，图像传感器的感光像素阵列的大小为M*N。即水平分辨率为M像素，垂直分辨率为N像素。拍摄装置的视场角可以分为水平视场角α与垂直视场角β。在一些实施例中，水平视场角α可以大于垂直视场角β。上述单位像素对应的视场角，可以是图像传感器的感光像素阵列在水平方向上单位像素对应的视场角，即α/M。单位像素对应的视场角，也可以是图像传感器的感光像素阵列在垂直方向上单位像素对应的视场角，即β/N。
作为例子，两次转动的转动角度差值在0.4至0.6倍的单位像素对应的视场角，可以是转动角度差值在0.4α/M至0.6α/M的角度范围内，也可以是转动角度差值在0.4β/N至0.6β/N的角度范围内。
作为例子，两次转动的转动角度差值为0.5倍的单位像素对应的视场角，可以是转动角度差值为0.5α/M，也可以是转动角度差值为0.5β/N。
在一些实施例中，两次转动的转动角度差值可以小于0.1度。
在一些实施例中，第一图像的空间相位信息与第二图像的空间相位信息不同。
可知，云台绕至少一个转动轴依次进行转动方向相反且转动角度不同的两次转动后，云台实际转过的角度小于单位像素对应的视场角。也即云台进行了主动可控的亚像素级抖动量的抖动。如此，本申请提供的一种图像处理方法，通过控制云台绕至少一个转动轴依次进行转动方向相反且转动角度不同的两次转动，来获得亚像素级抖动量的云台抖动，并在抖动前后获取第一图像以及至少一张第二图像。由于第一图像与第二图像有不相同的空间相位信息，因此可以基于超分辨率图像构建技术，利用两次转动的转动角度差值、第一图像以及至少一张第二图像来构建超高分辨率的第三图像。与相关技术相比，一方面，本实施例实现了利用搭载在云台上的拍摄装置获取超分辨率图像的技术构思，能够控制云台进行主动可控抖动，来获取包含指定空间相位信息的图像。如此，在构建超高分辨率图像时，无需进行相位估计，简化了计算量，从而提高了图像构建效率。另一方面，本实施例的方案适用范围较广，能适用于小尺寸驱动电机，不会对云台驱动电机造成损耗，也不会产生因小尺寸驱动电机额定力矩小而发热的问题。
在一些实施例中，上述步骤330第三图像的构建过程可以包括如图5所示的步骤：
步骤331：针对每张第二图像，根据所述转动角度差值，确定所述第一图像与所述第二图像的空间相位关系；
步骤332：基于所述空间相位关系，将所述第一图像的像素与所述第二图像的像素进行交错排列，构建所述第三图像。
如图6所示，在拍摄装置采集第一图像610后，控制云台进行两次方向相反且转动角度不同的转动，随后拍摄装置采集第二图像620。根据两次转动的转动角度差值，可以确定第一图像与第二图像的空间相位关系。从图6可以形象地得知，第二图像像素的空间相位，可以是第一图像像素 与像素之间的亚像素的空间相位。也即第二图像携带了第一图像的亚像素的图像信息，因此可以通过将第一图像的像素与第二图像的像素进行交错排列，构建出第三图像。
在一些实施例中，第一图像与第二图像的空间相位关系可以包括空间上的相对偏移方向和相对偏移量，如此，上述步骤331空间相位关系的确定过程包括如图7所示的步骤：
步骤3311：根据所述转动角度差值，确定所述云台绕所述至少一个转动轴的实际转动方向和实际转动角度；
步骤3312：基于所述实际转动方向，确定所述相对偏移方向；
步骤3313：基于所述实际转动角度，确定所述相对偏移量。
根据两次转动的转动角度差值，可以确定云台的实际转动方向和实际转动角度。基于云台的实际转动方向，可以确定第一图像与第二图像的相对偏移方向。作为例子，若云台在偏航方向上转动，则第一图像与第二图像的相对偏移方向为水平方向；若云台在俯仰方向上转动，则第一图像与第二图像的相对偏移方向为垂直方向。
基于云台的实际转动角度，可以确定第一图像与第二图像的相对偏移量。作为例子，若云台实际转动角度为0.5倍的单位像素对应的视场角，则第一图像与第二图像的相对偏移量为0.5个像素。
如此，通过两次转动的转动角度差值，可以确定出第一图像与第二图像的空间相位信息，然后基于空间相位信息，将第一图像与第二图像的像素进行交错排列，可以构建出分辨率分别高于第一图像与第二图像分辨率的第三图像。也即第三图像为超分辨率图像。
如图8A所示，在一些实施例中，云台可以包括第一转动轴810。云台绕第一转动轴810可以进行偏航转动。在拍摄装置采集第一图像后，云台可以绕第一转动轴810进行转动方向相反且转动角度不同的两次偏航转动，然后拍摄装置采集第二图像。
在一些实施例中，可以根据两次转动的转动角度差值，确定云台的实际转动方向和实际转动角度。作为例子，云台可以首先绕第一转动轴810在第一方向进行第一转动角度的偏航转动，随后绕第一转动轴810在第二方向进行第二转动角度的偏航转动。其中，第一方向与第二方向相反，例如第一方向可以是顺时针方向，第二方向可以是逆时针方向。同时，第一转动角度与第二转动角度均大于预设的角度阈值。角度阈值可以基于云台的驱动电机的额定力矩确定。
若第一转动角度大于第二转动角度，则可以确定云台实际上绕第一转动轴810在第一方向上进行偏航转动，转动角度为第一转动角度与第二转动角度之差。反之，若第一转动角度小于第二转动角度，则可以确定云台实际上绕第一转动轴810在第二方向上进行偏航转动。
在一些实施例中，为了获取包含指定空间相位信息的第二图像，可以让云台通过两次方向相反的偏航转动，以使云台在预设方向上进行预设角度的偏航转动，其中两次偏航转动的转动角度差值为该预设角度。
预设方向可以是第一方向，为了使云台在第一方向上进行预设角度的偏航转动，作为例子，云台可以首先绕第一转动轴810在第一方向进行第一转动角度的偏航转动，随后绕第一转动轴810在第二方向进行第二转动角度的偏航转动。
作为另一个例子，云台还可以首先绕第一转动轴810在第二方向进行第二转动角度的偏航转动，然后绕第一转动轴810在第一方向进行第一转动角度的偏航转动。其中，第一转动角度与第二转动角度之差为预设角度。
在云台进行两次方向相反的偏航转动后，拍摄装置可以采集第二图像。以云台实际转动角度等于0.5倍的单位像素对应的视场角为例，可知，第一图像与第二图像在水平方向上存在0.5个像素的相对偏移。如图9A所示，将第一图像910的像素与第二图像921的像素进行交错排列后，可以构建出第三图像931。其中，第一图像910与第二图像921的分辨率均为M*N，而第三图像931的分辨率为2M*N。
如图8B所示，在一些实施例中，云台可以包括第二转动轴820。云台绕第二转动轴820可以进行俯仰转动。在拍摄装置采集第一图像后，云台可以绕第二转动轴820进行转动方向相反且转动角度不同的两次俯仰转动，然后拍摄装置采集第二图像。
在一些实施例中，可以根据两次转动的转动角度差值，确定云台的实际转动方向和实际转动角度。作为例子，云台可以首先绕第二转动轴820在第三方向进行第三转动角度的俯仰转动，随后绕第二转动轴820在第四方向进行第四转动角度的偏航转动。其中，第三方向与第四方向相反，例如第三方向可以是顺时针方向，第四方向可以是逆时针方向。同时，第三转动角度与第四转动角度均大于预设的角度阈值。角度阈值可以基于云台的驱动电机的额定力矩确定。
若第三转动角度大于第四转动角度，则可以确定云台实际上绕第二转动轴820在第三方向上进行俯仰转动，转动角度为第三转动角度与第四转动角度之差。反之，若第三转动角度小于第四转动角度，则可以确定云台实际上绕第二转动轴820在第四方向上进行俯仰转动。
在一些实施例中，为了获取包含指定空间相位信息的第二图像，可以让云台通过两次方向相反的俯仰转动，以使云台在预设方向上进行预设角度的俯仰转动，其中两次俯仰转动的转动角度差值为该预设角度。
预设方向可以是第三方向，为了使云台在第三方向上进行预设角度的偏航转动，作为例子，云台可以首先绕第二转动轴820在第三方向进行第三转动角度的俯仰转动，随后绕第二转动轴820在第四方向进行第四转动角度的偏航转动。
作为另一个例子，云台还可以首先绕第二转动轴820在第四方向进行第四转动角度的俯仰转动，然后绕第二转动轴820在第三方向进行第三转动角度的偏航转动。其中，第三转动角度与第四转动角度之差为预设角度。
在云台进行两次方向相反的俯仰转动后，拍摄装置可以采集第二图像。以云台实际转动角度等于0.5倍的单位像素对应的视场角为例，可知，第一图像与第二图像在垂直方向上存在0.5个像素的相对偏移。如图9B所示，将第一图像910的像素与第二图像922的像素进行交错排列后，可以构建出的第三图像932。其中，第一图像910与第二图像921的分辨率均为M*N，而第三图像932的分辨率为M*2N。
在一些实施例中，第二图像的数量为多张，多张第二图像组成第二图像组，用于与第一图像构建第三图像。其中，可以在云台绕第一转动轴进行两次方向相反的偏航转动后，拍摄装置采集第二图像；可以在云台绕第二转动轴进行两次方向相反的俯仰转动后，拍摄装置采集第二图像；可以在云台绕第一转动轴和第二转动轴进行两次方向相反的转动后，拍摄装置采集第二图像。其中，上述多张第二图像没有获取的先后顺序。
作为例子，云台可以先绕第一转动轴进行两次方向相反的偏航转动，再绕第二转动轴进行两次方向相反的俯仰转动。
作为例子，云台可以先绕第一转动轴在第一方向上进行第一转动角度的偏航转动，然后绕第二转动轴在第三方向上进行第三转动角度的俯仰转动，接着绕第一转动轴在第二方向上进行第二转动角度的偏航转动，最后绕第二转动轴在第四方向上进行第四转动角度的俯仰转动。
上述的多张第二图像组成第二图像组，如图9C所示，将第一图像910的像素与第二图像组920中各个第二图像的像素进行交错排列后，可以构建出第三图像933，其中第二图像组920包括第二图像921、第二图像922以及第二图像923。第一图像910、第二图像921、第二图像922以及第二图像923的分辨率均为M*N，而第三图像933的分辨率为2M*2N。
本申请提供的一种图像处理方法，通过控制云台绕至少一个转动轴依次进行转动方向相反且转动角度不同的两次转动，来获得亚像素级抖动量的云台抖动，并在抖动前后获取第一图像以及至少一张第二图像。由于第一图像与第二图像有不相同的空间相位信息，因此可以基于超分辨率图像构建技术，利用两次转动的转动角度差值、第一图像以及至少一张第二图像来构建超高分辨率的第三图像。此外，通过控制云台进行偏航和/或俯仰方向上的转动，可以获得包括不同空间相位信息的第二图像，以此可以构建出不同分辨率的第三图像。
与相关技术相比，一方面，本实施例实现了利用搭载在云台上的拍摄装置获取超分辨率图像的技术构思，能够控制云台进行主动可控抖动，来获取包含指定空间相位信息的图像。如此，在构建超高分辨率图像时，无需进行相位估计，简化了计算量，从而提高了图像构建效率。另一方面，本实施例的方案适用范围较广，能适用于小尺寸驱动电机，不会对云台驱动电机造成损耗，也不会产生因小尺寸驱动电机额定力矩小而发热的问题。
此外，本申请还提供了一种图像处理方法，包括如图10所示的步骤：
步骤1010：获取拍摄装置采集的第一图像；所述拍摄装置搭载在云 台上，所述云台包括至少一个转动轴；
步骤1020：获取所述拍摄装置采集的第二图像组，所述第二图像组包括至少一张第二图像，所述第二图像是在所述拍摄装置采集所述第一图像后，所述云台绕所述至少一个转动轴依次进行转动方向相反且转动角度不同的两次转动后采集的；
步骤1030：针对每张第二图像，根据所述转动角度差值，确定所述云台绕所述至少一个转动轴的实际转动方向和实际转动角度；
步骤1040：基于所述实际转动方向，确定所述相对偏移方向；
步骤1050：基于所述实际转动角度，确定所述相对偏移量；
步骤1060：基于所述相对偏移方向与所述相对偏移量，将所述第一图像的像素与所述第二图像的像素进行交错排列，构建所述第三图像。
在需要利用搭载在云台上的拍摄装置获取超分辨率图像时，可以控制云台的转动，使拍摄装置朝向目标拍摄物。此时，云台处于第一位姿。拍摄装置按照设定的曝光参数，通过云台增稳拍摄分辨率为M*N的第一图像。其中，拍摄装置的水平视场角为α，水平分辨率为M，则单位像素对应的视场角为α/M。
随后，以第一位姿为基准，云台绕第一转动轴进行两次方向相反的偏航转动，以使云台在第一预设方向上进行了转动角度为0.5α/M的偏航转动。此时，云台处于第二位姿。拍摄装置按照相同的曝光参数，拍摄分辨率为M*N的第二图像。
随后，以第二位姿为基准，云台绕第二转动轴进行两次方向相反的俯仰转动，以使云台在第二预设方向上进行了转动角度为0.5α/M的俯仰转动。此时，云台处于第三位姿。拍摄装置按照相同的曝光参数，拍摄分辨率为M*N的第二图像。
随后，以第三位姿为基准，云台绕第一转动轴进行两次方向相反的偏航转动，以使云台在第三预设方向上进行了转动角度为0.5α/M的偏航转动。此时，云台处于第四位姿。拍摄装置按照相同的曝光参数，拍摄分辨率为M*N的第二图像。其中，第一预设方向与第二预设方向相反。
针对每张第二图像，根据转动角度差值，确定所述云台的实际转动方向和实际转动角度。并基于实际转动方向，确定第一图像与每张第二图像的相对偏移方向。基于实际转动角度，确定第一图像与每张第二图像的相对偏移量。最后基于相对偏移方向与相对偏移量，将上述第一图像与三张第二图像的像素进行交错排列，得到分辨率为2M*2N的第三图像。
本申请提供的一种图像处理方法，通过控制云台绕至少一个转动轴依次进行转动方向相反且转动角度不同的两次转动，来获得亚像素级抖动量的云台抖动，并在抖动前后获取第一图像以及至少一张第二图像。由于第一图像与第二图像有不相同的空间相位信息，因此可以基于超分辨率图像构建技术，利用两次转动的转动角度差值、第一图像以及至少一张第二图像来构建超高分辨率的第三图像。与相关技术相比，一方面，本实施例实现了利用搭载在云台上的拍摄装置获取超分辨率图像的技术构思，能够 控制云台进行主动可控抖动，来获取包含指定空间相位信息的图像。如此，在构建超高分辨率图像时，无需进行相位估计，简化了计算量，从而提高了图像构建效率。另一方面，本实施例的方案适用范围较广，能适用于小尺寸驱动电机，不会对云台驱动电机造成损耗，也不会产生因小尺寸驱动电机额定力矩小而发热的问题。
基于上述任意实施例所述的一种图像处理方法，本申请还提供了如图11所示的一种图像处理装置的结构示意图。如图11，在硬件层面，该图像处理装置包括处理器、内部总线、网络接口、内存以及非易失性存储器，当然还可能包括其他业务所需要的硬件。处理器从非易失性存储器中读取对应的计算机程序到内存中然后运行，以实现上述任意实施例所述的一种图像处理方法。
基于上述任意实施例所述的一种图像处理方法，本申请还提供了如图12所示的一种云台的结构示意图。如图12，在硬件层面，该云台包括至少一个转动轴、至少一个电机、处理器、内部总线、网络接口、内存以及非易失性存储器，当然还可能包括其他业务所需要的硬件。其中，至少一个电机用于驱动云台绕至少一个转动轴转动。处理器从非易失性存储器中读取对应的计算机程序到内存中然后运行，以实现上述任意实施例所述的一种图像处理方法。
基于上述任意实施例所述的一种图像处理方法，本申请还提供了如图13所示的一种可移动平台的结构示意图。如图13，在硬件层面，该可移动平台包括机身、云台、动力组件、处理器、内部总线、网络接口、内存以及非易失性存储器，当然还可能包括其他业务所需要的硬件。其中，云台用于携带拍摄装置，云台包括至少一个转动轴。动力组件用于驱动可移动平台在空间中运动。处理器从非易失性存储器中读取对应的计算机程序到内存中然后运行，以实现上述任意实施例所述的一种图像处理方法。
基于上述任意实施例所述的一种图像处理方法，本申请还提供了一种计算机程序产品，包括计算机程序，计算机程序被处理器执行时可用于执行上述任意实施例所述的一种图像处理方法。
基于上述任意实施例所述的一种图像处理方法，本申请还提供了一种计算机存储介质，存储介质存储有计算机程序，计算机程序被处理器执行时可用于执行上述任意实施例所述的一种图像处理方法。
对于装置实施例而言，由于其基本对应于方法实施例，所以相关之处参见方法实施例的部分说明即可。以上所描述的装置实施例仅仅是示意性的，其中所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部模块来实现本实施例方案的目的。本领域普通技术人员在不付出创造性劳动的情况下，即可以理解并实施。
需要说明的是，在本文中，诸如第一和第二等之类的关系术语仅仅用来将一个实体或者操作与另一个实体或操作区分开来，而不一定要求或 者暗示这些实体或操作之间存在任何这种实际的关系或者顺序。术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、物品或者设备不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、物品或者设备所固有的要素。在没有更多限制的情况下，由语句“包括一个……”限定的要素，并不排除在包括所述要素的过程、方法、物品或者设备中还存在另外的相同要素。
以上对本申请实施例所提供的方法和装置进行了详细介绍，本文中应用了具体个例对本申请的原理及实施方式进行了阐述，以上实施例的说明只是用于帮助理解本申请的方法及其核心思想；同时，对于本领域的一般技术人员，依据本申请的思想，在具体实施方式及应用范围上均会有改变之处，综上所述，本说明书内容不应理解为对本申请的限制。

Claims (16)

  1. 一种图像处理方法，其特征在于，所述方法包括：
    获取拍摄装置采集的第一图像；所述拍摄装置搭载在云台上，所述云台包括至少一个转动轴；
    获取所述拍摄装置采集的第二图像组，所述第二图像组包括至少一张第二图像，所述第二图像是在所述拍摄装置采集所述第一图像后，所述云台绕所述至少一个转动轴依次进行转动方向相反且转动角度不同的两次转动后采集的；
    根据所述两次转动的转动角度差值，所述第一图像与所述第二图像组构建第三图像，所述第三图像的分辨率分别高于所述第一图像和第二图像的分辨率。
  2. 根据权利要求1所述的方法，其特征在于，所述转动角度差值在预设的角度范围内。
  3. 根据权利要求2所述的方法，其特征在于，所述角度范围包括0.4至0.6倍的单位像素对应的视场角。
  4. 根据权利要求1所述的方法，其特征在于，所述转动角度差值小于0.1度。
  5. 根据权利要求1所述的方法，其特征在于，所述至少一个转动轴包括第一转动轴，所述云台绕所述至少一个转动轴依次进行转动方向相反且转动角度不同的两次转动，包括：
    所述云台绕所述第一转动轴进行转动方向相反且转动角度不同的两次偏航转动。
  6. 根据权利要求1所述的方法，其特征在于，所述至少一个转动轴包括第二转动轴，所述云台绕所述至少一个转动轴依次进行转动方向相反且转动角度不同的两次转动，包括：
    所述云台绕所述第二转动轴进行转动方向相反且转动角度不同的两次俯仰转动。
  7. 根据权利要求1所述的方法，其特征在于，所述第一图像的空间相位信息与第二图像的空间相位信息不同。
  8. 根据权利要求1所述方法，其特征在于，所述根据所述两次转动的转动角度差值，所述第一图像与所述第二图像组构建第三图像，包括：
    针对每张第二图像，根据所述转动角度差值，确定所述第一图像与所述第二图像的空间相位关系；
    基于所述空间相位关系，将所述第一图像的像素与所述第二图像的像素进行交错排列，构建所述第三图像。
  9. 根据权利要求8所述的方法，其特征在于，所述空间相位关系包括空间上的相对偏移方向和相对偏移量；所述根据所述转动角度差值，确定所述第一图像与所述第二图像的空间相位关系，包括：
    根据所述转动角度差值，确定所述云台绕所述至少一个转动轴的实际转动方向和实际转动角度；
    基于所述实际转动方向，确定所述相对偏移方向；
    基于所述实际转动角度，确定所述相对偏移量。
  10. 根据权利要求1所述的方法，其特征在于，所述第一图像与所述第二图像是所述拍摄装置至少在一类拍摄参数相同的情况下分别采集的图像。
  11. 根据权利要求10所述的方法，其特征在于，所述拍摄参数包括曝光参数。
  12. 一种图像处理装置，其特征在于，包括：
    处理器；
    用于存储处理器可执行指令的存储器；
    其中，所述处理器调用所述可执行指令时实现权利要求1-11任一所述方法的操作。
  13. 一种云台，搭载有拍摄装置，其特征在于，包括：
    至少一个转动轴；
    至少一个电机，用于驱动所述云台绕所述至少一个转动轴转动；
    处理器；
    用于存储处理器可执行指令的存储器；
    其中，所述处理器调用所述可执行指令时实现权利要求1-11任一所述方法的操作。
  14. 一种可移动平台，其特征在于，包括：
    机身；
    用于携带拍摄装置的云台，所述云台包括至少一个转动轴；
    动力组件，用于驱动所述可移动平台在空间中运动；
    处理器；
    用于存储处理器可执行指令的存储器；
    其中，所述处理器调用所述可执行指令时实现权利要求1-11任一所述方法的操作。
  15. 一种计算机程序产品，包括计算机程序，其特征在于，所述计算机程序被处理器执行时实现如权利要求1-11任一所述方法的步骤。
  16. 一种计算机可读存储介质，其特征在于，所述计算机可读存储介质上存储有若干计算机指令，所述计算机指令被执行时执行如权利要求1-11任一所述的方法。
PCT/CN2021/141316 2021-12-24 2021-12-24 一种图像处理方法、装置、云台、可移动平台 WO2023115562A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/CN2021/141316 WO2023115562A1 (zh) 2021-12-24 2021-12-24 一种图像处理方法、装置、云台、可移动平台

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2021/141316 WO2023115562A1 (zh) 2021-12-24 2021-12-24 一种图像处理方法、装置、云台、可移动平台

Publications (1)

Publication Number Publication Date
WO2023115562A1 true WO2023115562A1 (zh) 2023-06-29

Family

ID=86901116

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/141316 WO2023115562A1 (zh) 2021-12-24 2021-12-24 一种图像处理方法、装置、云台、可移动平台

Country Status (1)

Country Link
WO (1) WO2023115562A1 (zh)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106373087A (zh) * 2016-08-23 2017-02-01 大连理工大学 一种改进初始估计的图像超分辨率重建方法
US20180113462A1 (en) * 2016-10-22 2018-04-26 Gopro, Inc. Position-based soft stop for a 3-axis gimbal
CN109606718A (zh) * 2018-12-28 2019-04-12 成都纵横大鹏无人机科技有限公司 一种三轴稳定云台及无人机
CN110044262A (zh) * 2019-05-09 2019-07-23 哈尔滨理工大学 基于图像超分辨率重建的非接触式精密测量仪及测量方法
CN110113537A (zh) * 2019-06-12 2019-08-09 北京百度网讯科技有限公司 旋转拍摄系统、方法和装置

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106373087A (zh) * 2016-08-23 2017-02-01 大连理工大学 一种改进初始估计的图像超分辨率重建方法
US20180113462A1 (en) * 2016-10-22 2018-04-26 Gopro, Inc. Position-based soft stop for a 3-axis gimbal
CN109606718A (zh) * 2018-12-28 2019-04-12 成都纵横大鹏无人机科技有限公司 一种三轴稳定云台及无人机
CN110044262A (zh) * 2019-05-09 2019-07-23 哈尔滨理工大学 基于图像超分辨率重建的非接触式精密测量仪及测量方法
CN110113537A (zh) * 2019-06-12 2019-08-09 北京百度网讯科技有限公司 旋转拍摄系统、方法和装置

Similar Documents

Publication Publication Date Title
US20220092729A1 (en) Equatorial Stitching Of Hemispherical Images In A Spherical Image Capture System
JP5596972B2 (ja) 撮像装置の制御装置および制御方法
WO2019178872A1 (zh) 视频图像防抖方法和终端
WO2017020150A1 (zh) 一种图像处理方法、装置及摄像机
JP4914171B2 (ja) 撮像装置の制御方法及びカメラシステム
JP2018189730A (ja) 像ブレ補正装置及びその制御方法、プログラム、記憶媒体
CN102771121A (zh) 照相机平台系统
CN108520232A (zh) 立体环景影片产生方法及装置
CN105741233B (zh) 一种视频图像球面拼接方法及系统
JP6257439B2 (ja) 撮像装置及び撮像方法
US20060007344A1 (en) Variable speed, variable resolution digital cinema camera system
JP2019080226A (ja) 撮像装置、撮像装置の制御方法、及びプログラム
WO2021085246A1 (ja) 撮像支援装置、撮像支援システム、撮像システム、撮像支援方法、及びプログラム
WO2021085249A1 (ja) 撮像支援装置、撮像支援システム、撮像システム、撮像支援方法、及びプログラム
JP2008225522A (ja) 画像処理装置、カメラ装置、画像処理方法、およびプログラム
US20190214925A1 (en) Stabilizer Auto-Rotating Control Method
WO2023115562A1 (zh) 一种图像处理方法、装置、云台、可移动平台
JP2023052772A (ja) 撮像支援装置、撮像装置、撮像システム、撮像支援システム、撮像支援方法、及びプログラム
CN111862169A (zh) 目标跟拍方法、装置、云台相机及存储介质
JP7150456B2 (ja) 撮像システム、情報処理装置、情報処理装置の制御方法、及び、プログラム
CN112544065A (zh) 云台的控制方法和云台
JP4703504B2 (ja) 画像処理装置、画像処理方法、及びプログラム
JP2005244861A (ja) 撮像装置及び撮像系パラメータ校正方法
JP5987339B2 (ja) 撮像装置
JP2019080174A (ja) 画像処理装置、撮像システム、通信システム、画像処理方法、およびプログラム

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21968691

Country of ref document: EP

Kind code of ref document: A1