WO2019205103A1

WO2019205103A1 - Pan-tilt orientation correction method, pan-tilt orientation correction apparatus, pan-tilt, pan-tilt system, and unmanned aerial vehicle

Info

Publication number: WO2019205103A1
Application number: PCT/CN2018/084912
Authority: WO
Inventors: 伊星星; 许文; 李兵
Original assignee: 深圳市大疆创新科技有限公司
Priority date: 2018-04-27
Filing date: 2018-04-27
Publication date: 2019-10-31
Also published as: CN110382358A

Abstract

A pan-tilt orientation correction apparatus and a correction method therefor. An orientation of a pan-tilt (101) is corrected according to image data obtained by a photographing device (102) carried by the pan-tilt (101). The correction apparatus comprises a memory and a processor. The correction method comprises: obtaining a line sub-set of parallel lines in the image data (S3001); calculating an observation orientation of the pan-tilt (101) according to the line sub-set (S3002); and correcting a current orientation of the pan-tilt (101) according to the observation orientation (S3003). The correction apparatus and the correction method are applied to an unmanned aerial vehicle (100) and a pan-tilt system thereof.

Description

PTZ attitude correction method, pan/tilt attitude correction device, pan/tilt, pan/tilt system and drone

Technical field

Embodiments of the present invention relate to the field of cloud platforms, and in particular, to a pan-tilt attitude correction method, a pan-tilt attitude correction device, a pan/tilt head, a pan-tilt system, and a drone.

Background technique

At present, the PTZ is often used as a camera for stabilizing cameras on mobile devices such as drones and handheld poles. Take the common three-axis pan/tilt as an example. It can realize three directions by the motor set on the Yaw axis (yaw axis), Pitch axis (pitch axis) and Roll axis (roller axis). Actively stabilized, and an attitude sensor is provided on a load device (such as a camera mounted on a gimbal) to acquire the attitude of the load device in the Yaw, Pitch, and Roll directions in real time. When PTZ is pursuing miniaturization and is applied to price-sensitive consumer products, the attitude sensor often used is a MEMS inertial device. MEMS inertial devices generally have a large drift rate due to their implementation, which causes the sensing results to drift, and the amount of drift will accumulate over time. Therefore, when the time is a little longer, the measurement of the attitude sensing system will produce a drift amount, especially the drift amount of the roll axis. Since the deflection in the roll direction will cause the user to feel the overall rotation of the screen, the embodiment is more obvious. Affect the user experience.

Summary of the invention

The embodiment of the invention provides a PTZ attitude correction method, a PTZ attitude correction device, a PTZ, a PTZ system and a UAV, which can correct the attitude error caused by the drift of the PTZ attitude sensor, and can improve the user experience.

According to a first aspect of the present invention, a PTZ attitude correction method is provided, which corrects a posture of a PTZ according to image data acquired by a photographing device mounted on the PTZ, and includes: acquiring the image data a line sub-set of straight lines parallel to each other; calculating an observation posture of the pan/tilt according to the line sub-set; and correcting a current posture of the gimbal according to the observation posture.

According to a second aspect of the present invention, a pan/tilt attitude correction device is provided, which corrects a posture of a gimbal according to image data acquired by a photographing device mounted on the gimbal, and is characterized in that: a memory is stored for controlling the cloud An instruction set of the posture correction device; and one or more processors configured to perform processing of acquiring a line sub-set of mutually parallel straight lines in the image data according to the instruction set stored in the memory; The linear sub-set calculates the observation posture of the pan-tilt; and corrects the current posture of the gimbal according to the observation posture.

According to a third aspect of the present invention, a cloud platform comprising the above-described pan/tilt attitude correction device is provided.

According to a fourth aspect of the present invention, a cloud platform system comprising the above-described pan/tilt head and the photographing device mounted on the pan/tilt head is provided.

According to a fifth aspect of the invention, there is provided a drone equipped with the above-described pan/tilt head system.

DRAWINGS

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly described below. It is obvious that the drawings in the following description are some embodiments of the present invention. One of ordinary skill in the art can also obtain other drawings based on these drawings without paying for inventive labor.

FIG. 1 is a diagram showing an example of an application scenario of a pan/tilt provided by an embodiment of the present invention.

FIG. 2 is a diagram for explaining the concept of a subtraction point.

Fig. 3 is a flow chart showing a first embodiment of a pan/tilt attitude correcting method according to an embodiment of the present invention.

Fig. 4 is a flow chart showing a second embodiment of the pan/tilt attitude correcting method according to the embodiment of the present invention.

FIG. 5 is a flowchart for explaining step S4002.

FIG. 6 is a schematic diagram for explaining the calculation in step S4002.

Fig. 7 is a flowchart showing a step S5002 in the third embodiment of the pan/tilt attitude correcting method according to the embodiment of the present invention.

detailed description

The technical solutions in the embodiments of the present invention will be clearly described with reference to the accompanying drawings in the embodiments of the present invention. It is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative efforts are within the scope of the present invention.

It should be noted that when a component is referred to as being "fixed" to another component, it can be directly on the other component or the component can be present. When a component is considered to "connect" another component, it can be directly connected to another component or possibly a central component.

All technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs, unless otherwise defined. The terminology used in the description of the present invention is for the purpose of describing particular embodiments and is not intended to limit the invention. The term "and/or" used herein includes any and all combinations of one or more of the associated listed items.

Some embodiments of the present invention are described in detail below with reference to the accompanying drawings. The features of the embodiments and examples described below can be combined with each other without conflict.

In order to facilitate the understanding of the present invention, the main idea of the present invention and some background technologies involved in the solution are first described.

The main technical idea of the present invention is to correct the posture of the gimbal by using an image formed by a parallel line in space in a camera image (for example, an image acquired by a main camera provided on a gimbal). Specifically, by introducing information of a main camera (a shooting device mounted on a gimbal, or a shooting device integrated in a gimbal), an image erasing point feature is used to obtain a roll axis and a pitch axis angle of the gimbal, or a vertical The linear feature acquires the roll axis angle of the gimbal, thereby introducing new observation attitude information, and integrating with the current attitude information of the gimbal itself (which may be the posture information calculated by the IMU data set on the gimbal) to reduce the cloud The drift of the roll axis and the pitch axis improves the accuracy of the attitude control of the gimbal and improves the user experience. In some embodiments, the present invention will utilize a shadow point corresponding to the image formed by the parallel lines in the image in the image to correct the pan/tilt pose; in other embodiments, the present invention will directly utilize parallel in space. The image formed by the straight line is used to correct the pan-tilt attitude without going through the elimination point calculation process; in still other embodiments, the present invention combines the above two correction modes.

First, the application scenario of the pan/tilt provided by the embodiment of the present invention is described.

FIG. 1 is a diagram showing an example of an application scenario of a pan/tilt provided by some embodiments of the present invention. The drone 100 can be, for example, a multi-axis UAV, a fixed-wing UAV, or a vertical take-off drone. The multi-axis UAV is taken as an example, including the PTZ 101 and Shooting device 102. The drone 100 flies autonomously or under the control of a control terminal (not shown), and the subject 110 is photographed by the photographing device 102 mounted on the drone 100. The pan/tilt 101 is connected to the drone 100 and is used to mount the photographing device 102. Here, for the sake of easy understanding, the object 110 is depicted in the figure as an object including a parallel straight line such as a road, but the present invention is not limited thereto.

In other embodiments of the present invention, the pan/tilt head 101 and the photographing device 102 may not be mounted on the drone 100, but may be used separately, such as a handheld pan/tilt or a pan/tilt head (not shown). The present invention does not limit the specific application scenarios of the pan/tilt head 101 and the photographing device 102.

It can be understood that the pan/tilt head 101 is equipped with the photographing device 102, and may include a case where the photographing device 102 is integrated or detachably connected to the pan/tilt head 101 (for example, a pan-tilt camera with an interchangeable lens), and may also include the photographing device 102. In the case of the Yuntai 101 and the communication connection with the PTZ 101 (for example, a mobile phone PTZ), there is no limitation here. The pan/tilt has a plurality of rotatable shafts that can be rotated in various directions to adjust the photographing direction of the photographing device 102. In the embodiment of the present invention, the pan/tilt head includes at least a pitch axis and a roll axis, and the pan/tilt head can be moved in the pitch direction and the roll direction. The observation posture of the pan/tilt head 101 includes at least the pitch angle and the roll angle of the pan-tilt head 101.

The imaging device 102 is mounted on the pan/tilt head 101 for imaging a target. The photographing apparatus 102 may be various types of photographing apparatuses such as a camera that can take a high-definition image, a camera that shoots a continuous video, a night vision that photographs an infrared image, and the like.

Hereinafter, embodiments of the present invention will be specifically described by taking three embodiments as examples.

First embodiment

Hereinafter, a first embodiment in which the roll axis and the pitch axis angle of the pan/tilt are acquired using the image erasing point feature will be described.

FIG. 2 is a diagram for explaining the concept of a subtraction point. In the perspective projection process of camera imaging, parallel lines in three-dimensional space (such as mutually parallel rails, road edges, or building edge lines, etc.) intersect at a point in the camera's imaging picture, which is the elimination point. Two shading points v ₁ and v ₂ are taken as an example in FIG. ₂ . In perspective projection, the position of the shadow point depends on the direction of the parallel line in the three-dimensional space and the imaging plane. Therefore, according to the change of the relative positional relationship between the camera and the parallel lines in the three-dimensional space, the position of the shadow point in the imaging plane will also follow. Change.

Fig. 3 is a flow chart showing a first embodiment of a pan/tilt attitude correcting method according to an embodiment of the present invention. The pan/tilt attitude correction method of the first embodiment can be implemented by a controller mounted on the pan-tilt head 101, a controller mounted on the imaging device 102, a controller mounted on the drone 100, or a control terminal (not shown). The controller performs a program describing the method stored in a memory that it can access, thereby correcting the attitude of the gimbal. The pan/tilt attitude correction method according to the embodiment of the present invention corrects the posture of the gimbal according to the image data acquired by the imaging device 102 mounted on the pan-tilt 101, and includes the following steps:

In step S3001, a linear subset of straight lines parallel to each other in the image data is acquired;

In step S3002, an observation posture of the pan/tilt is calculated according to the linear subset;

In step S3003, the current posture of the pan/tilt is corrected according to the observed posture.

Here, in step S3001, straight lines parallel to each other refer to mutually parallel straight lines in a three-dimensional space, or objects to be photographed, such as rails, road edges, or building edge lines which are parallel to each other. And the linear subset of the mutually parallel straight lines, where the linear subset refers to the straight line corresponding to the mutually parallel straight lines in the imaged image, that is, the linear subset is a set of straight lines in the image data. To obtain a linear subset of mutually parallel straight lines in the image data, any algorithm that is existing or can perform similar functions can be used. Specifically, a set of straight lines in the image data may be first acquired, and then a subset of straight lines of the mutually parallel straight lines may be extracted from the set of straight lines; and the mutually parallel straight lines are extracted from the set of straight lines When the linear sub-sets are as described above, since the mutually parallel straight lines intersect in the image in the image, the parallel lines can be filtered according to the feature, that is, a set of straight lines intersecting at the same point can be It is considered to be parallel lines, and the specific calculation process will be further elaborated in the calculation process of the elimination point described later.

Wherein, taking a set of straight lines in the image data, any algorithm existing or capable of implementing similar functions may be used. As a specific example, an edge of the subject is detected by performing an edge operator operation on the image data, and a straight line fit is performed on the edge to obtain a line set in the image data, where image data processing is performed. Edge operators include, but are not limited to, Sobel operators, Roberts operators, and Prewitt operators.

As a specific example of straight line fitting the edge, the edge can be straight-line fitted by Hough transform.

In addition, the straight line subset of straight lines that are parallel to each other is extracted from the set of straight lines, and any algorithm existing or capable of performing similar functions may be used. As a specific example, a line sub-set of vertical lines parallel to each other is extracted from the set of straight lines according to a vertical line feature.

Here, as the vertical line feature, any geometric feature or mathematical feature capable of indicating that the straight line is a vertical line can be used. As an example, the vertical line feature includes a slope and a direction corresponding to a straight line in the image data corresponding to a vertical line in a three-dimensional space (such as a vertical edge of a building, etc.); specifically, when image data is processed When the edge line is fitted and the slope and direction of the edge line are calculated and the difference between the preset slope and the direction is within a certain threshold or a preset threshold, the edge line is considered to be a vertical line; that is, a vertical line Features include a certain or preset linear slope and range of directions. It should be noted that the edge line is considered to be a vertical line, and does not mean that the edge in the three-dimensional space corresponding to the edge line is vertical. For example, the threshold value may be set to be 20° from the vertical direction on the image data, so that a straight line which is considered to be offset by 20° from the vertical direction in the calculation process corresponds to a vertical line. The threshold here may also be other values, which are not limited herein, and are used to exclude those lines in the image data that are considered to be clearly not in a vertical line.

In step S3002, the observation pose of the pan/tilt is calculated according to the linear subset, and any algorithm existing or capable of implementing similar functions may be used. Here, as a specific example, the solution adopted by the embodiment of the present invention is: calculating a position of a shadow point of the image data according to the line subset, and calculating an observation posture of the cloud platform according to the position of the shadow point . Here, the observation posture of the gimbal refers to a posture calculated by the gimbal through the foregoing steps, that is, a posture of the gimbal calculated by processing the image captured by the main camera, which may be the same as the current posture of the gimbal. , can also be different. The current posture of the gimbal here refers to the attitude calculated by the gimbal through the data acquired by the attitude measuring sensor such as its own IMU.

In step S3003, the current posture of the pan/tilt is corrected according to the observed posture, and any correction algorithm existing or capable of implementing similar functions may be used. The solution adopted by the embodiment of the present invention is to correct the posture of the gimbal according to the attitude fusion between the observation posture and the current posture. The specific attitude fusion may adopt a Kalman filtering algorithm, but is not limited. Here, the current posture of the gimbal refers to a posture obtained by the PTZ currently calculated by the data acquired by the sensor (such as an IMU, a magnetic code meter, etc.) provided by the gimbal, and the observation posture of the gimbal may be the same as that of the gimbal. , can also be different.

In the embodiment of the present invention, the position of the erasing point of the image data is calculated by a Random Sample Concensus (RANSAC algorithm).

Specifically, the calculation process can be expressed as follows.

First, any straight line on the plane coordinate system can be represented by a straight line equation. In the present embodiment, for the aforementioned straight line that has been fitted in the image captured by the main camera, and filtered by the vertical line feature, it is considered to be vertical. A group of straight lines of lines can be considered as follows:

a _i *x+b _i *y+c _i =0, (i=1, 2...) (Equation 1)

Where i denotes a different straight line that is fitted, and a _i , b _i , c _i are known to be straight after fitting in the aforementioned steps.

The calculation model of the elimination point in the embodiment of the present invention is:

a _i *x ₀ +b _i *y ₀ +c _i =0 (Equation 2)

In a straight line in which image data has been fitted, a series of straight line subsets of vertical lines are extracted. The line sub-set includes a series of vertical lines corresponding to a series of a _i , b _i , c _i . At this time, x ₀ , y ₀ can be calculated by the RANSAC algorithm according to the series a _i , b _i , c _i .

Specifically, the inverse error can be introduced into the formula by the subtraction point to estimate the error size. When the error variance is less than a certain threshold, it is determined that (x ₀ , y ₀ ) is the elimination point of the parallel line of the group, and the equation (3) is used. Equations (4) and (5) calculate the observation attitude of the gimbal. Specifically, the angle of the gimbal in the roll direction, that is, the roll angle and the angle in the pitch direction, that is, the pitch angle are calculated.

In photographic geometry, the coordinates on the image plane are usually represented by homogeneous coordinates, so the elimination point vector is set to v=(x ₀ , y ₀ , 1) ^T , where (x ₀ , y ₀ ) is The coordinates of the shadow point.

Here, the intermediate vector R _{d is} calculated by Equation 3.

Where v is the aforementioned elimination point vector, K is the camera matrix, which can be regarded as the parameter related to the main camera itself, and R _d is the unit vector of the linear direction pointing to the elimination point, which is the calculated intermediate vector to facilitate subsequent Calculation.

The calculated R _{d is} substituted into Equations 4 and 5 below, and the angle in the roll direction, that is, the roll angle θ _roll and the angle in the pitch direction, that is, the pitch angle θ _pitch are calculated.

θ _pitch =arcsin(-R _d3 ) (Equation 5)

Where R _d =(R _d1 , R _d2 , R _d3 ,1) ^T , where R _d is also represented by homogeneous coordinates, and R _d1 , R _d2 , R _d3 represent R _d in three directions of the image coordinate system. Component.

So far, the present embodiment completes the process of calculating the attitude information of the pan tilt angle and the roll angle by the main camera image. It should be noted that since in this process, the preliminary edge fitting line is screened by the vertical line feature, there is a preset threshold; therefore, due to the setting of the preset threshold, it may result in screening out some The three-dimensional space does not belong to the vertical line, but it is considered to be a vertical line in the image data. In this case, the observation pose obtained by the calculation of the considered vertical line may have a large error. .

Therefore, in this embodiment, after the observation pose is calculated, it is compared with the current pose, and the difference between the observed pose and the current pose is calculated. If the difference between the two is within the allowable range, it is considered that this time The calculation of the observed pose is effective, and then the observed pose is merged with the current pose to correct the current attitude of the gimbal; if the difference between the two exceeds the allowable range, the calculation of the observed pose is considered invalid. Then, returning to step S3001, the calculation scheme of the present embodiment is re-executed. It can be understood that the range allowed here is preset, for example, the difference between the two is 5% or other values, and is not limited herein.

According to the solution of the embodiment, by introducing the information of the main camera (the photographing device 102), the observation posture is calculated using the image erasing point feature and the pan/tilt posture is corrected, which is equivalent to the attitude acquired by the attitude measuring sensor of the gimbal itself. In addition, an observation posture is obtained, which can reduce the drift of the roll direction and the pitch direction of the gimbal and improve the user experience.

Second embodiment

Hereinafter, a second embodiment in which the observation posture information of the roll axis angle of the pan/tilt is acquired using a vertical line will be described.

Fig. 4 is a flow chart showing a second embodiment of the pan/tilt attitude correcting method according to the embodiment of the present invention. The pan/tilt attitude correction method according to the second embodiment can be a controller mounted on the pan/tilt head 101, a controller mounted on the imaging device 102, a controller mounted on the drone 100, or a not-shown, similar to the first embodiment. The controller in the control terminal or the like executes a program describing the method stored in a memory that it can access, thereby correcting the posture of the gimbal. The pan/tilt attitude correction method according to the embodiment of the present invention corrects the posture of the pan-tilt based on the image data acquired by the photographing device 102 mounted on the pan-tilt 101, and includes the following steps:

In step S4001, a linear subset of straight lines parallel to each other in the image data is acquired. Here, it can be implemented in the same manner as step S3001 of the first embodiment, and details are not described herein again.

In step S4002, the observation posture of the pan/tilt is calculated based on the linear subset. Specifically, the roll angle is calculated based on a straight line subset of the vertical line and a vertical direction of the image data.

In step S4003, the current posture of the pan/tilt is corrected according to the observed posture. Here, it can be implemented in the same manner as step S3003 of the first embodiment, and details are not described herein again.

That is, the pan/tilt attitude correction method of the second embodiment is different from the first embodiment in that the observation posture of the pan/tilt is calculated using step S4002 different from step S3002 of the first embodiment.

Specifically, in the embodiment, step S4002 can be divided into three sub-steps S40021 to S40023. FIG. 5 is a flowchart for explaining step S4002.

Step S40021: It is determined that the current pan/tilt is in a horizontal state in the pitch direction.

In this step, the pan/tilt is determined to be in a horizontal state in the pitch direction by the current attitude of the gimbal. As a specific example, the judgment may be made by the pitch angle information in the current posture of the gimbal, using any means existing or capable of performing similar functions. For example, the controller mounted on the pan/tilt head 101, the controller mounted on the imaging device 102, the controller mounted on the drone 100, or a control terminal (not shown) can acquire the servo angle in the pitch direction, at least satisfying the above. The pan/tilt is determined to be in a horizontal state in the pitch direction when the difference between the servo angle of the pitch direction and the predetermined angle is less than a predetermined threshold and the difference between the pitch angle calculated by the previous predetermined time and the predetermined angle is less than a predetermined threshold.

Specifically, when the pitch angle of the current attitude of the pan/tilt deviates from 0° within a certain range (for example, the pitch angle is less than or equal to 2° as a predetermined threshold), it is determined that the current pan/tilt is in a horizontal state in the pitch direction. It should be understood that the pan/tilt is horizontal in the pitch direction, which means that the attitude information in the pitch direction of the pitch is horizontal, so that the following calculation steps can be performed, which is not actually horizontal, for example, the pan/tilt still has 1 The pitch angle of °, but still think that the gimbal is in the horizontal direction in the pitch direction; at the same time, the pan/tilt is horizontal in the pitch direction regardless of the angle in the roll direction of the gimbal, for example, the pan/tilt is tilted at 1°. Angle, but when shooting at a roll angle of 20° (ie “歪 neck”), the pan/tilt is also considered to be horizontal in the pitch direction. Further, the predetermined threshold may be a value that is set autonomously according to the user's needs, or a value that is set in advance in conjunction with control accuracy or the like.

Step S40022: Acquire a vertical line set in the line subset.

In this step, the method for obtaining the vertical line set in the line subset is the same as that in the previous embodiment, and details are not described herein again.

Step S40023: Calculate an observation posture of the pan/tilt according to the vertical line set.

In this step, reference may be made to FIG. 6, which is a schematic diagram for explaining the calculation in step S40023. Among them, the solid line frame indicates the image data observed in the current posture, indicating that there are several buildings in which a vertical line can be provided. Here, it is assumed that the horizontal direction in the coordinate system of the image data is the x-axis and the vertical direction corresponds to the y-axis. The dashed box indicates the range of image data in the horizontal pose. a, b, and c respectively represent the vertical lines extracted from the image, and θ _roll represents the roll angle of the pan/tilt in the roll direction. Here, it is assumed that the average direction vector of the linear sub-set of the vertical straight line on the x-axis and the y-axis is

Since the vertical straight line is actually a relationship perpendicular to the horizontal direction, the direction of the straight line sub-set of the vertical line should be equal to the vertical direction of the range of the image data (dashed frame) in the horizontal posture.

In step S4002, rapid integration is performed by the slope of the linear slope in the vicinity of infinity, and the roll angle θ _{roll in the} roll direction is calculated by the following Expression 6. That is, the roll angle θ _{roll in the} roll direction is:

In the present embodiment, since it is determined that the pan/tilt is in a horizontal state in the pitch direction, the slope of the vertical line in the image data represents the roll direction posture information of the pan/tilt, which can also be visually displayed from FIG. In this embodiment, by performing a different calculation process on the special case where the pan/tilt pitch angle is near 0°, the processing of the observation posture information can be simplified, thereby improving the efficiency of the attitude correction of the gimbal.

According to the solution of the present embodiment, by introducing the information of the main camera (the photographing device 102), when the pan-tilt tilt angle is in the vicinity of 0°, the roll direction is calculated only by calculating the angle between the vertical lines in the image. The roll angle allows the current observation pose to be calculated with a low amount of calculation.

Third embodiment

Hereinafter, a third embodiment in which the roll axis and the pitch axis angle of the pan/tilt head are acquired using, for example, the image erasing point feature, and the roll axis angle of the pan/tilt head is acquired using the vertical line feature will be described.

The third embodiment is different from the second embodiment in that step S5002 is used instead of step S4002 in the second embodiment.

FIG. 7 is a flow chart showing a detailed description of step S5002. Step S5002 includes the following steps:

In step S50021, it is determined that the pan/tilt is in a horizontal state in the pitch direction. In the case where the pan/tilt is in the horizontal state in the pitch direction, the process proceeds to step S50023, and otherwise proceeds to step S50022.

In step S50022, the observation posture of the gimbal is calculated based on the position of the erasing point. Here, the same method as in the step S3002 in the first embodiment may be adopted, that is, the position of the erasing point of the image data is calculated according to the linear subset, and the observation of the gimbal is calculated according to the position of the erasing point. Gesture, no longer repeat here.

In step S50023, the roll angle in the roll direction is calculated from the straight line subset of the vertical line and the vertical direction of the image data. Here, the same method as in the step S4002 in the second embodiment may be adopted, that is, the roll angle of the roll direction is calculated from the straight line subset of the vertical line and the vertical direction of the image data, This will not be repeated here.

Here, in step S50021, it is determined that the current pan/tilt is in a horizontal state in the pitch direction by using the same manner as step S40021 described above.

In addition, although the same method as that in step S3002 in the first embodiment is taken as an example in step S50022, the present embodiment is not limited thereto. Other methods of obtaining the observation posture of the pan/tilt including the roll direction of the roll direction and the pitch angle of the pitch direction may be used instead.

According to the solution of the present embodiment, by introducing the information of the main camera (the photographing device 102), by first prejudge the current pitch direction posture of the gimbal, only the image is calculated by determining that the pan tilt is in the horizontal state in the pitch direction. The angle of the vertical line is calculated as the roll angle in the roll direction. Otherwise, for example, the observation attitude of the pan/tilt is calculated according to the position of the shadow point, so that the current calculation can be calculated with good calculation and calculation accuracy. Observed posture.

Although the pitch direction and the roll direction are similarly mentioned in the above-described pan/tilt attitude correction method, the present invention is not limited thereto, and the pan/tilt head may be capable of supporting the photographing apparatus in the pitch direction, the roll direction, and the offset direction. Freely rotate in the direction of navigation.

In addition, in the solution of each of the above embodiments, when acquiring a linear subset of mutually parallel straight lines in the image data, a new one may be acquired when the straight line in the linear subset is less than a predetermined number. Image data and re-extraction of the line sub-collection.

Further, in the solution of each of the above embodiments, the image data can be continuously acquired in time series, thereby continuously calculating the observation posture of the pan/tilt.

Further, although the present invention has been described by taking an unmanned aerial vehicle as an example, the apparatus for mounting the pan/tilt is not limited thereto, and any device capable of carrying a pan/tilt, such as a hand-held vehicle, an automobile, or a wearable device, may be used.

In the several embodiments provided by the present invention, it should be understood that the disclosed apparatus and method may be implemented in other manners. For example, the device embodiments described above are merely illustrative. For example, the division of the unit is only a logical function division. In actual implementation, there may be another division manner, for example, multiple units or components may be combined or Can be integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interface, device or unit, and may be in an electrical, mechanical or other form.

The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of hardware plus software functional units.

The above-described integrated unit implemented in the form of a software functional unit can be stored in a computer readable storage medium. The above software functional unit is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor to perform the methods of the various embodiments of the present invention. Part of the steps. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, and the like, which can store program codes. .

A person skilled in the art can clearly understand that for the convenience and brevity of the description, only the division of each functional module described above is exemplified. In practical applications, the above function assignment can be completed by different functional modules as needed, that is, the device is installed. The internal structure is divided into different functional modules to perform all or part of the functions described above. For the specific working process of the device described above, refer to the corresponding process in the foregoing method embodiment, and details are not described herein again.

Finally, it should be noted that the above embodiments are merely illustrative of the technical solutions of the present invention, and are not intended to be limiting; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art will understand that The technical solutions described in the foregoing embodiments may be modified, or some or all of the technical features may be equivalently replaced; and the modifications or substitutions do not deviate from the technical solutions of the embodiments of the present invention. range.

Claims

A pan/tilt attitude correction method, which corrects a posture of a gimbal according to image data acquired by a photographing device mounted on the gimbal, and is characterized by comprising:

Obtaining a subset of lines of mutually parallel lines in the image data;

Calculating an observation posture of the pan/tilt according to the linear subset;

Correcting the current posture of the gimbal according to the observation posture.
The pan/tilt attitude correction method according to claim 1, wherein the acquiring a subset of straight lines of mutually parallel straight lines in the image data comprises:

Obtaining a set of straight lines in the image data;

A linear subset of the mutually parallel straight lines is extracted from the set of straight lines.
The pan/tilt attitude correction method according to claim 2, wherein the acquiring a set of straight lines in the image data comprises:

An edge of the subject is detected by performing an edge operator operation on the image data, and the edge is straight-line fitted to acquire a line set in the image data.
The pan/tilt attitude correction method according to claim 3, wherein the straight line fitting of the edge comprises:

The edges are fitted by a Hough transform.
The pan/tilt attitude correction method according to claim 2, wherein the extracting a line sub-set of mutually parallel straight lines from the set of straight lines comprises:

A line subset of vertical lines parallel to each other is extracted from the set of straight lines according to a vertical line feature.
The pan/tilt attitude correction method according to claim 1, wherein the extracting a line sub-set of mutually parallel straight lines from the set of straight lines comprises:

A line subset of vertical lines parallel to each other is extracted from the set of straight lines according to a vertical line feature.
The pan/tilt attitude correction method according to claim 5 or 6, wherein

The vertical line features include a slope and a direction.
The pan/tilt attitude correction method according to claim 7, wherein

The pan/tilt includes a pitch axis and a roll axis to move the pan/tilt in a pitch direction and a roll direction,

The observation posture of the pan/tilt includes a pitch angle of the pitch direction and a roll angle of the roll direction.
The pan/tilt posture correction method according to claim 8, wherein the calculating the observation posture of the gimbal comprises:

Determining that the gimbal is in a horizontal state in the pitch direction;

The roll angle is calculated from a straight line subset of the vertical line and a vertical direction of the image data.
The pan/tilt attitude correction method according to claim 9, wherein the determining that the gimbal is in a horizontal state in the pitch direction comprises:

Obtaining a pitch angle of the current pose,

When the pitch angle of the current posture deviates from 0° within a predetermined threshold, it is determined that the current pan/tilt is in a horizontal state in the pitch direction.
The pan/tilt posture correction method according to claim 1, wherein the calculating the observation posture of the pan/tilt according to the linear subset comprises:

Calculating a position of a shadow point of the image data according to the linear subset;

The observed attitude is calculated based on the position of the shadow point.
The pan-tilt attitude correction method according to claim 11, wherein calculating the position of the image-shadowing point of the image data according to the linear subset comprises:

A vector of the image point of the image data is calculated by the RANSAC algorithm.
The pan/tilt attitude correction method according to claim 1, wherein

The image data is acquired in time series, and the observed posture is continuously calculated.
The pan/tilt attitude correction method according to claim 1, wherein

When the straight line in the linear subset is less than the prescribed number, new image data is acquired and the extraction of the linear subset is performed again.
The pan/tilt posture correction method according to claim 1, wherein the correcting the current posture of the pan/tilt according to the observation posture comprises:

The posture of the observation posture is merged with the current posture to correct the posture of the gimbal.
The pan/tilt attitude correction method according to claim 15, wherein

Calculating the difference between the observed pose and the current pose,

If the difference between the observed posture and the current posture does not exceed the allowable range, the observed posture and the current posture are merged to correct the posture of the gimbal; or

If the difference between the observed posture and the current posture exceeds an allowable range, reacquiring a linear subset of mutually parallel straight lines in the image data, and calculating the observation of the gimbal according to the linear subset attitude.
The pan/tilt attitude correction method according to claim 1, wherein

The pan/tilt can freely rotate the mounted photographing device in the pitch direction, the roll direction, and the yaw direction.
The pan/tilt attitude correction method according to claim 1, wherein

The current posture of the gimbal is a posture calculated by the data acquired by the attitude measuring sensor of the gimbal.
A pan/tilt attitude correction device that corrects a posture of a gimbal according to image data acquired by a photographing device mounted on the gimbal, and is characterized by comprising:

a memory that stores an instruction set for controlling the pan/tilt attitude correction device; and

One or more processors configured to perform the following processing in accordance with the set of instructions stored by the memory:

Obtaining a subset of lines of mutually parallel lines in the image data;

Calculating an observation posture of the pan/tilt according to the linear subset;

Correcting the current posture of the gimbal according to the observation posture.
The pan/tilt attitude correction device according to claim 19, wherein the acquiring a subset of straight lines of mutually parallel straight lines in the image data comprises:

Obtaining a set of straight lines in the image data;

A linear subset of the mutually parallel straight lines is extracted from the set of straight lines.
The pan/tilt attitude correction device according to claim 20, wherein the acquiring a set of straight lines in the image data comprises:

An edge of the subject is detected by performing an edge operator operation on the image data, and the edge is straight-line fitted to acquire a line set in the image data.
The pan/tilt posture correction device according to claim 21, wherein the straight line fitting of the edge comprises:

The edges are fitted by a Hough transform.
The pan/tilt attitude correction device according to claim 20, wherein the extracting a line subset of straight lines that are parallel to each other from the set of straight lines comprises:

A line subset of vertical lines parallel to each other is extracted from the set of straight lines according to a vertical line feature.
The pan/tilt attitude correction device according to claim 20, wherein the extracting a line subset of straight lines that are parallel to each other from the set of straight lines comprises:

A line subset of vertical lines parallel to each other is extracted from the set of straight lines according to a vertical line feature.
A pan/tilt posture correcting device according to claim 23 or 24, wherein

The vertical line features include a slope and a direction.
The pan/tilt posture correcting device according to claim 25, characterized in that

The pan/tilt head includes a pitch axis and a roll axis to move the pan/tilt head in a pitch direction and a roll direction,

The observation posture of the pan/tilt includes a pitch angle of the pitch direction and a roll angle of the roll direction.
The pan/tilt posture correction device according to claim 22, wherein the calculating the observation posture of the gimbal comprises:

Determining that the gimbal is in a horizontal state in the pitch direction;

The roll angle is calculated from a straight line subset of the vertical line and a vertical direction of the image data.
The pan/tilt attitude correction device according to claim 27, wherein the determining that the gimbal is in a horizontal state in the pitch direction comprises:

Obtaining a pitch angle of the current pose,

When the pitch angle of the current posture deviates from 0° within a predetermined threshold, it is determined that the current pan/tilt is in a horizontal state in the pitch direction.
The pan/tilt posture correction device according to claim 19, wherein the calculating the observation posture of the pan/tilt according to the linear subset comprises:

Calculating a position of a shadow point of the image data according to the linear subset;

The observation attitude of the gimbal is calculated according to the position of the elimination point.
The pan/tilt attitude correction device according to claim 29, wherein calculating the position of the image-shadowing point of the image data according to the linear subset comprises:

A vector of the image point of the image data is calculated by the RANSAC algorithm.
The pan/tilt posture correcting device according to claim 19, wherein

The image data is acquired in time series, and the observation posture of the gimbal is continuously calculated.
The pan/tilt posture correcting device according to claim 19, wherein

When the straight line in the linear subset is less than the prescribed number, new image data is acquired and the extraction of the linear subset is performed again.
The pan/tilt posture correction device according to claim 19, wherein the correcting the current posture of the pan/tilt according to the observation posture comprises:

The posture of the gimbal is corrected by performing posture fusion based on the observed posture and the current posture.
The pan/tilt posture correcting device according to claim 33, wherein

Calculating the difference between the observed pose and the current pose,

If the difference between the observed posture and the current posture does not exceed the allowable range, the observed posture and the current posture are merged to correct the posture of the gimbal; or

If the difference between the observed posture and the current posture exceeds an allowable range, reacquiring a linear subset of mutually parallel straight lines in the image data, and calculating the observation of the gimbal according to the linear subset attitude.
The pan/tilt posture correcting device according to claim 19, wherein

The pan/tilt can freely rotate the mounted photographing device on the pitch axis, the roll axis, and the yaw axis.
The pan/tilt posture correcting device according to claim 19, wherein

The current posture of the gimbal is a posture calculated by the data acquired by the attitude measuring sensor of the gimbal.
A pan/tilt head, characterized in that the pan/tilt head comprises the pan/tilt attitude correction device according to any one of claims 19 to 36, and the pan/tilt head further comprises at least a pitch axis and a roll axis.
A pan/tilt system, comprising: the pan/tilt head according to claim 37; and the photographing device mounted on the pan/tilt head.
A drone, characterized in that the drone is equipped with the pan/tilt head system according to claim 38.