CN105551032B - The shaft tower image capturing system and its method of a kind of view-based access control model servo - Google Patents

Abstract

The invention discloses a tower image acquisition system based on visual servoing and a method thereof. The method includes step (1): according to inspection requirements, a camera on a platform is used to obtain video information of a tower and a frame of real-time image is intercepted from the video; Step (2): Identify and locate the position of the tower in the intercepted real-time image; Step (3): Determine whether the tower is located in the central position of the intercepted real-time image, if so, proceed to step (5); otherwise, determine the position of the tower relative to The deviation of the central position of the real-time image intercepted; Step (4): determine the direction of rotation of the cloud platform by the position of the pole tower in the image, then determine the amount of rotation of the platform according to the deviation obtained in step (3); after adjusting the platform, Get the real-time image of the tower at the current position again, and turn to step (2); step (5): start from a certain corner of the tower, adjust the position of the pan-tilt with an "S"-shaped path, and use the camera on the pan-tilt to collect Tower image.

Description

A tower image acquisition system and method based on visual servoing

technical field

The invention relates to the field of digital image processing, in particular to a visual servo-based tower image acquisition system and a method thereof.

Background technique

The tower is an important part of the overhead transmission line. Its function is to support the overhead line conductor and the overhead ground wire, and to make the connection between the conductor and the conductor, between the conductor and the overhead ground wire, between the conductor and the tower, and between the conductor and the ground and the cross There is sufficient safety distance between jumping objects. Due to the long-term exposure of the tower to the natural environment and the influence of natural or human factors, there are problems such as loosening and falling off of the screws, skewing of the tower, and exposed tower base. If these problematic towers are not regularly inspected and overhauled, it will affect the normal operation of the entire power grid and may even cause major safety accidents.

By carrying visible light and infrared equipment on the UAV to inspect the tower and collect image information, the problems existing in the tower can be found in time and the shortcomings of manual inspection can be effectively made up for. However, the current method of collecting image information of towers based on drones is that the operator observes the video information returned by the ground station and adjusts the on-board pan/tilt to complete the information collection of the tower. This requires the operator to observe the feedback information of the ground station in real time. Stronger. How to use tower recognition technology and visual servo technology to automatically complete tower information collection is particularly important.

Among the existing visual servo-based systems, the vision-servo-based UAV inspection platform control for transmission lines proposed by Shandong Electric Power Research Institute (Patent No.: CN 1029292880A) is designed to use visual servo technology to adjust the platform to realize transmission line inspection. tracking, but this patent only uses visual servo technology to realize the line tracking function, and does not describe the key technology of how to obtain high-quality images for towers.

Contents of the invention

In order to solve the shortcomings of the prior art, the present invention provides a visual servo-based tower image acquisition system and method thereof. The present invention utilizes tower identification technology to identify and locate the position of the tower in the video; then adjusts the position of the airborne pan/tilt through visual servo technology, and locates the high-definition camera to the tower; finally takes pictures to complete the information collection of the tower, thereby improving the information of the tower Accuracy of acquisition and quality of acquired images.

To achieve the above object, the present invention adopts the following technical solutions:

A visual servo-based tower image acquisition system, comprising:

Unmanned aerial vehicle, described unmanned aerial vehicle is loaded with cloud platform, and camera and video camera are installed on the cloud platform; Described camera is used for obtaining the video information of tower;

An image interception module, which is used to intercept a frame of real-time image from the video information of the tower;

A tower identification and positioning module, which is used to identify and locate the position of the tower in the intercepted real-time image;

Tower position judging module, which is used to judge whether the tower is located in the central position of the intercepted real-time image;

The pan-tilt control module is used to control the rotation of the pan-tilt by means of image-based visual servoing when the pole-tower is not located in the central position of the captured real-time image, and determines the direction of rotation of the pan-tilt by the position of the pole-tower in the image;

The image acquisition module determines the rotation amount of the pan/tilt according to the obtained deviation. When the tower is located in the center of the captured real-time image, start from a certain corner of the tower and adjust the position of the pan/tilt with an "S"-shaped path. camera for tower image acquisition.

The camera and the video camera are coaxial on the platform.

The camera is a monocular camera.

The tower identification and positioning module includes:

A straight line detection module, which is used to utilize the LSD straight line detection algorithm to carry out straight line detection to the acquired real-time image;

The tower positioning module is used to divide the real-time image into several image blocks, count the number of straight lines in each image block, and then judge whether there is a tower and determine the position of the tower.

A tower image acquisition method based on a visual servoing tower image acquisition system, comprising:

Step (1): According to the inspection requirements, use the camera on the platform to obtain the video information of the tower and intercept a frame of real-time image from the video;

Step (2): Identify and locate the position of the tower in the intercepted real-time image;

Step (3): determine whether the tower is located at the central position of the intercepted real-time image, if so, then enter step (5); otherwise, determine the deviation of the tower relative to the central position of the intercepted real-time image, and enter the next step;

Step (4): Use image-based visual servoing to control the rotation of the pan-tilt, determine the rotation direction of the pan-tilt by the position of the tower in the image, and then determine the amount of rotation of the pan-tilt according to the deviation obtained in step (3); adjust After the cloud platform, obtain the real-time image of the tower at the current position again, and proceed to the execution step (2);

Step (5): Starting from a certain corner of the tower, adjust the position of the pan-tilt with an "S"-shaped path, and at the same time use the camera on the pan-tilt to collect images of the tower.

The concrete process of described step (2) is:

First, the real-time image obtained by line detection using the LSD line detection algorithm;

Then, the real-time image is divided into several image blocks, the number of straight lines in each image block is counted, and then it is judged whether there is a tower and the position of the tower is determined.

Adopt the mode of image-based visual servoing to control the concrete process that the pan-tilt rotates in the described step (4):

Step (4.1): locate the image block containing the upper right corner of the tower component, and extract the SURF feature F ₁ of the image block;

Step (4.2): Determine the rotation direction of the pan/tilt according to the position of the image block. The rotation direction of the pan/tilt is the direction that makes the tower shift to the center of the image; then rotate the pan/tilt by the smallest unit to obtain the tower image at the current position , and extract its SURF feature F ₂ ;

Step (4.3): Match features F ₁ and F ₂ , and calculate the pixel-level offset of the matching point pair between F ₁ and F ₂ ;

Step (4.4): According to the linear mapping relationship between the characteristic offset and the pan-tilt rotation, the pan-tilt rotation is obtained.

In the step (4.3), RANSAC random sampling is used to eliminate mismatched feature pairs, and the matching matrix H of the features is obtained, and the matching equation of the two features can be obtained from the corresponding relationship: F ₂ =HF ₁ .

In the step (4.4), the linear mapping matrix between the characteristic offset and the pan/tilt rotation is the Jacobian matrix of the current image.

The beneficial effects of the present invention are:

(1) The present invention realizes the automatic acquisition of the tower image during the inspection process of the drone, which greatly reduces the labor intensity of the inspection personnel, and the present invention adopts an image-based visual servo control mode to realize the control of the cloud platform, By calculating the Jacobian matrix to describe the corresponding relationship between the image pixel information and the control amount of the pan/tilt;

(2) The present invention also sets "S" type path photographing mode according to the characteristics of the tower, which ensures the integrity of the tower information;

(3) The acquisition of the tower image in the present invention is completed based on a monocular camera, which requires less equipment and the overall system is simple.

Description of drawings

Fig. 1 is the flow chart of the tower image acquisition method based on visual servoing of the present invention;

Fig. 2 is to intercept a frame real-time image from the video information of obtaining tower;

Fig. 3 is the real-time image of the tower obtained after adjusting the cloud platform;

Fig. 4 a) is the first pole tower image collected in the present embodiment;

Fig. 4 b) is the second pole tower image collected in the present embodiment;

Fig. 4c) is the third pole tower image collected in the present embodiment;

Fig. 4 d) is the fourth tower image collected in the present embodiment;

Fig. 4 e) is the fifth pole tower image collected in the present embodiment;

Fig. 4 f) is the sixth tower image collected in the present embodiment;

Fig. 4g) is the seventh pole tower image collected in the present embodiment;

Fig. 4h) is the eighth tower image collected in the present embodiment;

Fig. 4i) is the ninth tower image collected in this embodiment.

detailed description

Below in conjunction with accompanying drawing and embodiment the present invention will be further described:

The tower image acquisition system based on visual servoing of the present invention includes:

Unmanned aerial vehicle, described unmanned aerial vehicle is loaded with cloud platform, and camera and video camera are installed on the cloud platform; Described camera is used for obtaining the video information of tower;

An image interception module, which is used to intercept a frame of real-time image from the video information of the tower;

A tower identification and positioning module, which is used to identify and locate the position of the tower in the intercepted real-time image;

Tower position judging module, which is used to judge whether the tower is located in the central position of the intercepted real-time image;

The pan-tilt control module is used to control the rotation of the pan-tilt by means of image-based visual servoing when the pole-tower is not located in the central position of the captured real-time image, and determines the direction of rotation of the pan-tilt by the position of the pole-tower in the image;

Image acquisition module: determine the rotation amount of the pan/tilt according to the obtained deviation; when the tower is located at the center of the captured real-time image, start from a certain corner of the tower and adjust the position of the pan/tilt with an "S"-shaped path, and at the same time use the top of the pan/tilt The camera collects tower images.

Wherein, the camera and the video camera are coaxial on the platform.

The camera can adopt a monocular camera.

Further, the tower identification and positioning module includes:

A line detection module, which is used to utilize the LSD line detection algorithm to carry out the real-time image obtained by line detection;

The tower positioning module is used to divide the real-time image into several image blocks, count the number of straight lines in each image block, and then judge whether there is a tower and determine the position of the tower.

When using the image-based visual servo method to control the rotation of the pan-tilt, the eye-in-hand method is used. The pan-tilt has m degrees of freedom, and the angular velocity of the pan-tilt rotation is q=[q ₁ ,…,q _p ], the linear velocity at the end is r=[r ₁ ,…,r _m ], the two have the following relationship:

r＝J _r q

in,

The position transformation of the pan/tilt is converted into the change of image parameters through the camera perspective projection matrix, so as to obtain the transformation relationship between the image feature space and the position space of the end of the pan/tilt.

Assuming that the coordinates of the current image are (x, y), the final transformation relationship is:

in, f is the focal length of the camera;

u=[ν _x , ν _y , ν _z , ω _x , ω _y , ω _z ] ^T , v _x , v _y , v _z represent the linear velocity of the camera rotating along the x, y and z axes respectively, ω _x , ω _y , ω _z represent the angular velocity of the camera along the x, y and z axes, respectively.

J _Image is the Jacobian matrix of the image, and the relationship between the rate of change of the image feature and the speed of the pan/tilt is expressed by the composite Jacobian matrix as J=J _Image ·J _r .

As shown in Figure 1, the tower image acquisition method of the tower image acquisition system based on visual servoing of the present invention comprises:

Step (1): According to the inspection requirements, use the camera on the platform to obtain the video information of the tower and intercept a frame of real-time image from the video, as shown in Figure 2;

Step (2): Identify and locate the position of the tower in the intercepted real-time image;

Step (3): determine whether the tower is located at the central position of the intercepted real-time image, if so, then enter step (5); otherwise, determine the deviation of the tower relative to the central position of the intercepted real-time image, and enter the next step;

Step (4): Use image-based visual servoing to control the rotation of the pan-tilt, determine the rotation direction of the pan-tilt by the position of the tower in the image, and then determine the amount of rotation of the pan-tilt according to the deviation obtained in step (3); adjust After the cloud platform, obtain the real-time image of the tower at the current position again, and proceed to the execution step (2);

Step (5): Starting from a certain corner of the tower, adjust the position of the pan-tilt with an "S"-shaped path, and at the same time use the camera on the pan-tilt to collect images of the pole and tower, and the acquired images are shown in Figures 4a)-4i).

Further, the specific process of identifying and locating the position of the tower in the intercepted real-time image in step (2) is:

First, the real-time image obtained by line detection using the LSD line detection algorithm;

Then, the real-time image is divided into several image blocks, the number of straight lines in each image block is counted, and then it is judged whether there is a tower and the position of the tower is determined.

When performing visual servoing, the present invention can be completed in two forms: position-based visual servo control and image-based visual servo control. The position-based visual servo control needs to calculate the position and orientation of the object in real time, and predict The trajectory of the end effector of the manipulator. In this way, in the presence of noise, it is easy for the feature points to leave the field of view. Image-based visual servoing is not sensitive to quantization errors and measurement errors, mainly because when the manipulator moves in the direction where the image feature point error decreases, the probability of feature points leaving the field of view is very low, so the servo control in step (4) The process is controlled by image-based visual servoing.

Obtain the gimbal information and the relative information of the tower in the video when the drone is hovering, and adjust the step (4) so that the photo conditions are met at this time.

Wherein, in step (4), the specific process of controlling the rotation of the pan-tilt by means of image-based visual servoing is:

Step (4.1): locate the image block containing a certain corner of the tower component, and extract the SURF feature F ₁ of the image block;

Step (4.2): Determine the rotation direction of the pan/tilt according to the position of the image block. The rotation direction of the pan/tilt is the direction that makes the tower shift to the center of the image; then rotate the pan/tilt by the smallest unit to obtain the tower image at the current position , and extract its SURF feature F ₂ ;

Step (4.3): Match features F ₁ and F ₂ , and calculate the pixel-level offset of the matching point pair between F ₁ and F ₂ ;

Step (4.4): According to the linear mapping relationship between the characteristic offset and the pan-tilt rotation, the pan-tilt rotation is obtained.

Furthermore, in step (4.3), RANSAC random sampling is used to eliminate mismatched feature pairs to obtain feature matching matrix H, and the matching equation of two features can be obtained from the corresponding relationship: F ₂ =HF ₁ .

Furthermore, in step (4.4), the linear mapping matrix between the feature offset and the pan/tilt rotation is the Jacobian matrix of the current image.

In step (4.3), the offset (P _x , P _y ) of the matching point pair between F ₁ and F ₂ at the pixel level is calculated as:

(P _x , P _y ) = J _Image *u.

In this embodiment, the real-time image is divided into 7*9 image blocks. First, adjust the cloud platform so that the position of the upper right corner of the tower in the image acquired in real time is adjusted to the center position of the entire image;

Secondly, using 2 image blocks in the horizontal direction and 1.5 image blocks in the vertical direction as the rotation step, the pixel-level rotation step is converted into the direction and angle of the pan/tilt adjustment through the image Jacobian matrix;

Finally, starting from a certain corner of the tower, adjust the pan/tilt in an "S" shape and start the camera to shoot. A total of 9 images are taken for each level of the tower, as shown in Figure 4a)-Figure 4i).

Wherein, the real-time image can also be divided into other numbers of image blocks, and correspondingly, the step size of the pan-tilt rotation in the horizontal direction and the vertical direction can also be adjusted according to the actual situation.

Although the specific implementation of the present invention has been described above in conjunction with the accompanying drawings, it does not limit the protection scope of the present invention. Those skilled in the art should understand that on the basis of the technical solution of the present invention, those skilled in the art do not need to pay creative work Various modifications or variations that can be made are still within the protection scope of the present invention.

Claims (4)

1. A tower image acquisition method of a tower image acquisition system based on visual servo is characterized by comprising the following steps:

step (1): according to the inspection requirement, a camera on a holder is adopted to obtain the video information of the tower and a frame of real-time image is intercepted from the video;

step (2): identifying and positioning the position of the tower in the intercepted real-time image;

and (3): judging whether the tower is located at the central position of the intercepted real-time image, if so, entering the step (5); otherwise, determining the deviation of the tower relative to the central position of the intercepted real-time image, and entering the next step;

and (4): controlling the rotation of the cradle head in an image-based visual servo mode, determining the rotation direction of the cradle head through the position of the tower in the image, and then determining the rotation quantity of the cradle head according to the deviation obtained in the step (3); after the cradle head is adjusted, acquiring the real-time image of the tower at the current position again, and executing the step (2);

and (5): adjusting the position of a cradle head by an S-shaped path from a certain angle of the tower, and simultaneously acquiring images of the tower by a camera on the cradle head;

the specific process of controlling the rotation of the holder by adopting the visual servo mode based on the image in the step (4) is as follows:

step (4.1): locating an image block containing a certain corner of a tower component, and extracting SURF (speeded up robust feature) F of the image block₁；

Step (4.2): determining the rotation direction of the holder according to the position of the image block, wherein the rotation direction of the holder is the direction which enables the tower to deviate towards the center of the image; then, the cradle head is rotated by the minimum unit to obtain the tower image at the current position, and the SURF characteristic F of the tower image is extracted₂；

Step (4.3): matching feature F₁And F₂And calculate F₁And F₂The offset of the matching point pair at the pixel level;

step (4.4): and obtaining the rotation quantity of the holder according to the linear mapping relation between the characteristic offset and the rotation quantity of the holder.

2. The tower image acquisition method as claimed in claim 1, wherein the specific process of the step (2) is as follows:

firstly, carrying out line detection by using an LSD (least squares-based) line detection algorithm to obtain a real-time image;

then, the real-time image is divided into a plurality of image blocks, the number of straight lines in each image block is counted, and then whether a tower exists or not is judged, and the position of the tower is determined.

3. The tower image acquisition method as claimed in claim 1, wherein in the step (4.3), RANSAC random sampling is used to eliminate mismatching feature pairs, a matching matrix H of features is obtained, and a matching equation of two features obtained from a corresponding relationship is as follows: f₂＝HF₁。

4. The tower image acquisition method according to claim 1, wherein in the step (4.4), the linear mapping matrix between the characteristic offset and the pan-tilt rotation amount is a Jacobian matrix of the current image.