CN116912715A - Unmanned aerial vehicle vision servo control method and system for fan blade inspection - Google Patents
Unmanned aerial vehicle vision servo control method and system for fan blade inspection Download PDFInfo
- Publication number
- CN116912715A CN116912715A CN202310708227.0A CN202310708227A CN116912715A CN 116912715 A CN116912715 A CN 116912715A CN 202310708227 A CN202310708227 A CN 202310708227A CN 116912715 A CN116912715 A CN 116912715A
- Authority
- CN
- China
- Prior art keywords
- fan
- camera
- point
- aerial vehicle
- unmanned aerial
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 60
- 238000007689 inspection Methods 0.000 title claims abstract description 55
- 230000000007 visual effect Effects 0.000 claims abstract description 26
- 230000002787 reinforcement Effects 0.000 claims abstract description 24
- 230000006870 function Effects 0.000 claims abstract description 17
- 239000013598 vector Substances 0.000 claims description 35
- 238000004422 calculation algorithm Methods 0.000 claims description 14
- 230000009471 action Effects 0.000 claims description 12
- 238000012549 training Methods 0.000 claims description 12
- 230000011218 segmentation Effects 0.000 claims description 11
- 238000012423 maintenance Methods 0.000 claims description 6
- 238000012937 correction Methods 0.000 claims description 5
- 238000013144 data compression Methods 0.000 claims description 4
- 238000003709 image segmentation Methods 0.000 claims description 4
- 230000003287 optical effect Effects 0.000 claims description 4
- 238000004590 computer program Methods 0.000 claims description 3
- 230000001788 irregular Effects 0.000 abstract description 2
- 230000006872 improvement Effects 0.000 description 6
- 230000008569 process Effects 0.000 description 6
- 238000005516 engineering process Methods 0.000 description 3
- 238000003384 imaging method Methods 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 239000003795 chemical substances by application Substances 0.000 description 2
- 238000011217 control strategy Methods 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000012935 Averaging Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000010248 power generation Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 239000004576 sand Substances 0.000 description 1
- 239000002344 surface layer Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/10—Terrestrial scenes
- G06V20/17—Terrestrial scenes taken from planes or by drones
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
- G06F17/10—Complex mathematical operations
- G06F17/11—Complex mathematical operations for solving equations, e.g. nonlinear equations, general mathematical optimization problems
- G06F17/13—Differential equations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/136—Segmentation; Edge detection involving thresholding
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/194—Segmentation; Edge detection involving foreground-background segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20021—Dividing image into blocks, subimages or windows
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Mathematical Physics (AREA)
- Pure & Applied Mathematics (AREA)
- Computational Mathematics (AREA)
- Mathematical Analysis (AREA)
- Mathematical Optimization (AREA)
- Data Mining & Analysis (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Multimedia (AREA)
- Operations Research (AREA)
- Algebra (AREA)
- Remote Sensing (AREA)
- Databases & Information Systems (AREA)
- Software Systems (AREA)
- General Engineering & Computer Science (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses an unmanned aerial vehicle vision servo control method and system for fan blade inspection, wherein through the shooting function of an unmanned aerial vehicle, panoramic information of fan blades is shot at different angles, camera pose information during shooting is recorded, coordinates of key points of a fan are solved according to multiple groups of information, and path planning of unmanned aerial vehicle inspection starting points and directions for the fan blades is completed; and finally, performing visual servo by using a reinforcement learning method, enabling the unmanned aerial vehicle to be close to the blade, attaching the curved surface of the blade to fly and photographing. According to the method, under the condition that the length and direction information of the fan blade is not required to be obtained, the positioning of key points of the fan is automatically completed, high-precision visual tracking is performed, and the problem of routing inspection route deviation caused by irregular shapes of the blade and short-time wind gusts is solved.
Description
Technical Field
The invention belongs to the technical field of unmanned aerial vehicle inspection, and mainly relates to an unmanned aerial vehicle vision servo control method and system for fan blade inspection.
Background
Wind power generation has been used on a large scale worldwide as a renewable energy source with mature technology. However, the surface layer of the fan blade is easy to wear, so that the damage such as sand holes, scratches and the like is caused, and the details of the blade surface can be captured and analyzed at a short distance through automatic detection of the unmanned aerial vehicle. The detection of early defects of the blades can effectively reduce the later maintenance cost, and the design of the unmanned aerial vehicle inspection method for the fan blades has autonomy, stability, adaptability and economy.
Many achievements are achieved based on unmanned aerial vehicle autonomous completion task research, a few autonomous inspection schemes are proposed, and the unmanned aerial vehicle autonomous inspection scheme is based on fan appearance and priori pose data to complete position control by using a positioning system, and is based on various three-dimensional sensors, such as: millimeter wave radars, lidars, ultrasonic sensors, multi-camera and the like are inspected after local three-dimensional reconstruction is completed, and direct inspection is performed after linear fitting is performed on the blade gestures of a fan, and most of the methods depend on priori data and additional sensors, however, the fan is usually installed in a region with severe environment, and the method has a great challenge on flying stability. At the same time, these additional sensors are often expensive and difficult to maintain.
In order to improve the adaptability of the unmanned aerial vehicle in a complex environment, reduce maintenance cost and improve reliability, a fan blade unmanned aerial vehicle inspection method with autonomy, stability, adaptability and economy is designed by using a machine vision method, and the method has great significance.
Disclosure of Invention
Aiming at the problems of low inspection stability and low image quality of inspection shooting caused by the fact that the unmanned aerial vehicle is lack of adjustment during the inspection of the fan blades due to the fact that priori information is relied on in the automatic inspection of the unmanned aerial vehicle with the fan blades in the prior art, the invention provides the unmanned aerial vehicle vision servo control method and system for the inspection of the fan blades, through the shooting function of the unmanned aerial vehicle, panoramic information of the fan blades is shot at different angles, camera pose information during shooting is recorded, coordinates of key points of the fan are solved according to multiple groups of information, and path planning of inspection starting points and directions of the unmanned aerial vehicle facing the fan blades is completed; and finally, performing visual servo by using a reinforcement learning method, enabling the unmanned aerial vehicle to be close to the blade, attaching the curved surface of the blade to fly and photographing. According to the method, under the condition that the length and direction information of the fan blade is not required to be obtained, the positioning of key points of the fan is automatically completed, high-precision visual tracking is performed, and the problem of routing inspection route deviation caused by irregular shapes of the blade and short-time wind gusts is solved.
In order to achieve the above purpose, the technical scheme adopted by the invention is as follows: a visual servo control method of an unmanned aerial vehicle for fan blade inspection comprises the following steps:
s1: position information acquisition: shooting panoramic information of the fan blades at different angles and recording camera pose information during shooting by virtue of a shooting function of the unmanned aerial vehicle, wherein the panoramic information and the camera pose information at least comprise three groups, are respectively shot at different positions of the fan blades, and each position comprises the whole blade of the fan;
s2: solving coordinates of key points of a fan: solving and obtaining coordinates of key points of a fan according to the plurality of groups of panoramic information and camera pose information obtained in the step S1, wherein the key points of the fan are She Jiandian of fan blades; the straight line equation corresponding to the straight line Li of the key point in space is:
wherein t is a parameter, x i The X value of the three-dimensional direction vector of the straight line Li is given by Xi, the X value of the coordinate of the camera is given by Xi when the camera acquires an image, the Y value of the three-dimensional direction vector of the straight line Li is given by Yi, the Y value of the coordinate of the camera is given by Yi when the camera acquires an image, the z_i is given by Z value of the three-dimensional direction vector of the straight line Li, and the Z value of the coordinate of the camera is given by Zi when the camera acquires an image;
s3: path planning: according to the fan key point coordinates obtained by solving in the step S2, finishing path planning of the unmanned aerial vehicle inspection starting point and direction facing the fan blade; the path planning is required to ensure that the largest local image of the fan blade occupies 50% -75% of the area in the field of view of the camera;
s4: track implementation: and (3) performing visual servo by using a reinforcement learning method according to the path obtained in the step (S3), enabling the unmanned aerial vehicle to be close to the blade, enabling the unmanned aerial vehicle to fly by attaching the curved surface of the blade, and photographing.
As an improvement of the present invention, the step S2 specifically includes:
s21: separating pixels belonging to the fan blades in the image from panoramic information by using a semantic segmentation model to obtain a panoramic segmentation map of the fan blades;
s22: extracting pixel coordinates of points on three blade tips of the fan according to the panoramic segmentation map obtained in the step S21, and taking the pixel coordinates as pixel coordinates of key points;
s23: according to the pixel coordinate values extracted in the step S22, the camera pose when the image is shot and the parameters of the camera, the coordinates of the key points of the fan blade in the space are solved, specifically: plane S 1 For the camera to shoot the image plane of the point C at the position of the point A, the point B on the plane is the center of the image shot by the camera, namely the optical axis of the camera at the position A and the plane S 1 Point B in the image at the pixel coordinates ofGo (go)/(go)>The orientation of the camera is the orientation of the vector AB;
plane S 2 For the camera to shoot the image plane where the point C is located at the point D, the point C is located at the plane S 1 And plane S 2 The pixel coordinates of the point C on the plane S1 are the a1 th row and the b1 st column, so that the pixel distance from the center point of the camera to the image plane can be obtained in the pixel coordinate system as shown in the following formula,
wherein θ is the angle of the FOV angle of the camera; with camera A as origin, vectorA direction vector of straight line L1, vector +.>The values of (2) are specifically:
wherein a is 1 Is the line number of the point C on the plane S1, b 1 For the column number of the point C on the plane S1, W is the course angle of the camera in the local space coordinate system o-xyz;
since the straight line is known to pass through the point a, the point a coordinates are (X 0 ，Y 0 ，Z 0 ) The parameter equation for the straight line Li can be found as follows:
wherein t is a parameter, L i And as a straight line equation corresponding to a certain key point of the fan in space, the intersection point of a plurality of space straight lines is the space position of the key point.
As an improvement of the present invention, the step S3 specifically includes:
s31: according to the three-dimensional coordinate information of the key points, the direction of the fan and the coordinates of the center point of the fan blade are obtained;
s32: determining a starting point and a finishing point of a visual servo and giving a patrol direction within a fixed distance from the front side and the back side of the fan blade according to the fan orientation and the fan blade center point coordinates obtained in the step S31; the fixed distance is determined by both the fan blade size and the FOV angle of the camera.
As another improvement of the present invention, the solving method of the fan direction in the step S31 specifically includes: taking the difference between three-dimensional coordinates of two blades in three blades of the fan to obtain one vector, taking the difference between three-dimensional coordinates of the other two blades again to obtain the other vector, and calculating an outer product of the two vectors to obtain a direction vector of a plane, namely the direction of the fan;
the fan blade center point coordinate solving method specifically comprises the following steps: and (5) calculating the average value of the three-dimensional coordinates of the three blades of the fan.
As a further improvement of the present invention, the step S4 specifically includes:
s41: inputting the shot image into an image segmentation algorithm, and separating a blade image from a background image;
s42: after data compression is carried out on the segmented image, the segmented image is used as a state to be input into a reinforcement learning model for training, the output action of the reinforcement learning model is a maintenance forward value perpendicular to the inspection direction, the correction action is carried out, and the reward function is calculated;
s43: updating reinforcement learning parameters according to the rewarding value, the state and the action, and optimizing a reinforcement learning model; when the reward value reaches the threshold value, training is completed, and visual servo is performed by using the corrected model after training, so that track implementation is completed.
As a further improvement of the present invention, the reward function in step S42 is specifically:
w＝-(d 2 +k·s)
where w is the prize function value, d is the offset value in visual servoing, s is the number of servoing steps, and k is the coefficient.
In the step S1, the edges of the image shot by the unmanned aerial vehicle camera are cut by 5% and then used, and the image calibration card is used for calibrating the camera, wherein the FOV value of the camera is modified by cutting and calibration.
In order to achieve the above purpose, the invention also adopts the technical scheme that: an unmanned aerial vehicle vision servo control system for fan blade inspection, comprising a computer program which when executed by a processor performs the steps of any of the methods described above.
Compared with the prior art, the invention has the beneficial effects that: the unmanned aerial vehicle vision servo control method and system for fan blade inspection provided by the invention have high autonomy, and are good in robustness and high in quality of acquired pictures in the inspection process. According to the invention, under the condition that the blade direction of the length of the fan blade and the shutdown angle of the blade are unknown, the capacity of key points of the fan blade is autonomously calculated, the expected inspection path of each blade is calculated through the key points, meanwhile, the unmanned aerial vehicle position is continuously corrected in the inspection process according to the image shot by a camera by using a visual servo method, the unmanned aerial vehicle position can be corrected under the condition that the shape of the fan blade is greatly changed and short-time gusts are encountered, high-quality fan blade pictures are shot, and compared with the existing automatic inspection method of the fan unmanned aerial vehicle, the unmanned aerial vehicle inspection method has higher autonomy level, applicability and robustness and higher engineering value.
Drawings
FIG. 1 is a flow chart of the steps of the method of the present invention;
FIG. 2 is a schematic diagram of positioning spatial key points based on a monocular camera in step S2 of the present invention;
FIG. 3 is a schematic diagram of visual servoing during inspection according to the method of the present invention.
Detailed Description
The present invention is further illustrated in the following drawings and detailed description, which are to be understood as being merely illustrative of the invention and not limiting the scope of the invention.
Example 1
An unmanned aerial vehicle vision servo control method for fan blade inspection, as shown in fig. 1, comprises the following steps:
s1: position information acquisition: the unmanned aerial vehicle flies to at least three different positions in front of the fan blade, the panoramic information of the fan blade is shot at different angles, and the position and pose information of the camera during shooting is recorded.
S2: solving coordinates of key points of a fan: according to the pictures with the number of pictures and the camera pose information solving algorithm when the pictures are taken, the coordinates taking three tip points of the fan blade as key points are obtained; the method specifically comprises the following steps:
s21: separating pixels belonging to the fan blade in the image from the background by utilizing a semantic segmentation model to obtain a panoramic segmentation map of the fan blade;
s22: according to the panoramic segmentation map, and by combining with obvious external characteristics of the fan, extracting pixel coordinates of points on three blade tips of the fan, and taking the pixel coordinates as pixel coordinates of key points;
s23: and solving coordinates of key points of the fan blade in space according to the extracted pixel coordinate values, the camera pose when the image is shot and the parameters of the camera. Midpoint a (X in the local spatial coordinate system o-xyz 0 ，Y 0 ，Z 0 ) And point D is the camera shooting position in space. When the camera resolution is a row and B column (unit is pixel, the same applies below), the angle of FOV angle of the camera is θ degree, and the camera position is located at a (X 0 ，Y 0 ，Z 0 ) When the camera shoots horizontally, the positive direction of the X axis is 0 degree, and the course angle is W. Plane S 1 For the camera to shoot the image plane of the point C at the position of the point A, the point B on the plane is the center of the image shot by the camera, namely the optical axis of the camera at the position A and the plane S 1 Point B in the image at the pixel coordinates ofGo (go)/(go)>Columns. Thus, the orientation of the camera is the orientation of vector AB. Plane S 2 For the camera to shoot the image plane where the point C is located at the point D, the point C is the target point to be solved, and the point C is located at the plane S 1 And plane S 2 The pixel coordinates of the point C on the plane S1 are the a1 th row and the b1 st column, so that the pixel distance from the center point of the camera to the image plane can be obtained in the pixel coordinate system as shown in the following formula,
wherein, taking camera A as origin, vectorA direction vector of straight line L1, vector +.>Wherein W is the course angle of the camera in the local spatial coordinate system o-xyz, and the remaining parameters are obtained according to the linear imaging model of the camera.
Since the straight line is known to pass through the point a, the point a coordinates are (X 0 ，Y 0 ，Z 0 ) The parameter equation for the straight line L can be found as follows:
(t is a parameter)
L 1 In one picture, a straight line equation corresponding to a certain key point of the fan in space is solved for the same key point in other pictures by using the same method, and finally, at least three non-parallel equations of space straight lines can be obtained, wherein the space positions of the key points are the intersection points of the space straight lines.
S3: path planning: according to three key point coordinates of the fan, path planning aiming at the unmanned aerial vehicle inspection starting point and direction of the blade is carried out; the method specifically comprises the following steps:
s31: according to the three-dimensional coordinate information of the three key points, the direction of the fan and the coordinates of the center point of the fan blade can be obtained.
S32: according to the coordinates of the center point of the fan, the coordinates of the tip of the fan and the orientation of the fan, the starting point and the end point of the visual servo are determined at a certain distance from the front surface and the back surface of the three blades of the fan, the inspection direction is given, the distance from the blades of the fan is determined by the size of the blades of the fan and the FOV angle of the camera, and the largest local image of the blades of the fan occupies 50% -75% of the field of view of the camera.
S4: track implementation: according to the path obtained in the step S3, a reinforcement learning method is used for visual servo, so that the unmanned aerial vehicle is close to the blade, and the curved surface of the attached blade flies and photographs, and the steps specifically comprise:
s41: inputting the shot image into an image segmentation algorithm, and separating a blade image from a background;
s42: after data compression is carried out on the segmented image, the segmented image is used as a state to be input into a reinforcement learning model for training, the output action of reinforcement learning is a maintenance forward value perpendicular to the inspection direction, the correction action is carried out, and a reward function is calculated and designed as follows:
w＝-(d 2 +k·s)
wherein w is a reward function value, d is an offset value in visual servo, s is a servo step number, k is a coefficient, k.s is used for solving the problem that the flight time is too long due to a servo algorithm when the penalty direction calculation is inaccurate, and reinforcement learning is finished when a servo camera loses a target. To penalize the decision of such agents in this case, the bonus function is then additionally given a negative number of large absolute value, the specific value being determined according to the specific parameters of the system.
S43: updating reinforcement learning parameters according to the rewarding value, the state and the action to obtain a better control strategy; when the reward value reaches the threshold value, training is considered to be completed, and a corrected model after training is used for visual servo, so that errors in the inspection process are reduced, and the purpose of fitting the curved surface of the blade to fly is achieved.
Example 2
According to the unmanned aerial vehicle vision servo control method for inspection of the fan blade, the unmanned aerial vehicle flies to three different positions in front of the fan blade, panoramic information of the fan blade is shot at different angles, camera pose information during shooting is recorded, and panoramic refers to three key points of the blade tip of the fan, wherein the whole blade of the fan needs to be contained in order to solve the blade. According to the three pictures and the camera pose information solving algorithm when shooting, three key point coordinates of the fan blade are obtained; planning the starting point and the direction of the visual servo according to the coordinates of three key points of the fan; visual servoing is performed by using a reinforcement learning method, so that the unmanned aerial vehicle is close to the blade, and flies and photographs in a shape of the attached blade. The method specifically comprises the following steps:
s1: the unmanned aerial vehicle flies to three different positions in front of the fan blade, shoots the panoramic information of the fan blade at different angles, and records the pose information of the camera during shooting.
In this embodiment, the unmanned aerial vehicle carries the RTK and acquires latitude and longitude information, and acquires orientation information through the 9-axis IMU. And establishing a northeast coordinate system by using the flying spot as an origin and the position of the airplane as a positive direction during taking off through a conversion algorithm of the longitude and latitude high coordinate system and the northeast coordinate system, wherein the coordinate system is a space local coordinate system in the example. The camera is connected with the unmanned aerial vehicle through the two-axis cradle head, and pose information of the camera is pose information in the coordinate system.
In this embodiment, the fan height is 80 meters, the blade length is 40 meters, and the camera FOV angle that unmanned aerial vehicle carried is 90 degrees. Based on the information, the unmanned aerial vehicle flies to 45 meters in front of the fan blade to shoot a fan blade option image.
S2: and solving three key point coordinates of the fan according to the three photographed pictures and a camera pose information solving algorithm during photographing.
The fan blade panorama is image-segmented using a semantic segmentation algorithm, the entire image is classified into two classes, wherein the pixel value of the fan blade portion is set to 255, appearing white, the remaining pixel values are set to 0, appearing black. In the binarized picture, the uppermost white pixel points are selected, the coordinate mean value of the uppermost white pixel points is calculated as the first key point coordinate, the leftmost white pixel points are selected, the coordinate mean value of the leftmost white pixel points is calculated as the second key point coordinate, the rightmost white pixel points are selected, the coordinate mean value of the rightmost white pixel points is calculated as the third key point coordinate, the noise point is eliminated according to the outline of the fan, whether the noise point is located at the special conditions such as the image edge or not is judged, and interference data of non-key points are eliminated;
and solving coordinates of key points of the fan blade in space according to the extracted pixel coordinate values, the camera pose when the image is shot and the parameters of the camera. As shown in FIG. 2, a point A (X) is located at the o-xyz midpoint of the local spatial coordinate system 0 ，Y 0 ，Z 0 ) And point D is two points in space where the camera takes a picture. When the camera resolution is a row and B column (unit is pixel, the same applies below), the angle of FOV angle of the camera is θ degree, and the camera position is located at a (X 0 ，Y 0 ，Z 0 ) When the camera shoots horizontally, the positive direction of the X axis is 0 degree, and the course angle is W. Plane S 1 For the camera to shoot the image plane of the point C at the position of the point A, the point B on the plane is the center of the image shot by the camera, namely the optical axis of the camera at the position A and the plane S 1 Point B in the image at the pixel coordinates ofGo (go)/(go)>Columns. Thus, the orientation of the camera is the orientation of vector AB. Plane S 2 For the camera to shoot the image plane where the point C is located at the point D, the point C is the target point to be solved, and the point C is located at the plane S 1 And plane S 2 The pixel coordinate of the point C on the plane S1 is the a1 st row and the b1 st column, and the pixel distance from the camera center point to the image plane can be obtained in the pixel coordinate system as shown in the following formula.
In this embodiment, when the camera resolution is 1024 x 1024 pixels, the FOV angle of the camera is 90 degrees,a pixel.
With camera A as origin, vectorA direction vector of straight line L1, vector +.>Wherein W is the course angle of the camera in the local spatial coordinate system o-xyz, and the remaining parameters are obtained according to the linear imaging model of the camera.
Since the straight line is known to pass through the point a, the point a coordinates are (X 0 ，Y 0 ，Z 0 ) The parameter equation capable of obtaining the straight line L is as follows, and the straight line L is obtained when i is 1 1 Is a function of the equation (c).
Wherein t is a parameter, L 1 In one picture, a straight line equation corresponding to a certain key point of the fan in space is solved for the same key point in the other two pictures by using the same method, and finally, three non-parallel equations of space straight lines can be obtained, and the space positions of the three key points of the fan blade are the intersection points of the three space straight lines.
S3: and according to the fan key point coordinates obtained by solving, finishing the path planning of the unmanned aerial vehicle inspection starting point and direction facing the fan blade.
As shown in fig. 3, according to three-dimensional coordinate information of three key points, the direction of the fan and the coordinates of the center point of the fan blade can be obtained.
In this example, the step of obtaining the orientation may number 3 points, for example, the key point of the blade on the upper side of the fan is the 1-point, the key point of the blade on the left side is the 2-point, the key point of the blade on the right side is the 3-point, the coordinates of the 1-point and the 2-point are taken to make a difference, a vector is obtained, the coordinates of the 1-point and the 3-point are taken to make a difference, another vector is obtained, the two vectors are subjected to an outer product, the direction vector perpendicular to the plane formed by the three points can be obtained, and the obtaining of the center point is directly performed by three-point averaging.
According to the coordinates of the center point of the fan, the coordinates of the tip of the fan and the orientation of the fan, the starting point and the end point of the visual servo are determined at a certain distance from the front surface and the back surface of the three blades of the fan, the inspection direction is given, the distance from the blades of the fan is determined by the size of the blades of the fan and the FOV angle of the camera, and the largest local image of the blades of the fan occupies 50% -75% of the field of view of the camera.
In the example, the resolution of the camera is 1024×1024 pixels, the FOV of the camera is 90 degrees, the blade length is 40 meters, and the distance between the unmanned aerial vehicle and the fan blade is set to 2.5 meters
S4: inputting the shot image into an image segmentation algorithm, and separating the image at the close range of the blade from the background;
the fan blade panorama is image-segmented using a semantic segmentation algorithm, the entire image is classified into two classes, wherein the pixel value of the fan blade portion is set to 255, appearing white, the remaining pixel values are set to 0, appearing black.
After data compression is carried out on the segmented image, the segmented image is used as a state to be input into a reinforcement learning model for training, the output action of reinforcement learning is a maintenance forward value perpendicular to the inspection direction, the correction action is carried out, and a reward function is calculated and designed as follows:
w＝-(d 2 +k·s)
wherein w is a reward function value, d is an offset value in visual servo, s is a servo step number, k is a coefficient, when k.s is used for punishing inaccurate direction calculation, the servo algorithm causes the problem that the flight time is too long, when a servo camera loses a target, the round of reinforcement learning is finished, in order to punish the decision of the intelligent agent under the condition, the reward function additionally gives a negative number with a large absolute value, and the specific value is determined according to specific parameters of the system.
In this example, a camera inspection of 1024×1024 pixels is used, and a penalty coefficient is set to-90000.
The reinforcement learning algorithm used in the embodiment may be a DDPG algorithm, the input state variable is image information after continuous several frames of compression, the output correction variable is a vector perpendicular to the inspection path, and the input image is set to be continuous four frames of images.
Updating reinforcement learning parameters according to the rewarding value, the state and the action to obtain a better control strategy;
when the reward value reaches the threshold value, training is considered to be completed, and a corrected model after training is used for visual servo, so that errors in the inspection process are reduced, and the purpose of fitting the shape of the blade is achieved.
Example 3
This embodiment differs from embodiments 1 and 3 in that: the unmanned aerial vehicle in the method can be further provided with positioning equipment based on UWB. Therefore, the establishment of the space local coordinate system can be based on the space local coordinate system positioned by the UWV equipment, and the space local coordinate system does not need to be converted into a northeast day coordinate system through a longitude and latitude coordinate system.
In addition, because the edge of the inspection camera carried by the unmanned aerial vehicle has large distortion, the direct use can cause positioning errors. Therefore, in the flight process, the image edge shot by the camera is cut out by 5% to be distorted by a large part, and the camera is calibrated by using the image calibration card. In this example, the coordinates of the pixels in the image used for positioning are corrected by the calibration model, and the FOV value of the camera is corrected by clipping and calibration.
In summary, the method utilizes the sensor carried by the unmanned aerial vehicle and the inspection camera to determine the coordinate position of the key point of the fan blade under the condition of not depending on priori data such as the dimension, the shape and the like of the fan blade. According to the determined key point coordinates, the method adopts a visual servo technology to correct the inspection track in the process of inspecting along the blade, and solves the problems of unstable imaging precision, detail missing and the like of the fan blade in a dynamic environment. In the feedback control link of visual servo, the method adopts reinforcement learning technology, and compared with the traditional PID manual parameter adjustment, the method realizes better control performance and wider adaptability. Compared with the prior art, the method has higher autonomy and robustness.
It should be noted that the foregoing merely illustrates the technical idea of the present invention and is not intended to limit the scope of the present invention, and that a person skilled in the art may make several improvements and modifications without departing from the principles of the present invention, which fall within the scope of the claims of the present invention.
Claims (8)
1. The unmanned aerial vehicle vision servo control method for fan blade inspection is characterized by comprising the following steps:
s1: position information acquisition: shooting panoramic information of the fan blades at different angles and recording camera pose information during shooting by virtue of a shooting function of the unmanned aerial vehicle, wherein the panoramic information and the camera pose information at least comprise three groups, are respectively shot at different positions of the fan blades, and each position comprises the whole blade of the fan;
s2: solving coordinates of key points of a fan: solving and obtaining coordinates of key points of a fan according to the plurality of groups of panoramic information and camera pose information obtained in the step S1, wherein the key points of the fan are She Jiandian of fan blades; straight line L passing through key point i The corresponding linear equation in space is:
wherein t is a parameter, x i Is a straight line L i X value, X of three-dimensional direction vector i When the camera acquires an image, the X value and the y value of the camera coordinate i Is a straight line L i Y-value, Y of three-dimensional direction vector i When the camera acquires an image, the Y value and z of the camera coordinate i Is a straight line L i Z value, Z of three-dimensional direction vector i When an image is acquired for a camera, the Z value of the camera coordinate;
the intersection point of the plurality of space straight lines is the space position of the key point;
s3: path planning: according to the fan key point coordinates obtained by solving in the step S2, finishing path planning of the unmanned aerial vehicle inspection starting point and direction facing the fan blade; the path planning is required to ensure that the largest local image of the fan blade occupies 50% -75% of the area in the field of view of the camera;
s4: track implementation: and (3) performing visual servo by using a reinforcement learning method according to the path obtained in the step (S3), enabling the unmanned aerial vehicle to be close to the blade, enabling the unmanned aerial vehicle to fly by attaching the curved surface of the blade, and photographing.
2. The unmanned aerial vehicle vision servo control method for fan blade inspection according to claim 1, wherein the unmanned aerial vehicle vision servo control method comprises the following steps: the step S2 specifically includes:
s21: separating pixels belonging to the fan blades in the image from panoramic information by using a semantic segmentation model to obtain a panoramic segmentation map of the fan blades;
s22: extracting pixel coordinates of points on three blade tips of the fan according to the panoramic segmentation map obtained in the step S21, and taking the pixel coordinates as pixel coordinates of key points;
s23: according to the pixel coordinate values extracted in the step S22, the camera pose when the image is shot and the parameters of the camera, the coordinates of the key points of the fan blade in the space are solved, specifically: plane S 1 For the camera to shoot the image plane of the point C at the position of the point A, the point B on the plane is the center of the image shot by the camera, namely the optical axis of the camera at the position A and the plane S 1 Point B in the image at the pixel coordinates ofGo (go)/(go)>The orientation of the camera is the orientation of the vector AB;
plane S 2 For the camera to shoot the image plane where the point C is located at the point D, the point C is located at the plane S 1 And plane S 2 Plane pixel coordinates of point C on plane S1Is a (a) 1 Line b 1 The pixel distance from the camera center point to the image plane can be found in the pixel coordinate system as shown in the following formula,
wherein θ is the angle of the FOV angle of the camera; with camera A as origin, vectorA direction vector of straight line L1, vector +.>The values of (2) are specifically:
wherein a is 1 Is the line number of the point C on the plane S1, b 1 Column number of point C on plane S1; w is the course angle of the camera in the local space coordinate system o-xyz;
since the straight line is known to pass through the point a, the point a coordinates are (X 0 ,Y 0 ,Z 0 ) The parameter equation for the straight line Li can be found as follows:
wherein t is a parameter, L i For a straight line equation corresponding to a certain key point of the fan in space, the intersection point of a plurality of space straight lines is the closing pointSpatial location of the key points.
3. The unmanned aerial vehicle vision servo control method for fan blade inspection according to claim 2, wherein: the step S3 specifically includes:
s31: according to the three-dimensional coordinate information of the key points, the direction of the fan and the coordinates of the center point of the fan blade are obtained;
s32: determining a starting point and a finishing point of a visual servo and giving a patrol direction within a fixed distance from the front side and the back side of the fan blade according to the fan orientation and the fan blade center point coordinates obtained in the step S31; the fixed distance is determined by both the fan blade size and the FOV angle of the camera.
4. The unmanned aerial vehicle vision servo control method for fan blade inspection according to claim 3, wherein: the solving method of the fan orientation in the step S31 specifically includes: taking the difference between three-dimensional coordinates of two blades in three blades of the fan to obtain one vector, taking the difference between three-dimensional coordinates of the other two blades again to obtain the other vector, and calculating an outer product of the two vectors to obtain a direction vector of a plane, namely the direction of the fan;
the fan blade center point coordinate solving method specifically comprises the following steps: and (5) calculating the average value of the three-dimensional coordinates of the three blades of the fan.
5. The unmanned aerial vehicle vision servo control method for fan blade inspection according to claim 4, wherein: the step S4 specifically includes:
s41: inputting the shot image into an image segmentation algorithm, and separating a blade image from a background image;
s42: after data compression is carried out on the segmented image, the segmented image is used as a state to be input into a reinforcement learning model for training, the output action of the reinforcement learning model is a maintenance forward value perpendicular to the inspection direction, the correction action is carried out, and the reward function is calculated;
s43: updating reinforcement learning parameters according to the rewarding value, the state and the action, and optimizing a reinforcement learning model; when the reward value reaches the threshold value, training is completed, and visual servo is performed by using the corrected model after training, so that track implementation is completed.
6. The unmanned aerial vehicle vision servo control method for fan blade inspection according to claim 5, wherein: the reward function in step S42 specifically includes:
w＝-(d2+k·s)
where w is the prize function value, d is the offset value in visual servoing, s is the number of servoing steps, and k is the coefficient.
7. The unmanned aerial vehicle vision servo control method for fan blade inspection according to claim 1, wherein the unmanned aerial vehicle vision servo control method comprises the following steps: in the step S1, the edges of the images shot by the unmanned aerial vehicle camera are cut by 5% and then used, and the camera is calibrated by using an image calibration card, wherein the FOV value of the camera is modified by cutting and calibration.
8. Unmanned aerial vehicle vision servo control system towards fan blade inspection, including computer program, its characterized in that: the computer program, when executed by a processor, implements the steps of the method as described in any of the above.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310708227.0A CN116912715A (en) | 2023-06-15 | 2023-06-15 | Unmanned aerial vehicle vision servo control method and system for fan blade inspection |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310708227.0A CN116912715A (en) | 2023-06-15 | 2023-06-15 | Unmanned aerial vehicle vision servo control method and system for fan blade inspection |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116912715A true CN116912715A (en) | 2023-10-20 |
Family
ID=88354051
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310708227.0A Pending CN116912715A (en) | 2023-06-15 | 2023-06-15 | Unmanned aerial vehicle vision servo control method and system for fan blade inspection |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116912715A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117212077A (en) * | 2023-11-08 | 2023-12-12 | 云南滇能智慧能源有限公司 | Wind wheel fault monitoring method, device and equipment of wind turbine and storage medium |
| CN117536797A (en) * | 2023-10-24 | 2024-02-09 | 华能安徽怀宁风力发电有限责任公司 | Unmanned aerial vehicle-based fan blade inspection system and method |
| CN117893933A (en) * | 2024-03-14 | 2024-04-16 | 国网上海市电力公司 | Unmanned inspection fault detection method and system for power transmission and transformation equipment |
-
2023
- 2023-06-15 CN CN202310708227.0A patent/CN116912715A/en active Pending
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117536797A (en) * | 2023-10-24 | 2024-02-09 | 华能安徽怀宁风力发电有限责任公司 | Unmanned aerial vehicle-based fan blade inspection system and method |
| CN117536797B (en) * | 2023-10-24 | 2024-05-31 | 华能安徽怀宁风力发电有限责任公司 | Unmanned aerial vehicle-based fan blade inspection system and method |
| CN117212077A (en) * | 2023-11-08 | 2023-12-12 | 云南滇能智慧能源有限公司 | Wind wheel fault monitoring method, device and equipment of wind turbine and storage medium |
| CN117212077B (en) * | 2023-11-08 | 2024-02-06 | 云南滇能智慧能源有限公司 | Wind wheel fault monitoring method, device and equipment of wind turbine and storage medium |
| CN117893933A (en) * | 2024-03-14 | 2024-04-16 | 国网上海市电力公司 | Unmanned inspection fault detection method and system for power transmission and transformation equipment |
| CN117893933B (en) * | 2024-03-14 | 2024-05-24 | 国网上海市电力公司 | Unmanned inspection fault detection method and system for power transmission and transformation equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN116912715A (en) | Unmanned aerial vehicle vision servo control method and system for fan blade inspection | |
| CN107844750B (en) | Water surface panoramic image target detection and identification method | |
| CN113436260B (en) | Mobile robot pose estimation method and system based on multi-sensor tight coupling | |
| WO2020014909A1 (en) | Photographing method and device and unmanned aerial vehicle | |
| CN109145864A (en) | Determine method, apparatus, storage medium and the terminal device of visibility region | |
| CN115761550A (en) | Water surface target detection method based on laser radar point cloud and camera image fusion | |
| CN109492525B (en) | Method for measuring engineering parameters of base station antenna | |
| US20230105991A1 (en) | Method of imaging a wind turbine rotor blade | |
| CN110998241A (en) | System and method for calibrating an optical system of a movable object | |
| KR20210105487A (en) | Method for estimation of river bed change rate using hyperspectral image | |
| Cvišić et al. | Recalibrating the KITTI dataset camera setup for improved odometry accuracy | |
| KR20200084972A (en) | Method for acquisition of hyperspectral image using an unmanned aerial vehicle | |
| CN105976324B (en) | Vehicle image splicing method | |
| CN115240089A (en) | Vehicle detection method of aerial remote sensing image | |
| CN113593035A (en) | Motion control decision generation method and device, electronic equipment and storage medium | |
| CN114463521B (en) | Building target point cloud rapid generation method for air-ground image data fusion | |
| CN116202489A (en) | Method and system for co-locating power transmission line inspection machine and pole tower and storage medium | |
| CN117671033A (en) | Quick calibration method and system for main point of camera image based on night light tracking | |
| CN116977902B (en) | Target tracking method and system for on-board photoelectric stabilized platform of coastal defense | |
| CN110414384B (en) | Intelligent rice and wheat harvester leading line tracking method | |
| CN117095315A (en) | High-precision spillway bottom plate concrete structure surface crack characteristic detection method | |
| CN116402904A (en) | Combined calibration method based on laser radar inter-camera and monocular camera | |
| CN116486290A (en) | Unmanned aerial vehicle monitoring and tracking method and device, electronic equipment and storage medium | |
| CN110989645A (en) | Target space attitude processing method based on compound eye imaging principle | |
| CN115792912A (en) | Method and system for sensing environment of unmanned surface vehicle based on fusion of vision and millimeter wave radar under weak observation condition |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |