CN109961398B - Fan blade image segmentation and grid optimization splicing method - Google Patents

Fan blade image segmentation and grid optimization splicing method Download PDF

Info

Publication number
CN109961398B
CN109961398B CN201910124247.7A CN201910124247A CN109961398B CN 109961398 B CN109961398 B CN 109961398B CN 201910124247 A CN201910124247 A CN 201910124247A CN 109961398 B CN109961398 B CN 109961398B
Authority
CN
China
Prior art keywords
image
grid
fan blade
registered
images
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201910124247.7A
Other languages
Chinese (zh)
Other versions
CN109961398A (en
Inventor
徐进
丁显
刘泉
曹移明
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Luneng New Energy Group Co ltd
Original Assignee
Luneng New Energy Group Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Luneng New Energy Group Co ltd filed Critical Luneng New Energy Group Co ltd
Priority to CN201910124247.7A priority Critical patent/CN109961398B/en
Publication of CN109961398A publication Critical patent/CN109961398A/en
Application granted granted Critical
Publication of CN109961398B publication Critical patent/CN109961398B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformation in the plane of the image
    • G06T3/40—Scaling the whole image or part thereof
    • G06T3/4038—Scaling the whole image or part thereof for image mosaicing, i.e. plane images composed of plane sub-images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30108—Industrial image inspection
    • G06T2207/30164—Workpiece; Machine component
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02E—REDUCTION OF GREENHOUSE GAS [GHG] EMISSIONS, RELATED TO ENERGY GENERATION, TRANSMISSION OR DISTRIBUTION
    • Y02E10/00—Energy generation through renewable energy sources
    • Y02E10/70—Wind energy
    • Y02E10/72—Wind turbines with rotation axis in wind direction

Abstract

The invention discloses a fan blade image segmentation and grid optimized splicing method, which comprises the following steps: s1, continuously acquiring images of single-sided fan blades to form a group of original images to be spliced; s2, performing image foreground segmentation on each original image by utilizing a U-net algorithm, and extracting a main body part of the fan blade to form a group of images to be registered; s3, gridding each image to be registered, establishing an index from 1 to m for a grid vertex of each image to be registered, then representing x and y coordinates of the grid vertex as a vector V with 2m dimensionality, defining a global energy function related to V, minimizing the energy function to obtain an optimal solution of the grid vertex, and S4, completing image splicing according to the optimal solution of the grid vertex. The method has the advantages of solving the problem of target splicing failure caused by overlarge depth of field of the target and the background, realizing natural splicing of a plurality of fan blades, reducing distortion of visual effect of the spliced image and realizing continuity and reality.

Description

Fan blade image segmentation and grid optimization splicing method
Technical Field
The invention relates to the field of image stitching. More specifically, the invention relates to a fan blade image segmentation and grid optimization splicing method.
Background
The fan blade is a key component of the wind turbine generator, and due to long-term exposure in the external natural environment, the surface of the fan blade has common defects of sand holes, cracks, peeling and the like, so that the efficiency of wind power generation and the service life of the fan are influenced. At present, the defect detection of fan blades mainly depends on manual work, and the problems of low working efficiency, danger and high cost exist. Along with the popularization and application of unmanned aerial vehicle technique, utilize unmanned aerial vehicle to replace the manual work to patrol and examine the operation, closely observe the blade through high definition resolution camera, can discover in advance, in time handle defective blade, guarantee system equipment safe operation. According to incomplete statistics, the method is at least 3 times faster than the traditional manual detection, the cost is saved by nearly 50%, and the downtime of the generator is reduced by 2/3.
When the unmanned aerial vehicle carries out fan blade inspection operation, in order to meet the resolution requirement of defect identification, a shooting camera cannot realize complete blade coverage through single shooting, generally, single-side shooting of a single fan blade requires 20-40 different photos, in addition, the blade inspection is divided into a windward side, a leeward side, a front edge and a rear edge, and numerous scattered pictures are not beneficial to management of the blade inspection operation; blade defects are analyzed through pictures, and specific defect positions cannot be located, so that the shot scattered fan blades need to be spliced. The problem of the fan blade splicing process has two points of particularity: 1) The main body of the fan blade has single characteristic relative to the background, and most characteristic points in the image come from the background, which is not beneficial to maintaining the blade characteristic in the splicing process; 2) When the unmanned aerial vehicle acquires the fan blade image, the viewpoint position is obviously translated, so that great parallax can be caused, the single-point perspective assumption of the traditional image splicing method is destroyed, and the problem of multi-viewpoint image splicing is solved, so that great registration error can occur when the image is directly spliced, and serious distortion or ghost image phenomena can be generated. Based on this, provide an image concatenation algorithm based on unmanned aerial vehicle patrols and examines operation to the fan, the discrete fan blade undistorted concatenation that shoots with unmanned aerial vehicle becomes complete blade image by a wide margin, and it is the problem that the present urgent need be solved to provide the technological means for blade fault identification, location and patrol and examine photo management.
Disclosure of Invention
An object of the present invention is to solve at least the above problems and to provide at least the advantages described later.
The invention also aims to provide a fan blade image segmentation and grid optimization splicing method, which uses a deep learning U-net algorithm and a linear retention constrained fan blade splicing method to solve the problem of target splicing failure caused by overlarge depth of field of a target and a background; the gridding splicing technology of the straight line maintaining technology is utilized to effectively protect the straight line characteristics of the edge of the fan, the natural splicing of a plurality of fan blades is realized, and the visual effect of the spliced image is small in distortion and continuous and real.
To achieve these objects and other advantages and in accordance with the purpose of the invention, as embodied and broadly described herein, there is provided a fan blade image segmentation and mesh optimization stitching method, comprising the steps of:
s1, continuously collecting images of a single-sided fan blade to form a group of original images to be spliced;
s2, performing image foreground segmentation on each original image of the group of original images by using a U-net algorithm, and extracting a main body part of the fan blade to form a group of images to be registered;
s3, gridding each image to be registered, establishing an index from 1 to m for a grid vertex of each image to be registered, and then expressing x and y coordinates of the grid vertex as a vector V with dimension of 2m, wherein V = [ x ] of the vector V 1 y 1 x 2 y 2 … x m y m ]Defining a global energy function about V, and minimizing the energy function to obtain an optimal solution of the mesh vertex
Figure BDA0001971700750000021
Comprises the following steps:
Figure BDA0001971700750000022
wherein psi a (V) is an alignment error term; psi l (V) is a local similarity term; psi g (V) is a global similarity regularization term; psi line (V) is a line preservation regularization term; lambda [ alpha ] l 、λ g 、λ line Adjusting weights of the local similarity item, the global similarity item and the straight line keeping constraint item respectively;
and S4, according to the optimal solution of the grid vertex, carrying out image mapping and pixel fusion to complete image splicing.
Preferably, the number of the group of images to be registered is N, the group of images to be registered is numbered as i, i = 1-N, the SIFT algorithm is utilized to extract the characteristic points of each image to be registered and match the characteristic points of the overlapped area to obtain a coarse matching characteristic point pair set, and the random sampling consistency algorithm is utilized to eliminate mismatching pairs in the coarse matching characteristic point pair set to obtain an effective matching characteristic point pair set;
determining a quadrilateral mesh in which the feature points in the effective matching feature point pair set are located, describing the positions of the corresponding feature points by using vertex coordinates of the quadrilateral mesh in a linear mode, and determining a mesh vertex matching pair to obtain a mesh vertex matching pair set;
alignment error term psi a (V) is:
Figure BDA0001971700750000023
wherein J is a set of image pairs with overlapping regions;
Figure BDA0001971700750000024
is->
Figure BDA0001971700750000025
Linear weighting of the grid vertex coordinates of the quadrangle; m ij Matching a set of pairs for the mesh vertices; />
Figure BDA0001971700750000031
And/or>
Figure BDA0001971700750000032
Is a pair of valid matching pairs of feature points.
Preferably, the local similarity term ψ l (V) is:
Figure BDA0001971700750000033
wherein the content of the first and second substances,
Figure BDA0001971700750000034
for the grid vertex coordinates before the geometric transformation, <' >>
Figure BDA0001971700750000035
The coordinates of the vertexes of the mesh after the geometric transformation,
Figure BDA0001971700750000036
for the grid diagonal vector before the geometric transformation, <' >>
Figure BDA0001971700750000037
Transforming the diagonal vector of the mesh for geometric projection；/>
Figure BDA0001971700750000038
Is->
Figure BDA0001971700750000039
A transformation matrix of E i A set of diagonal vectors representing the entire grid.
Preferably, the global similarity term ψ g (V) is:
Figure BDA00019717007500000310
wherein the content of the first and second substances,
Figure BDA00019717007500000311
is a weight function; />
Figure BDA00019717007500000312
And/or>
Figure BDA00019717007500000313
Is a diagonal similarity transformation function; s is i And theta i Are global geometric transformation parameters.
Preferably, the method comprises the steps of extracting straight lines L in the fan blade image by using an LSD algorithm to obtain a straight line set L, collecting n sampling points on each straight line L to obtain a sampling point set of each straight line
Figure BDA00019717007500000314
Straight line hold constraint term psi line (V) is:
Figure BDA00019717007500000315
/>
wherein [ a ] l ,b l ] ⊥ A unit normal vector that is a straight line;
Figure BDA00019717007500000316
are like points>
Figure BDA00019717007500000317
The coordinates of the four vertices of the located grid are linearly weighted.
Preferably, step S2 specifically includes the steps of:
s2a, collecting at least 80 original images of the fan blade, and manually dividing the original images by adopting a method of manually marking a main body part of the fan blade to obtain a calibration image;
s2b, building a deep learning network based on U-net, loading the original image serving as an input image and the calibration image serving as an output image into the deep learning network, and training to obtain a fan blade model;
and S2c, taking each original image in the group of original images in the step S1 as an input image, and processing by using a fan blade segmentation model to obtain a corresponding output image, namely the image to be registered.
The invention at least comprises the following beneficial effects:
firstly, the routing inspection shooting of the fan blade by the unmanned aerial vehicle is multi-viewpoint shooting, the shooting points are translated greatly, so that large parallax can be caused, obvious background and foreground field depth exist, and splicing is interfered;
secondly, by utilizing the gridding splicing technology of the straight line maintaining technology and optimizing the straight line criterion, the straight line characteristics at the edge of the fan are effectively protected, so that only one-time optimization solution is needed in the grid optimization stage, the natural splicing of a plurality of fan blades is realized, the visual effect of the spliced image is small in distortion, continuous and real, the calculation mode is direct, the application range is wide, and the reliability is higher.
Additional advantages, objects, and features of the invention will be set forth in part in the description which follows and in part will become apparent to those having ordinary skill in the art upon examination of the following or may be learned from practice of the invention.
Drawings
FIG. 1 is a schematic flow diagram of a fan blade image segmentation and grid optimization stitching method according to the present invention;
FIG. 2 illustrates two original images according to one embodiment of the present invention;
FIG. 3 illustrates two images to be registered according to an embodiment of the present invention;
fig. 4 is a schematic diagram of a coarse matching result of feature points of two to-be-registered pictures with an overlapping area in one embodiment of the present invention;
fig. 5 is a schematic diagram of a result of rejecting mismatching pairs of feature points of two to-be-registered pictures with an overlapping area in one embodiment of the present invention;
fig. 6 is a schematic diagram of an image after completing image stitching according to an embodiment of the present invention.
Detailed Description
The present invention is further described in detail below with reference to examples so that those skilled in the art can practice the invention with reference to the description.
As shown in FIG. 1, the invention provides a fan blade image segmentation and grid optimization splicing method, which comprises the following steps:
s1, continuously acquiring images of a single-sided fan blade by an unmanned aerial vehicle to form a group of original images to be spliced;
s2, performing image foreground segmentation on each original image of the group of original images by utilizing a U-net algorithm, and extracting a main body part of the fan blade to form a group of images to be registered;
s3, establishing uniform quadrilateral grids for each image to be registered, namely gridding each image to be registered, establishing indexes from 1 to m for grid vertexes of each image to be registered, and then expressing x and y coordinates of the grid vertexes as a 2 m-dimensional vector V, V = [ x ] dimension 1 y 1 x 2 y 2 … x m y m ]Defining a global energy function about V, and minimizing the energy function to obtain an optimal solution of the mesh vertex
Figure BDA0001971700750000053
The global energy function for V is: v = ψ a (V)+λ l ψ l (V)+λ g ψ g (V)+λ line ψ line (V)
Figure BDA0001971700750000051
Comprises the following steps: />
Figure BDA0001971700750000052
Wherein psi a (V) is an alignment error term; psi l (V) is a local similarity term; psi g (V) is a global similarity regularization term; psi line (V) is a line preservation regularization term; lambda [ alpha ] l 、λ g 、λ line Adjusting weights of the local similarity item, the global similarity item and the straight line keeping constraint item respectively;
and S4, according to the optimal solution of the grid vertex, carrying out image mapping and pixel fusion to complete image splicing.
In the technical scheme, the unmanned aerial vehicle continuously collects the single-side fan blade, in the continuous collection process, the distance between the unmanned aerial vehicle and the blade is basically kept unchanged, an overlapping area exists between any two adjacent collected original images, the adjusting weights of the local similarity item, the global similarity item and the straight line keeping constraint item depend on the characteristics of details and structural problems in the images, for one type of images, the adjusting weights are unchanged, and in the step S3, psi a (V) is an alignment error term used to regulate better alignment of the overlapping regions of the images to be registered,. Psi l (V) is a local similarity item, which is used for transmitting the geometric transformation of the grids in the overlapping area to the whole grid under the condition of ensuring that each grid undergoes similar geometric transformation as much as possible, and ensuring the continuity of the geometric transformation of the grids; psi g (V) a global similarity regularization term which controls the meshes of the non-overlapping regions to be dominated by global similarity transformation and the meshes of the overlapping regions to be dominated by alignment, ensuring that the overlapping regions are sufficiently aligned and the non-overlapping regions are able to try to ensure that they undergo a similarity transformation, reducing projection distortion; psi line (V) is a straight lineKeeping a regular term, which is to keep the linear structural characteristics of the blade; step S4 specifically includes: mapping all images to a uniform coordinate system according to the obtained optimal solution of the grid vertex (coordinate), so that matching points of different images to be registered are positioned at the same position, solving weighted average to perform pixel fusion, and completing image splicing; the technical scheme is adopted, a deep learning U-net algorithm and a straight line keeping constraint fan blade splicing method are used, and the problem that the splicing of the target is failed due to the fact that the depth of field of the target and the background is too large is solved; the gridding splicing technology of the straight line maintaining technology is utilized to effectively protect the straight line characteristics of the edge of the fan, natural splicing of a plurality of fan blades is realized, the visual effect distortion of the spliced image is small, the spliced image is continuous and real, the calculation mode is direct, the application range is wide, and the reliability is higher.
In another technical scheme, a group of images to be registered is N, the group of images to be registered is numbered as i, i = 1-N, the SIFT algorithm is utilized to extract the characteristic points of each image to be registered and match the characteristic points of an overlapping area to obtain a coarse matching characteristic point pair set, the error of image geometric correction can be caused by the mismatching pairs existing in the coarse matching characteristic point pair set, and the mismatching pairs in the coarse matching characteristic point pair set are eliminated by utilizing the random sampling consistency algorithm to obtain an effective matching characteristic point pair set;
determining a quadrilateral mesh in which the feature points in the effective matching feature point pair set are located, describing the positions of the corresponding feature points by using vertex coordinates of the quadrilateral mesh in a linear mode, and determining a mesh vertex matching pair to obtain a mesh vertex matching pair set;
the alignment error term psi a (V) is:
Figure BDA0001971700750000061
/>
j is an image pair set with an overlapping area, and for an image to be registered obtained by foreground segmentation of continuously shot original images, any two adjacent images to be registered are opposite to each other as a pair of images with overlapping areas;
Figure BDA0001971700750000062
is->
Figure BDA0001971700750000063
Linear weighting of the grid vertex coordinates of the quadrangle; m ij Matching a set of pairs for the mesh vertices; />
Figure BDA0001971700750000064
And/or>
Figure BDA0001971700750000065
Is a pair of valid matching pairs of feature points. In the above technical solution, obtaining the rough matching feature point pair set specifically includes the following steps:
step one, generating a scale space L (x, y, sigma) of each image to be registered under different scales by convolution of the image to be registered and a Gaussian kernel function: l (x, y, σ) = G (x, y, σ) × I (x, y), wherein,
Figure BDA0001971700750000066
is a Gaussian kernel function; σ is the variance of a Gaussian normal distribution, also called scale space factor; (x, y) is an index of the current pixel point of the image to be registered; i (x, y) represents an image to be registered; * Representing a convolution operation;
discretizing and sampling the scale space, generating a series of Gaussian images to form a Gaussian pyramid, subtracting scale space functions of two adjacent layers in the Gaussian pyramid to obtain a Gaussian difference pyramid D (x, y, sigma), and further constructing a difference Gaussian scale space, wherein D (x, y, sigma) = (G (x, y, k sigma) -G (x, y, sigma))) I (x, y) = L (x, y, k sigma) -L (x, y, sigma);
detecting an extreme point of the scale space, and accurately positioning the extreme point: comparing each pixel point in the difference Gaussian scale space with 8 pixel points on the same layer, 9 pixel points on the upper layer and 9 pixel points on the lower layer for 26 pixel points, and if the current pixel point is a maximum value or a minimum value, locating the current pixel point at a key point;
because the influence of the edge and the noise on the difference Gaussian scale space is large, further screening is carried out on key points, the feature points with scale invariance are accurately positioned, and low-contrast points and edge response points need to be filtered out to accurately position extreme points to obtain the feature points;
step three, determining the gradient magnitude m (x, y) and the direction theta (x, y) of the Gaussian pyramid characteristic point (x, y), and calculating the main direction of the characteristic point by using the gradient magnitude and the direction of the characteristic point to ensure the rotation invariance of the characteristic point, wherein,
Figure BDA0001971700750000071
Figure BDA0001971700750000072
l (x, y) is the scale of the characteristic point (x, y);
sampling in a neighborhood window taking the feature point as a center, and counting the gradient direction of a neighborhood pixel by using a histogram, wherein the orientation of the gradient histogram is 0-360 degrees, wherein each 10 degrees is a column, and the main peak value of the histogram represents the main direction of the neighborhood gradient at the feature point, namely the main direction of the feature point;
after determining the main direction of the feature points, taking an 8 × 8 window by taking the feature points as the center, further calculating gradient direction histograms of 8 directions on each 4 × 4 small block, drawing an accumulated value of each gradient direction, and finally forming 128-dimensional SIFT feature description vectors, namely generating feature descriptors;
matching the feature points of the two spatially adjacent images to be registered by using a nearest neighbor distance and next Nearest Neighbor Distance Ratio (NNDR) criterion, wherein point sets in matching in the two spatially adjacent images to be registered form a rough matching feature point pair set after matching is completed; by adopting the scheme, the positions of the obtained effective matching feature points of the effective matching feature point pairs are linearly described by the coordinates of the vertices of the quadrilateral grids where the effective matching feature points are located, and a grid vertex matching pair set is obtained, so that the vertices of the grids participate in the calculation of the alignment error items, the images to be aligned can be aligned in the overlapping area better, and the registration effect among the images is further improved.
In another embodiment, the local similarity term ψ l (V) is:
Figure BDA0001971700750000073
wherein, the first and the second end of the pipe are connected with each other,
Figure BDA0001971700750000074
for the grid vertex coordinates before the geometric transformation, <' >>
Figure BDA0001971700750000075
For the mesh vertex coordinates after the geometric transformation,
Figure BDA0001971700750000076
for the grid diagonal vector before the geometric transformation, <' >>
Figure BDA0001971700750000077
The grid diagonal vector after the geometric projection transformation is obtained; />
Figure BDA0001971700750000078
Is->
Figure BDA0001971700750000079
Transformation matrix of (E) i A set of diagonal vectors representing the entire grid. By adopting the scheme, the geometric transformation of the meshes in the overlapping area is spread to the whole mesh under the condition that each mesh undergoes similar geometric transformation as much as possible, and the continuity of the geometric transformation of the meshes is ensured.
In another solution, the global similarity term ψ g (V) is:
Figure BDA00019717007500000710
wherein the content of the first and second substances,
Figure BDA0001971700750000081
the weight function is positively correlated with the distance between the grid edge and the overlapping area, a higher weight value is given to a grid diagonal line far away from the overlapping area, and the constraint of the alignment item of the overlapping area plays a leading role; />
Figure BDA0001971700750000082
And &>
Figure BDA0001971700750000083
Is a diagonal similarity transformation function; s i And theta i The parameters are global geometric transformation parameters, specifically rotation parameters and scale parameters. With this scheme, the grids of the non-overlapping regions are controlled to be dominated by global semblance transformation, while the grids of the overlapping regions are dominated by alignment, ensuring that the overlapping regions are sufficiently aligned, while the non-overlapping regions can try to ensure that a semblance transformation is performed, reducing projection distortion.
In another technical scheme, a straight line L in a fan blade image is extracted by using an LSD algorithm to obtain a straight line set L, n sampling points are collected on each straight line L to obtain a sampling point set of each straight line
Figure BDA0001971700750000084
Straight line hold constraint term psi line (V) is:
Figure BDA0001971700750000085
wherein [ a ] l ,b l ] ⊥ A unit normal vector that is a straight line;
Figure BDA0001971700750000086
are like points>
Figure BDA0001971700750000087
The coordinates of the four vertices of the located grid are linearly weighted. By adopting the scheme, the straight line criterion is optimized, so that only one-time optimization solution is needed in the grid optimization stage, the straight line characteristics at the edge of the fan are effectively protected by utilizing the gridding splicing technology of the straight line maintaining technology, the natural splicing of a plurality of fan blades is realized, and the visual effect of the spliced image is small in distortion and continuous and real.
In another technical solution, the step S2 specifically includes the following steps:
s2a, collecting at least 80 original images of the fan blade, and manually dividing the original images by adopting a method of manually marking a main body part of the fan blade to obtain a calibration image;
s2b, building a deep learning network based on U-net, loading the original image serving as an input image and the calibration image serving as an output image into the deep learning network, and training to obtain a fan blade model;
and S2c, taking each original image in the group of original images in the step S1 as an input image, and processing by using a fan blade segmentation model to obtain a corresponding output image, namely the image to be registered. By adopting the scheme, the problem of overlarge depth of field of the background of the original image is solved by using a deep learning U-net algorithm.
Example 1
As shown in fig. 2 to 6, a fan blade image segmentation and mesh optimization stitching method includes the following steps:
s1, continuously acquiring 23 images of a single-sided fan blade by an unmanned aerial vehicle to form a group of original images to be spliced, wherein part of the original images (subjected to gray processing) are shown in FIG. 2;
s2, performing image foreground segmentation on each original image of the group of original images by using a U-net algorithm, extracting a main body part of the fan blade to form a group of images to be registered, wherein the images to be registered corresponding to the original images shown in the figure 2 (after gray processing) are shown in the figure 3;
extracting feature points of each image to be registered by utilizing an SIFT algorithm and matching the feature points of the overlapping area to obtain a rough matching feature point pair set, wherein the rough matching result of the feature points of the two images to be registered with the overlapping area is shown in FIG. 4; removing mismatching pairs in the rough matching characteristic point pair set by using a random sampling consistency algorithm to obtain an effective matching characteristic point pair set, wherein the results of removing mismatching pairs from the characteristic points of two to-be-registered pictures with overlapping regions are shown in fig. 5
S3, establishing a uniform quadrilateral grid for each image to be registered by taking pixels as units, wherein the size of the uniform quadrilateral grid is 40 multiplied by 40 pixels, the resolution of a downsampled picture is 800 multiplied by 600 pixels, and the value of a weight coefficient is multiplied by lambda l ＝0.25、λ g ＝0.75、λ line The optimization solving problem of the large-scale sparse matrix is effectively solved through CGLS iteration, according to statistics, 72934 sparse items are totally obtained when 23 images are subjected to grid optimization solving, 16928 vertexes participate in optimization, the iteration time is 2.6 seconds, and the total processing time is 69.5 seconds;
and S4, performing image mapping and pixel fusion according to the optimal solution of the grid vertex to complete image splicing, which is specifically shown in FIG. 6.
While embodiments of the invention have been described above, it is not limited to the applications set forth in the description and the embodiments, which are fully applicable in various fields of endeavor to which the invention pertains, and further modifications may readily be made by those skilled in the art, it being understood that the invention is not limited to the details shown and described herein without departing from the general concept defined by the appended claims and their equivalents.

Claims (5)

1. The fan blade image segmentation and grid optimization splicing method is characterized by comprising the following steps of:
s1, continuously acquiring images of single-sided fan blades to form a group of original images to be spliced;
s2, performing image foreground segmentation on each original image of the group of original images by utilizing a U-net algorithm, and extracting a main body part of the fan blade to form a group of images to be registered;
s3, gridding each image to be registered, establishing an index from 1 to m for the grid vertex of each image to be registered, and then setting the networkThe x, y coordinates of the grid vertices are represented as a 2 m-dimensional vector V, V = [ x = [ ] 1 y 1 x 2 y 2 …x m y m ]Defining a global energy function related to V, and minimizing the energy function to obtain an optimal solution of the mesh vertex
Figure FDA0003986054020000011
Comprises the following steps:
Figure FDA0003986054020000012
wherein psi a (V) is an alignment error term; psi l (V) is a local similarity term; psi g (V) is a global similarity regularization term; psi line (V) is a line maintenance regularization term; lambda [ alpha ] l 、λ g 、λ line Adjusting weights of the local similarity item, the global similarity item and the straight line keeping constraint item respectively;
s4, according to the optimal solution of the grid vertex, image mapping and pixel fusion are carried out, and image splicing is completed;
the method comprises the steps that a group of images to be registered are N, the number of the group of images to be registered is i, i = 1-N, the SIFT algorithm is used for extracting the characteristic points of each image to be registered and matching the characteristic points of an overlapping area to obtain a coarse matching characteristic point pair set, and the random sampling consistency algorithm is used for eliminating mismatching pairs in the coarse matching characteristic point pair set to obtain an effective matching characteristic point pair set;
determining a quadrilateral mesh in which the feature points in the effective matching feature point pair set are located, describing the positions of the corresponding feature points by using vertex coordinates of the quadrilateral mesh in a linear mode, and determining a mesh vertex matching pair to obtain a mesh vertex matching pair set;
alignment error term psi a (V) is:
Figure FDA0003986054020000013
wherein J is an image having an overlap regionA pair set;
Figure FDA0003986054020000014
is->
Figure FDA0003986054020000015
Linear weighting of the grid vertex coordinates of the quadrangle; m ij Matching a set of pairs for the mesh vertices; />
Figure FDA0003986054020000016
And/or>
Figure FDA0003986054020000017
Is a pair of valid matching pairs of feature points.
2. The fan blade image segmentation and grid optimized stitching method of claim 1, wherein the local similarity term ψ l (V) is:
Figure FDA0003986054020000021
wherein, the first and the second end of the pipe are connected with each other,
Figure FDA0003986054020000022
for the grid vertex coordinates before the geometric transformation, <' >>
Figure FDA0003986054020000023
For the mesh vertex coordinates after the geometric transformation,
Figure FDA0003986054020000024
for grid diagonal vectors prior to geometric transformation>
Figure FDA0003986054020000025
The grid diagonal vector after the geometric projection transformation is obtained; />
Figure FDA0003986054020000026
Is->
Figure FDA0003986054020000027
Transformation matrix of (E) i A set of diagonal vectors representing the entire grid.
3. The fan blade image segmentation and grid optimized stitching method of claim 2, wherein the global similarity term ψ g (V) is:
Figure FDA0003986054020000028
wherein, the first and the second end of the pipe are connected with each other,
Figure FDA0003986054020000029
is a weight function; />
Figure FDA00039860540200000210
And &>
Figure FDA00039860540200000211
Is a diagonal similarity transformation function; s i And theta i Are global geometric transformation parameters.
4. The fan blade image segmentation and grid optimized splicing method according to claim 2, wherein the LSD algorithm is used for extracting straight lines L in the fan blade image to obtain a straight line set L, and n sampling points are collected on each straight line L to obtain a sampling point set of each straight line
Figure FDA00039860540200000212
Straight line hold constraint term psi line (V) is:
Figure FDA00039860540200000213
wherein [ a ] l ,b l ] ⊥ A unit normal vector that is a straight line;
Figure FDA00039860540200000214
is a sample point>
Figure FDA00039860540200000215
The coordinates of the four vertexes of the grid are linearly weighted.
5. The fan blade image segmentation and grid optimization splicing method according to claim 1, wherein the step S2 specifically comprises the following steps:
s2a, collecting at least 80 original images of the fan blade, and manually dividing the original images by adopting a method of manually marking a main body part of the fan blade to obtain a calibration image;
s2b, building a deep learning network based on U-net, loading the original image serving as an input image and the calibration image serving as an output image into the deep learning network, and training to obtain a fan blade model;
and S2c, taking each original image in the group of original images in the step S1 as an input image, and processing by using a fan blade segmentation model to obtain a corresponding output image, namely the image to be registered.
CN201910124247.7A 2019-02-18 2019-02-18 Fan blade image segmentation and grid optimization splicing method Active CN109961398B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910124247.7A CN109961398B (en) 2019-02-18 2019-02-18 Fan blade image segmentation and grid optimization splicing method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910124247.7A CN109961398B (en) 2019-02-18 2019-02-18 Fan blade image segmentation and grid optimization splicing method

Publications (2)

Publication Number Publication Date
CN109961398A CN109961398A (en) 2019-07-02
CN109961398B true CN109961398B (en) 2023-04-07

Family

ID=67023584

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910124247.7A Active CN109961398B (en) 2019-02-18 2019-02-18 Fan blade image segmentation and grid optimization splicing method

Country Status (1)

Country Link
CN (1) CN109961398B (en)

Families Citing this family (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111507904B (en) * 2020-04-22 2023-06-02 华中科技大学 Image stitching method and device for microscopic printing patterns
CN112085694B (en) * 2020-06-30 2023-10-31 数笙智能科技(上海)有限公司 Automatic wind energy fan blade system of patrolling and examining of artificial intelligence
CN112211793A (en) * 2020-08-25 2021-01-12 华北电力大学（保定） Wind turbine generator fault automatic identification method based on video image analysis
CN112365399A (en) * 2020-10-09 2021-02-12 北京星闪世图科技有限公司 Fan blade image panoramic stitching method and system based on deep learning
CN112435163B (en) * 2020-11-18 2022-10-18 大连理工大学 Unmanned aerial vehicle aerial image splicing method based on linear feature protection and grid optimization
CN112381718B (en) * 2020-11-24 2021-07-30 中国科学院自动化研究所 Automatic stitching method, system and device for microscopic sequence images based on affine transformation
CN112529019B (en) * 2020-12-24 2024-02-09 华中科技大学 Image stitching method and system based on fusion of linear features and key point features
CN112862683B (en) * 2021-02-07 2022-12-06 同济大学 Adjacent image splicing method based on elastic registration and grid optimization
KR102467590B1 (en) * 2022-08-23 2022-11-17 (주)니어스랩 Apparatus and method for image stitching based on artificial intelligence for inspecting an wind turbine

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105389787A (en) * 2015-09-30 2016-03-09 华为技术有限公司 Panorama image stitching method and device

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105786431A (en) * 2016-03-04 2016-07-20 京东方科技集团股份有限公司 Jointing screen and method and device for achieving automatic distribution of display content of jointing screen
CN107067370A (en) * 2017-04-12 2017-08-18 长沙全度影像科技有限公司 A kind of image split-joint method based on distortion of the mesh
CN107248142B (en) * 2017-05-11 2019-10-29 大连理工大学 A kind of cultural relic fragments method for automatically split-jointing
CN107301620B (en) * 2017-06-02 2019-08-13 西安电子科技大学 Method for panoramic imaging based on camera array
CN108280805B (en) * 2018-01-30 2021-07-20 北京理工大学 Image splicing method based on manifold optimization
CN108416732A (en) * 2018-02-02 2018-08-17 重庆邮电大学 A kind of Panorama Mosaic method based on image registration and multi-resolution Fusion

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105389787A (en) * 2015-09-30 2016-03-09 华为技术有限公司 Panorama image stitching method and device

Also Published As

Publication number Publication date
CN109961398A (en) 2019-07-02

Similar Documents

Publication Publication Date Title
CN109961398B (en) Fan blade image segmentation and grid optimization splicing method
CN107133943B (en) A kind of visible detection method of stockbridge damper defects detection
WO2018028103A1 (en) Unmanned aerial vehicle power line inspection method based on characteristics of human vision
CN110070530B (en) Transmission line icing detection method based on deep neural network
CN113065558A (en) Lightweight small target detection method combined with attention mechanism
CN109410207A (en) A kind of unmanned plane line walking image transmission line faultlocating method based on NCC feature
CN110163177B (en) Unmanned aerial vehicle automatic sensing and identifying method for wind turbine generator blades
CN109859171A (en) A kind of flooring defect automatic testing method based on computer vision and deep learning
CN108010030A (en) A kind of Aerial Images insulator real-time detection method based on deep learning
CN110189315B (en) Unmanned aerial vehicle automatic positioning method for blades of wind turbine generator
CN111008961B (en) Transmission line equipment defect detection method and system, equipment and medium thereof
CN107679495B (en) Detection method for movable engineering vehicles around power transmission line
CN109544501A (en) A kind of transmission facility defect inspection method based on unmanned plane multi-source image characteristic matching
CN114973032A (en) Photovoltaic panel hot spot detection method and device based on deep convolutional neural network
CN113643365B (en) Camera pose estimation method, device, equipment and readable storage medium
CN113569981A (en) Power inspection bird nest detection method based on single-stage target detection network
CN111861866A (en) Panoramic reconstruction method for substation equipment inspection image
CN112258490A (en) Low-emissivity coating intelligent damage detection method based on optical and infrared image fusion
CN111683221B (en) Real-time video monitoring method and system for natural resources embedded with vector red line data
CN115908354A (en) Photovoltaic panel defect detection method based on double-scale strategy and improved YOLOV5 network
Yang et al. Towards accurate image stitching for drone-based wind turbine blade inspection
CN114445615A (en) Rotary insulator target detection method based on scale invariant feature pyramid structure
Bellavia et al. Image orientation with a hybrid pipeline robust to rotations and wide-baselines
CN110321869A (en) Personnel&#39;s detection and extracting method based on Multiscale Fusion network
CN112686120A (en) Power transmission line anomaly detection method based on unmanned aerial vehicle aerial image

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant