CN112580711B - Video image fusion method based on wind turbine generator fault recognition - Google Patents
Video image fusion method based on wind turbine generator fault recognition Download PDFInfo
- Publication number
- CN112580711B CN112580711B CN202011471166.3A CN202011471166A CN112580711B CN 112580711 B CN112580711 B CN 112580711B CN 202011471166 A CN202011471166 A CN 202011471166A CN 112580711 B CN112580711 B CN 112580711B
- Authority
- CN
- China
- Prior art keywords
- image
- fusion
- entropy
- level
- pyramid
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000007500 overflow downdraw method Methods 0.000 title claims abstract description 15
- 230000004927 fusion Effects 0.000 claims abstract description 70
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 27
- 238000009432 framing Methods 0.000 claims abstract description 23
- 238000004364 calculation method Methods 0.000 claims abstract description 11
- 238000000354 decomposition reaction Methods 0.000 claims abstract description 8
- 230000003993 interaction Effects 0.000 claims abstract description 7
- 238000006243 chemical reaction Methods 0.000 claims abstract description 5
- 238000007499 fusion processing Methods 0.000 claims abstract description 4
- 238000000034 method Methods 0.000 claims description 25
- 238000012545 processing Methods 0.000 claims description 21
- 230000006870 function Effects 0.000 claims description 17
- 230000008569 process Effects 0.000 claims description 9
- 238000004590 computer program Methods 0.000 claims description 8
- 230000009466 transformation Effects 0.000 claims description 6
- 238000010276 construction Methods 0.000 claims description 5
- 230000000694 effects Effects 0.000 claims description 4
- 238000001914 filtration Methods 0.000 claims description 4
- 238000005286 illumination Methods 0.000 claims description 4
- 230000006835 compression Effects 0.000 claims description 3
- 238000007906 compression Methods 0.000 claims description 3
- 238000011156 evaluation Methods 0.000 claims description 3
- 238000013441 quality evaluation Methods 0.000 claims description 3
- 238000000844 transformation Methods 0.000 claims description 3
- 230000000739 chaotic effect Effects 0.000 claims description 2
- 238000004088 simulation Methods 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 3
- 230000006399 behavior Effects 0.000 description 2
- 230000015572 biosynthetic process Effects 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 238000004880 explosion Methods 0.000 description 2
- 238000003384 imaging method Methods 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 description 1
- 230000002159 abnormal effect Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 230000002349 favourable effect Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000002265 prevention Effects 0.000 description 1
- 230000000644 propagated effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
- 230000002123 temporal effect Effects 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/251—Fusion techniques of input or preprocessed data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/40—Software arrangements specially adapted for pattern recognition, e.g. user interfaces or toolboxes therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
- G06V10/462—Salient features, e.g. scale invariant feature transforms [SIFT]
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02E—REDUCTION OF GREENHOUSE GAS [GHG] EMISSIONS, RELATED TO ENERGY GENERATION, TRANSMISSION OR DISTRIBUTION
- Y02E10/00—Energy generation through renewable energy sources
- Y02E10/70—Wind energy
- Y02E10/72—Wind turbines with rotation axis in wind direction
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Cheminformatics (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Human Computer Interaction (AREA)
- Multimedia (AREA)
- Image Processing (AREA)
Abstract
The disclosure provides a video image fusion method based on wind turbine generator fault identification, comprising the following steps: collecting video image information of a wind turbine generator; carrying out image framing treatment on the video image information; performing gray level conversion on the image subjected to framing treatment, and performing pixel level image fusion under wavelet decomposition to obtain a first fusion image; carrying out pixel-level image fusion under the Laplacian pyramid on the image subjected to framing treatment to obtain a second fusion image; a SIFT algorithm is adopted on the image after framing treatment, and a third fusion image under the characteristic level is obtained; calculating entropy, joint entropy and root mean square error of the three fusion images; designing a GUI man-machine interaction interface, embodying three fusion processing algorithms and index quantity calculation in the interface, obtaining a final fusion image through the obtained index value, and carrying out fault identification on the wind turbine generator by using the final fusion image. The fault state of the fan external equipment can be better identified, and the fan external equipment is not influenced by a shielding object under a single angle.
Description
Technical Field
The disclosure belongs to the technical field of wind turbine generator fault identification, and particularly relates to a video image fusion method based on wind turbine generator fault identification.
Background
The safety of the external equipment of the wind power equipment is not only that the external equipment is effectively treated after the formation and the explosion of the crisis, but also that the possibility factor of wind power accidents possibly generated by early warning is predicted, so that the formation and the explosion of the crisis are fundamentally prevented. The external equipment of the wind turbine generator system must adhere to the principle of combining prevention and emergency, and particularly, the advanced prediction and recognition capability of the system should be paid attention to. The method is favorable for preparing, early warning and other basic works for enhancing emergency management of the crisis event, improves the early warning and precaution capability of the crisis event, and fully realizes the organic combination of early warning and emergency, normal management and abnormal management. In addition, because the impellers, the blades and the tower of the wind turbine generator are in the sites with high wind power and high wind speed for a long time, the external environment is bad, and therefore the fault probability of external equipment is high.
In addition, in recent years, video applications have become more and more widespread. With the continuous progress of technology, the improvement of living standard of people and the arrival of networked age, video becomes an important means for people to know the current events and perform entertainment activities. Meanwhile, video monitoring is not available in various aspects such as public security case breaking, traffic monitoring, safety production and the like. Because video has continuity that a single image does not have, video images have a tremendous amount of information, which has both spatial and temporal resolution capabilities, but it is very complex and difficult to extract and mine out such information. And the video is used as an information source with strong continuity, and the processing means is far less than the processing of a single image, so that the conversion of the video into the image for processing is imperative.
Disclosure of Invention
The video image fusion method based on wind turbine generator fault identification aims at solving at least one of the technical problems existing in the prior art.
In one aspect of the disclosure, a video image fusion method based on wind turbine fault identification is provided, the method comprising:
collecting video image information of a wind turbine generator;
carrying out image framing treatment on the video image information to obtain multiple images of multiple sources or single source;
performing gray level conversion on the image subjected to the framing treatment, and performing pixel level image fusion under wavelet decomposition on the converted gray level image to obtain a first fusion image;
carrying out pixel-level image fusion under the Laplacian pyramid on the image subjected to the framing treatment to obtain a second fusion image;
a SIFT algorithm is adopted on the image after framing treatment, and a third fusion image under the characteristic level is obtained;
calculating entropy, joint entropy and root mean square error of the first fusion image, the second fusion image and the third fusion image;
designing a GUI man-machine interaction interface, embodying three fusion processing algorithms and calculation of index quantity in the GUI man-machine interaction interface, obtaining a final fusion image through the obtained index value, and carrying out fault identification on the wind turbine by using the final fusion image.
In some optional embodiments, the performing pixel-level image fusion under wavelet decomposition on the converted gray-scale image to obtain a first fused image includes:
and projecting the converted gray level image onto a group of wavelet functions, and decomposing the gray level image into superposition of the group of wavelet functions to obtain the first fusion image.
In some optional embodiments, the performing pixel-level image fusion under the laplacian pyramid on the image after the framing process to obtain a second fused image includes:
taking the image after framing as an original image, taking the original image as a 0 th layer of a Gaussian pyramid, and assuming the original image as G0, and convolving the original image by using a Gaussian kernel w, wherein the Gaussian kernel is represented by the following formula (1):
the image obtained by downsampling after convolution is used as a layer 1G 1 of the image tower, so that after the image is processed, the lower layer image can be changed into 4 times of the upper layer image in size; taking the obtained image as an input image, and repeatedly convoluting and downsampling to obtain images of the 1 st layer to the N th layer to form a pyramid-type Gaussian image tower;
convolving and upsampling the image of the upper layer of the Gaussian pyramid to obtain a predicted image, namely the following formula (2):
the magnification operator Expand can be expressed as the following formulas (3) to (5):
0<l≤N,0≤i<R l ,0≤j<C l (4)
wherein G is l Representing the first level of the pyramid of the original image, G l * Representing the first level of the predicted image pyramid, i, j representing the behavior i, j, respectively, of the image, m, N representing the number of rows and columns, respectively, of the Gaussian kernel, N representing the highest level of the pyramid, R l Represents the maximum number of lines, C, in the image l Representing the maximum number of columns in the image;
the formulas (3) to (5) realize that even rows and columns deleted in the Gaussian construction process are inserted into 0, and then the Gaussian kernel w is used for convolution, namely filtering processing is carried out, so that an image with the same size as that before downsampling is obtained as a predicted image;
subtracting the predicted image from the next layer image to obtain a difference image, and repeatedly iterating to obtain a series of decomposed images which are arranged into pyramids, namely Laplacian pyramids;
fusing all layers of unified processing to obtain a corresponding image tower for image reconstruction to obtain the second fused image, wherein the unified processing is as follows: the layers except the top layer take the largest absolute value, and the coefficients of the highest layer are averaged.
In some optional embodiments, the performing a SIFT algorithm on the image after framing to obtain a third fused image at a feature level includes:
and (3) detecting a scale space extremum: searching image positions on all scales, and identifying potential interest points which are unchanged for the scales and the rotation through a Gaussian differential function;
positioning key points: determining the position and the scale at each candidate position by fitting a fine model, and selecting key points according to the stability degree of the key points;
and (3) direction determination: assigning one or more directions to each keypoint location based on the direction of the gradient of the image portion; all subsequent manipulations of the image data are transformed with respect to the orientation, scale and position of the keypoints, thereby providing invariance to these transformations;
key point description: measuring gradients of the image local at selected scales within a neighborhood around each keypoint, the gradients being transformed into a representation that allows for deformation and illumination variation of relatively large local shapes;
and performing image fusion based on a decomposed four-step SIFT algorithm to obtain the third fusion image under the feature level.
In some optional embodiments, the calculating entropy, joint entropy, and root mean square error of the first fused image, the second fused image, and the third fused image includes:
entropy is taken as a parameter for thermodynamically representing the state of an object, and describes the chaotic degree of a system; the use of entropy for image evaluation refers to the degree of information enrichment contained in an image, and if the entropy increases, it means that the image obtains a larger amount of information by processing, and the entropy of image a is defined as the following formula (6):
wherein n is the gray level of the image, generally 256, and pA is the proportion of the pixel point with the gray level of j in the image;
the joint entropy generally characterizes the amount of information transferred from the source image to the fused image, i.e. the joint information between the two images, the joint entropy of image a being defined as the following equation (7):
E(M,N)＝-∑ m,n p MN (m,n)log 2 p MN (m,n) (7)
according to the definition of the joint entropy between the two images, another calculation mode of mutual information between the two images can be obtained; the entropy of the image M is E (M), the entropy of the image N is E (N), and the joint entropy between the two images is E (M, N), then the mutual information between the image M and the image N can be calculated by the following formula (8):
MI(M,N)＝E(M)+E(N)-E(M,N) (8)
the peak signal-to-noise ratio is used as a quality evaluation index based on a reference image, is mainly used for judging the fidelity of the image, and is also often used as a measuring method for the signal reconstruction quality in the fields of image compression and the like; it has a great relation to the mean square error, often defined simply by mean square error; the effect and quality of image fusion are shown in that the peak signal-to-noise ratio is large and the mean square error is small; the mean square error between image M and image N can be defined as the following equation (9):
the mean square error can be used to derive the peak signal-to-noise ratio as follows (10):
in another aspect of the present disclosure, there is provided an electronic device including:
one or more processors;
a storage unit for storing one or more programs which, when executed by the one or more processors, enable the one or more processors to implement the method according to the preceding description.
In another aspect of the disclosure, a computer readable storage medium is provided, on which a computer program is stored, which computer program, when being executed by a processor, is capable of implementing the method according to the preceding description.
According to the video image fusion method based on wind turbine generator fault identification, aiming at faults of a fan blade, an impeller and a tower, a video sensor can be installed from the outside; the state images of the external equipment under multiple angles are fused through video image framing to synthesize a panoramic image of the external equipment, so that the fault state of the external equipment of the fan can be better identified, and the influence of a shielding object under a single angle is avoided; under the condition that the sensor shakes due to the large external wind force, the final fault identification image can still be well obtained. In addition, in scientific research and engineering application in the field of automatic control, a great deal of complicated calculation and simulation curve drawing tasks exist, and the time and the workload for calculating and drawing the simulation curve can be greatly saved by adopting GUI interface design programming, and errors in manual fault identification can be reduced.
Drawings
FIG. 1 is a block diagram schematically illustrating an example electronic device for implementing a video image fusion method based on wind turbine fault identification in an embodiment of the disclosure;
FIG. 2 is a flowchart of a video image fusion method based on wind turbine fault identification in another embodiment of the present disclosure;
FIG. 3 is a flowchart of an image fusion algorithm based on wavelet transform according to another embodiment of the present disclosure;
FIG. 4 is a flowchart of a Laplacian pyramid-based image fusion algorithm in accordance with another embodiment of the present disclosure;
fig. 5 is a flow chart of a feature-level-based image fusion algorithm according to another embodiment of the present disclosure.
Detailed Description
In order that those skilled in the art will better understand the technical solutions of the present disclosure, the present disclosure will be described in further detail with reference to the accompanying drawings and detailed description.
First, an example electronic device for implementing a video image fusion method based on wind turbine fault recognition according to an embodiment of the present disclosure is described with reference to fig. 1.
As shown in fig. 1, electronic device 200 includes one or more processors 210, one or more storage devices 220, one or more input devices 230, one or more output devices 240, etc., interconnected by a bus system 250 and/or other forms of connection mechanisms. It should be noted that the components and structures of the electronic device shown in fig. 1 are exemplary only and not limiting, as the electronic device may have other components and structures as desired.
Processor 210 may be a Central Processing Unit (CPU), or may be a processing unit that is comprised of multiple processing cores, or has data processing capabilities and/or instruction execution capabilities, and may control other components in electronic device 200 to perform desired functions.
The storage 220 may include one or more computer program products that may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory. The volatile memory may include, for example, random Access Memory (RAM) and/or cache memory (cache), and the like. The non-volatile memory may include, for example, read Only Memory (ROM), hard disk, flash memory, and the like. One or more computer program instructions may be stored on the computer readable storage medium that can be executed by a processor to perform client functions and/or other desired functions in the disclosed embodiments (implemented by the processor) as described below. Various applications and various data, such as various data used and/or generated by the applications, may also be stored in the computer readable storage medium.
The input device 230 may be a device used by a user to input instructions and may include one or more of a keyboard, a mouse, a microphone, a touch screen, and the like.
The output device 240 may output various information (e.g., images or sounds) to the outside (e.g., a user), and may include one or more of a display, a speaker, and the like.
Next, a video image fusion method based on wind turbine generator fault recognition according to another embodiment of the present disclosure will be described with reference to fig. 2.
As shown in fig. 2, a video image fusion method S100 based on wind turbine generator fault recognition, the method includes:
s110, collecting video image information of the wind turbine generator.
Specifically, in this step, image acquisition devices such as a camera may be used to collect video image information of the wind turbine, for example, a micro camera may be installed around the wind turbine external blade, the impeller and the tower, and multiple cameras may be used to perform image fusion on the external device video.
And S120, carrying out image framing processing on the video image information to obtain multiple images or multiple images with single source.
S130, carrying out gray level conversion on the image subjected to the framing treatment, and carrying out pixel level image fusion under wavelet decomposition on the converted gray level image to obtain a first fusion image.
In particular, wavelet transformation refers to projecting a signal onto a set of wavelet functions and decomposing the signal into a superposition of the series of wavelet functions, which are scaled and translated from the basic wavelet functions. In this step, the converted gray-scale image is projected onto a set of wavelet functions, and the gray-scale image is decomposed into a superposition of the set of wavelet functions, resulting in the first fused image, as shown in fig. 3.
And S140, performing pixel-level image fusion under the Laplacian pyramid on the image subjected to the framing treatment to obtain a second fusion image.
Specifically, in this step, a gaussian pyramid decomposition is first performed on the source image. The fusion principle of the image pyramid algorithm is that each image participating in fusion is decomposed into the same pyramid, the same pyramid is arranged from top to bottom according to the sequence from low resolution to high resolution, then the pyramids decomposed by all the images are fused on the corresponding layer according to a certain rule, the corresponding image tower is obtained, and the image reconstruction is carried out, so that the last fused image is obtained. Embodiments of the present disclosure employ a laplacian pyramid algorithm, where the laplacian pyramid is optimized from a gaussian pyramid.
The gaussian pyramid is the most basic image tower, the image after framing is taken as an original image, the original image is taken as a 0 th layer of the gaussian pyramid, the original image is assumed to be G0, and the original image is convolved by using a gaussian kernel w, wherein the gaussian kernel is represented by the following formula (1):
the image obtained by downsampling (removing even lines and columns) after convolution is used as a layer 1G 1 of the image tower, so that after the image is processed, the lower layer image can be 4 times as large as the upper layer image; taking the obtained image as an input image, and repeatedly convoluting and downsampling to obtain images of the 1 st layer to the N th layer to form a pyramid-type Gaussian image tower;
however, because the gaussian pyramid needs to be rolled and downsampled in the construction process, some high-frequency information is lost, and thus the new pyramid algorithm, namely the laplacian pyramid, is adopted in the embodiment of the disclosure. The construction of the laplacian pyramid is based on a gaussian pyramid. To construct a laplacian pyramid, the previous layer of image of the gaussian pyramid needs to be convolved and up-sampled to obtain a predicted image, namely the following formula (2):
the magnification operator Expand can be expressed as the following formulas (3) to (5):
0<l≤N,0≤i<R l ,0≤j<C l (4)
wherein G is l Representing the first level of the pyramid of the original image, G l * Representing the first level of the predicted image pyramid, i, j representing the behavior i, j, respectively, of the image, m, N representing the number of rows and columns, respectively, of the Gaussian kernel, N representing the highest level of the pyramid, R l Represents the maximum number of lines, C, in the image l Representing the maximum number of columns in the image;
the formulas (3) to (5) realize that even rows and columns deleted in the Gaussian construction process are inserted into 0, and then the Gaussian kernel w is used for convolution, namely filtering processing is carried out, so that an image with the same size as that before downsampling is obtained as a predicted image;
subtracting the predicted image from the next layer image to obtain a difference image, and repeatedly iterating to obtain a series of decomposed images which are arranged into pyramids, namely Laplacian pyramids;
fusing all layers of unified processing to obtain a corresponding image tower for image reconstruction to obtain the second fused image, wherein the unified processing is as follows: the layers except the top layer take the largest absolute value, and the coefficients of the highest layer are averaged.
A specific flow of image fusion using a laplacian pyramid is shown in fig. 4.
And S150, adopting a SIFT algorithm to the image subjected to the framing processing to obtain a third fusion image under the characteristic level.
Specifically, in this step, when image fusion is performed, the image registration and the target tracking and recognition performance are affected by factors such as the self-state of the target, the environment in which the scene is located, and the imaging characteristics of the imaging equipment. And then the SIFT algorithm can solve the problem of image registration failure caused by object shielding, illumination influence, sundry scene, noise, rotation, scaling and translation of the object and affine projection transformation of the image to a certain extent.
The SIFT algorithm searches feature points in different scale spaces, and only through gaussian blur can different scale spaces be obtained. Gaussian blur corresponds to an image filtering which is also present in pyramid decomposition, with a unique linear kernel (gaussian convolution kernel) to achieve the scaling. The scale space of the image is defined by convolving the source image with a two-dimensional gaussian function of variable scale:
L(i,j,σ)＝G(i,j,kσ)*I(i,j)
where G (I, j, kσ) is a variable-scale gaussian function, I (I, j) is a source image, and k is a scale variation. The stability of the detected feature points is ensured by generating a Gaussian difference scale space by adopting different Gaussian difference kernels and image convolution:
D(i,j,σ)＝(G(i,j,kσ)-G(i,j,σ))*I(i,j)＝L(i,j,kσ)-L(i,j,σ)
a gaussian pyramid is constructed by a scale-space approach plus downsampling. In order to ensure the accuracy of feature point detection, the detected feature points need to be compared with 8 adjacent pixel points of the Gaussian pyramid layer and 9 adjacent pixel points of the upper layer and the lower layer. Feature point detection is achieved through a SIFT algorithm, and panoramic fusion of images can be achieved by fusing the obtained feature points. Fig. 5 clearly shows the process of achieving feature level fusion of images using SIFT algorithm.
More specifically, in the present embodiment, the SIFT algorithm is decomposed into four steps:
(1) And (3) detecting a scale space extremum: searching image positions on all scales, and identifying potential interest points which are unchanged for the scales and the rotation through a Gaussian differential function;
(2) Positioning key points: determining the position and the scale at each candidate position by fitting a fine model, and selecting key points according to the stability degree of the key points;
(3) And (3) direction determination: assigning one or more directions to each keypoint location based on the direction of the gradient of the image portion; all subsequent manipulations of the image data are transformed with respect to the orientation, scale and position of the keypoints, thereby providing invariance to these transformations;
(4) Key point description: within a neighborhood around each keypoint, gradients of the image local are measured on a selected scale, which gradients are transformed into a representation that allows for deformation and illumination variation of a relatively large local shape.
And performing image fusion based on a decomposed four-step SIFT algorithm to obtain the third fusion image under the feature level.
And S160, calculating entropy, joint entropy and root mean square error of the first fusion image, the second fusion image and the third fusion image.
Specifically, entropy is taken as a parameter for thermodynamically representing the state of an object, and describes the degree of confusion of a system; the use of entropy for image evaluation refers to the degree of information enrichment contained in an image, and if the entropy increases, it means that the image obtains a larger amount of information by processing, and the entropy of image a is defined as the following formula (6):
wherein n is the gray level of the image, generally 256, and pA is the proportion of the pixel point with the gray level of j in the image;
the joint entropy generally characterizes the amount of information transferred from the source image to the fused image, i.e. the joint information between the two images, the joint entropy of image a being defined as the following equation (7):
E(M,N)＝-∑ m,n p MN (m,n)log 2 p MN (m,n) (7)
according to the definition of the joint entropy between the two images, another calculation mode of mutual information between the two images can be obtained; the entropy of the image M is E (M), the entropy of the image N is E (N), and the joint entropy between the two images is E (M, N), then the mutual information between the image M and the image N can be calculated by the following formula (8):
MI(M,N)＝E(M)+E(N)-E(M,N) (8)
the peak signal-to-noise ratio is used as a quality evaluation index based on a reference image, is mainly used for judging the fidelity of the image, and is also often used as a measuring method for the signal reconstruction quality in the fields of image compression and the like; it has a great relation to the mean square error, often defined simply by mean square error; the effect and quality of image fusion are shown in that the peak signal-to-noise ratio is large and the mean square error is small; the mean square error between image M and image N can be defined as the following equation (9):
the mean square error can be used to derive the peak signal-to-noise ratio as follows (10):
the three indexes are calculated based on a single image, a source image and a reference image respectively, and the performance of the fusion image is judged more comprehensively by integrating the three indexes.
S170, designing a GUI man-machine interaction interface, embodying three fusion processing algorithms and index quantity calculation in the GUI man-machine interaction interface, obtaining a final fusion image through the obtained index value, and carrying out fault identification on the wind turbine generator by using the final fusion image.
According to the video image fusion method based on wind turbine generator fault recognition, a multi-source video is adopted to perform feature level image fusion aiming at the situation that a target object is blocked in the video shooting process, and a SIFT algorithm is specifically adopted; aiming at sensor shake in the video shooting process, pixel-level image fusion is adopted, and specifically, wavelet transformation and Laplacian pyramid image fusion algorithm are adopted. In addition, the fusion image with the maximum information quantity is obtained through calculation of three index quantities of entropy, joint entropy and root mean square error. And finally, constructing two types of video image fusion algorithms and the calculation of three index quantities into a GUI interface, so that the GUI interface is convenient to use in the actual fault identification process.
According to the video image fusion method based on wind turbine generator fault identification, aiming at faults of a fan blade, an impeller and a tower, a video sensor can be installed from the outside; the state images of the external equipment under multiple angles are fused through video image framing to synthesize a panoramic image of the external equipment, so that the fault state of the external equipment of the fan can be better identified, and the influence of a shielding object under a single angle is avoided; under the condition that the sensor shakes due to the large external wind force, the final fault identification image can still be well obtained. In addition, in scientific research and engineering application in the field of automatic control, a great deal of complicated calculation and simulation curve drawing tasks exist, and the time and the workload for calculating and drawing the simulation curve can be greatly saved by adopting GUI interface design programming, and errors in manual fault identification can be reduced.
In another aspect of the present disclosure, there is provided an electronic device including:
one or more processors;
and a storage unit for storing one or more programs, which when executed by the one or more processors, enable the one or more processors to implement the method according to the preceding description.
In another aspect of the disclosure, a computer readable storage medium is provided, on which a computer program is stored, which computer program, when being executed by a processor, is capable of implementing the method according to the preceding description.
Wherein the computer readable medium may be embodied in the apparatus, device, system of the present disclosure or may exist alone.
Wherein the computer readable storage medium may be any tangible medium that can contain, or store a program that can be an electronic, magnetic, optical, electromagnetic, infrared, semiconductor system, apparatus, device, more specific examples of which include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, an optical fiber, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.
The computer-readable storage medium may also include a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein, specific examples of which include, but are not limited to, electromagnetic signals, optical signals, or any suitable combination thereof.
It is to be understood that the above embodiments are merely exemplary embodiments employed to illustrate the principles of the present disclosure, however, the present disclosure is not limited thereto. Various modifications and improvements may be made by those skilled in the art without departing from the spirit and substance of the disclosure, and are also considered to be within the scope of the disclosure.
Claims (7)
1. The video image fusion method based on wind turbine generator fault identification is characterized by comprising the following steps:
collecting video image information of a wind turbine generator;
carrying out image framing treatment on the video image information to obtain multiple images of multiple sources or single source;
performing gray level conversion on the image subjected to the framing treatment, and performing pixel level image fusion under wavelet decomposition on the converted gray level image to obtain a first fusion image;
carrying out pixel-level image fusion under the Laplacian pyramid on the image subjected to the framing treatment to obtain a second fusion image;
a SIFT algorithm is adopted on the image after framing treatment, and a third fusion image under the characteristic level is obtained;
calculating entropy, joint entropy and root mean square error of the first fusion image, the second fusion image and the third fusion image;
designing a GUI man-machine interaction interface, embodying three fusion processing algorithms and calculation of index quantity in the GUI man-machine interaction interface, taking a fusion image with the maximum information quantity obtained through judgment as a final fusion image, and carrying out fault identification on the wind turbine generator by using the final fusion image.
2. The method according to claim 1, wherein the performing pixel-level image fusion under wavelet decomposition on the converted gray-scale image to obtain a first fused image includes:
and projecting the converted gray level image onto a group of wavelet functions, and decomposing the gray level image into superposition of the group of wavelet functions to obtain the first fusion image.
3. The method according to claim 1, wherein the performing pixel-level image fusion under a laplacian pyramid on the frame-processed image to obtain a second fused image includes:
taking the image after framing as an original image, taking the original image as a 0 th layer of a Gaussian pyramid, and assuming the original image as G0, and convolving the original image by using a Gaussian kernel w, wherein the Gaussian kernel is represented by the following formula (1):
the image obtained by downsampling after convolution is used as a layer 1G 1 of the image tower, so that after the image is processed, the lower layer image can be changed into 4 times of the upper layer image in size; taking the obtained image as an input image, and repeatedly convoluting and downsampling to obtain images of the 1 st layer to the N th layer to form a pyramid-type Gaussian image tower;
convolving and upsampling the image of the upper layer of the Gaussian pyramid to obtain a predicted image, namely the following formula (2):
the magnification operator Expand can be expressed as the following formulas (3) to (5):
0<l≤N,0≤i<R l ,0≤j<C l (4)
wherein G is l Representing the first level of the pyramid of the original image, G l * Representing the first level of the pyramid of the predicted image, i, j representing the behavior i and j of the pixel points in the image, m, N representing the number of rows and columns in the Gaussian kernel, N representing the highest level of the pyramid, R l Represents the maximum number of lines, C, in the image l Representing the maximum number of columns in the image;
the formulas (3) to (5) realize that even rows and columns deleted in the Gaussian construction process are inserted into 0, and then the Gaussian kernel w is used for convolution, namely filtering processing is carried out, so that an image with the same size as that before downsampling is obtained as a predicted image;
subtracting the predicted image from the next layer image to obtain a difference image, and repeatedly iterating to obtain a series of decomposed images which are arranged into pyramids, namely Laplacian pyramids;
fusing all layers of unified processing to obtain a corresponding image tower for image reconstruction to obtain the second fused image, wherein the unified processing is as follows: the layers except the top layer take the largest absolute value, and the coefficients of the highest layer are averaged.
4. A method according to any one of claims 1 to 3, wherein the performing a SIFT algorithm on the framed image to obtain a third fused image at a feature level comprises:
and (3) detecting a scale space extremum: searching image positions on all scales, and identifying potential interest points which are unchanged for the scales and the rotation through a Gaussian differential function;
positioning key points: determining the position and the scale at each candidate position by fitting a fine model, and selecting key points according to the stability degree of the key points;
and (3) direction determination: assigning one or more directions to each keypoint location based on the direction of the gradient of the image portion; all subsequent manipulations of the image data are transformed with respect to the orientation, scale and position of the keypoints, thereby providing invariance to these transformations;
key point description: measuring gradients of the image local at selected scales within a neighborhood around each keypoint, the gradients being transformed into a representation that allows deformation of the local shape and illumination variation;
and performing image fusion based on a decomposed four-step SIFT algorithm to obtain the third fusion image under the feature level.
5. A method according to any one of claims 1 to 3, wherein said calculating entropy, joint entropy and root mean square error of said first, second and third fused images comprises:
entropy is taken as a parameter for thermodynamically representing the state of an object, and describes the chaotic degree of a system; the use of entropy for image evaluation refers to the degree of information enrichment contained in an image, and if the entropy increases, it means that the image obtains a larger amount of information by processing, and the entropy of image a is defined as the following formula (6):
wherein n is the gray level of the image, generally 256, and pA is the proportion of the pixel point with the gray level of j in the image;
the joint entropy generally characterizes the amount of information transferred from the source image to the fused image, i.e. the joint information between the two images, the joint entropy of image a being defined as the following equation (7):
E(M,N)＝-∑ m,n p MN (m,n)log 2 p MN (m,n) (7)
according to the definition of the joint entropy between the two images, another calculation mode of mutual information between the two images can be obtained; the entropy of the image M is E (M), the entropy of the image N is E (N), and the joint entropy between the two images is E (M, N), then the mutual information between the image M and the image N can be calculated by the following formula (8):
MI(M,N)＝E(M)+E(N)-E(M,N) (8)
the peak signal-to-noise ratio is used as a quality evaluation index based on a reference image and is used for judging the fidelity of the image, and is also frequently used as a measuring method for the signal reconstruction quality in the fields of image compression and the like; it has a great relation to the mean square error, often defined simply by mean square error; the effect and quality of image fusion are shown in that the peak signal-to-noise ratio is large and the mean square error is small; the mean square error between image M and image N can be defined as the following equation (9):
the mean square error can be used to derive the peak signal-to-noise ratio as follows (10):
6. an electronic device, comprising:
one or more processors;
a storage unit for storing one or more programs, which when executed by the one or more processors, enable the one or more processors to implement the method of any one of claims 1 to 5.
7. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, is capable of realizing the method according to any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011471166.3A CN112580711B (en) | 2020-12-14 | 2020-12-14 | Video image fusion method based on wind turbine generator fault recognition |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011471166.3A CN112580711B (en) | 2020-12-14 | 2020-12-14 | Video image fusion method based on wind turbine generator fault recognition |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112580711A CN112580711A (en) | 2021-03-30 |
| CN112580711B true CN112580711B (en) | 2024-03-12 |
Family
ID=75135839
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011471166.3A Active CN112580711B (en) | 2020-12-14 | 2020-12-14 | Video image fusion method based on wind turbine generator fault recognition |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112580711B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11841926B2 (en) * | 2021-02-10 | 2023-12-12 | Apple Inc. | Image fusion processor circuit for dual-mode image fusion architecture |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102324033A (en) * | 2011-09-20 | 2012-01-18 | 吴建华 | Wind-powered electricity generation safe and intelligent early warning emergency system image processing method |
| CN106339998A (en) * | 2016-08-18 | 2017-01-18 | 南京理工大学 | Multi-focus image fusion method based on contrast pyramid transformation |
| CN108344574A (en) * | 2018-04-28 | 2018-07-31 | 湖南科技大学 | A kind of Wind turbines Method for Bearing Fault Diagnosis for combining adaptation network based on depth |
| CN111612030A (en) * | 2020-03-30 | 2020-09-01 | 华电电力科学研究院有限公司 | Wind turbine generator blade surface fault identification and classification method based on deep learning |
| CN111696067A (en) * | 2020-06-16 | 2020-09-22 | 桂林电子科技大学 | Gem image fusion method based on image fusion system |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8406569B2 (en) * | 2009-01-19 | 2013-03-26 | Sharp Laboratories Of America, Inc. | Methods and systems for enhanced dynamic range images and video from multiple exposures |
-
2020
- 2020-12-14 CN CN202011471166.3A patent/CN112580711B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102324033A (en) * | 2011-09-20 | 2012-01-18 | 吴建华 | Wind-powered electricity generation safe and intelligent early warning emergency system image processing method |
| CN106339998A (en) * | 2016-08-18 | 2017-01-18 | 南京理工大学 | Multi-focus image fusion method based on contrast pyramid transformation |
| CN108344574A (en) * | 2018-04-28 | 2018-07-31 | 湖南科技大学 | A kind of Wind turbines Method for Bearing Fault Diagnosis for combining adaptation network based on depth |
| CN111612030A (en) * | 2020-03-30 | 2020-09-01 | 华电电力科学研究院有限公司 | Wind turbine generator blade surface fault identification and classification method based on deep learning |
| CN111696067A (en) * | 2020-06-16 | 2020-09-22 | 桂林电子科技大学 | Gem image fusion method based on image fusion system |
Non-Patent Citations (2)
| Title |
|---|
| CNN Based Fault Recognition with Multi-Scale Fusion Attention Mechanism;Bo Wang等;《IOP Conference Series: Earth and Environmental Science》;第474卷;正文1-5 * |
| 基于时频图像识别的旋转机械多特征融合故障诊断方法研究;王维刚;《CNKI优秀博士学位论文全文库（工程科技II辑）》(第2期);C029-51 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112580711A (en) | 2021-03-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109671020B (en) | Image processing method, device, electronic equipment and computer storage medium | |
| WO2018166438A1 (en) | Image processing method and device and electronic device | |
| CN110998671B (en) | Three-dimensional reconstruction method, device, system and storage medium | |
| Mohammed et al. | Boosting image forgery detection using resampling features and copy-move analysis | |
| CN113920538B (en) | Object detection method, device, equipment, storage medium and computer program product | |
| CN111415300A (en) | Splicing method and system for panoramic image | |
| CN115937794B (en) | Small target object detection method and device, electronic equipment and storage medium | |
| CN112270246B (en) | Video behavior recognition method and device, storage medium and electronic equipment | |
| Flenner et al. | Resampling forgery detection using deep learning and a-contrario analysis | |
| CN117197669A (en) | Remote sensing image target detection method and system for improving YOLOV8 algorithm | |
| CN112580711B (en) | Video image fusion method based on wind turbine generator fault recognition | |
| CN112419342A (en) | Image processing method, image processing device, electronic equipment and computer readable medium | |
| CN116503399A (en) | Insulator pollution flashover detection method based on YOLO-AFPS | |
| CN108229281B (en) | Neural network generation method, face detection device and electronic equipment | |
| CN114926734A (en) | Solid waste detection device and method based on feature aggregation and attention fusion | |
| CN112800932B (en) | Method for detecting remarkable ship target in offshore background and electronic equipment | |
| CN113688928A (en) | Image matching method and device, electronic equipment and computer readable medium | |
| CN117593702A (en) | Remote monitoring method, device, equipment and storage medium | |
| CN115861922B (en) | Sparse smoke detection method and device, computer equipment and storage medium | |
| Choi et al. | Deep‐learning‐based nuclear power plant fault detection using remote light‐emitting diode array data transmission | |
| Xu et al. | A two-stage noise level estimation using automatic feature extraction and mapping model | |
| CN116777788A (en) | Remote sensing image cloud removal method based on image fusion feature learning reconstruction | |
| CN116152633A (en) | Detection method and system of target detection network based on spatial feature representation | |
| Patel et al. | ThermISRnet: an efficient thermal image super-resolution network | |
| CN113869163B (en) | Target tracking method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |