CN112580711A - Video image fusion method based on wind turbine generator fault recognition - Google Patents
Video image fusion method based on wind turbine generator fault recognition Download PDFInfo
- Publication number
- CN112580711A CN112580711A CN202011471166.3A CN202011471166A CN112580711A CN 112580711 A CN112580711 A CN 112580711A CN 202011471166 A CN202011471166 A CN 202011471166A CN 112580711 A CN112580711 A CN 112580711A
- Authority
- CN
- China
- Prior art keywords
- image
- fusion
- entropy
- level
- pyramid
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000007500 overflow downdraw method Methods 0.000 title claims abstract description 15
- 230000004927 fusion Effects 0.000 claims abstract description 63
- 238000012545 processing Methods 0.000 claims abstract description 38
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 27
- 238000009432 framing Methods 0.000 claims abstract description 21
- 238000000354 decomposition reaction Methods 0.000 claims abstract description 11
- 238000004364 calculation method Methods 0.000 claims abstract description 9
- 230000003993 interaction Effects 0.000 claims abstract description 7
- 238000006243 chemical reaction Methods 0.000 claims abstract description 4
- 238000007499 fusion processing Methods 0.000 claims abstract description 4
- 238000000034 method Methods 0.000 claims description 26
- 230000006870 function Effects 0.000 claims description 15
- 238000003860 storage Methods 0.000 claims description 13
- 230000008569 process Effects 0.000 claims description 9
- 230000009466 transformation Effects 0.000 claims description 6
- 238000004590 computer program Methods 0.000 claims description 5
- 238000010276 construction Methods 0.000 claims description 5
- 238000001514 detection method Methods 0.000 claims description 5
- 230000000694 effects Effects 0.000 claims description 4
- 238000001914 filtration Methods 0.000 claims description 4
- 238000005286 illumination Methods 0.000 claims description 4
- 230000006835 compression Effects 0.000 claims description 3
- 238000007906 compression Methods 0.000 claims description 3
- 238000011156 evaluation Methods 0.000 claims description 3
- 238000013441 quality evaluation Methods 0.000 claims description 3
- 238000000844 transformation Methods 0.000 claims description 3
- 238000005070 sampling Methods 0.000 claims description 2
- 238000004088 simulation Methods 0.000 description 4
- 238000013461 design Methods 0.000 description 2
- 238000003384 imaging method Methods 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 description 1
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000005065 mining Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 230000002265 prevention Effects 0.000 description 1
- 230000000644 propagated effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000005728 strengthening Methods 0.000 description 1
- 230000002123 temporal effect Effects 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/251—Fusion techniques of input or preprocessed data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/40—Software arrangements specially adapted for pattern recognition, e.g. user interfaces or toolboxes therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
- G06V10/462—Salient features, e.g. scale invariant feature transforms [SIFT]
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02E—REDUCTION OF GREENHOUSE GAS [GHG] EMISSIONS, RELATED TO ENERGY GENERATION, TRANSMISSION OR DISTRIBUTION
- Y02E10/00—Energy generation through renewable energy sources
- Y02E10/70—Wind energy
- Y02E10/72—Wind turbines with rotation axis in wind direction
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Computational Biology (AREA)
- General Engineering & Computer Science (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Multimedia (AREA)
- Human Computer Interaction (AREA)
- Image Processing (AREA)
Abstract
The invention provides a video image fusion method based on wind turbine generator fault identification, which comprises the following steps: collecting video image information of a wind turbine generator; performing image framing processing on the video image information; performing gray level conversion on the image subjected to the framing processing, and performing pixel level image fusion under wavelet decomposition to obtain a first fusion image; performing pixel-level image fusion under a Laplacian pyramid on the image subjected to the framing processing to obtain a second fusion image; adopting an SIFT algorithm to the image subjected to framing processing to obtain a third fusion image under the characteristic level; calculating the entropy, the joint entropy and the root-mean-square error of the three fused images; designing a GUI (graphical user interface) man-machine interaction interface, embodying the three fusion processing algorithms and the calculation of the index quantity in the interface, obtaining a final fusion image through the obtained index value, and identifying the fault of the wind turbine generator by using the final fusion image. The fault state of the fan external equipment can be better identified, and the influence of a shelter under a single angle is avoided.
Description
Technical Field
The disclosure belongs to the technical field of wind turbine generator fault identification, and particularly relates to a video image fusion method based on wind turbine generator fault identification.
Background
The safety of the external equipment of the wind power equipment is not only effectively treated after the crisis is formed and outbreak, but also is more important to predict and early warn possible factors which may cause wind power accidents, so that the formation and outbreak of the crisis are fundamentally prevented. The external equipment of the wind turbine generator must adhere to the principle of combining prevention and emergency, and particularly the advance prediction identification capability of the system should be emphasized. The method is beneficial to doing basic work such as preparation, early warning and the like for strengthening emergency management of the crisis event, improving the early warning and precautionary capability of the crisis event and fully realizing the organic combination of the early warning and emergency, normal state management and abnormal state management. In addition, the impeller, the blades and the tower of the wind turbine generator are in a place with high wind power and high wind speed for a long time, and the external environment is severe, so that the fault probability of external equipment is high.
In addition, in recent years, video applications have become more widespread. With the continuous progress of science and technology, the improvement of living standard of people and the coming of the networking era, video gradually becomes an important means for people to know matters and carry out entertainment activities. Meanwhile, video monitoring is not available in various aspects such as public security solution, traffic monitoring, safety production and the like. Since video has continuity which a single image does not have, video images have a huge amount of information, and the video images have both spatial resolution and temporal resolution, but actually extracting and mining the information is complex and difficult. Moreover, the video is used as an information source with strong continuity, and the processing means is far less than that of a single image, so that the video is converted into the image to be processed imperatively.
Disclosure of Invention
The present disclosure is directed to at least one of the technical problems in the prior art, and provides a video image fusion method based on wind turbine generator fault identification.
One aspect of the present disclosure provides a video image fusion method based on wind turbine generator fault identification, where the method includes:
collecting video image information of a wind turbine generator;
performing image framing processing on the video image information to obtain a multi-source or single-source multi-image;
performing gray level conversion on the image subjected to the framing processing, and performing pixel level image fusion under wavelet decomposition on the converted gray level image to obtain a first fusion image;
performing pixel-level image fusion under a Laplacian pyramid on the image subjected to the framing processing to obtain a second fusion image;
adopting an SIFT algorithm to the image subjected to framing processing to obtain a third fusion image under a characteristic level;
calculating entropy, joint entropy and root mean square error of the first fused image, the second fused image and the third fused image;
designing a GUI (graphical user interface) man-machine interaction interface, embodying three fusion processing algorithms and the calculation of an index quantity in the GUI man-machine interaction interface, obtaining a final fusion image through the obtained index value, and carrying out fault identification on the wind turbine generator by utilizing the final fusion image.
In some optional embodiments, the performing pixel-level image fusion under wavelet decomposition on the converted grayscale image to obtain a first fused image includes:
and projecting the converted gray level image onto a group of wavelet functions, and decomposing the gray level image into superposition of the group of wavelet functions to obtain the first fusion image.
In some optional embodiments, the performing pixel-level image fusion under a laplacian pyramid on the image after the framing processing to obtain a second fused image includes:
the image after the frame division processing is used as an original image, the original image is used as the 0 th layer of the Gaussian pyramid, the original image is assumed to be G0, and the original image is convoluted by a Gaussian kernel w, wherein the Gaussian kernel is expressed by the following formula (1):
the convolution is followed by down-sampling the resulting image as layer 1G 1 of the image tower, so that after the image is processed, the lower layer image becomes 4 times the size of the upper layer image; taking the obtained image as an input image, and repeatedly performing convolution and downsampling to obtain images of the 1 st layer to the Nth layer, wherein the images become a pyramid-type Gaussian image tower;
convolving and upsampling the upper layer image of the Gaussian pyramid to obtain a predicted image, namely the predicted image is represented by the following formula (2):
the enlargement operator Expand can be expressed by the following formulas (3) to (5):
0<l≤N,0≤i<Rl,0≤j<Cl (4)
wherein G islRepresenting the first level of the original image pyramid, Gl *Representing the ith level of the pyramid of the predicted image, i, j respectively representing the pixel points of the row i and the column j in the image, m, N respectively representing the number of rows and the column in the Gaussian kernel, N representing the highest level of the pyramid, RlRepresenting the maximum number of lines in the image, ClRepresents the sameMaximum number of columns in the image;
the equations (3) to (5) realize that even rows and columns deleted in the Gaussian construction process are inserted into 0, and then the Gaussian kernel w is used for convolution, namely, filtering processing is carried out, and an image with the same size as that before downsampling is obtained and is a predicted image;
subtracting the predicted image from the next layer of image to obtain a difference image, and repeating iteration to obtain a series of decomposition images which are arranged into a pyramid, namely a Laplacian pyramid;
and performing unified processing on all layers for fusion to obtain a corresponding image tower for image reconstruction to obtain the second fusion image, wherein the unified processing comprises the following steps: the maximum absolute value of each layer except the top layer is obtained, and the coefficient of the highest layer is averaged.
In some optional embodiments, the obtaining a third fused image at a feature level by using a SIFT algorithm on the image after the framing processing includes:
and (3) detection of extreme values in the scale space: searching image positions on all scales, and identifying potential interest points which are invariable in scale and rotation through a Gaussian differential function;
key point positioning: at each candidate location, determining the location and scale by fitting a refined model, the key points being selected based on their degree of stability;
direction determination: assigning one or more directions to each keypoint location based on the local gradient direction of the image; all subsequent operations on the image data are transformed with respect to the orientation, scale and location of the keypoints, thereby providing invariance to these transformations;
description of key points: measuring local gradients of the image at a selected scale in a neighborhood around each keypoint, the gradients being transformed into a representation that allows for relatively large local shape distortions and illumination variations;
and performing image fusion based on a decomposed four-step SIFT algorithm to obtain the third fusion image under the characteristic level.
In some optional embodiments, the calculating of entropy, joint entropy, and root mean square error for the first fused image, the second fused image, and the third fused image comprises:
the entropy is used as a parameter for thermodynamically representing the state of an object and describes the chaos degree of a system; the entropy used for image evaluation refers to the degree of information richness contained in an image, and if the entropy is increased, the image obtains more information amount through processing, and the entropy of the image A is defined as the following formula (6):
wherein n is the gray level of the image, generally 256, and pA is the proportion of the pixel point with the gray level j in the image;
joint entropy generally represents the amount of information transferred from a source image to a fused image, i.e., the joint information between two images, and the joint entropy of image a is defined by the following equation (7):
E(M,N)＝-∑m,npMN(m,n)log2pMN(m,n) (7)
according to the definition of the joint entropy between the two images, another calculation mode of mutual information between the two images can be obtained; the entropy of the image M is E (M), the entropy of the image N is E (N), and the joint entropy between the two images is E (M, N), then the mutual information between the image M and the image N can be calculated by the following formula (8):
MI(M,N)＝E(M)+E(N)-E(M,N) (8)
the peak signal-to-noise ratio is used as a quality evaluation index based on a reference image, is mainly used for judging the fidelity of the image, and is also often used as a measuring method of signal reconstruction quality in the fields of image compression and the like; it has a large relationship with the mean square error, often defined simply by the mean square error; the effect and quality of image fusion are reflected in that the peak signal-to-noise ratio is large, and the mean square error is small; the mean square error between image M and image N may be defined as the following equation (9):
the peak signal-to-noise ratio can be derived from the mean square error as the following equation (10):
in another aspect of the present disclosure, an electronic device is provided, including:
one or more processors;
a storage unit for storing one or more programs which, when executed by the one or more processors, enable the one or more processors to implement the method according to the preceding description.
In another aspect of the disclosure, a computer-readable storage medium is provided, on which a computer program is stored, which, when being executed by a processor, is adapted to carry out the method according to the above.
According to the video image fusion method based on the wind turbine generator fault identification, aiming at the faults of the fan blade, the impeller and the tower, a video sensor can be installed from the outside; by framing the video images, state images of the external equipment at multiple angles are fused to synthesize a panoramic image of the external equipment, so that the fault state of the external equipment of the fan can be better identified, and the influence of a shelter at a single angle is avoided; under the condition that the external wind force is large, so that the sensor shakes, the final fault identification image can still be well obtained. In addition, in scientific research and engineering application in the field of automatic control, a large number of complex tasks of calculating and drawing simulation curves exist, the time and the workload for calculating and drawing the simulation curves can be greatly saved by adopting GUI interface design programming, and errors in manual fault identification can be reduced.
Drawings
FIG. 1 is a block diagram illustrating components of an exemplary electronic device for implementing a video image fusion method based on wind turbine generator fault identification according to an embodiment of the present disclosure;
FIG. 2 is a flowchart of a video image fusion method based on wind turbine generator fault identification according to another embodiment of the present disclosure;
FIG. 3 is a flowchart of an image fusion algorithm based on wavelet transform according to another embodiment of the present disclosure;
FIG. 4 is a flow chart of an image fusion algorithm based on Laplacian pyramid according to another embodiment of the disclosure;
fig. 5 is a flowchart of an image fusion algorithm based on feature level according to another embodiment of the present disclosure.
Detailed Description
For a better understanding of the technical aspects of the present disclosure, reference is made to the following detailed description taken in conjunction with the accompanying drawings.
First, an example electronic device for implementing a video image fusion method based on wind turbine generator fault identification according to an embodiment of the present disclosure is described with reference to fig. 1.
As shown in FIG. 1, electronic device 200 includes one or more processors 210, one or more memory devices 220, one or more input devices 230, one or more output devices 240, and the like, interconnected by a bus system 250 and/or other form of connection mechanism. It should be noted that the components and structures of the electronic device shown in fig. 1 are exemplary only, and not limiting, and the electronic device may have other components and structures as desired.
The processor 210 may be a Central Processing Unit (CPU), or may be made up of multiple processing cores, or other forms of processing unit having data processing capabilities and/or instruction execution capabilities, and may control other components in the electronic device 200 to perform desired functions.
Storage 220 may include one or more computer program products that may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory. The volatile memory may include, for example, Random Access Memory (RAM), cache memory (cache), and/or the like. The non-volatile memory may include, for example, Read Only Memory (ROM), hard disk, flash memory, etc. On which one or more computer program instructions may be stored that a processor may execute to implement the client functionality (implemented by the processor) in the embodiments of the disclosure described below and/or other desired functionality. Various applications and various data, such as various data used and/or generated by the applications, may also be stored in the computer-readable storage medium.
The input device 230 may be a device used by a user to input instructions and may include one or more of a keyboard, a mouse, a microphone, a touch screen, and the like.
The output device 240 may output various information (e.g., images or sounds) to an outside (e.g., a user), and may include one or more of a display, a speaker, and the like.
Next, a video image fusion method based on wind turbine generator fault identification according to another embodiment of the present disclosure will be described with reference to fig. 2.
As shown in fig. 2, a video image fusion method S100 based on wind turbine generator fault identification includes:
and S110, collecting video image information of the wind turbine generator.
Specifically, in this step, video image information of the wind turbine generator system may be acquired by using image acquisition devices such as cameras, for example, micro cameras may be installed around the wind turbine external blade, the impeller, and the tower, and a plurality of cameras are used to perform image fusion on the external device video.
And S120, performing image framing processing on the video image information to obtain a multi-source or single-source multi-image.
S130, performing gray scale conversion on the image subjected to the framing processing, and performing pixel level image fusion under wavelet decomposition on the converted gray scale image to obtain a first fusion image.
In particular, wavelet transform refers to projecting a signal onto a set of wavelet functions and decomposing the signal into a superposition of the series of wavelet functions, which are scaled and shifted from the basic wavelet functions. In this step, the converted grayscale image is projected onto a set of wavelet functions, and the grayscale image is decomposed into a superposition of the set of wavelet functions, resulting in the first fused image, as shown in fig. 3.
And S140, carrying out pixel-level image fusion under a Laplacian pyramid on the image subjected to the framing processing to obtain a second fusion image.
Specifically, in this step, the source image is first subjected to gaussian pyramid decomposition. The fusion principle of the image pyramid algorithm is that each image participating in fusion is decomposed into the same pyramid, the pyramids are arranged from top to bottom according to the sequence of the resolution from low to high, then the pyramids decomposed from all the images are fused in the corresponding layers according to a certain rule, the corresponding image towers are obtained, image reconstruction is carried out on the corresponding image towers, and the final fused image is obtained. The laplacian pyramid algorithm is adopted in the disclosed embodiments, and the laplacian pyramid is optimized from a gaussian pyramid.
The gaussian pyramid is the most basic image pyramid, and the frame-processed image is used as an original image, the original image is used as the 0 th layer of the gaussian pyramid, the original image is assumed to be G0, and the original image is convolved by a gaussian kernel w, wherein the gaussian kernel is expressed by the following formula (1):
the image obtained by downsampling (removing even number of rows and columns) after convolution is used as the layer 1G 1 of the image tower, so that after the image is processed, the lower layer image becomes 4 times the size of the upper layer image; taking the obtained image as an input image, and repeatedly performing convolution and downsampling to obtain images of the 1 st layer to the Nth layer, wherein the images become a pyramid-type Gaussian image tower;
however, since the gaussian pyramid needs to be convolved and downsampled in the construction process, and some high-frequency information is lost, the embodiment of the present disclosure adopts a new pyramid algorithm, that is, a laplacian pyramid. The construction of the Laplacian pyramid is based on the Gaussian pyramid. To construct the laplacian pyramid, first, the previous layer of image of the laplacian pyramid needs to be convolved and upsampled to obtain a predicted image, which is the following formula (2):
the enlargement operator Expand can be expressed by the following formulas (3) to (5):
0<l≤N,0≤i<Rl,0≤j<Cl (4)
wherein G islRepresenting the first level of the original image pyramid, Gl *Representing the ith level of the pyramid of the predicted image, i, j respectively representing the pixel points of the row i and the column j in the image, m, N respectively representing the number of rows and the column in the Gaussian kernel, N representing the highest level of the pyramid, RlRepresenting the maximum number of lines in the image, ClRepresenting the maximum number of columns in the image;
the equations (3) to (5) realize that even rows and columns deleted in the Gaussian construction process are inserted into 0, and then the Gaussian kernel w is used for convolution, namely, filtering processing is carried out, and an image with the same size as that before downsampling is obtained and is a predicted image;
subtracting the predicted image from the next layer of image to obtain a difference image, and repeating iteration to obtain a series of decomposition images which are arranged into a pyramid, namely a Laplacian pyramid;
and performing unified processing on all layers for fusion to obtain a corresponding image tower for image reconstruction to obtain the second fusion image, wherein the unified processing comprises the following steps: the maximum absolute value of each layer except the top layer is obtained, and the coefficient of the highest layer is averaged.
A specific process of image fusion using the laplacian pyramid is shown in fig. 4.
S150, adopting an SIFT algorithm to the image after the framing processing to obtain a third fusion image under the characteristic level.
Specifically, in this step, when image fusion is performed, the image registration and target tracking and identification performance may be affected by factors such as the self state of the target, the environment of the scene, and the imaging characteristics of the imaging equipment. Then, the SIFT algorithm can solve the problem of image registration failure caused by target shielding, illumination influence, sundry scenes, noise, rotation, scaling, translation, image affine transformation and projection transformation of the target to a certain extent.
The SIFT algorithm searches for feature points in different scale spaces, and the different scale spaces can be obtained only through gaussian blur. Gaussian blur corresponds to an image filtering, which is also present in pyramid decomposition, using a unique linear kernel (gaussian convolution kernel) to implement the scaling. The scale space of the image is defined by the convolution of the source image with a two-dimensional gaussian function of variable scale:
L(i,j,σ)＝G(i,j,kσ)*I(i,j)
wherein G (I, j, k sigma) is a Gaussian function with variable scale, I (I, j) is a source image, and k is the scale variation. The stability of the detected feature points is ensured by generating a gaussian difference scale space by using different gaussian difference kernels and image convolutions:
D(i,j,σ)＝(G(i,j,kσ)-G(i,j,σ))*I(i,j)＝L(i,j,kσ)-L(i,j,σ)
and constructing a Gaussian pyramid by a scale space method and downsampling. In order to ensure the accuracy of feature point detection, the detected feature points need to be compared with 8 adjacent pixel points of the local layer of the gaussian pyramid and 9 pixel points of the upper and lower layers of the gaussian pyramid. Feature point detection is realized through an SIFT algorithm, and the obtained feature points are fused, namely, panoramic fusion of images can be realized. Fig. 5 clearly shows the process of implementing feature level fusion of images using the SIFT algorithm.
More specifically, in the present embodiment, the SIFT algorithm is decomposed into four steps:
(1) and (3) detection of extreme values in the scale space: searching image positions on all scales, and identifying potential interest points which are invariable in scale and rotation through a Gaussian differential function;
(2) key point positioning: at each candidate location, determining the location and scale by fitting a refined model, the key points being selected based on their degree of stability;
(3) direction determination: assigning one or more directions to each keypoint location based on the local gradient direction of the image; all subsequent operations on the image data are transformed with respect to the orientation, scale and location of the keypoints, thereby providing invariance to these transformations;
(4) description of key points: local gradients of the image are measured at a selected scale in a neighborhood around each keypoint, and these gradients are transformed into a representation that allows for relatively large local shape distortions and illumination variations.
And performing image fusion based on a decomposed four-step SIFT algorithm to obtain the third fusion image under the characteristic level.
And S160, calculating entropy, joint entropy and root mean square error of the first fused image, the second fused image and the third fused image.
Specifically, the entropy is used as a parameter for thermodynamically representing the state of an object and describes the chaos degree of a system; the entropy used for image evaluation refers to the degree of information richness contained in an image, and if the entropy is increased, the image obtains more information amount through processing, and the entropy of the image A is defined as the following formula (6):
wherein n is the gray level of the image, generally 256, and pA is the proportion of the pixel point with the gray level j in the image;
joint entropy generally represents the amount of information transferred from a source image to a fused image, i.e., the joint information between two images, and the joint entropy of image a is defined by the following equation (7):
E(M,N)＝-∑m,npMN(m,n)log2 pMN(m,n) (7)
according to the definition of the joint entropy between the two images, another calculation mode of mutual information between the two images can be obtained; the entropy of the image M is E (M), the entropy of the image N is E (N), and the joint entropy between the two images is E (M, N), then the mutual information between the image M and the image N can be calculated by the following formula (8):
MI(M,N)＝E(M)+E(N)-E(M,N) (8)
the peak signal-to-noise ratio is used as a quality evaluation index based on a reference image, is mainly used for judging the fidelity of the image, and is also often used as a measuring method of signal reconstruction quality in the fields of image compression and the like; it has a large relationship with the mean square error, often defined simply by the mean square error; the effect and quality of image fusion are reflected in that the peak signal-to-noise ratio is large, and the mean square error is small; the mean square error between image M and image N may be defined as the following equation (9):
the peak signal-to-noise ratio can be derived from the mean square error as the following equation (10):
the three types of indexes are calculated based on a single image, a source image and a reference image respectively, and the performance of the fused image is judged to be more comprehensive by integrating the three types of indexes.
S170, designing a GUI (graphical user interface) man-machine interaction interface, embodying the three fusion processing algorithms and the calculation of the index quantity in the GUI man-machine interaction interface, obtaining a final fusion image through the obtained index value, and performing fault identification on the wind turbine generator by using the final fusion image.
According to the video image fusion method based on wind turbine generator fault recognition, aiming at the situation that a target object is shielded in the video shooting process, a multi-source video is adopted for carrying out feature level image fusion, and an SIFT algorithm is specifically adopted; aiming at the sensor shake in the video shooting process, pixel-level image fusion is adopted, and wavelet transformation and a Laplace pyramid image fusion algorithm are specifically adopted. In addition, the fused image with the largest information amount is judged and obtained through calculation of three index quantities, namely entropy, joint entropy and root-mean-square error. And finally, two types of video image fusion algorithms and three index quantity calculation are built into a GUI interface, so that the method is convenient to use in the actual fault identification process.
According to the video image fusion method based on the wind turbine generator fault identification, aiming at the faults of the fan blade, the impeller and the tower, a video sensor can be installed from the outside; by framing the video images, state images of the external equipment at multiple angles are fused to synthesize a panoramic image of the external equipment, so that the fault state of the external equipment of the fan can be better identified, and the influence of a shelter at a single angle is avoided; under the condition that the external wind force is large, so that the sensor shakes, the final fault identification image can still be well obtained. In addition, in scientific research and engineering application in the field of automatic control, a large number of complex tasks of calculating and drawing simulation curves exist, the time and the workload for calculating and drawing the simulation curves can be greatly saved by adopting GUI interface design programming, and errors in manual fault identification can be reduced.
In another aspect of the present disclosure, an electronic device is provided, including:
one or more processors;
a storage unit for storing one or more programs which, when executed by the one or more processors, enable the one or more processors to carry out a method according to the preceding description.
In another aspect of the disclosure, a computer-readable storage medium is provided, on which a computer program is stored, which, when being executed by a processor, is adapted to carry out the method according to the above.
The computer readable medium may be included in the apparatus, device, system, or may exist separately.
The computer readable storage medium may be any tangible medium that can contain or store a program, and may be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, more specific examples of which include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, an optical fiber, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.
The computer readable storage medium may also include a propagated data signal with computer readable program code embodied therein, for example, in a non-transitory form, such as in a carrier wave or in a carrier wave, wherein the carrier wave is any suitable carrier wave or carrier wave for carrying the program code.
It is to be understood that the above embodiments are merely exemplary embodiments that are employed to illustrate the principles of the present disclosure, and that the present disclosure is not limited thereto. It will be apparent to those skilled in the art that various changes and modifications can be made therein without departing from the spirit and scope of the disclosure, and these are to be considered as the scope of the disclosure.
Claims (7)
1. A video image fusion method based on wind turbine generator fault identification is characterized by comprising the following steps:
collecting video image information of a wind turbine generator;
performing image framing processing on the video image information to obtain a multi-source or single-source multi-image;
performing gray level conversion on the image subjected to the framing processing, and performing pixel level image fusion under wavelet decomposition on the converted gray level image to obtain a first fusion image;
performing pixel-level image fusion under a Laplacian pyramid on the image subjected to the framing processing to obtain a second fusion image;
adopting an SIFT algorithm to the image subjected to framing processing to obtain a third fusion image under a characteristic level;
calculating entropy, joint entropy and root mean square error of the first fused image, the second fused image and the third fused image;
designing a GUI (graphical user interface) man-machine interaction interface, embodying three fusion processing algorithms and the calculation of an index quantity in the GUI man-machine interaction interface, obtaining a final fusion image through the obtained index value, and carrying out fault identification on the wind turbine generator by utilizing the final fusion image.
2. The method according to claim 1, wherein the pixel-level image fusion under wavelet decomposition of the converted grayscale image to obtain a first fused image comprises:
and projecting the converted gray level image onto a group of wavelet functions, and decomposing the gray level image into superposition of the group of wavelet functions to obtain the first fusion image.
3. The method according to claim 1, wherein the performing pixel-level image fusion under a laplacian pyramid on the framed image to obtain a second fused image comprises:
the image after the frame division processing is used as an original image, the original image is used as the 0 th layer of the Gaussian pyramid, the original image is assumed to be G0, and the original image is convoluted by a Gaussian kernel w, wherein the Gaussian kernel is expressed by the following formula (1):
the convolution is followed by down-sampling the resulting image as layer 1G 1 of the image tower, so that after the image is processed, the lower layer image becomes 4 times the size of the upper layer image; taking the obtained image as an input image, and repeatedly performing convolution and downsampling to obtain images of the 1 st layer to the Nth layer, wherein the images become a pyramid-type Gaussian image tower;
convolving and upsampling the upper layer image of the Gaussian pyramid to obtain a predicted image, namely the predicted image is represented by the following formula (2):
the enlargement operator Expand can be expressed by the following formulas (3) to (5):
0＜l≤N，0≤i＜Rl，0≤j＜Cl (4)
wherein G islRepresenting the first level of the original image pyramid, Gl *Representing the ith level of the pyramid of the predicted image, i, j respectively representing the pixel points of the row i and the column j in the image, m, N respectively representing the number of rows and the column in the Gaussian kernel, N representing the highest level of the pyramid, RlRepresenting the maximum number of lines in the image, ClRepresenting the maximum number of columns in the image;
the equations (3) to (5) realize that even rows and columns deleted in the Gaussian construction process are inserted into 0, and then the Gaussian kernel w is used for convolution, namely, filtering processing is carried out, and an image with the same size as that before downsampling is obtained and is a predicted image;
subtracting the predicted image from the next layer of image to obtain a difference image, and repeating iteration to obtain a series of decomposition images which are arranged into a pyramid, namely a Laplacian pyramid;
and performing unified processing on all layers for fusion to obtain a corresponding image tower for image reconstruction to obtain the second fusion image, wherein the unified processing comprises the following steps: the maximum absolute value of each layer except the top layer is obtained, and the coefficient of the highest layer is averaged.
4. The method according to any one of claims 1 to 3, wherein the obtaining a third fused image at a feature level by using a SIFT algorithm on the image after the framing processing comprises:
and (3) detection of extreme values in the scale space: searching image positions on all scales, and identifying potential interest points which are invariable in scale and rotation through a Gaussian differential function;
key point positioning: at each candidate location, determining the location and scale by fitting a refined model, the key points being selected based on their degree of stability;
direction determination: assigning one or more directions to each keypoint location based on the local gradient direction of the image; all subsequent operations on the image data are transformed with respect to the orientation, scale and location of the keypoints, thereby providing invariance to these transformations;
description of key points: measuring local gradients of the image at a selected scale in a neighborhood around each keypoint, the gradients being transformed into a representation that allows for relatively large local shape distortions and illumination variations;
and performing image fusion based on a decomposed four-step SIFT algorithm to obtain the third fusion image under the characteristic level.
5. The method according to any one of claims 1 to 3, wherein the calculating of entropy, joint entropy and root mean square error for the first fused image, the second fused image and the third fused image comprises:
the entropy is used as a parameter for thermodynamically representing the state of an object and describes the chaos degree of a system; the entropy used for image evaluation refers to the degree of information richness contained in an image, and if the entropy is increased, the image obtains more information amount through processing, and the entropy of the image A is defined as the following formula (6):
wherein n is the gray level of the image, generally 256, and pA is the proportion of the pixel point with the gray level j in the image;
joint entropy generally represents the amount of information transferred from a source image to a fused image, i.e., the joint information between two images, and the joint entropy of image a is defined by the following equation (7):
E(M，N)＝-∑m，npMN(m，n)log2pMN(m，n) (7)
according to the definition of the joint entropy between the two images, another calculation mode of mutual information between the two images can be obtained; the entropy of the image M is E (M), the entropy of the image N is E (N), and the joint entropy between the two images is E (M, N), then the mutual information between the image M and the image N can be calculated by the following formula (8):
MI(M，N)＝E(M)+E(N)-E(M，N) (8)
the peak signal-to-noise ratio is used as a quality evaluation index based on a reference image, is mainly used for judging the fidelity of the image, and is also often used as a measuring method of signal reconstruction quality in the fields of image compression and the like; it has a large relationship with the mean square error, often defined simply by the mean square error; the effect and quality of image fusion are reflected in that the peak signal-to-noise ratio is large, and the mean square error is small; the mean square error between image M and image N may be defined as the following equation (9):
the peak signal-to-noise ratio can be derived from the mean square error as the following equation (10):
6. an electronic device, comprising:
one or more processors;
a storage unit to store one or more programs that, when executed by the one or more processors, cause the one or more processors to implement the method of any of claims 1 to 5.
7. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, is able to carry out a method according to any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011471166.3A CN112580711B (en) | 2020-12-14 | 2020-12-14 | Video image fusion method based on wind turbine generator fault recognition |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011471166.3A CN112580711B (en) | 2020-12-14 | 2020-12-14 | Video image fusion method based on wind turbine generator fault recognition |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112580711A true CN112580711A (en) | 2021-03-30 |
| CN112580711B CN112580711B (en) | 2024-03-12 |
Family
ID=75135839
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011471166.3A Active CN112580711B (en) | 2020-12-14 | 2020-12-14 | Video image fusion method based on wind turbine generator fault recognition |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112580711B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20220253651A1 (en) * | 2021-02-10 | 2022-08-11 | Apple Inc. | Image fusion processor circuit for dual-mode image fusion architecture |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100183071A1 (en) * | 2009-01-19 | 2010-07-22 | Segall Christopher A | Methods and Systems for Enhanced Dynamic Range Images and Video from Multiple Exposures |
| CN102324033A (en) * | 2011-09-20 | 2012-01-18 | 吴建华 | Wind-powered electricity generation safe and intelligent early warning emergency system image processing method |
| CN106339998A (en) * | 2016-08-18 | 2017-01-18 | 南京理工大学 | Multi-focus image fusion method based on contrast pyramid transformation |
| CN108344574A (en) * | 2018-04-28 | 2018-07-31 | 湖南科技大学 | A kind of Wind turbines Method for Bearing Fault Diagnosis for combining adaptation network based on depth |
| CN111612030A (en) * | 2020-03-30 | 2020-09-01 | 华电电力科学研究院有限公司 | Wind turbine generator blade surface fault identification and classification method based on deep learning |
| CN111696067A (en) * | 2020-06-16 | 2020-09-22 | 桂林电子科技大学 | Gem image fusion method based on image fusion system |
-
2020
- 2020-12-14 CN CN202011471166.3A patent/CN112580711B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100183071A1 (en) * | 2009-01-19 | 2010-07-22 | Segall Christopher A | Methods and Systems for Enhanced Dynamic Range Images and Video from Multiple Exposures |
| CN102324033A (en) * | 2011-09-20 | 2012-01-18 | 吴建华 | Wind-powered electricity generation safe and intelligent early warning emergency system image processing method |
| CN106339998A (en) * | 2016-08-18 | 2017-01-18 | 南京理工大学 | Multi-focus image fusion method based on contrast pyramid transformation |
| CN108344574A (en) * | 2018-04-28 | 2018-07-31 | 湖南科技大学 | A kind of Wind turbines Method for Bearing Fault Diagnosis for combining adaptation network based on depth |
| CN111612030A (en) * | 2020-03-30 | 2020-09-01 | 华电电力科学研究院有限公司 | Wind turbine generator blade surface fault identification and classification method based on deep learning |
| CN111696067A (en) * | 2020-06-16 | 2020-09-22 | 桂林电子科技大学 | Gem image fusion method based on image fusion system |
Non-Patent Citations (2)
| Title |
|---|
| BO WANG等: "CNN Based Fault Recognition with Multi-Scale Fusion Attention Mechanism", 《IOP CONFERENCE SERIES: EARTH AND ENVIRONMENTAL SCIENCE》 * |
| 王维刚: "基于时频图像识别的旋转机械多特征融合故障诊断方法研究", 《CNKI优秀博士学位论文全文库（工程科技II辑）》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20220253651A1 (en) * | 2021-02-10 | 2022-08-11 | Apple Inc. | Image fusion processor circuit for dual-mode image fusion architecture |
| US11841926B2 (en) * | 2021-02-10 | 2023-12-12 | Apple Inc. | Image fusion processor circuit for dual-mode image fusion architecture |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112580711B (en) | 2024-03-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109671020B (en) | Image processing method, device, electronic equipment and computer storage medium | |
| CN113920538B (en) | Object detection method, device, equipment, storage medium and computer program product | |
| CN112270246B (en) | Video behavior recognition method and device, storage medium and electronic equipment | |
| CN112419372B (en) | Image processing method, device, electronic equipment and storage medium | |
| CN111415300A (en) | Splicing method and system for panoramic image | |
| CN115937794B (en) | Small target object detection method and device, electronic equipment and storage medium | |
| CN114140346A (en) | Image processing method and device | |
| Li et al. | ConvTransNet: A CNN–transformer network for change detection with multiscale global–local representations | |
| Fahmy et al. | Micro‐movement magnification in video signals using complex wavelet analysis | |
| CN112419342A (en) | Image processing method, image processing device, electronic equipment and computer readable medium | |
| CN117593702B (en) | Remote monitoring method, device, equipment and storage medium | |
| Zhu et al. | A lightweight encoder–decoder network for automatic pavement crack detection | |
| CN112580711B (en) | Video image fusion method based on wind turbine generator fault recognition | |
| CN112800932B (en) | Method for detecting remarkable ship target in offshore background and electronic equipment | |
| CN106778822B (en) | Image straight line detection method based on funnel transformation | |
| Xu et al. | A two-stage noise level estimation using automatic feature extraction and mapping model | |
| CN115861922A (en) | Sparse smoke and fire detection method and device, computer equipment and storage medium | |
| CN113780305B (en) | Significance target detection method based on interaction of two clues | |
| CN115330930A (en) | Three-dimensional reconstruction method and system based on sparse to dense feature matching network | |
| CN111967292B (en) | Lightweight SAR image ship detection method | |
| CN116543246A (en) | Training method of image denoising model, image denoising method, device and equipment | |
| CN113688928A (en) | Image matching method and device, electronic equipment and computer readable medium | |
| CN112396602A (en) | Steel coating detection method based on interframe cross-scale similarity polymerization | |
| JP4914260B2 (en) | Template matching apparatus and method | |
| Skłodowski et al. | Movement tracking in terrain conditions accelerated with CUDA |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |