CN114596315A

CN114596315A - Aircraft ground detection icing method, device and system and computer equipment

Info

Publication number: CN114596315A
Application number: CN202210500175.3A
Authority: CN
Inventors: 李彪; 邢志伟; 阚犇; 朱书杰; 王立文; 陈飞
Original assignee: Civil Aviation University of China
Current assignee: Civil Aviation University of China
Priority date: 2022-05-10
Filing date: 2022-05-10
Publication date: 2022-06-07

Abstract

The invention belongs to the technology of airplane ground ice accretion detection, and discloses a method, a device and a system for airplane ground ice accretion detection and computer equipment. Acquiring images of the airplane ground ice accretion area by using a visual sensor, identifying and classifying the airplane ground ice accretion images, quantizing the ice thickness image information into numerical values, building a detection system facing the airplane ground ice accretion images by combining ice thickness calculation and ice type identification results, and building a standardized flow facing the airplane ground ice accretion integrated detection; based on a standardized detection flow and a safety standard, the airplane ground ice accretion detection equipment based on the visual sensor is designed, and the type and the thickness of the airplane ground ice accretion are identified. The aircraft icing detection process designed by the invention has the characteristics of quick response, high accuracy and good robustness, and the designed aircraft icing detection equipment is little influenced by the environment, has timely and good detection result and has good application prospect.

Description

Aircraft ground detection icing method, device and system and computer equipment

Technical Field

The invention belongs to the technical field of airplane ground icing detection, and particularly relates to a method, a device and a system for detecting icing on the ground of an airplane and computer equipment.

Background

The airplane plays an important role in long-distance transportation, and can achieve high speed and high efficiency no matter a passenger goes out or materials are transported. While bringing great convenience to people, the aircraft is highly valued by airlines and consumers for flight safety. The ice accretion of the airplane can cause serious influence on the performance of the airplane, and the ice accretion is very easy to accumulate on the surface of the airplane facing to the airspeed direction, such as a nose, an airspeed head, a windshield, an engine air inlet, a fairing front edge, a wing front edge, a vertical tail front edge and the like. The icing of the airspeed head and the antenna can directly damage the normal functions of the airspeed head and the antenna, and serious problems of instrument failure, communication interruption and the like are caused. The ice accretion of the airplane can destroy the normal lift-drag characteristic and the operation stability of the airplane, and the airplane is very easy to stall in the low-speed flight, particularly in the take-off and landing processes. The gap of the empennage is accumulated with ice, which not only causes the control accuracy and amplitude to deviate from normal values, but also can cause the mechanism to be stuck in serious conditions, and leads the deck to lose the moving capability. Therefore, the detection of the ground ice accumulation condition of the airplane is particularly important for flight safety, and the safe operation of the airplane can be ensured and the personal and property safety of passengers can be ensured only by timely and accurately detecting the type and the thickness of the ice accumulation on the surface of the airplane and deicing the airplane. The ice accretion detection method widely applied at present mainly comprises a sensor detection method and an image feature extraction method.

The ice accretion detection sensor is widely applied mainly in an optical fiber type, an ultrasonic type, a microwave type, an infrared reflection type and the like: (1) the optical fiber type: determining the ice thickness according to the difference of the light intensity of the light reflected and scattered back in different ice thicknesses; (2) the ultrasonic wave formula: when the ice layer is formed on the surface, the piezoelectric device receives ultrasonic waves reflected by an ice and air interface, and the ice thickness is calculated according to the time delay of the reflection and the reception of the ultrasonic waves in the ice; (3) the microwave formula: the surface of the waveguide tube is provided with the waveguide tube, and the phase constant of ice gathered on the surface of the waveguide tube insulating layer can be changed, so that the resonance frequency is reduced, and the thickness is obtained according to the offset of the resonance frequency; (4) infrared reflection: based on the fact that all solids can reflect infrared energy when the temperature is higher than absolute zero, the infrared energy reflects the surface temperature of the detection piece, the temperature difference value is very close to that of the detection piece when the detection piece is frozen, and ice accumulation information is obtained through detection of the temperature difference value.

The image feature extraction method is characterized in that according to an image processing technology, useful information in an ice and snow image is extracted through digital signal processing, and distinguishing features such as color features, contour features and texture features are extracted from the image through algorithms such as machine learning and the like to identify various ice types. The method only needs to collect the image information of key parts such as the wings, the empennage, the propeller and the like of the airplane, does not need to directly contact the surface of the airplane, does not damage the aerodynamic characteristics of the airplane, and has good and quick protection performance on the airplane and high accuracy. However, the method has high requirements on image quality, and needs good hardware equipment, otherwise, the quality of the acquired image is too low, so that the detection result has great deviation from the actual result.

Through the above analysis, the problems and defects of the prior art are as follows:

(1) the size and the range of the accumulated ice detected by the accumulated ice detection sensor are limited, the sensor can only be installed at a key part which is easy to generate the accumulated ice, other regions with small probability of the accumulated ice cannot be comprehensively detected, the sensor is influenced by environmental factors to cause lower accuracy, the pneumatic characteristic of the airplane can be influenced when the sensor is installed on the surface of the airplane, and the sensor is high in damage, maintenance and replacement cost and difficult to widely use.

(2) The image feature extraction method has high requirements on image quality, is limited by hardware restriction precision, has poor effect on various feature extraction algorithms and has low robustness.

(3) There is no standard integrated ice accretion detection flow and convenient and fast ice accretion detection equipment.

(4) In the prior art, the airplane icing detection has slow response, low accuracy and poor robustness; the aircraft icing detection equipment is easily influenced by the environment, has slow detection result and poor effect, and the detection cost of the prior art is high.

The difficulty in solving the above problems and defects is: the information acquired by the ice accumulation sensor is used for detecting that the icing condition of the airplane is greatly influenced by environmental factors, and the sensor is easy to damage, so that the detection result is wrong; the traditional high-precision camera is set up to detect the icing condition of the airplane, so that the real-time performance is poor and the cost is too high.

The significance of solving the problems and the defects is as follows: the invention designs a standardized airplane ice accretion detection flow and detection equipment with convenience, rapidness and high accuracy, can obtain the information of the type and the thickness of the ice accretion only by utilizing the equipment to collect images of an airplane ice accretion area and through a built-in integrated image processing system, and has the advantages of quick response and high precision of the whole detection flow, good robustness and high reliability.

The invention designs a standardized airplane ground ice accretion detection flow and equipment, which are not influenced by environment, can conveniently, quickly and accurately realize airplane ground ice accretion detection and have great significance for ensuring airplane ground deicing and passenger personal and property safety.

The invention solves the problems that the existing airplane ground ice accumulation detection method is greatly influenced by the environment, the detection flow is not standard, the precision is not high, the detection cost is overhigh due to no integrated equipment, and the like; a standardized flow and equipment for detecting the ice accretion on the ground of an airplane based on a vision sensor are provided.

Disclosure of Invention

In order to overcome the problems in the related art, the disclosed embodiments of the present invention provide a method for detecting aircraft ground icing, an apparatus for detecting aircraft ground icing, a system for detecting aircraft ground icing, a storage medium for receiving a user input program, and a computer device. The technical scheme is as follows:

the aircraft ground detection icing method comprises the following steps: identifying and classifying the distribution conditions of the ground ice accretion areas of the airplane under different external conditions, and quantizing the ice accretion area information into numerical values;

constructing an integrated detection process facing the aircraft ground ice accumulation based on the multi-element ice accumulation information of the aircraft surface;

and constructing the airplane ground ice accretion detection equipment based on the visual sensor based on the detection process, and identifying the type and the thickness of the airplane ground ice accretion.

In one embodiment, the aircraft ground icing detection method specifically comprises the following steps:

acquiring ground ice accretion image information of an airplane by using a binocular vision sensor;

performing graying, denoising, morphological processing and affine transformation preprocessing on the collected ice accretion image;

converting the information quantity of the ice accretion thickness image into an actual numerical value by a zooming principle;

marking the icing area and type of the collected icing image, dividing the image data set into a training set, a verification set and a test set, and training an icing type identification model based on a YOLOv5 algorithm;

inputting the test set into a trained ice accretion type recognition model, outputting a recognition result, and evaluating the accuracy and the recall rate of the accuracy of the ice accretion type recognition model;

and step six, combining ice thickness calculation and ice type identification to detect the ice accretion condition on the ground of the airplane.

In one embodiment, in the first step, the binocular vision sensor is calibrated in relative position, and three-dimensional information of the ice accretion area is restored:

rotation matrix for relative position relationship

And translation vector

Expressed, the binocular stereo calibration is solved

And

the process of (2); the known left and right extraocular parameters are

Arbitrarily choose a space point of the checkerboard

The coordinate value of which in the world coordinate system is

The coordinates of the point projected by the binocular vision sensor in the coordinate systems of the left and right vision sensors are

And

and the conversion relation between the visual sensor coordinate system and the world coordinate system is known as follows:

in the formula

And

the following relations exist:

thus obtaining the relative position relation of the binocular vision sensor; said rotationMatrix array

And translation vector

The following formula:

shooting an ice accumulation area of the airplane by using a binocular vision sensor to obtain two images in different directions, and then reversely solving three-dimensional information of the ice accumulation area by calculating parallax; two identical vision sensors are combined into a binocular vision system which is transversely arranged in parallel,

optical axis of vision sensor

And

optical axis of vision sensor

Are parallel to each other and are provided with a plurality of parallel grooves,

shaft and

the axes are collinear; a certain spatial point in the ice accumulation region

In two vision sensor coordinate systems

Shaft and

the axial coordinate values are identical, only

The axis coordinate values differ by a distance

；

World coordinate system and

the coordinate systems of the vision sensors are the same if the space points

In that

The coordinates in the vision sensor coordinate system are

Then it is obtained at

The coordinates in the vision sensor coordinate system are

According to the relation of central projection, the method comprises the following steps:

in the formula (I), the compound is shown in the specification,

and

are respectively image points

And

pixel coordinates of (2), known

Is an internal parameter of the vision sensor;

from the above

、

、

、

Obtaining the following formula:

in the formula (I), the compound is shown in the specification,

is the parallax error;

a certain spatial point is corresponding to two images of the known ice accumulation area

Pixel coordinates of

And

calculating spatial points using intrinsic parameters of vision sensor

Three-dimensional coordinates of

By the use of

Solving three-dimensional point cloud data of the surface of the ice accretion area, and restoring three-dimensional information of the ice accretion area;

the collected ice accretion image information comprises: all of the ice accretion images acquired from above the ice accretion area, and all of the ice accretion images acquired from the side.

In one embodiment, in the second step, the graying, denoising, morphological processing, and affine transformation preprocessing of the collected ice accretion image includes:

1) carrying out graying processing on the image: the maximum value method is adopted, the maximum value of the gray scale in three components of the RGB ice type image is set as the gray scale value of the whole image, and the formula is as follows:

in the formula (I), the compound is shown in the specification,

as a gray scale image

The gray value of (d);

as an original image in

The component value of (c);

2) denoising each ice type image by adopting bilateral filtering: carrying out weighted average on all image pixel points of the ice image by utilizing bilateral filtering, and combining the weighted average values according to the neighborhood pixel values to obtain the pixel value of a central pixel point, wherein the formula is as follows:

wherein the content of the first and second substances,

is an output pixel value, which is the pixel value of the central pixel point,

is the value of a neighborhood of pixels that,

so as to make

Neighborhood pixel value of center point

Wherein a weighting coefficient of the bilateral filter weight function is determined by a product of a domain kernel and a value domain kernel;

domain core

Sum-value domain kernel

Is defined as follows:

wherein the content of the first and second substances,

in order to be the radius of the filtering,

and

multiplying to obtain a bilateral filtering weight function

The following formula:

3) performing morphological processing on the ice accretion image by using the operation expansion and corrosion:

the operation expansion fills the recesses and holes at the edges of the ice accretion image by using structural elements, traverses the ice accretion image by taking the original point of the structural elements as the center, judges whether a target pixel value is superposed with a pixel point of 1, and then executes and operation, wherein an output image is formed by a set of structural original point positions which are all contained in the original image, and fills the internal recesses and holes of the ice accretion image;

in order to be the original image, the image is processed,

is a structural element and is characterized in that,

quilt

The swelling is defined as:

in the formula (I), the compound is shown in the specification,

is a set of real integers which are,

is composed of

Is reflected by the light of (a) the light source,

is an empty set;

the corrosion utilizes structural elements to eliminate burrs and noise points at the edge of the ice accretion image and utilizes the structural elements

For the original image

Go through the traversal if

If the structure element contains complete structure element, the structure element is reserved

The original point, the output image is the set of all the points satisfying the condition;

quilt

Corrosion, defined as:

in the formula (I), the compound is shown in the specification,

is a set

The complement of (a) is to be added,

is a set of real integers

To the collection

Translation of (2);

4) carrying out affine change on the ice accretion image, expanding a data set, and clockwise rotating the ice accretion image according to 45 degrees every time to obtain seven ice accretion images with different angles.

In one embodiment, in step three, the converting the information amount of the ice accretion thickness image into an actual numerical value by the scaling principle comprises:

any point on the edge of the accumulated ice

The height in the image to the lower edge of the wing is

The thickness of the wing in the image is

At a magnification of the microscope

Thickness of accumulated ice in the image

The actual thickness of accumulated ice is

。

In one embodiment, in step four, the method for building the ice accretion type identification model comprises the following steps:

firstly, labeling an icing region and a type of an acquired icing image by using a labeling tool LabelImg, randomly classifying a labeled data set into a training set, a verification set and a test set according to a ratio of 8:1:1, wherein the training set is used for data samples of model fitting, the verification set is a sample set reserved in a model training process and is used for adjusting hyper-parameters of a model and evaluating the capability of the model, and the test set is used for evaluating the generalization capability of a final model of the model;

and secondly, training an ice accretion type recognition model by using the classified data set:

i) modifying data and model configuration files: modifying the type names into open ice, frost ice, mixed ice and non-accumulated ice, and modifying the category number into 4;

ii) selecting a pre-training model: selecting YOLOv5x with the highest model network depth and network width;

iii) modifying configuration parameters: the number of samples for one training is set to be 16, the number of iterations is set to be 300, and the learning rate is set to be 0.0001;

iv) after preparation, starting training;

in the fifth step, the evaluation of the accuracy and the recall rate of the precision of the ice accretion type identification model comprises the following steps:

and (3) evaluating the accuracy:

number of images indicating correct recognition of ice type versus non-icing

In the formula, TP represents the number of correctly recognized ice accretion images as a certain ice type; FP indicates the number of identified false ice types; FN represents the number of unrecognized ice types; TN indicates the number of correctly identified non-accretions;

recall evaluation:

indicating how many correct ice type images are identified among all the ice accretion images for recall;

in the formula (I), the compound is shown in the specification,

in order to correctly classify the number of images,

is the total number of training set images.

Another object of the present invention is to provide an aircraft ground detection icing device implementing the aircraft ground detection icing method, the aircraft ground detection icing device including:

the image acquisition system is used for acquiring ice accretion image information from the upper part and the side surface of an ice accretion area of the airplane by adopting a binocular electron microscope;

an image processing system comprising: the ice accretion thickness calculating module is used for extracting the height of the required part in the image to calculate the actual thickness of the accumulated ice and obtaining the actual ice accretion thickness through a scaling principle; the ice accretion type identification module is used for constructing an airplane ground ice accretion type identification model by combining the collected, processed and classified image data sets, and detecting and identifying airplane ice accretion type information;

the data transmission system is used for transmitting the detected and identified aircraft ice accretion to a remote monitoring center through an LoRa wireless transmission module so as to obtain reference information for subsequent aircraft deicing work;

and the motor driving system is used for controlling the mechanical arm to shoot and collect the ice accretion images of different parts of the airplane at different angles.

Another object of the present invention is to provide an aircraft ground detection icing system for implementing the aircraft ground detection icing method, the aircraft ground detection icing system comprising:

the ice accretion image information acquisition module is used for acquiring the ground ice accretion image information of the airplane by using the binocular vision sensor;

the icing image data set forming module is used for carrying out graying, denoising, morphological processing and affine transformation preprocessing on the icing image collected under different external conditions to form an icing image data set;

the ice accretion thickness image information quantification module is used for converting the ice accretion thickness image information quantity into an actual numerical value by utilizing a scaling principle;

the icing type identification model building module is used for labeling the icing areas and types in the data set image in batches and building an icing type identification model based on a YOLOv5 algorithm;

the aircraft ground ice accretion detection device building module is used for building a flow of aircraft ground ice accretion detection by combining ice accretion type identification and ice thickness calculation and building aircraft ground ice accretion detection devices.

Another object of the present invention is to provide a program storage medium for receiving a user input, the stored computer program causing an electronic device to execute the method for detecting ice accretion on the ground of an aircraft.

Another object of the present invention is to provide a computer device comprising a memory and a processor, the memory storing a computer program which, when executed by the processor, causes the processor to perform the aircraft ground detection icing method.

Aiming at the technical problems in the prior art and the difficulty in solving the problems, the technical problems to be solved by the technical scheme of the invention are closely combined with the technical scheme to be protected and the results, data and the like in the research and development process, the technical problems to be solved by the technical scheme of the invention are deeply analyzed in detail, and some creative technical effects are brought after the problems are solved. The specific description is as follows:

compared with the existing sensor icing detection technology, the method for detecting the collected images of the icing area of the airplane belongs to non-invasive detection, and the aerodynamic characteristics of the airplane cannot be influenced by damage to the airplane; compared with the existing feature extraction detection technology, the required image processing operation is simpler and easier to operate, the requirements can be met only by basic gray processing, denoising and enhancement, and complex algorithm support is not needed.

Secondly, the quantification of the image information of the ice accretion thickness can be completed only by a simple ice thickness conversion formula, and the ice thickness is measured by using the comparison sensor by using the principles of sound transmission, piezoelectric effect and the like, so that the measurement error is reduced.

Thirdly, the building process of the ice accretion type recognition model is simple and easy to operate, only the processed data set needs to be put into a YOLOv5 algorithm model for training, a complex and difficult parameter adjusting process is not needed, the generalization capability of the recognition model is strong, and the ice accretion type recognition effect is good; compared with the existing icing detection method, the method combines ice thickness calculation and ice type identification to standardize and integrate the icing detection process, and can accurately detect the ground icing condition of the airplane in a short time.

Fourthly, the invention also designs the accumulated ice detection equipment based on the standard integrated accumulated ice detection process, so that the ground accumulated ice detection of the airplane is not influenced by environmental factors, the detection speed is high, the precision is high, and the problems of easy damage, high cost and the like of the traditional sensor detection method are solved.

Fifthly, acquiring images of the airplane ground ice accretion area by using a visual sensor, identifying and classifying the airplane ground ice accretion images, quantizing the ice thickness image information into numerical values, building a detection system facing the airplane ground ice accretion images by combining ice thickness calculation and ice type identification results, and designing a standardized flow facing the airplane ground ice accretion integrated detection; based on a standardized detection flow and a safety standard, the airplane ground ice accretion detection equipment based on the visual sensor is designed, and the type and the thickness of the airplane ground ice accretion are identified. The aircraft icing detection process designed by the invention has the characteristics of quick response, high accuracy and good robustness, and the designed aircraft icing detection equipment is little influenced by the environment, has timely and good detection result and has good application prospect.

Sixth, the commercial value after the technical scheme of the invention is converted is that the ground ice accumulation detection of the airplane is not influenced by environmental factors, the detection speed is high, the precision is high, the problem that the traditional sensor detection method is easy to damage is solved, and a decision basis is provided for ensuring the ground deicing of the airplane. The method helps various large aircraft manufacturers to save the cost of embedding the ice accretion sensor in the aircraft wing, provides creative points for aircraft ground ice accretion detection and deicing research and development manufacturers, and further advances the non-contact ice accretion detection technology.

And seventhly, the ice type identification and the ice thickness calculation are integrated, the standardization and integration of the ice accretion detection process are realized, the ground ice accretion condition of the airplane is researched and judged in a short time, and the technical blank is filled.

Eighth, according to the traditional aircraft ground icing detection method based on image understanding, complex image processing operation needs to be carried out on images, feature extraction algorithms needed by machine learning are complex, a series of convolutional neural networks used for image recognition, such as YOLO (sparse polar organic solvent) and the like, can be used for autonomously learning icing image features, complex algorithms of machine learning are omitted, and the problem that small differences among ice types are difficult to distinguish can be better solved as long as the ice type image data samples are large enough.

Ninth, the invention well eliminates the prejudice of using the convolutional neural network to detect the ground ice deposition of the airplane at present by solving the problems that the traditional sensor is easy to damage and the image feature extraction algorithm is complicated.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure.

FIG. 1 is a flow chart of a method for detecting ice accretion on the ground of an aircraft according to embodiment 1 of the present invention;

FIG. 2 is a flowchart of a method for detecting ice accretion on the ground of an aircraft according to embodiment 2 of the present invention;

FIG. 3 is a schematic diagram of three-dimensional reconstruction of an ice accretion region provided in embodiment 3 of the present invention;

FIG. 4A is a diagram illustrating the effect of morphological processing on an original image A provided in embodiment 4 of the present invention;

FIG. 4B is a diagram illustrating the morphological processing effect of the structural element B provided in embodiment 4 of the present invention;

FIG. 4C is a graph showing the effect of morphological processing on the erosion image provided in example 4 of the present invention;

FIG. 4D is a diagram illustrating the effect of morphological processing on the dilated image provided in embodiment 4 of the present invention;

FIG. 5 is a diagram illustrating the effect of the image preprocessing method provided in embodiment 4 of the present invention;

fig. 6 is a schematic diagram of calculation of the thickness of the ice accretion pixel to be converted into the actual thickness by the ice accretion detection method provided in embodiment 5 of the present invention;

FIG. 7 is a block diagram of an aircraft ground detection icing assembly provided in accordance with embodiment 9 of the present invention;

FIG. 8 is a schematic diagram of an aircraft ground detection icing assembly provided in accordance with embodiment 9 of the present invention;

FIG. 9 is a schematic view of an aircraft ground detection icing system provided in accordance with embodiment 10 of the present invention;

in the figure: 1. an image acquisition system; 2. an image processing system; 3. a data transmission system; 4. a motor drive system; 5. an ice accretion image information acquisition module; 6. an icing image dataset forming module; 7. an icing thickness image information quantification module; 8. building a model for identifying the type of the ice accretion; 9. the airplane ground ice accretion detection device is constructed by a module.

Detailed Description

In order to make the aforementioned objects, features and advantages of the present invention comprehensible, embodiments accompanied with figures are described in detail below. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. This invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein, but rather should be construed as broadly as the present invention is capable of modification in various respects, all without departing from the spirit and scope of the present invention.

Example 1

As shown in fig. 1, the method for detecting ice accretion on the ground of an aircraft according to an embodiment of the present invention includes the following steps:

s101, acquiring information of the ground ice accretion image of the airplane by using a binocular vision sensor, preprocessing the image and marking the ice accretion characteristic.

S102, quantizing the ice thickness image information into numerical values through a scaling principle, randomly classifying the ice accretion images processed by the ice accretion images to form a data set, and building an ice accretion type identification model based on a YOLOv5 algorithm to realize a standardized flow of airplane ground ice accretion thickness and type detection.

And S103, finally, designing airplane ground ice accretion detection equipment based on a visual sensor to realize integrated detection of the airplane ground ice accretion condition.

In the embodiment of the method for detecting the ice accretion on the ground of the airplane, the following steps can be carried out: identifying and classifying the distribution conditions of the ground ice accretion areas of the airplane under different external conditions, and quantizing the ice accretion area information into numerical values; designing a standardized flow facing the integrated detection of the ice accretion on the ground of the airplane based on the multivariate ice accretion information on the surface of the airplane; based on a standardized detection flow and a safety standard, the airplane ground ice accretion detection equipment based on the visual sensor is designed, and the type and the thickness of the airplane ground ice accretion are identified.

Example 2

According to the aircraft ground detection icing method provided in embodiment 1, as shown in fig. 2, the aircraft ground detection icing method provided in this embodiment of the present invention further describes embodiment 1 in detail, and the specific steps include:

(1) acquiring ground ice accretion image information of the airplane by using a binocular vision sensor;

(2) preprocessing collected ice accretion images such as graying, denoising, morphological processing, affine transformation and the like;

(3) quantifying the ice accretion thickness image information into numerical values by a scaling principle;

(4) marking the icing area and type of the collected icing image, dividing an image data set into a training set, a verification set and a test set, and training an icing type identification model based on a YOLOv5 algorithm;

(5) inputting the test set into the trained model, outputting a recognition result, and evaluating the accuracy and recall rate of the model precision;

(6) and the ice thickness calculation and the ice type identification are combined to realize the standardization and the integrated detection of the ground ice accretion condition of the airplane.

Example 3

On the basis of the embodiment 2, as a preferred embodiment, as shown in the principle of three-dimensional reconstruction of the ice accretion area in fig. 3, in the step (1), three-dimensional reconstruction of the ice accretion area is performed by using binocular stereo vision (calibration of relative positions of binocular vision sensors is performed, and three-dimensional information of the ice accretion area is restored):

calibrating the relative position of the binocular vision sensor, and restoring the three-dimensional information of the ice accretion area:

rotation matrix for relative position relationship

And translation vector

Expressed, the binocular stereo calibration is solved

And

the process of (2); the known left and right extraocular parameters are

Arbitrarily choose a space point of the checkerboard

The coordinate value of which in the world coordinate system is

And

in the formula

And

the following relationships exist:

thus obtaining the relative position relation of the binocular vision sensor; the rotation matrix

And translation vector

The following formula:

the method comprises the steps of shooting an ice accumulation area of an airplane by using a binocular vision sensor to obtain two images in different directions, and then reversely solving three-dimensional information of the ice accumulation area by calculating parallax. Two identical vision sensors are combined into a binocular vision system which is transversely arranged in parallel,

optical axis of vision sensor

And

optical axis of vision sensor

shaft and

the axes are collinear; a certain spatial point in the ice accumulation area

In two vision sensor coordinate systems

Shaft and

the axial coordinate values are identical, only

The axis coordinate values differ by a distance

。

Assume the world coordinate system and

the coordinate systems of the vision sensors are the same if the space points

In that

The coordinates in the vision sensor coordinate system are

Then it is obtained at

The coordinates in the vision sensor coordinate system are

From the relationship of the central projection, it can be known that:

in the formula (I), the compound is shown in the specification,

and

are respectively image points

And

pixel coordinates of (2) are known

Is an internal parameter of the vision sensor;

by the formula

、

、

、

Obtaining:

in the formula (I), the compound is shown in the specification,

is the parallax.

In summary, it is known that a certain spatial point corresponds to two images of the ice accumulation region

Pixel coordinates of

And

calculating spatial points using intrinsic parameters of vision sensor

Three-dimensional coordinates of

And then, the three-dimensional point cloud data of the ice accumulation area surface is obtained by using the formula, and the three-dimensional information of the three-dimensional point cloud data is restored.

The collecting of the ice accretion image information comprises: the image is collected from the upper part of the ice accumulation area by using a binocular electron microscope, so that the characteristics of all ice accumulation images are displayed, the image identification is convenient, and the image is collected from the side face so as to simplify the ice thickness calculation.

Example 4

FIG. 4A is an original image, FIG. 4B is a structural element, FIG. 4C is an erosion image, and FIG. 4D is a morphological processing effect diagram of an expansion image; on the basis of the embodiment 2, as a preferred embodiment, in the step (2), in order to improve the quality and the quantity of the ice accretion image data set and avoid the influence of color information on the detection effect, the image is processed into a gray image, the image is subjected to denoising processing by bilateral filtering, the ice type gray image is expanded and corroded to eliminate the pits and holes in the image so as to achieve the purpose of enhancement, and finally, the image is subjected to affine transformation to expand the data set.

In the step (2), the effect of the image preprocessing method is shown in fig. 5.

Firstly, carrying out gray processing on an image: the maximum value method is adopted, the maximum value of the gray scale in three components of the RGB ice type image is set as the gray scale value of the whole image, and the formula is as follows:

in the formula (I), the compound is shown in the specification,

as a gray scale image

The gray value of (d);

as an original image in

The component value of (c).

Then, carrying out denoising treatment on each ice type image by adopting bilateral filtering:

carrying out weighted average on all image pixel points of the ice image by utilizing bilateral filtering, combining the weighted average values according to the neighborhood pixel values to obtain the pixel value of a central pixel point, and obtaining the following formula:

wherein, among others,

is an output pixel value, which is the pixel value of the central pixel point,

is the value of a neighborhood of pixels that,

so as to make

Neighborhood pixel value of center point

Wherein the weighting coefficients of the bilateral filter weight function are determined by the product of a domain kernel and a value domain kernel.

Domain core

Sum value domain kernel

Is defined as follows:

wherein the content of the first and second substances,

in order to be the radius of the filtering,

and

multiplying together to obtainTo bilateral filtering weight function

The following formula:

and then performing morphological processing on the image by utilizing expansion and corrosion to achieve the purpose of data enhancement:

and expanding, namely filling the recesses and holes at the edges of the ice accretion image by using structural elements, traversing the ice accretion image by taking the original point of the structural elements as the center, judging whether the target pixel value is superposed with the pixel point of 1, and then executing AND operation, wherein the output image is formed by a set of structural original point positions which are all contained in the original image, and the internal recesses and holes of the ice accretion image can be filled. Suppose that

In order to be the original image, the image is processed,

is a structural element and is characterized in that,

quilt

The swelling is defined as:

in the formula (I), the compound is shown in the specification,

is a set of real integers which are,

is composed of

Is reflected by the light of (a) the light source,

is an empty set;

the corrosion utilizes structural elements to eliminate burrs and noise points at the edge of the ice accretion image. By means of structural elements

For the original image

Go through the traversal if

Origin, output image is the set of all the points that satisfy the condition.

Quilt

The corrosion can be defined as:

in the formula (I), the compound is shown in the specification,

is a set

The complement of (a) is set up,

is a set of real integers

Pair set

To (3) is performed.

Finally, carrying out affine change on the ice accretion image, and further expanding a data set: and rotating the ice accretion image clockwise according to 45 degrees every time to obtain seven ice accretion images with different angles, thereby achieving the purpose of data amplification.

Example 5

On the basis of the embodiment 2, as a preferred embodiment, the step (3) converts the thickness of the ice accretion in the image into an actual thickness, and the calculation principle is as shown in fig. 6.

Any point on the edge of the accumulated ice

The height in the image to the lower edge of the wing is

The thickness of the wing in the image is

At a magnification of the microscope

Thickness of accumulated ice in the image

The actual thickness of accumulated ice is

。

Example 6

On the basis of the embodiment 2, as a preferred embodiment, in the step (4), in order to realize the ice accretion type recognition, the ice accretion region and type recognition is performed on the collected ice accretion image, the labeled image is randomly classified into a training set, a verification set and a test set, the training set and the verification set are used for training an ice accretion type recognition model, and the test set is used for evaluating the generalization ability of the model. And modifying algorithm built-in parameters according to the ice accretion type and the image scale information, and training an ice accretion type recognition model based on YOLOv 5.

In the step (4), the method for building the ice accretion type identification model comprises the following steps:

firstly, labeling the icing area and type of an acquired icing image by using a labeling tool LabelImg, randomly classifying the labeled data set into a training set, a verification set and a test set according to the ratio of 8:1:1, wherein the training set (train) is a data sample for model fitting, the verification set (val) is a sample set reserved in the model training process and can be used for adjusting the hyper-parameters of the model and primarily evaluating the capability of the model, and the test set (test) is used for evaluating the generalization capability of the final model of the model.

Then, training an ice accretion type recognition model by using the classified data set:

a) modifying data and model configuration files: the type names of the ice-making materials are modified into open ice, frost ice, mixed ice and non-accumulated ice, and the category number is modified into 4;

b) selecting a pre-training model: selecting YOLOv5x with the highest model network depth (depth _ multiple) and network width (width _ multiple);

c) modifying the relevant configuration parameters: the number of training samples (batch _ size) at one time is set to 16, the number of iterations (epoch) is set to 300, and the learning rate (learning _ rate) is set to 0.0001;

d) after the preparation is finished, the training is started.

Example 7

On the basis of the embodiment 2, as a preferred embodiment, in the step (5), in order to evaluate the generalization ability of the icing type identification model, the accuracy and recall rate of the icing type identification model are evaluated.

The step (5) of evaluating the accuracy and the recall rate of the trained icing type identification model comprises the following steps:

TP represents the number of correctly identified ice accretion images as a certain ice type; FP indicates the number of identified false ice types; FN represents the number of unrecognized ice types; TN indicates the number of correctly identified non-accretions;

a) accuracy rate (

): number of images representing correct recognition of ice type versus non-icing:

（17）

in the formula

In order to correctly classify the number of images,

is the total number of training set images.

b) Recall rate: (

): indicating how many correct ice type images were identified in all of the ice accretion images.

（18）

In the invention, the ice thickness calculation and the ice type identification in the 5 steps are combined to establish a normalized and integrated detection flow of the ice thickness and the ice type facing the key part of the airplane.

Example 8

On the basis of the embodiment 2, as a preferred embodiment, in the step (6), in order to realize the normalized integrated detection of the aircraft ground icing condition, the ice thickness calculation method and the icing type identification model are obtained through the steps (1) to (4), and the integrated detection process of the aircraft ground icing condition based on the visual sensor is designed by combining the ice thickness calculation method and the icing type identification model.

Example 9

To implement the method of embodiment 1, as shown in fig. 7, an aircraft ground icing detecting device (an aircraft ground icing detecting apparatus based on a vision sensor) provided by an embodiment of the present invention includes:

the image acquisition system 1 is used for acquiring ice accretion image information from the upper part and the side surface of an ice accretion area of the airplane by adopting a binocular electron microscope;

the image processing system 2 comprises an accumulated ice thickness calculating module, extracts the thickness of the required part in the image and calculates the actual thickness of the accumulated ice by a scaling principle; and the ice accretion type identification module is used for constructing an airplane ground ice accretion type identification model by combining the acquired, processed and classified image data sets, and detecting and identifying airplane ice accretion type information.

The data transmission system 3 is used for transmitting the detected and identified aircraft ice accretion to a remote monitoring center through an LoRa wireless transmission module so as to obtain reference information for subsequent aircraft deicing work;

and the motor driving system 4 is used for controlling various running states of the equipment and controlling the mechanical arm to shoot and collect ice accretion images of different parts of the airplane at different angles.

As shown in fig. 8, the principle of the device for detecting ice accretion on the ground of an airplane according to an embodiment of the present invention includes: the image acquisition system 1 is provided with a visual sensor, and dynamically acquired airplane ice accretion images are transmitted to the image processing system 2 through a USB (universal serial bus) to be detected and processed;

the image processing system 2 receives and stores the collected ice accretion images in an internal high-speed memory SDRAM, the ice accretion thickness corresponding to an image shooting area is obtained by the images one by one through an ice accretion thickness conversion algorithm, and whether the ice accretion in the images is open ice, frost ice or mixed ice is detected through an ice accretion type detection model;

after the thickness and the type of the ice accretion image are detected, the ice accretion information of the airplane is transmitted back to the remote monitoring center through the LoRa wireless transmission module, and under the broadcast transparent transmission mode, the module does not change data and protocols and transmits and receives the information. The module has the characteristics of high sensitivity, low power consumption, strong anti-interference line, long transmission distance and support of dispersed acquisition, and data transmission can be stably carried out under different external conditions.

The aircraft ground detection icing device provided by the embodiment of the invention further comprises a battery module (not shown in the figure) for supplying power to each image acquisition system 1, the image processing system 2, the data transmission system 3 and the motor driving system 4.

Example 10

To implement the method of embodiment 1, as shown in fig. 9, an aircraft ground ice accretion detection system (an aircraft ground ice accretion detection system based on a vision sensor) provided by an embodiment of the present invention includes:

the ice accretion image information acquisition module 5 is used for acquiring the ground ice accretion image information of the airplane by using a visual sensor;

the icing image data set forming module 6 is used for carrying out preprocessing operations such as graying, denoising, enhancing, transforming and the like on the icing image acquired under different external conditions to form an icing image data set;

the ice accretion thickness image information quantization module 7 is used for quantizing the ice accretion thickness image information into numerical values by utilizing a scaling principle;

the icing type identification model building module 8 is used for labeling the icing areas and types in the data set image in batches and building an icing type identification model based on a YOLOv5 algorithm;

the aircraft ground ice accretion detection device building module 9 is used for building the aircraft ground ice accretion detection device based on the visual sensor by combining the standardized flow of the aircraft ground ice accretion integrated detection of the ice accretion type identification and the ice thickness calculation.

For the information interaction, execution process and other contents between the above-mentioned devices/units, because the embodiments of the method of the present invention are based on the same concept, the specific functions and technical effects thereof can be referred to the method embodiments specifically, and are not described herein again.

It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-mentioned division of the functional units and modules is illustrated, and in practical applications, the above-mentioned function distribution may be performed by different functional units and modules according to needs, that is, the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-mentioned functions. Each functional unit and module in the embodiments may be integrated in one processing unit, or each unit may exist alone physically, or two or more units are integrated in one unit, and the integrated unit may be implemented in a form of hardware, or in a form of software functional unit. In addition, specific names of the functional units and modules are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present invention. For the specific working processes of the units and modules in the system, reference may be made to the corresponding processes in the foregoing method embodiments, which are not described herein again.

An embodiment of the present invention further provides a computer device, where the computer device includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, the processor implementing the steps of any of the various method embodiments described above when executing the computer program.

Embodiments of the present invention further provide a computer-readable storage medium, where a computer program is stored, and when the computer program is executed by a processor, the steps in the above method embodiments may be implemented.

The embodiment of the present invention further provides an information data processing terminal, where the information data processing terminal is configured to provide a user input interface to implement the steps in the above method embodiments when implemented on an electronic device, and the information data processing terminal is not limited to a mobile phone, a computer, or a switch.

The embodiment of the present invention further provides a server, where the server is configured to provide a user input interface to implement the steps in the above method embodiments when implemented on an electronic device.

Embodiments of the present invention provide a computer program product, which, when running on an electronic device, enables the electronic device to implement the steps in the above method embodiments when executed.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, all or part of the flow of the method according to the embodiments of the present invention may be implemented by a computer program, which may be stored in a computer-readable storage medium and used for instructing related hardware to implement the steps of the embodiments of the method according to the embodiments of the present invention. Wherein the computer program comprises computer program code, which may be in the form of source code, object code, an executable file or some intermediate form, etc. The computer readable medium may include at least: any entity or device capable of carrying computer program code to a photographing apparatus/terminal apparatus, a recording medium, computer Memory, Read-Only Memory (ROM), Random Access Memory (RAM), electrical carrier wave signal, telecommunication signal, and software distribution medium. Such as a usb-drive, a removable hard drive, a magnetic or optical disk, etc.

In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and reference may be made to the related descriptions of other embodiments for parts that are not described or illustrated in a certain embodiment.

Evidence of the relevant effects of the examples:

by combining all the technical schemes, the invention has the advantages and positive effects that:

compared with the sensor detection and feature extraction method, the method has the advantages of small ice thickness calculation error and high identification accuracy. As shown in the table below.

The method comprises the steps of collecting images of airplane ground ice accretion areas by using a visual sensor, identifying and classifying the airplane ground ice accretion images, quantizing the ice thickness image information into numerical values, building a detection system facing the airplane ground ice accretion images by combining ice thickness calculation and ice type identification results, and designing a standardized flow facing airplane ground ice accretion integrated detection; based on a standardized detection process and a safety standard, the airplane ground icing detection equipment based on the visual sensor is designed, and the type and the thickness of the airplane ground icing are identified. The aircraft icing detection process designed by the invention has the characteristics of quick response, high accuracy and good robustness, and the designed aircraft icing detection equipment is little influenced by the environment, has timely and good detection result and has good application prospect.

The above description is only for the purpose of illustrating the preferred embodiments of the present invention, and the scope of the present invention is not limited thereto, and any modification, equivalent replacement, and improvement made by those skilled in the art within the technical scope of the present invention disclosed herein, which is within the spirit and principle of the present invention, should be covered by the present invention.

Claims

1. An aircraft ground ice accretion detection method, characterized in that it comprises:

inputting a test set into the trained icing type identification model, outputting an identification result, and evaluating the accuracy and the recall rate of the accuracy of the icing type identification model;

2. The aircraft ground ice accretion detection method of claim 1, wherein in step one, the collecting of aircraft ground ice accretion image information by using a binocular vision sensor comprises:

calibrating the relative position of the binocular vision sensor, and restoring the three-dimensional information of the ice accretion area;

rotation matrix for relative position relationship

And translation vector

To express, binocular stereo scaling as solving

And

the process of (2); the left and right extraocular parameters are

Arbitrarily choose a space point of the checkerboard

Points in space

The coordinate value in the world coordinate system is

The point projected by the binocular vision sensor is at the left and right vision sensorsThe coordinates of the device in the coordinate system are

And

from the transformation relationship between the visual sensor coordinate system and the world coordinate system, it can be known that:

in the formula

And

the following relationships exist:

thereby obtaining the relative position relation of the binocular vision sensors, and the rotation matrix

And translation vector

The following formula:

optical axis of vision sensor

And

optical axis of vision sensor

shaft and

the axes are collinear; a certain spatial point in the ice accumulation area

In two vision sensor coordinate systems

Shaft and

the axial coordinate values are identical, only

The axis coordinate values differ by a distance

；

World coordinate system and

the coordinate systems of the vision sensors are the same, and the space points

In that

The coordinates in the vision sensor coordinate system are

Then it is obtained at

The coordinates in the vision sensor coordinate system are

According to the relation of central projection:

in the formula (I), the compound is shown in the specification,

and

are respectively image points

And

pixel coordinates of (2) are known

Is an internal parameter of the vision sensor;

by the formula:

、

、

、

obtaining:

in the formula (I), the compound is shown in the specification,

is the parallax error;

Pixel coordinates of

And

calculating spatial points using intrinsic parameters of vision sensor

Three-dimensional coordinates of

By the use of

3. The aircraft ground detection icing method of claim 1, wherein in step two, the graying, denoising, morphological processing and affine transformation preprocessing of the collected icing image comprises:

in the formula (I), the compound is shown in the specification,

as a gray scale image

The gray value of (d);

as an original image in

The component value of (a);

wherein the content of the first and second substances,

is an output pixel value, which is the pixel value of the central pixel point,

is a value of a neighboring pixel (or a neighboring pixel),

so as to make

Neighborhood pixel value of center point

domain core

Sum-value domain kernel

Is defined as follows:

wherein the content of the first and second substances,

in order to be the radius of the filtering,

and

multiplying to obtain a bilateral filtering weight function

The following formula:

in order to be the original image, the image is processed,

is a structural element and is characterized in that,

quilt

The swelling is defined as:

in the formula (I), the compound is shown in the specification,

is a set of real integers which are,

is composed of

Is reflected by the light of (a) the light source,

is an empty set;

For the original image

Go through the traversal if

quilt

Corrosion, defined as:

in the formula (I), the compound is shown in the specification,

is a set

The complement of (a) is to be added,

is a set of real integers

Pair set

The translation of (2);

4. The aircraft ground ice accretion detecting method of claim 1, wherein in step three, converting the information content of the ice accretion thickness image into an actual numerical value by a scaling principle comprises:

any point on the edge of the accumulated ice

The height in the image to the lower edge of the wing is

The thickness of the wing in the image is

At the time of collection, the magnification of the microscope is

Thickness of accumulated ice in the image

The actual thickness of accumulated ice is

。

5. The aircraft ground detection icing method according to claim 1, wherein in step four, the building method of the icing type identification model comprises the following steps:

firstly, marking the ice accumulation region and type of the collected ice accumulation image by using a marking tool LabelImg, and randomly classifying the marked data set into a training set, a verification set and a test set according to the ratio of 8:1:1, wherein the training set is used for data samples of model fitting; the verification set is a sample set reserved in the model training process and is used for adjusting the hyper-parameters of the model and evaluating the capability of the model; the test set is used for evaluating the generalization capability of the model final model;

and secondly, training an ice accretion type recognition model by using the classified data set.

6. The aircraft ground detection icing method of claim 5, wherein in step four, the training of the icing type identification model using the classified data set comprises:

iv) after preparation, start training.

7. An aircraft ground detection icing method according to claim 1, wherein in step five, the accuracy and recall of the icing type identification model precision is evaluated as:

and (3) evaluating the accuracy:

for accuracy, the number of images representing correct identification of ice type versus non-icing is defined as:

in the formula, TP represents the number of correctly identified ice accretion images as a certain ice type, FP represents the number of identified error ice types, FN represents the number of unrecognized ice types, and TN represents the number of correctly identified non-accretion ice;

recall evaluation:

for recall, it is indicated how many correct ice type images were identified in all of the ice accrued images, defined as:

in the formula (I), the compound is shown in the specification,

in order to correctly classify the number of images,

is the total number of training set images.

8. An aircraft ground detection icing device for implementing the aircraft ground detection icing method according to any one of claims 1 to 7, wherein the aircraft ground detection icing device comprises:

the image acquisition system (1) is used for acquiring ice accretion image information from the upper part and the side surface of an ice accretion area of the airplane by adopting a binocular electron microscope;

image processing system (2) comprising: the ice accretion thickness calculating module is used for extracting the height of the required part in the image to calculate the actual thickness of the accumulated ice and obtaining the actual ice accretion thickness through a scaling principle; the ice accretion type identification module is used for constructing an airplane ground ice accretion type identification model by combining the collected, processed and classified image data sets, and detecting and identifying airplane ice accretion type information;

the data transmission system (3) is used for transmitting the detected and identified aircraft ice accretion to a remote monitoring center through an LoRa wireless transmission module so as to obtain reference information for subsequent aircraft deicing operation;

and the motor driving system (4) is used for controlling the mechanical arm to shoot and collect ice accretion images of different parts of the airplane at different angles.

9. An aircraft ground detection icing system for implementing the aircraft ground detection icing method of any one of claims 1 to 7, wherein the aircraft ground detection icing system comprises:

the ice accretion image information acquisition module (5) is used for acquiring the ground ice accretion image information of the airplane by utilizing a binocular vision sensor;

the icing image data set forming module (6) is used for carrying out graying, denoising, morphological processing and affine transformation preprocessing on the icing image acquired under different external conditions to form an icing image data set;

the ice accretion thickness image information quantization module (7) converts the ice accretion thickness image information quantity into an actual numerical value by utilizing a scaling principle;

the icing type identification model building module (8) is used for labeling the icing areas and types in the data set image in batches and building an icing type identification model based on a YOLOv5 algorithm;

the airplane ground ice accretion detection device building module (9) is used for building a flow of airplane ground ice accretion detection by combining ice accretion type identification and ice thickness calculation and building airplane ground ice accretion detection devices.

10. A computer arrangement, characterized in that the computer arrangement comprises a memory and a processor, the memory storing a computer program which, when executed by the processor, causes the processor to carry out the method of aircraft ground detection of icing according to any one of claims 1 to 7.