CN113487529B - Cloud map target detection method for meteorological satellite based on yolk - Google Patents

Abstract

The invention discloses a Yolo-based meteorological satellite cloud image target detection method. The multiple visible light nephograms are divided into: the first visible light nephogram, the second visible light nephogram and the third visible light nephogram; wherein, the invalid information ratio of the first visible light nephogram ψ ₁ satisfies ψ ₁ <ψ _min ; the invalidity of the second visible light nephogram The information proportion ψ ₂ satisfies ψ _min ≤ψ ₂ ≤ψ _max ; the invalid information proportion ψ ₃ of the third visible light cloud image satisfies ψ ₃ >ψ _max ; ψ _min is the lower limit of the invalid information threshold, and ψ _max is the upper limit of the invalid information threshold; three , fuse the first visible cloud image with its corresponding infrared cloud image to obtain a fusion cloud image; compose the infrared cloud image corresponding to the first visible cloud image, the fusion cloud image and the third visible cloud image to form a cloud atlas to be detected; 4. Use the Yolo algorithm to detect Target detection is performed on the cloud images in the cloud atlas, and typical weather phenomena in the cloud images are identified.

Description

A Yolo-based meteorological satellite cloud image target detection method

technical field

The invention belongs to the technical field of meteorological satellite cloud image identification and detection, in particular to a Yolo-based meteorological satellite cloud image target detection method.

Background technique

Meteorological satellites are an important meteorological tool, and among the data transmitted by meteorological satellites, the most important are meteorological cloud image files, which play an important role in weather forecasting, especially precipitation analysis. Meteorological satellites can obtain daytime visible light cloud images, day and night infrared cloud images and water vapor distribution maps, etc. and transmit them by fax, and provide domestic and foreign meteorological data utilization stations for reception and utilization. For weather forecasting, the content of satellite cloud image analysis includes distinguishing cloud images of different channels, such as infrared cloud images and visible light cloud images, as well as analyzing the distribution of large-scale clouds and their corresponding weather systems. In the process of analyzing satellite cloud images, there are some basic features, such as patterns (structures), which can be related to different weather systems or physical system processes through different types of cloud systems or cloud clusters, such as typhoons, cyclones, and low vortices. The iso-cloud system has a spiral structure, while the cloud system of the front, the jet stream and the tropical convergence zone has a belt-like structure ^[1] . For the analysis of such cloud systems, it is basically judged by the shape of the cloud images of different channels. However, there is no better satellite cloud image analysis method, mainly through manual visual interpretation, which is not only slow but also low in accuracy.

At present, there are various channels in satellite cloud images, but generally only infrared cloud images are used for cloud image analysis. Because meteorological satellites can detect infrared cloud images during the day and night, it is obtained by detecting the wavelength of the infrared region of objects and atmospheric radiation. As long as the cloud system has temperature, it can be detected, regardless of illumination. During the imaging process, areas with higher temperatures have higher brightness, and areas with lower temperatures have lower brightness. The temperature of the cloud layer is significantly different from the temperature of the surrounding atmosphere, so that we can judge the shape of the cloud by infrared radiation. Similar to it is the visible light cloud map, which is obtained by directly measuring the reflection of light by the cloud layer. The advantage of the visible light cloud map is that the shape and texture of the clouds it detects are better (better than the infrared cloud map), but the visible light cloud map is not enough. Routine detection is not possible in this case. In fact, in actual production, there are cases where the utilization of other channels other than the infrared cloud image is low.

Therefore, it is of great significance to seek a high-precision real-time satellite cloud image detection algorithm that can make full use of satellite cloud image information.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a Yolo-based meteorological satellite cloud image target detection method, which uses the Yolo algorithm to detect and identify typical weather phenomena in the cloud image after fusing the infrared cloud image and the visible light cloud image. ; It can effectively use the information in the visible light cloud map to improve the detection speed and detection accuracy of the satellite cloud map.

The technical scheme provided by the present invention is:

A Yolo-based meteorological satellite cloud image target detection method, comprising the following steps:

Step 1, extracting multiple infrared cloud images and multiple visible light cloud images from the original satellite cloud image data collected by the meteorological satellite;

Wherein, the infrared cloud map and the visible light cloud map are in a one-to-one correspondence;

Step 2: Counting invalid information in the visible light nephogram, and dividing the multiple visible light nephograms into: a first visible light nephogram, a second visible light nephogram and a third visible light nephogram according to the proportion of invalid information in the visible light nephogram;

Wherein, the invalid information ratio ψ ₁ of the first visible light nephogram satisfies ψ ₁ <ψ _min ;

The invalid information ratio ψ ₂ of the second visible light nephogram satisfies ψ _min ≤ψ ₂ ≤ψ _max ;

The invalid information ratio ψ ₃ of the third visible light nephogram satisfies ψ ₃ >ψ _max ;

In the formula, ψ _min is the lower limit of the invalid information threshold, and ψ _max is the upper limit of the invalid information threshold;

Step 3, fuse the second visible nephogram with its corresponding infrared nephogram to obtain a fusion nephogram; and

Composing the infrared cloud map corresponding to the third visible light cloud map, the fusion cloud map and the first visible light cloud map to form a cloud atlas to be detected;

Step 4: Use the Yolo algorithm to perform target detection on the cloud images in the detected cloud image set, and identify typical weather phenomena in the cloud images.

Preferably, in the second step, use the open-source opencv library to read all the pixels of the visible light cloud image, and determine the proportion of the number of useless pixels in the visible light cloud image to the total pixel points to obtain the visible light cloud image The percentage of invalid information in .

Preferably, ψ _min =25% and ψ _max =75%.

Preferably, in the step 3, the second visible light cloud image is fused with the corresponding infrared cloud image, including:

Step 1, scaling the second visible light cloud image and the infrared cloud image to be fused to the same size;

Step 2. After the second visible light cloud image and the infrared cloud image are respectively expressed at multiple scales by using the Laplace pyramid, layered fusion is performed on the second visible light cloud image and the infrared cloud image;

Among them, when merging at each level, the region where the useful pixels in the second visible light cloud image are located is fused with other regions in the infrared cloud image according to the position information to obtain a fusion cloud image;

The other area refers to the area in the infrared cloud image corresponding to the location of the useless pixel in the light cloud image.

Preferably, before the step 4, the method further includes: creating a training data set in the COCO data set format according to the satellite cloud image data, training the Yolo algorithm model, and testing the speed and detection accuracy of the training model to obtain the Yolo algorithm meteorological data. Cloud image detection model.

Preferably, the Yolo-based meteorological satellite cloud image target detection method further includes deploying the Yolo algorithm meteorological cloud image detection model on the Web side, using the Flask framework as the front-end framework and the Vue framework as the back-end framework; The function of uploading cloud images on the web side for detection achieves the effect of real-time detection.

Preferably, in the fourth step, the Yolov3 or Yolov5 algorithm is used to perform target detection on the cloud images in the detection cloud atlas.

Preferably, in the fourth step, the Yolov5 algorithm is used to perform target detection on the cloud images in the detected cloud image set.

Preferably, the typical weather phenomena include six weather phenomena: cold front, warm front, typhoon, cyclone, strong convection and torrent cloud system.

The beneficial effects of the present invention are:

(1) The meteorological satellite cloud image target detection method based on Yolo provided by the present invention, the visible light cloud image and the infrared cloud image are fused, not only absorbs the advantage of the clear texture of the visible light cloud image, but also makes up the visible light cloud image from the infrared cloud image. The defect that cannot be measured at night; Combining the two cloud images can obtain more complete and accurate meteorological data.

(2) The Yolo-based meteorological satellite cloud image target detection method provided by the present invention can effectively improve the detection speed and detection accuracy of the satellite cloud image, and achieve the effect of real-time monitoring.

Description of drawings

FIG. 1 is a visible light cloud diagram with useful information in the right half according to the present invention.

FIG. 2 is a visible light cloud diagram with some useful information on the left side before fusion according to the present invention.

FIG. 3 is an infrared cloud image before fusion according to the present invention.

FIG. 4 is a cloud image after fusion using the fusion method in the present invention.

FIG. 5 is an algorithm structure diagram of YOLOv5 according to the present invention.

FIG. 6 is a structural diagram of the CBL according to the present invention.

FIG. 7 is a structural diagram of the Res unit according to the present invention.

FIG. 8 is a structural diagram of the SPP according to the present invention.

FIG. 9 is the main flow chart of the YOLOv5 algorithm according to the present invention.

Figure 10 is a recall image according to the present invention.

Figure 11 is a mAP 0.5 image according to the present invention.

Figure 12 is a mAP 0.5:0.95 image according to the present invention.

Figure 13 is a precision image according to the present invention.

FIG. 14 is a schematic diagram of a meteorological cloud image detection result according to the present invention.

FIG. 15 is a schematic diagram of the MVVM framework according to the present invention.

Detailed ways

The present invention will be further described in detail below with reference to the accompanying drawings, so that those skilled in the art can implement it with reference to the description.

The invention provides a Yolo-based meteorological satellite cloud image target detection method, which is used for infrared channel (wavelength 10.5-12.5 micrometers) and visible light channel (wavelength 0.4-0.78 micrometers) to detect the characteristics of different weather phenomena, and perform image fusion of the two channels , and then use the YOLO algorithm to identify typical weather phenomena in the cloud map.

1. Fusion method of visible light cloud image and infrared cloud image.

First of all, by observing the VIS image (visible light cloud map), it can be seen that there are two kinds of data in the picture that cannot obtain useful information. One is that there is no data that cannot be measured by light, and the background shows a large area of blue; the other is damage. , with a large area of yellow in the background. For both kinds of data, we take the direct discard and use the infrared nephogram as the detection dataset instead.

For part of the visible light cloud map, there is a large area of background information, but there is still a lot of useful information (meteorological information). Unable to detect. If the information in the right half is simply discarded, the cloud image resources will be wasted. Therefore, an image fusion algorithm should be used, and the infrared cloud image should be used to complete the missing information in the left half of the visible cloud image, and then the fused image should be enhanced.

In this embodiment, the image pixel thresholding processing method is used to judge the images, and judge which images are directly discarded, which images are fused, and which images are directly used.

The processing method of image pixel thresholding is as follows: first, use the imread function in python's cv2 function library to read the two images (visible light cloud image and infrared cloud image), and then use the resize function to zoom the image, in order to make the infrared cloud image and the infrared cloud image. The size of the visible light cloud map is the same, which is convenient for future processing. After that, read all the pixels of the cloud image in the image, convert the image data into pure data format, and determine the proportion of useless pixels in the overall image. If it exceeds a certain threshold, it means that there is too much useless information in the visible cloud image. The picture is discarded directly, and the infrared cloud image is used as a substitute. If it is lower than a certain threshold, it means that there is not much useless information in the visible light cloud image and can be used directly. If it is between the two thresholds, it means that the useful information and useless information in the image occupy the In a similar way, the infrared cloud map can be used to fill in the useless information in the visible light map.

In this embodiment, the proportion of useless information in the visible light channel pictures of the 118 satellite cloud images is as follows: 47 pictures are below 10%, 7 pictures are below 20%, 11 pictures are below 30%, 6 pictures are below 40%, 6 5 pictures below 50%, 5 pictures below 60%, 2 pictures below 70%, 4 pictures below 80%, 4 pictures below 90%, 26 pictures below 99%; it can be seen that there are quite a few pictures in the picture A lot of useless information. When the lower threshold is 25% and the upper threshold is 75%, 63 images contain less useless information, 33 images contain more useless information, and 22 images have similar proportions. For the visible light cloud image with more useless information (higher than the upper threshold limit), the infrared cloud image is directly used for replacement and fusion. The visible light nephogram and the infrared nephogram should be deeply integrated.

The invention adopts the improved Laplace fusion algorithm to deeply fuse the visible light cloud image and the infrared cloud image.

The image pyramid is a representation of different resolutions for the same image, and can express images of multiple scales, so the image pyramid can be used for image scaling, image segmentation and other fields. The principle of its multi-scale representation of images is to perform continuous convolution operations and downsampling operations. In the process of each loop operation, the scale of the image will be reduced, and its expressive ability will also be enhanced. The image closer to the top of the pyramid has a higher degree of abstraction to the image and also loses more information. In order to make up for the defect of too much information loss in the downsampling process, the Laplacian algorithm can improve the traditional pyramid, by obtaining the image of the pyramid at each layer, using the upsampling of the image of the previous layer and the image of this layer to linearly The computation generates a new image so that the details of the image are preserved.

In the specific process of upsampling, the image size is first increased by interpolation. The specific operation is to insert blank placeholders by judging the parity of the number of rows and columns, and then use the convolution kernel for filtering. In this way, the result obtained by upsampling is the same as the size of the lower layer, and the Laplacian pyramid is obtained by linear operation of the two.

The way of image fusion in the Laplacian pyramid is to first decompose the image into different scales, and then fuse them separately at each level. First, the fusion is performed from the top pyramid. The method of fusion is to calculate the gradient of the pixels at the same position in the two pictures, and then compare them with the surrounding pixels. If one of the two pictures has a larger pixel gradient, then The pixels of the corresponding image are used in the fused image, because the large gradient indicates that the image edge texture and other information are richer and should be preserved. In the processing of other levels, the gradient information is obtained by combining the image information of the upper layer and performing matrix operations, and then comparing, and selecting the larger gradient in the two images as the fusion result.

The present invention is to perform fusion of infrared cloud image and visible light image. In the visible light image, the illumination is continuous, so the useless information is basically concentrated in a certain area of the image, such as all in the left half or the right half. Therefore, the present invention improves the traditional fusion algorithm according to the characteristics of the visible light cloud image. First, the Laplace operation is performed on the visible light image and the infrared cloud image, and then when each level is fused, the position information is used to determine which image pixel information in the original image is used for the fused image. In this embodiment, combined with 75% and 25% selected in the experiment of the previous stage as the fusion threshold, this stage uses 1/4 of the visible light cloud image and 3/4 of the infrared cloud image for fusion. Specifically, the improved method is to abandon the traditional mode of using gradient determination in the fusion process of each layer, but directly use 1/4 of the visible light cloud image as the pixels in the first half of the image after fusion, and 3/4 of the infrared cloud image as the fusion image. The pixels in the second half of the image are then processed by layered fusion.

Since the image size of the data set of the present invention is 1222*916, the fusion needs to input the image size of the nth power of 2, so the experiment first scales the image to 512*512, and then increases it to 1222*916 after the fusion is completed.

The visible light image and infrared cloud image before fusion are shown in Figure 2 and Figure 3, respectively, and the image after fusion is shown in Figure 4. It can be seen that the fused pictures are stitched naturally, the information of the two pictures are preserved, and the processing effect is better.

After the method provided by the invention is used for the fusion of the visible light cloud image and the infrared cloud image, not only the utilization rate of the image resources is improved, but also the image information is enriched, which lays the groundwork for the next image detection and identification.

2. The YOLO algorithm is used to perform target detection and modeling on satellite cloud images to identify six typical weather phenomena, including cold fronts, warm fronts, typhoons, cyclones, strong convection and torrent cloud systems.

1. Satellite image HDF5 data preprocessing

hdf is a new type of data format, it has many advantages, firstly, all data can be stored in a single hdf file, secondly, data in different formats can be stored in one hdf file, such as symbolic data, numerical data, graphic data, etc. Users can divide the stored data into different levels, and add descriptive information to each level. Layering can be carried out according to the user's own needs, which is more flexible. At the same time, hdf files are more compatible with other formats, and new data modes can be added. No additional data conversion operations are required to transfer hdf files between different platforms, because it is a platform-independent format, and the same hdf files can be manipulated on different platforms.

In satellite cloud image data, hdf format is used more. First of all, the hdf file can store the picture information of the image, and the unit information of the image can also be explained. Secondly, the information of different channels of the satellite cloud image can be stored hierarchically in the hdf file, which is convenient for users to read. The experimental cloud image data adopted in the present invention adopts data in hdf format.

The satellite cloud image data used in the present invention is to intercept cloud images in the range of 0-60 degrees north latitude and 70-150 degrees east longitude from the full disk image, and generate equal longitude and latitude projection files (hdf files). Each hdf file contains 14 datasets, storing 14 channels of isolatitude and longitude projection data. The dataset names are: LonLatChannelB04, LonLatChannelB05, LonLatChannelB06, LonLatChannelB09, LonLatChannelB10, LonLatChannelB11, LonLatChannelB12, LonLatChannelB14, LonLatChannelB16, LonLatChannelIR1, LonLatChannelIR2, LonLatChannelIR3, LonLatChannelIR4, LonLatChannelVIS. The last 3 characters of the dataset name represent the channel name, such as B04, IR1, VIS. The data in the first row and first column of the data set represents the data at 60 degrees north latitude and 70 degrees east longitude, and the data in the last row and last column of the data set represents the data at 0 degrees north latitude and 150 degrees east longitude. Among them, the data set of the VIS channel is 3666 rows and 4888 columns, and the other channel data sets are 916 rows and 1222 columns. The data type of all datasets is a 16-bit big-endian unsigned integer. The unit of data in B04, B05, B06 and VIS datasets is ALBEDO (%), and the unit of B09, B10, B11, B12, B14, B16, IR1, IR2, IR3, IR4 datasets is KELVIN.

Using python's h5py library, the steps are as follows:

(1) First use h5py.File("pathname") to read the file

(2) Read the file of the channel according to the dictionary format

(3) Use the matplotlib library to display pictures. The image can be displayed simply by the function pyplot.imshow() and the function pyplot().

Take the "H8_ALL_PROJ_L2_20190114_0500.hdf" file as an example, read and display the B04 channel image. Among them, the image width is 1222, the height is 916, and the indexed color mode is used.

Afterwards, the displayed image needs to be saved at the original resolution for later neural network training. If you directly use the savefig function in matplotlib.pyplot to save the picture, then the saved picture not only contains the coordinate axis, but also contains the upper, lower, left and right empty edges, which will make the training input inconsistent with our expectations, and the resolution of the picture is larger than the real one The data shows that markup files are also misplaced. Therefore, some settings of the matlabplotlib library are carried out in the present invention.

First, use the set_size_inches function to set the size of the picture, then use the remove coordinate axis display, then use the subplots_adjust function and the margins function to remove the blanks around the picture, and finally use the savefig function to save the picture, and get the ideal picture.

Finally, read each file and each of their channel data in a loop, and save them as files.

The marker file provided by the relevant meteorological experts is a marker file in csv format, including seven fields of type, name, top, left, bottom, right, and filename. For training, six fields of type, top, left, bottom, right, and filename are required.

In the "H8_ALL_PROJ_L2_20190114_0500.hdf" file, some samples can be intercepted and displayed briefly.

2. Build a target detection marker dataset

The YOLO algorithm is based on a feedforward convolutional neural network. YOLOv5 uses CSPDaeknet (cross-stage local area network) as Backbone, and uses PANET (path aggregation network) and SPP (spatial pyramid pooling) as Neck, and uses the same Head as YOLOv3. In YOLOv5, the activation function of each part is different. Leaky ReLU is used in the middle layer and hidden layer, and Sigmod activation function is used in the detection layer. These activation functions are computationally less expensive than the v4 algorithm. YOLOv5 is configured with two optimization functions, and the initialization hyperparameters used for the corresponding training are given respectively. Stochastic gradient optimization, which trains better on large datasets, while Adam optimizer trains better on smaller custom datasets

The YOLOv5 network mainly consists of 4 components:

1) Input layer: perform data enhancement processing on the input image

2) Backbone: Aggregate images at different resolutions to form richer features of images

3) Neck: Mixing the input image features

4) Prediction: Predict the bounding box and the confidence of the category

The algorithm structure is shown in Figure 5.

The individual component structures and tensors are calculated as follows:

(1) CBL (as shown in Figure 6): Combine traditional convolution operations, etc. to generate new network components, which are the most basic components in the network.

(2) Res unit (as shown in Figure 7): There are different numbers of continuous residual structures in the structure, and more res_units can allow the network to be built deeper

(3) Concat: Combine two tensors into one tensor according to a certain dimension.

(4) Add: Add two tensors on the same dimension to generate results of the same size

(5) Spp structure (as shown in Figure 8): First, sampling operations are performed at different sizes, and then different results are fused into one block by calculation, and finally convolution is performed again to modify the size of the output.

In this embodiment, the single-stage detection YOLOv5 algorithm is adopted, and the main process is shown in FIG. 9 .

The data set training of YOLOv5 requires the COCO data set. The present invention first produces the VOC2007 data set, and then converts it.

Create a new folder to store the entire dataset:

There are many tools for labeling pictures. The present invention selects the labelimg open source tool. After saving, a file in xml format with the same name is generated to save the label information. The object field is used to save the information of each sample. There are two important data in the file. The bndbox saves the location of the target, and the name saves the category of the target.

Since the present invention uses a large amount of marked data, it will be cumbersome to use this tool to mark each item one by one, so the present invention uses a simple python program to read the required 6 fields from the csv file that stores the mark in the preceding paragraph and Generate the corresponding xml tag file according to the file format shown in the figure above.

Then put the xml annotation file into the corresponding folder.

The name of the training file used by each category is stored in the txt file. The present invention uses a simple python program to generate, and then the data needs to be converted into a coco data set, which is also realized by a python file. The corresponding image name or label file name is stored in each train or val directory.

3. Training parameter settings

After the data set is prepared, the model is further configured with parameters. In this example, the YOLOv5x model with the highest accuracy of the YOLOv5 network is used.

(1) Set the model configuration file

Modify the category of the object to be detected to 6 according to the yolov5x.yaml file.

(2) Set the training profile

According to the coco.yaml file, modify the training folder and verification folder paths, the category of the object and the label of each category.

4. Training and evaluation

The training environment is configured as follows:

(1) Operating system: CentOS Linux release 7.8.2003 (Core)

(2) Graphics card: 8 TITAN X graphics cards (2 of which were used in the experiment)

(3) cpu: 56

(4) Memory: 256G

(5) cuda version: 11.0

(6)pytorch version: 1.7.1

The algorithm sets hyperparameters before training.

Training process:

(1) Set the parameters of DDP mode

You can set the number of global processes, process number, etc.

(2) Set whether to resume training

(3) Load processing hyperparameter list,

(4) Model training

First, to obtain the path of the training log, you need to read the result file. The result file includes the training results saved in each iteration. After analyzing the results of the previous operation, you can better perform the next experiment. After that, some model settings need to be performed, such as setting the parameters for saving the ground truth, which will generally be saved when it is smaller than 3, and log files will be generated during training, and the algorithm will be saved in the evolve folder. Set the path to save the weight files generated in each iteration, and set the path to save the results of each training. Then get some information from the program, such as the training round, the batch of data set input, the total batch of training (distributed training), the weight file generated after training a picture, and the program number (distributed training). Then you need to save some settings at runtime, save the hyperparameter file hyp and the command parameter opt of the project. Identify whether the type of cuda uses GPU for training, and obtain the number of categories and category names of the recognized targets. If single_cls is set in the command line parameters, the targets to be detected belong to one category. Then read the model parameter file in pt format. If you choose to use pre-training, you can call the script to automatically download the pre-trained model from the network, and then create the model, or you can use the command line or configuration file to set parameters. The difference between using the file or the command line is whether the recovery parameters are set. If the model will be created according to the configuration file; the parameter setting will lead to the setting of the keywords in the anchor box parameters, that is, whether to load the predefined anchor box, if the training is set. Restoring parameters will continue training on the basis of the last training weight parameters; the file has provided the user with the default training parameters of the dataset, the default is to use the custom weight in the file, if no parameters are set, the user can directly set the anchor box and then Load the pre-training weights for training, the user-defined settings will be overwritten by the file settings, and the file settings have a higher priority; so before the program runs, you need to set the parameters to decide which parameters to use. If the weight file is used, the training process is also performed. If so, the position of the anchor box in the weight file is discarded and the anchor box defined by the user is used; if the recovery parameters are set, the parameters are not discarded, and the weight and the anchor box configuration file are loaded together. It is also necessary to set the frozen layer model, set the image size batch_size of the simulated processing batch, select the appropriate optimizer, and set the optimization method of the pg0 group, set the parameter weight of the training process, and optimize the bn for batch normalization of features. mode, set the optimization mode of the bias parameter biases, and then print the optimized information. You also need to set the learning rate decay to measure the speed, and the model will use cosine annealing to decay and then resume training. Initialize the first generation of training data to choose whether to continue training with the best previous training results. Then save the weight of the best parameter file during the training process as best.pt according to best_gitness. Then make a judgment. If the previous training process has been completed, then perform a new round of experiments, set parameters, set the loading optimizer and best_fitness, load the training result result.txt, and load the training round. If the recovery parameters are set, the weights should be backed up. Although the current recovery parameters can be approximately perfect without errors, other problems may occur during the parameter recovery process and the previous weights may be lost. Using additional backup operations can make the data more secure. . If the set number of training epoches is relatively large, the newly set epoches is the number of rounds that need to be retrained instead of the total number of rounds. Then read the total step size of the model and the model input image resolution from the model structure file, and check the input image resolution set by the user to ensure that the step size gs is divisible. Then check whether the distributed training is set, set the rank process number and whether to use the DataParallel mode that only supports native multi-card, when the rank is set to -1 and there is only one gpu, distributed training will be skipped. After synchronizing BatchNorm, use cross-card synchronization bn. Create an EMA exponential moving average for the model. If the number of GPU processes is greater than 1, this index is not created. Then create the dataloader of the training set, get the largest category value in the label, and compare it with the number of categories. If it is greater than the number of categories, there is a problem. After this start setting up the model. Set the coefficient of classification loss, set the number of categories, and hyperparameters according to the number of categories of your own dataset. Then set the value of giou as the coefficient of the label in the objectness loss, initialize the image sampling weight according to the labels, and obtain the name of the category. Then start to set the frequency of the categories, stitch the labels of all samples together, and visualize them after statistics. Get the categories of all samples, and then set the length, width and center position for visualization based on this statistic. Calculates the aspect ratio value of the default anchor point anchor and the dataset label box. If the number of label boxes satisfying the above conditions is less than 99% of the total, then some unsupervised machine learning methods are used to obtain new anchor settings.

Then start the training of the model. Set the number of iterations for warm-up training, initialize mAP and results. Set the number of rounds for the learning rate decay. The purpose is that after the training is interrupted, the recovery parameters can continue the training and can also connect to the previous training for the learning rate decay. Then print the resolution of the training and test input images, the number of cpu processes called when loading the image, and determine the number of serial numbers that epoches start with. Then enter the loop training for each epoch. If the picture sampling strategy is set, according to the previously initialized picture sampling weight model.class_weights and maps and the number of categories contained in each picture, the picture index indices are generated through random.choices to select the sampling. If DDP mode is set, the sampling policy is broadcast. Then initialize the average loss information printed during training. If the data is scrambled in DDP mode, the random sampling data of ddp.sampler is based on epoch+seed as the random seed, so each time the epoch is different, the random number seed is also different. Then use tqdm to create a progress bar to facilitate the display of information during training. Then calculate the number of iterations. Use warm-up training in the first nw iterations, and choose accumulate and learning rate in a certain way. The learning rate of the bias parameter is decreased from 0.1 to the baseline learning rate lr*lf(epoch), and the other learning rates are increased from 0 to lr*lf(epoch). lf is the decay coefficient of the cosine annealing set before, and the parameter momentmum gradually increases from 0.9. Then multi-scale training is performed, and the size is randomly selected from imgsz*0.5, imgsz*1.5+gs. Then carry out mixed-precision forward propagation, and calculate the loss including classification loss, injectness loss, box regression loss, and total loss value. The value of loss is stored loss_items as a tuple containing classification loss, objectness loss, box regression loss and total loss. Then do backpropagation. The model backpropagates a certain number of times, then updates the parameters once according to the accumulated gradient and then performs learning rate decay. If it is a single GPU or cpu for training, mAP needs to be updated, including updating the EMA attribute and adding the include attribute. Determine whether the epoch is the last round. Then the test set is tested, and indicators such as mAP are calculated. At the same time, it is responsible for writing the indicators into the result file. If the bucket command line parameter is set, the result file needs to be uploaded to the network. Then set the visualization tool Tensorbord, add model metrics, training loss and other information to the tensorboard display. Update the best mAP by best_fitness. After that, save the model. In addition, you need to save information such as epoch, results, optimizer, etc. The optimizer will be saved after the last round is completed. Other parameters, such as model saves the EMA model. After the model training is completed, use the strip_optimizer function to remove the optimizer from the ckpt, and in order to reduce the variable storage, perform the model.half operation on the model to change the Float32 model to Float16, which can reduce the model size and improve the inference speed. After the end, visualize the results file. Finally free the video memory of the graphics card.

Evolve the hyperparameters:

In the conda environment, use the train.py file, set the device to "cuda0, 1", set the parameter configuration file, and set the batch size of model iteration to 16.

The results were obtained after training for 300 epochs of 1 hour 13 minutes 35 seconds.

The experimental recall image is shown in Figure 10, and the experiment reaches a maximum of more than 96%. The mAP_0.5 image is shown in Figure 11, and the highest is more than 97%; the mAP_0.5:0.95 image is shown in Figure 12. The accuracy image is shown in Fig. 13, the highest accuracy is close to 99%.

mAP is mean AP. When evaluating the results of the neural network, this value reflects the goodness of the model. The higher the mAP value, the higher the accuracy and the better performance of the corresponding classification algorithm. mAP is calculated by first calculating the AP value for each target detection and then taking a weighted average of them. The minimum value of mAP is 0 and the maximum value is 1. The larger the value, the better. This criterion is the most important in the target detection algorithm. The YOLOv5x model used in this experiment has an average precision value close to 99% for all categories. At the same time, the precision, recall rate and PR curve of this experiment have achieved good results. While achieving good training results, our training speed is very fast, which can reach around 30ms.

In another embodiment, experiments were conducted using the YOLOv3 algorithm under the same conditions. The YOLOv3 algorithm was slightly weaker than YOLOv5 in terms of precision and recall, but the training time of YOLOv3 was shorter, using 49 minutes and 35 seconds under the same conditions. . That is, in the same case of 300 iterations, the training time of the YOLOv3 algorithm is reduced by 24 minutes compared with the YOLOv5 algorithm, and the recall rate is increased by 1.43%, but the accuracy of the YOLOv3 algorithm is 3.75% lower than that of the YOLOv5 algorithm. So if the accuracy requirements are not particularly high, you can choose to use the YOLOv3 algorithm for faster training.

5. Results testing and analysis

The testing process is as follows. First, determine whether the test is called during training, and if so, obtain the equipment during training. Otherwise, call it directly and select the device. Delete the previous test_batch0_gt.jpg and test_batch0_pred.jpg. Then load the model and check if the resolution of the input image is divisible by 32. If the device is not a cpu and the number of GPUs is 1, convert the model from Float32 to FLoat16 to improve the speed of forward propagation. Then configure it, load the data configuration information, and set the iou threshold, which is taken every 0.05 from 0.5 to 0.95. Then set the Dataloader and set the rect parameter to true because the test evaluation of yolov5 is based on rectangle inference.

Initialize test parameters and path index information. Set the display information of the tqdm progress bar. Initialize indicator, time. Initialize loss for test set, initialize dictionary stats for json file, ap. Then disable gradient propagation, let the model do forward propagation, and calculate the time of forward propagation cumulatively. Calculate the loss. If it is a test performed during training, it is necessary to calculate and return the GIoU, obj, and cls losses of the test set through the training results. Then set the confidence threshold and iou threshold and perform NMS processing, and calculate the cumulative processing time of NMS.

Then make statistics for each picture, write the prediction information to the txt file, generate a json file dictionary, count tp, etc. First obtain the label information of the si-th image, including class, x, y, w, h. Get the label category, count the number of test images, if the prediction is empty, add empty information to stats and save the prediction result to a txt file. By obtaining the length and width of the corresponding image, and then setting the path of the txt file according to the image name. Adjust the coordinates of the prediction box to the coordinates based on its original length and width. Save the categories and coordinates to a txt file. Correct the predicted coordinates to the inside of the image. Save a dictionary of json files in coco format. First get the picture id, get the frame, adjust the frame to the size of the original image, and convert it to xywh format. The channel order of the format has changed. The frame coordinate format in coco's json format is xyzh, where xy is the upper-left corner coordinate, that is, the coordinate format of coco's json format is: upper-left corner coordinate + length and width, so it is necessary to set The coordinates of the center point are converted to the upper left corner. The prediction evaluation is then initialized. Store the detected target, obtain the box in xyxy format and multiply it by wh, process each class in the picture separately, calculate the index of the label box and the index of the prediction box. Calculate the iou value of the prediction box and select the largest ious value. Then add the detected target to detected to get true positives under different iou thresholds. Then count the structure of each image into stats. Draw the ground truth and prediction box of the first batch of pictures and save. The information from the stats list is then stitched together. Print the indicator results and show the indicators for each category in detail. Print the time spent on forward propagation, the time in nms, and the total time. Using the previously saved json format prediction results, and evaluating the indicators through cocoapi, the labels of the test set also need to be converted into coco json format. First get the image id, get the json file path of the prediction box and open it. Get and initialize a json file of test set labels. File to initialize prediction boxes. Create an evaluator. Evaluate and present the results. Finally, return the test indicator results.

The present invention tests all the pictures once, and can frame the types of clouds and their confidence in the pictures, wherein 0 means cold front, 1 means warm front, 2 means typhoon, 3 means cyclone, 4 means strong convection, 5 indicates the jet stream system. The cloud image detection effect is shown in Figure 14. In Figure 14, the two positions are circled with confidence boxes, the detected target belongs to the category 3 (cyclone), and the probability values of the display prediction are 0.87 and 0.91, respectively.

3. Model deployment and application

1. Model web deployment

(1) Flask framework

Flask is a lightweight framework in the python language, which provides the most basic functions of a web framework and is more flexible and flexible than other frameworks. Its core is relatively simple, and only provides basic communication functions. Users need to choose and expand to achieve specific needs. Separating functionality from the framework eliminates the need for users to install unnecessary functionality every time, but to add them at their own discretion.

Flask has a huge advantage, it allows users to develop more options, and the design style is minimalist, allowing users to build applications with greater freedom. First of all, the design of Flask is lightweight and modular, and a variety of rich extensions can be used to design programs that meet the needs. Secondly, the basic UI design of Flask's API is beautiful and coherent, so that a simple design can produce a beautiful interface. The Flask model is small, thanks to the fact that the python platform is easier to deploy in the actual environment. In addition, Flask supports various http request processing functions. The framework is highly flexible and flexible in configuration, which can meet various output requirements.

The present invention needs to deploy the neural network framework on the web side, and only needs to use some network functions, so Flask in the back-end framework becomes a better choice, and the required functions can be added to the micro-framework. In addition, since the neural network framework platform used in the present invention is written in the python language, the two can be more conveniently and natively combined by using Flask.

(2) Vue framework

Vue adopts MVVM architecture, namely View (view), ViewModel (logic), Model (data), as shown in Figure 15. In the MVVM framework, the view layer and the data layer do not exchange data directly, but through the ViewModel layer. Vue uses two-way data binding technology, ViewModel is responsible for monitoring user data, and then notifying the view layer when changes occur. When the user operates the view layer through the ViewModel, it will also notify the corresponding data for persistence operation. The MVVM architecture provides a hierarchical architecture abstraction for front-end pages. It can also combine other js libraries such as Ajax to realize data communication and persistence, enriching the functions and experience of the system. Vue takes ViewModel in MVVM as the core, and adopts two-way data binding technology to ensure the consistency of M-layer and V-layer data, and it is a lightweight framework, which makes the development process simple and efficient. Vue adopts a unique data-driven approach, which changes the way the previous DOM view is manually updated. As long as the data changes, the view will automatically change. In addition, Vue makes the view componentized, a whole web page is divided into different blocks, each block is a component. The network is composed of multiple components spliced and nested, which achieves better maintainability and reusability of the code.

The Vue framework has many features. First of all, it is a lightweight front-end framework that can automatically track and calculate expressions and computed properties in templates. It uses the MVVM architecture and has a rich component system with diversified interfaces. Secondly, it implements two-way binding of data, uses templates and instructions to perform data operations in the document object model, and enables DOM to achieve automatic real-time rendering. Vue has a rich instruction system, and almost all events in the developing page can be completed using the instruction description in Vue. Directives can be combined with variable values to change and are bound to DOM applications accordingly. Vue also has a powerful component function, extending the elements in the original HTML. It enables code reuse through encapsulation. At the same time, there is an inherited communication method between components. The parent and child components transmit messages in attribute parameters, and the data is unidirectionally transmitted from the parent component to the child component. The child component responds to the message to update the data of the parent component. In this way, the development of components is closely related to html and javascript. Users can choose the components they need to integrate, which speeds up the development process and reduces the amount of code. At the same time, the development of components supports hot reloading. Vue uses the Vue-router plugin, which can build applications with only one page. The routing plug-in sets the user's path to web browsing, and maps the user's access to the path to a custom component. Compared with the traditional way of using hyperlinks, this setting realizes a single-page application

When the user accesses, Vue displays the view, and the user sends a request to the server to upload the image again. The server receives the client parsing and parsing, flask saves the received image to the upload directory, then calls the related functions of yolov5 to realize image recognition, and finally returns it to Vue to update the view.

The trained model is deployed on the Web side, using the Flask framework as the front-end framework and the Vue framework as the back-end framework, to detect images through the trained weight files. The function of uploading cloud images for detection on the web side is completed, and the effect of real-time detection is achieved.

The multi-channel image fusion method for satellite cloud images provided by the present invention, for the characteristics of different weather phenomena in the infrared channel (wavelength 10.5-12.5 microns) and visible light channel (wavelength 0.4-0.78 microns), the fusion of two channel images is performed: a) using The open source opencv library counts invalid information in the visible light channel cloud image and sets a threshold for filtering. Discard the pictures with too much invalid information and completely retain the pictures with less invalid information, and fuse the pictures in the middle area; b) Laplacian operation is performed on the infrared cloud image and the visible light cloud image respectively, and established on the basis of the Gaussian image pyramid Laplacian Image Pyramid. A new fusion strategy is then proposed. According to the threshold in a), set the ratio and relative position of the two cloud image fusion, perform position-based pixel-level fusion for each layer, and then perform linear filling on the image to obtain the fusion result.

On the basis of image fusion, the YOLO algorithm is used to detect and model satellite cloud images to identify six typical weather phenomena, including cold fronts, warm fronts, typhoons, cyclones, strong convection and torrent cloud systems. The cloud images of each channel are extracted from the original satellite cloud image data (HDF5 format), and the visible light cloud image and the infrared cloud image are fused. Then create a training dataset in COCO dataset format, perform model training, and test the trained model. In the environment used in this embodiment, the YOLOv5 model can achieve a picture detection speed of about 30FPS and a detection accuracy of more than 95%, which well meets the requirements of satellite cloud image detection speed and accuracy.

The invention also deploys the trained model on the Web side, uses the Flask framework as the front-end framework, and the Vue framework as the back-end framework, and detects pictures through the trained weight file. The function of uploading cloud images for detection on the web side is completed, and the effect of real-time detection is achieved.

Although the embodiment of the present invention has been disclosed as above, it is not limited to the application listed in the description and the embodiment, and it can be applied to various fields suitable for the present invention. For those skilled in the art, it can be easily Therefore, the invention is not limited to the specific details and illustrations shown and described herein without departing from the general concept defined by the appended claims and the scope of equivalents.

Claims (9)

1. A meteorological satellite cloud target detection method based on yolk is characterized by comprising the following steps:

extracting a plurality of infrared cloud pictures and a plurality of visible light cloud pictures from original satellite cloud picture data acquired by a meteorological satellite;

the infrared cloud pictures and the visible light cloud pictures are in one-to-one correspondence;

counting invalid information in the visible light cloud pictures, and dividing the visible light cloud pictures into a plurality of pieces according to the proportion of the invalid information in the visible light cloud pictures: a first visible light cloud picture, a second visible light cloud picture and a third visible light cloud picture;

wherein the invalid information proportion psi of the first visible light cloud picture ₁ Satisfy psi ₁ ＜ψ _min ；

The invalid information of the second visible light cloud picture accounts for psi ₂ Satisfy psi _min ≤ψ ₂ ≤ψ _max ；

The ratio psi of invalid information of the third visible light cloud picture ₃ Satisfy psi ₃ ＞ψ _max ；

In the formula, /) _min For a lower threshold of invalid information, # _max An invalid information threshold upper limit;

step three, fusing the second visible light cloud picture with the infrared cloud picture corresponding to the second visible light cloud picture to obtain a fused cloud picture; and

forming a cloud atlas to be detected by the infrared cloud atlas, the fused cloud atlas and the first visible light cloud atlas corresponding to the third visible light cloud atlas;

and step four, performing target detection on the cloud pictures in the detected cloud picture set by adopting a Yolo algorithm, and identifying typical weather phenomena in the cloud pictures.

2. The method for detecting the cloud target of the meteorological satellite based on the Yolo of claim 1, wherein in the second step, an open source opencv library system is used for reading all the pixel points of the visible cloud image, and the number of the useless pixel points in the visible cloud image is judged to be the proportion of the total pixel points, so that the invalid information proportion in the visible cloud image is obtained.

3. The method for detecting cloud target of meteorological satellite based on Yolo according to claim 2, characterized in that psi _min ＝25％，ψ _max ＝75％。

4. The method for detecting the targets by the cloud images of the meteorological satellite based on the yolk according to the claim 2 or the claim 3, wherein in the third step, the second visible light cloud image is fused with the corresponding infrared cloud image, and the method comprises the following steps:

step 1, zooming the second visible light cloud picture and the infrared cloud picture to be fused into the same size;

step 2, after the second visible light cloud picture and the infrared cloud picture are respectively subjected to multi-scale expression by adopting a Laplace pyramid, the second visible light cloud picture and the infrared cloud picture are subjected to layered fusion;

and when fusion is carried out on each layer, the region where the useful pixel points in the second visible light cloud picture are located is fused with other regions in the infrared cloud picture according to the position information, so that a fused cloud picture is obtained.

5. The method for detecting the cloud target of the meteorological satellite based on the yolk according to claim 4, further comprising before the fourth step: and manufacturing a training data set in a COCO data set format according to the satellite cloud picture data, training the Yolo algorithm model, and testing the speed and the detection precision of the training model to obtain the Yolo algorithm meteorological cloud picture detection model.

6. The method for detecting the targets of the cloud images of the meteorological satellites based on the yolk according to claim 5, further comprising the steps of carrying out Web-side deployment on the yolk algorithm cloud image detection model, wherein a flash frame is used as a front-end frame, and an Vue frame is used as a back-end frame; the cloud picture is uploaded at the web end to be detected, and the effect of real-time detection is achieved.

7. The method for detecting the target of the cloud images of the meteorological satellite based on the yolk according to claim 6, wherein in the fourth step, the yolk 3 or yolk 5 algorithm is adopted to detect the target of the cloud images in the detected cloud image set.

8. The method for detecting cloud target based on Yolo weather satellite images as claimed in claim 7, wherein in the fourth step, the Yolov5 algorithm is adopted to perform target detection on the cloud images in the detected cloud image set.

9. The yolk-based weather satellite cloud target detection method of claim 8, wherein the typical weather phenomenon comprises: the cold front, warm front, typhoon, cyclone, strong convection and torrent clouds are 6 weather phenomena.

Images