CN108595484B

CN108595484B - Marine atmosphere waveguide data acquisition and visualization processing method

Info

Publication number: CN108595484B
Application number: CN201810206210.4A
Authority: CN
Inventors: 杨嘉琛; 赵启明
Original assignee: Tianjin University
Current assignee: Tianjin University
Priority date: 2018-03-13
Filing date: 2018-03-13
Publication date: 2021-03-26
Anticipated expiration: 2038-03-13
Also published as: CN108595484A

Abstract

The invention relates to a marine atmospheric waveguide data acquisition and visualization processing method, which comprises the following steps: and data interaction and information acquisition among the atmospheric waveguide multi-sensors. Preprocessing and distributed fusion of data. And carrying out initial control and abnormal value marking operation on the original data file. And (3) carrying out continuity check on the data which is not marked by the abnormal value in the data file processed in the third step by using a filtering-mean square error control method. The vertical variation of the data was checked for consistency. And (4) carrying out curve fitting and interpolation by using a BP neural network, inputting the original data processed in the fifth step as a training array, and carrying out data interpolation according to the obtained fitting curve. And carrying out average calculation on the data of the multiple groups of sensors. Visual graphical display and visual processing of data: and carrying out graphical drawing according to the data processed by the steps.

Description

Marine atmosphere waveguide data acquisition and visualization processing method

Technical Field

The invention belongs to the field of ocean data processing, and relates to ocean atmospheric waveguide data acquisition and visualization processing.

Background

The close relation between big data and ocean science and technology enables ocean big data to be processed into a hot spot which is concerned at home and abroad, and ocean big data acquisition and processing are the basis and the premise for developing and utilizing the ocean, are the key for promoting the further development of the ocean science and technology and obtaining the international competitive advantage, and have important significance for realizing the strong national strategy of the ocean in China. However, the marine information data is rich and complex, the data volume is large and most of heterogeneous data, and the application of the large data processing in the marine information processing is very small although the large data processing has been widely applied in other fields, so that the application of the large data acquisition and processing method in the marine field has become a key problem.

At present, in marine data acquisition and processing, atmospheric waveguide data is an important part, and has important significance for evaluating and predicting electromagnetic wave propagation and marine detection communication systems. However, as for the atmospheric waveguide data acquisition and processing system, the research at home and abroad is still incomplete, and with the rise of data visualization processing, a set of complete atmospheric waveguide data acquisition and visualization processing system has become a hotspot and difficulty of the current research.

Disclosure of Invention

The invention aims to provide an atmospheric waveguide data acquisition and visualization processing method, which is used for performing visualization data processing on raw data on the basis of multi-sensor information interaction and distributed fusion acquired data. The technical scheme is as follows:

a marine atmospheric waveguide data acquisition and visualization processing method comprises the following steps:

the first step is as follows: and data interaction and information acquisition among the atmospheric waveguide multi-sensors.

The second step is that: preprocessing and distributed fusion of data. Binary data acquired by each sensor is converted into a required data type, and a distributed fusion mode is adopted for each sensor to obtain an original data file of the atmospheric waveguide.

The third step: and carrying out initial control and abnormal value marking operation on the original data file.

The fourth step: and (3) carrying out continuity test on data which is not marked by the abnormal value in the data file processed in the third step by using a filtering-mean square error control method, including calculation of a sliding average value and a mean square error, and if the relation between the sliding average value and the mean square error is judged to be abnormal, marking the abnormal data.

The fifth step: the vertical variation of the data was checked for consistency. According to the phenomenon that meteorological elements including temperature are continuously distributed in the vertical direction, three layers of data including a bottom layer, a middle layer and a top layer of the data are selected to perform mean square error calculation, and whether the requirement of consistency is met is judged.

And a sixth step: and (4) carrying out curve fitting and interpolation by using a BP neural network, inputting the original data processed in the fifth step as a training array, and carrying out data interpolation according to the obtained fitting curve.

The seventh step: average of the values: and carrying out average calculation on the data of the multiple groups of sensors.

Eighth step: visual graphical display and visual processing of data: and performing graphical drawing according to the data processed by the steps, drawing graphs including a line graph, a block diagram, a wind speed graph and a contour graph, and further processing the data according to the visual display of the graphs.

The ninth step: and outputting the standardized file.

Preferably, in the fifth step, for the temperature element, the temperature varies linearly with height in the vertical direction, i.e.

DX＝|X₁-X₂|

DX is the difference between the element values of the height levels 1 and 2.

Determining the element consistency range:

a. setting DX as 2;

b. selecting three-layer temperature element X in observation data_1i、X_2i，X_3iI is 1, …, M, which represents the data of bottom layer, middle layer and top layer, the mean square error is calculated for the corresponding time point element values of the bottom layer and the middle layer:

the points marked as abnormal and the points with the difference value of more than 2 between the upper layer and the lower layer are not counted and accumulated, the point number is M, and the mean square error Y is obtained₁

With Y₁M is taken as the upper limit Y, and m is 1.5 or 2 times.

c. The mean square deviations of the top layer temperature data and the middle layer temperature data are respectively calculated by the method, and the mean square deviations of the bottom layer temperature data and the top layer temperature data are recorded as Y₂、Y₃

Checking consistency

a. If | X_1i-X_2i|＜Y₁Description of X_1i、X_2iAnd the requirement of vertical consistency inspection is met.

b. If | X_1i-X_2i|＞Y₁Then continue to compare | X_3i-X_2iThe value of | if | X_3i-X_2i|＜Y₂

Then X is considered_1iAnd marking the data of the layer without meeting the requirement of vertical consistency check.

The atmospheric waveguide acquisition and data visualization processing method provided by the invention fully considers the richness and the heterogeneity of data, adds a data interpolation method based on a BP neural network on the basis of combining data fusion and data processing, performs visual display and processing of data, has good data processing effect after verification of real data, can be applied to actual atmospheric waveguide data acquisition and processing, and has a certain reference value for acquisition and processing modes of other ocean data.

Drawings

FIG. 1 schematic diagram of multi-sensor interactive acquisition and preprocessing

FIG. 2 data processing flow diagram

FIG. 3 is a schematic view of data visualization display and processing

Detailed Description

The invention utilizes multi-sensor information interaction and distributed fusion to acquire data, processes the data by a data fitting and interpolation method based on a BP neural network, performs graphical display and visual processing on the data on the basis, and finally performs standard file generation operation on the processed data. The specific technical scheme is as follows:

the first step is as follows: and data interaction and information acquisition among the atmospheric waveguide multi-sensors. The information of the ocean atmospheric waveguide is collected through the temperature sensor, the pressure sensor, the wind speed sensor and other sensors, and data sharing and cooperative control can be achieved through data interaction among the sensors.

The second step is that: preprocessing and distributed fusion of data. Because the data collected by the sensor is binary data, the binary data is converted into the required data type for the convenience of the subsequent data processing operation. And a distributed fusion mode is adopted for each sensor, and finally, an original data file of the atmospheric waveguide is obtained.

The third step: and carrying out initial control and abnormal value marking operation on the original data file. And deleting data with position information loss such as GPS (global positioning system) caused by abnormal input of a ship body information common signal, and deleting data which exceeds a reasonable value range of the data and has too large mutation.

The fourth step: and (3) carrying out continuity test on the data by using a filtering-mean square error control method, wherein the continuity test comprises the calculation of a moving average value and a mean square error, and the marking treatment is carried out on abnormal data. Specifically, a certain element X in the observation data is selected_i(i 1, …, N), running average of data not labeled with outliers

(i ═ 1, …, N), the mean square error of the corresponding elements before and after filtering was calculated:

when in use

When the mean square error is larger than m times, X is considered to be_iOutliers are marked.

The fifth step: the vertical variation of the data was checked for consistency. Because meteorological elements are generally continuously distributed in the vertical direction, three continuous layers of data are selected for mean square error calculation, and whether the requirement of consistency is met is judged. Specifically, taking the temperature element as an example, the temperature linearly varies with the height in the vertical direction, i.e.

DX＝|X₁-X₂| (2)

DX is the difference between the element values of the height levels 1 and 2.

Determining the element consistency range:

a. first, suppose DX is 2;

b. selecting three-layer temperature element X in observation data_1i(i＝1,…,M)、X_2i(i＝1,…,M)，X_3i(i ═ 1, …, M), representing the bottom layer, middle layer, top layer data, respectively, the mean square error was first calculated for the corresponding time point element values of the bottom layer and middle layer:

the points marked as abnormal and the points with the difference value of the upper layer and the lower layer larger than 2 are not counted and accumulated, the number of the points is M, and the mean square error Y is obtained₁

With Y₁M times as the upper limit Y, and a large number of practices prove that the value of m is 1.5 or 2 times.

Checking consistency

c. If | X_1i-X_2i|＜Y₁Description of X_1i、X_2iAnd the requirement of vertical consistency inspection is met.

d. If | X_1i-X_2i|＞Y₁Then continue to compare | X_3i-X_2iThe value of | if | X_3i-X_2i|＜Y₂

And a sixth step: and carrying out curve fitting and interpolation by using the BP neural network, inputting the original data as a training array, and carrying out data interpolation according to the obtained fitting curve. In particular, the amount of the solvent to be used,

suppose the input data is a vector of length N, p (N) ═ p₁,p₂,…,p_N]Then the output vector can be represented as o (n) ═ o₁,o₂,…,o_N]. Setting the excitation function of the BP neural network as a Tan-sigmoid function, and recording the function as

And adopting Levenberg-Marquardt as a training method, and calculating the stepping amount according to the following formula:

wherein J_kIs the jacobian matrix of the function f. And (3) taking the processed data as input, setting a proper number of neurons, and training the neurons through a neural network to obtain fitting.

The seventh step: average of the values. Since the sensor performs one set of measurement every 3 seconds, the average data in minutes is often needed in practical application, and the average calculation is needed for multiple sets of data. In particular, the amount of the solvent to be used,

because the source is regular, every 3 seconds is a set of data, the accumulation of n to m points is first calculated (skipping over the points marked as outliers, the number of points is counted as k), and then divided by the accumulated number to obtain the average of the values, as follows:

eighth step: and (4) visual graphic display and visual processing of data. And performing graphical drawing according to the processed data, wherein the graphical drawing comprises drawing of other graphs such as a line graph, a block diagram, a wind velocity diagram, a contour diagram and the like, and further processing the data according to the visual display of the graphs.

The ninth step: and outputting the standardized file. Based on the data processing, the processed data is sorted and standardized to be output.

Claims

1. A marine atmospheric waveguide data acquisition and visualization processing method comprises the following steps:

the first step is as follows: data interaction and information acquisition among the atmospheric waveguide multi-sensors;

the second step is that: preprocessing and distributed fusion of data: converting binary data acquired by each sensor into a required data type, and obtaining an original data file of the atmospheric waveguide by adopting a distributed fusion mode for each sensor;

the third step: carrying out initial control and abnormal value marking operation on the original data file;

the fourth step: carrying out continuity test on data which is not marked by the abnormal value in the data file processed in the third step by using a filtering-mean square error control method, wherein the continuity test comprises calculation of a sliding average value and a mean square error, and if the relation between the sliding average value and the mean square error is judged to be abnormal, marking the abnormal data;

the fifth step: the vertical change of the data was checked for consistency: according to the phenomenon that meteorological elements including temperature are continuously distributed in the vertical direction, three layers of data of a bottom layer, a middle layer and a top layer are respectively selected in the vertical direction, mean square error calculation is carried out, and whether the requirement of consistency is met is judged;

and a sixth step: performing curve fitting and interpolation by using a BP neural network, inputting the original data processed in the fifth step as a training array, and performing data interpolation according to the obtained fitting curve;

the seventh step: average of the values: carrying out average calculation on data of a plurality of groups of sensors;

eighth step: visual graphical display and visual processing of data: carrying out graphical drawing according to the data processed by the steps, drawing graphs including a line graph, a block diagram, a wind speed graph and a contour graph, and further processing the data according to the visual display of the graphs;

the ninth step: and outputting the standardized file.