CN114444580A - Big data processing method based on convolutional neural network - Google Patents
Big data processing method based on convolutional neural network Download PDFInfo
- Publication number
- CN114444580A CN114444580A CN202210008247.2A CN202210008247A CN114444580A CN 114444580 A CN114444580 A CN 114444580A CN 202210008247 A CN202210008247 A CN 202210008247A CN 114444580 A CN114444580 A CN 114444580A
- Authority
- CN
- China
- Prior art keywords
- big data
- convolution
- processing
- data
- convolved
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/22—Matching criteria, e.g. proximity measures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computational Linguistics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Evolutionary Biology (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Image Analysis (AREA)
Abstract
The invention realizes a big data processing method based on a convolutional neural network, wherein the function processing of highly dense track data in a terminal area can be processed, and the movement of multiple targets in a longer time sequence can be tracked by combining a multi-target detection model of deep learning and a target tracking frame of a filtering method through multi-module improvement; the method comprises the steps of training optical flow information and a detail enhancement series method based on image features, so that the information tracked by a final frame contains more details of the target; finally, the frame is used for tracking the big data information after dynamic convolution in a plurality of scenes, and a plurality of evaluation methods are designed, so that the accuracy and the efficiency of the tracking method are effectively proved, the local data processing of the big data of the digraph and the undirected graph is supported, the connection among the big data characteristics after convolution is easily increased and deleted, namely, the connection among the big data characteristics after convolution is easily maintained.
Description
Technical Field
The invention belongs to the technical field of informatization, and particularly relates to a big data processing method based on a convolutional neural network.
Background
The centralized processing system of the navigation plan is a centralized data processing system which completes centralized acceptance, centralized processing and unified release on dynamic messages of the navigation plan submitted by a shipping company. And on the basis of completely unifying the advance navigation plan, the format and the content of the navigation plan are verified and audited and distributed to the user to pass. The system mainly aims to improve the data quality of the navigation plan through centralized message acceptance and audit, and meet the data requirements of users such as control users, traffic management users, statistical clearing units and military parties.
With the increase of the types of the data recorded by the airline references, the realization of the comprehensive data playback in the ground security equipment is one of the important requirements of users. The navigation data and the multi-channel audio and video data can be processed and played comprehensively. At present, ground playback equipment of a navigation parameter recording system is relatively independent, the playback process of data does not show the processing characteristic, and in addition, even if multiple kinds of data are played back simultaneously, the processing performance among the data is poor, and a perfect processing mechanism is not provided.
Disclosure of Invention
The invention aims to provide a big data processing method based on a convolutional neural network, which can better process the data playback process of big data such as navigation data playback, audio data playback and video and the like, can be applied to the processing and playback of various data in a local area network, and solves the processing problem and the comprehensive playback problem among various navigation parameter data.
The invention relates to a big data processing method based on a convolutional neural network, which is characterized by comprising the following steps:
step 001, preprocessing the newly added batch big data to obtain a time-aligned track pair, extracting input features as input of a convolutional neural network, then defining a convolutional kernel function, an activation function, a pooling layer, a full-link layer and an output layer of the convolutional neural network, matching the current track pair with a track in a collision avoidance knowledge base, and obtaining big data after convolution;
step 002, completing function processing on the convolved big data through cosine similarity, completing convolution distribution function processing of data to be processed, decomposing the grid big data, and decomposing and describing the grid big data as the relation between the convolved big data characteristic and the convolved big data characteristic;
step 003, the time series functionalization processing is finished on the convolved big data processed by the convolution distribution function by using the sequence relation mapping, and the relation storage data between the convolved big data characteristic and the convolved big data characteristic is stored by using an Oracle table of a relational database;
step 004, initializing parameters including a big data feature identifier after starting convolution to be processed, data to be processed, column members in a column family, processing depth and a queue, and finishing dynamic processing of big data after convolution by a big data dynamic functionalization processing method after convolution of a learning model and a big data feature functionalization processing method after convolution of shadow tracking;
and 005, inquiring the big data characteristics after the convolution is started according to the big data characteristic identification after the convolution is started, inserting the big data characteristics after the convolution is started into the queue, finishing the processing when the current processing depth is greater than the initialized processing depth and the queue is empty, and processing the column members in the column family in the Oracle table corresponding to the big data characteristics after the convolution at the head of the queue.
Further, in step 002, the function processing is completed on the convolved big data through the cosine similarity, the convolution distribution function processing of the data to be processed is completed, the grid big data is decomposed, and the grid big data is decomposed and described as a relation between the convolved big data feature and the convolved big data feature, and the method further includes:
combining the multiple function views with the time information to obtain a ship group in an active state within a period of time;
acquiring the relation between data in convolutions with different characteristics and the occupation conditions of the two areas so as to reflect the passing condition of the convolution of the navigation program;
classifying the marine program and displaying differences within the same class according to the efficiency of processing the marine program at intervals between the ships;
generating a trajectory based on the raw data, normalizing the data to organize the data in time series;
reflecting the relation among the large data features after convolution through the large data feature features after convolution, and distinguishing the large data features after different convolutions by using the large data feature marks after convolution;
processing the convolution characteristic of the shipping device, generating a spiral structure view and a safety view to display the convolution utilization rate of the navigation program and the state of the ship interval, and generating a track;
defining an identifier capable of distinguishing each convolved big data feature in the whole situation as a convolved big data feature identifier, wherein the convolved big data feature identifiers correspond to the convolved big data features one by one;
constructing a similarity matrix between the tracks through a rapid dynamic time warping algorithm, and then classifying the tracks through spectral clustering; and extracting the characteristics of the navigation track data, and reconstructing the navigation program which actually passes through in the terminal area according to the irregular navigation track data.
Further, the step 003 of performing time series functionalization processing on the convolved big data processed by the convolution distribution function by using sequence relation mapping, and storing the linkage storage data between the convolved big data feature and the convolved big data feature by using an Oracle table of a relational database, further includes:
representing delay data in convolution according to a time sequence relation, displaying a function of delay change of different times and a relation between areas to obtain and compare delay propagation;
converting the convolved big data feature identification into a recorded row keyword; the contact characteristics, the processing characteristics and other characteristics are respectively converted into a contact column family, a column family and other column families of the Oracle table; the data or data fields to be processed are stored in respective column members of a column family;
and defining a function form of the delay influence factor, and acquiring no weight of the function influence factor changing along with time in different areas.
Further, in step 004, initializing parameters including feature identifiers of the big data after starting convolution to be processed, the data to be processed, column members in a column family, processing depth and a queue, and completing dynamic processing of the big data after convolution by a big data dynamic functionalization processing method after convolution of a learning model and a big data feature functionalization processing method after convolution of shadow tracking, the method further includes:
marking the peak value of the scalar field corresponding to the density map as a target in a training set, and identifying a low-density target in the gray map by using the model according to a training result;
calculating optical flows of all pixels in the images based on a dense optical flow method, selecting an optical flow algorithm to obtain an optical flow field of a pair of images, and calculating a motion field between the pair of images;
setting parameters to generate dynamic Berlin noise movement, wherein the initial parameters generate a density map animation comprising a small number of density lumps, the significance of the objects is high, and the grabbing interval is short;
resetting parameters, increasing the size of a single density cluster and the number of density clusters contained in a single area, and expanding the grabbing range and the interval of data collection;
expanding the collected data set, wherein the expanded data set is also used for model training, and repeating the steps until enough data are collected after a certain amount of data are obtained;
decomposing a multi-target tracking task into target detection and motion detection through a multi-target tracking model, extracting image characteristics, and selecting ResNet to replace VGG in an original model;
predicting a possible state of the moving object, and comparing the predicted object state with a target measurement state at the next time. The algorithm corrects the matching relation according to the difference between the target predicted state and the real state, finally confirms the process that the position of the target moves along with the time, and then further describes the moving trend of the target.
Further, in step 005, according to the big data feature identifier after the start of convolution for processing, querying the big data feature after the start of convolution for processing, inserting the big data feature after the start of convolution for processing into the queue, where the current processing depth is greater than the initialized processing depth and the queue is empty-time to end processing, and processing the column members in the column family in the Oracle table corresponding to the big data feature after the head of the queue convolution, the method further includes:
when the feature depth of the big data after the convolution of the queue head is larger than the processing depth, adding the big data feature after the convolution of the queue head, which is not accessed, of the big data after the convolution of the queue head, into the queue, recording the current processing depth of the big data feature after the convolution of the queue head and the big data feature after the convolution of the queue head, and deleting the big data feature elements after the convolution of the queue head;
and judging whether the current processing depth is greater than the initialized processing depth and the queue is empty or not, if so, finishing the processing, and otherwise, turning to the step 003.
The invention realizes a big data processing method based on a convolutional neural network, wherein the function processing of highly dense track data in a terminal area can be processed, and the movement of multiple targets in a longer time sequence can be tracked by combining a multi-target detection model of deep learning and a target tracking frame of a filtering method through multi-module improvement; the method comprises the steps of training optical flow information and a detail enhancement series method based on image features, so that the information tracked by a final frame contains more details of the target; finally, the frame is used for tracking the big data information after dynamic convolution in a plurality of scenes, and a plurality of evaluation methods are designed, so that the accuracy and the efficiency of the tracking method are effectively proved, the local data processing of the big data of the digraph and the undirected graph is supported, the connection among the big data characteristics after convolution is easily increased and deleted, namely, the connection among the big data characteristics after convolution is easily maintained.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings through which the embodiments or the prior art descriptions need to pass are briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a flowchart of a big data processing method based on a convolutional neural network according to the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be described clearly and completely with reference to the accompanying drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In the description of the present invention, it is to be understood that the terms "central," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "radial," "circumferential," and the like are used in the orientations and positional relationships indicated in the drawings for convenience in describing the invention and to simplify the description, but are not intended to indicate or imply that the device or element so referred to must have a particular orientation, be constructed in a particular orientation, and be operated in a particular manner, and are not to be construed as limiting the invention. Furthermore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the present invention, "a plurality" means two or more unless otherwise specified.
In the description of the present invention, it should be noted that, unless otherwise explicitly specified or limited, the terms "mounted," "connected," and "connected" are to be construed broadly, e.g., as meaning either a fixed connection, a removable connection, or an integral connection; can be mechanically or electrically connected; they may be connected directly or indirectly through intervening media, or they may be interconnected between two elements. The specific meanings of the above terms in the present invention can be understood in specific cases to those skilled in the art.
The invention relates to a big data processing method based on a convolutional neural network, which is characterized by comprising the following steps:
step 001, preprocessing the newly added batch big data to obtain a time-aligned track pair, extracting input features as input of a convolutional neural network, then defining a convolutional kernel function, an activation function, a pooling layer, a full-link layer and an output layer of the convolutional neural network, matching the current track pair with a track in a collision avoidance knowledge base, and obtaining big data after convolution;
step 002, completing function processing on the convolved big data through cosine similarity, completing convolution distribution function processing of data to be processed, decomposing the grid big data, and decomposing and describing the grid big data as the relation between the convolved big data characteristic and the convolved big data characteristic;
step 003, the time series functionalization processing is finished on the convolved big data processed by the convolution distribution function by using the sequence relation mapping, and the relation storage data between the convolved big data characteristic and the convolved big data characteristic is stored by using an Oracle table of a relational database;
step 004, initializing parameters including a big data feature identifier after starting convolution to be processed, data to be processed, column members in a column family, processing depth and a queue, and finishing dynamic processing of big data after convolution by a big data dynamic functionalization processing method after convolution of a learning model and a big data feature functionalization processing method after convolution of shadow tracking;
and 005, inquiring the big data characteristics after the convolution is started according to the big data characteristic identification after the convolution is started, inserting the big data characteristics after the convolution is started into the queue, finishing the processing when the current processing depth is greater than the initialized processing depth and the queue is empty, and processing the column members in the column family in the Oracle table corresponding to the big data characteristics after the convolution at the head of the queue.
In the description herein, references to the description of the term "one embodiment," "some embodiments," "an illustrative embodiment," "an example," "a specific example," or "some examples" or the like mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
While embodiments of the invention have been shown and described, it will be understood by those of ordinary skill in the art that: various changes, modifications, substitutions and alterations can be made to the embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.
Further, in step 002, the function processing is completed on the convolved big data through the cosine similarity, the convolution distribution function processing of the data to be processed is completed, the grid big data is decomposed, and the grid big data is decomposed and described as a relation between the convolved big data feature and the convolved big data feature, and the method further includes:
combining the multiple function views with the time information to obtain a ship group in an active state within a period of time;
acquiring the relation between data in convolutions with different characteristics and the occupation conditions of the two areas so as to reflect the passing condition of the convolution of the navigation program;
classifying the marine program and displaying differences within the same class according to the efficiency of processing the marine program at intervals between the ships;
generating a trajectory based on the raw data, normalizing the data to organize the data in time series;
reflecting the relation among the large data features after convolution through the large data feature features after convolution, and distinguishing the large data features after different convolutions by using the large data feature marks after convolution;
processing the convolution characteristic of the shipping device, generating a spiral structure view and a safety view to display the convolution utilization rate of the navigation program and the state of the ship interval, and generating a track;
defining an identifier capable of distinguishing each convolved big data feature in the whole situation as a convolved big data feature identifier, wherein the convolved big data feature identifiers correspond to the convolved big data features one by one;
constructing a similarity matrix between the tracks through a rapid dynamic time warping algorithm, and then classifying the tracks through spectral clustering; and extracting the characteristics of the navigation track data, and reconstructing the navigation program which actually passes through in the terminal area according to the irregular navigation track data.
Data processing and functionalization are completed based on cosine similarity, data construction comprises time, flight number, longitude, latitude, altitude, ground speed and course, the signals are collected according to a time sequence, and the coverage area of the signals is larger than the actual terminal area; which part of the collected data belongs to the terminal area under study needs to be acquired and the ship data on the high altitude routes can interfere with the modeling. Second, different vessels transmit signals at different frequencies. Some vessels have data that is too small to model over time and should be correctly excluded. Finally, vessel data collected over a period of time may be part of the complete marine process, and it is also a challenge how to correctly classify the trajectories based on the partial data. In addition, as described above, since the spatial domain processing in the terminal area is microscopic compared to the route processing, the screening of data cannot be too extensive, but the data needs to be simplified in order to avoid unnecessary computation overhead; data are unified, and the track is rasterized into a grid through a Bresenham algorithm to resample the track; the similarity matrix is constructed by calculating the similarity between any two tracks, and the track data is sequenced according to the flight number by the following algorithm:
aX，Y＝Dist(Xi，Yj)+min[D(Xi-1，Yj)，D(Xi，Yj-1)，D(Xi-1，Yj-1)]
wherein a is an element in a distance matrix, X and Y represent data of two time series, the data correspond to tracks acquired by different ships, i and j respectively mean lengths corresponding to the X data and the Y data, the calculated distances are based on Euclidean distances, and a dynamic programming technology is adopted in the calculation process. The greater the difference between the two sequences X and Y, aX，YThe higher will be the value of (c). And further constructing similarity matrixes of all tracks in the terminal area, namely sim matrixes for short. Each element in sim is the difference between the maximum value in the distance matrix calculated by the above formula and the corresponding element. Sim is set as a kernel function (e.g., RBF kernel function), and then eigenvalues are computed through a diagonal matrix and a laplace matrix. In the next step, the data is re-clustered from a new matrix consisting of feature vectors.
Based on the airspace information overview of a statistical chart, functionally improving parallel coordinate axes to represent the passing of different time-interval airspace heights; based on the safety monitoring of the relative position vector view, the center circle is set as the minimum safety interval (5 km) in the civil aviation system, and the area between the two circles will change in proportion according to the difference of the set values. The safe interval value may be increased if the user wants to observe the data in a larger convolution scale. Conversely, if the value of the horizontal safety interval is decreased, the same inter-element distance in the view will increase. In this view, the center of the view corresponds to the position of the sea runway, and all vessel positions are switched to polar coordinates. Discrete color coding rules are selected based on convolution functionalization of the helical structure model. Through the color change of the spiral structure, the user can know the average speed of the ship in different areas. And (4) safe interval processing, namely finding certain ships too close to each other in the same height by reducing the height difference between flights. Subsequently, the safety interval standard is reduced to 10 km, which is generally used as a general shipping safety standard.
Further, the step 003 of performing time series functionalization processing on the convolved big data processed by the convolution distribution function by using sequence relation mapping, and storing the linkage storage data between the convolved big data feature and the convolved big data feature by using an Oracle table of a relational database, further includes:
representing delay data in convolution according to a time sequence relation, displaying functions of delay change at different times and displaying the relation among areas so as to obtain and compare delay propagation;
converting the convolved big data feature identification into a recorded row keyword; the contact characteristics, the processing characteristics and other characteristics are respectively converted into a contact column family, a column family and other column families of the Oracle table; the data or data fields to be processed are stored in respective column members of a column family;
and defining a function form of the delay influence factor, and acquiring no weight of the function influence factor changing along with time in different areas.
The main information in the data includes: flight number, departure sea area, destination, estimated departure time, actual departure time, estimated arrival time, actual arrival time, and delay time. According to the dynamic delay expression designed by the Poisson distribution, a Poisson distribution taking the concerned sea area as the center is generated in a rapid Poisson disc sampling mode, and each data point contained in the distribution indicates a delay event. To ensure that each region has a consistent distribution shape, a poisson disk is pre-generated in the system as a template to ensure that the data points in this work all follow the same poisson distribution.
The center of the initial poisson distribution is at the sea location of interest. When the delay level is stepped, the center of the distribution will move towards the source data direction causing the delay, and the newly generated points will be distributed around the updated distribution center. The last step is carried out on all relevant sea areas; each iteration generates a new point based on the existing poisson distribution shape or around the new distribution due to a delayed level change. If a delay step occurs, the distance traveled by the nuclei is the same as the interval of the poisson distribution, and it is checked whether the new nucleus location overlaps with an existing nucleus.
Generating a delay change field, and generating dynamic delay Poisson distribution based on delay events of a continuous time sequence; the contour map is switched according to 4 delay levels to display the delay condition at time t. In the experiment, when the color thermodynamic diagram was converted into a contour map, the density of data generation was taken as a scalar value to calculate the contour. The variation fields between two successive delayed heating powers are calculated by StreamMap and then displayed by the distribution of arrows. The trend of the delay variation is indicated by an arrow and the vector field is projected to the contour plane. The density of the arrows indicating the field change varies according to the length of the boundary, thereby ensuring that the direction of change can be clearly shown regardless of the length of the field lines.
Further, in step 004, initializing parameters including feature identifiers of the big data after starting convolution to be processed, the data to be processed, column members in a column family, processing depth and a queue, and completing dynamic processing of the big data after convolution by a big data dynamic functionalization processing method after convolution of a learning model and a big data feature functionalization processing method after convolution of shadow tracking, the method further includes:
marking the peak value of the scalar field corresponding to the density map as a target in a training set, and identifying a low-density target in the gray map by using the model according to a training result;
calculating optical flows of all pixels in the images based on a dense optical flow method, selecting an optical flow algorithm to obtain an optical flow field of a pair of images, and calculating a motion field between the pair of images;
setting parameters to generate dynamic Berlin noise movement, wherein the initial parameters generate a density map animation comprising a small number of density lumps, the significance of the objects is high, and the grabbing interval is short;
resetting parameters, increasing the size of a single density cluster and the number of density clusters contained in a single area, and expanding the grabbing range and the interval of data collection;
expanding the collected data set, wherein the expanded data set is also used for model training, and repeating the steps until enough data are collected after a certain amount of data are obtained;
decomposing a multi-target tracking task into target detection and motion detection through a multi-target tracking model, extracting image characteristics, and selecting ResNet to replace VGG in an original model;
predicting a possible state of the moving object, and comparing the predicted object state with a target measurement state at the next time. The algorithm corrects the matching relation according to the difference between the target predicted state and the real state, finally confirms the process that the position of the target moves along with the time, and then further describes the moving trend of the target.
Creating a density map animation and obtaining a data set in the form of the density map from the animation, and cutting the image and adjusting the size of the image to be used as data to be learned. And taking the multiple targets in the two finally divided pictures as two non-intersecting sets in the bipartite graph, wherein each target can be regarded as vertex processing in the graph, and thus, the target matching problem is converted into the vertex matching problem. The algorithm finally recursively converts all vertices in the set to saturation points to ensure that all vertices have a matching relationship.
Further, in step 005, according to the big data feature identifier after the start of convolution for processing, querying the big data feature after the start of convolution for processing, inserting the big data feature after the start of convolution for processing into the queue, where the current processing depth is greater than the initialized processing depth and the queue is empty-time to end processing, and processing the column members in the column family in the Oracle table corresponding to the big data feature after the head of the queue convolution, the method further includes:
when the feature depth of the big data after the convolution of the queue head is larger than the processing depth, adding the big data feature after the convolution of the queue head, which is not accessed, of the big data after the convolution of the queue head, into the queue, recording the current processing depth of the big data feature after the convolution of the queue head and the big data feature after the convolution of the queue head, and deleting the big data feature elements after the convolution of the queue head;
and judging whether the current processing depth is greater than the initialized processing depth and the queue is empty or not, if so, finishing the processing, and otherwise, turning to the step 003.
The present invention is not limited to the above preferred embodiments, but rather, any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (5)
1. A big data processing method based on a convolutional neural network is characterized by comprising the following steps:
step 001, preprocessing the newly added batch big data to obtain a time-aligned track pair, extracting input features as input of a convolutional neural network, then defining a convolutional kernel function, an activation function, a pooling layer, a full-link layer and an output layer of the convolutional neural network, matching the current track pair with a track in a collision avoidance knowledge base, and obtaining big data after convolution;
step 002, performing function processing on the convolved big data through cosine similarity, performing convolution distribution function processing on the data to be processed, decomposing the grid big data, and decomposing and describing the grid big data as the relation between the convolved big data characteristic and the convolved big data characteristic;
step 003, the time series functionalization processing is finished on the convolved big data processed by the convolution distribution function by using the sequence relation mapping, and the relation storage data between the convolved big data characteristic and the convolved big data characteristic is stored by using an Oracle table of a relational database;
step 004, initializing parameters including a big data feature identifier after starting convolution to be processed, data to be processed, column members in a column family, processing depth and a queue, and finishing dynamic processing of big data after convolution by a big data dynamic functionalization processing method after convolution of a learning model and a big data feature functionalization processing method after convolution of shadow tracking;
and 005, inquiring the big data characteristics after the convolution is started according to the big data characteristic identification after the convolution is started, inserting the big data characteristics after the convolution is started into the queue, finishing the processing when the current processing depth is greater than the initialized processing depth and the queue is empty, and processing the column members in the column family in the Oracle table corresponding to the big data characteristics after the convolution at the head of the queue.
2. The big data processing method based on the convolutional neural network as claimed in claim 1, comprising:
in the step 002, the function processing is completed on the convolved big data through the cosine similarity, the convolution distribution function processing of the data to be processed is completed, the grid big data is decomposed, and the grid big data is decomposed and described as the relation between the convolved big data feature and the convolved big data feature, and the method further includes:
combining the multiple function views with the time information to obtain a ship group in an active state within a period of time;
acquiring the relation between data in convolutions with different characteristics and the occupation conditions of the two areas so as to reflect the passing condition of the convolution of the navigation program;
classifying the marine program and displaying differences within the same class according to the efficiency of processing the marine program at intervals between the ships;
generating a trajectory based on the raw data, normalizing the data to organize the data in time series;
reflecting the relation among the large data features after convolution through the large data feature features after convolution, and distinguishing the large data features after different convolutions by using the large data feature marks after convolution;
processing the convolution characteristic of the aircraft, generating a spiral structure view and a safety view to display the convolution utilization rate of the navigation program and the state of the ship interval, and generating a track;
defining an identifier capable of distinguishing each convolved big data feature in the whole situation as a convolved big data feature identifier, wherein the convolved big data feature identifiers correspond to the convolved big data features one by one;
constructing a similarity matrix between the tracks through a rapid dynamic time warping algorithm, and then classifying the tracks through spectral clustering;
and extracting the characteristics of the navigation track data, and reconstructing the navigation program which actually passes through in the terminal area according to the irregular navigation track data.
3. The big data processing method based on the convolutional neural network as claimed in claim 1, comprising:
the step 003 of performing time series functionalization processing on the convolved big data processed by the convolution distribution function by using sequence relation mapping, and storing the linkage storage data between the convolved big data feature and the convolved big data feature by using an Oracle table of a relational database, further includes:
representing delay data in convolution according to a time sequence relation, displaying a function of delay change of different times and a relation between areas to obtain and compare delay propagation;
converting the convolved big data feature identification into a recorded row keyword; the contact characteristics, the processing characteristics and other characteristics are respectively converted into a contact column family, a column family and other column families of the Oracle table; the data or data fields to be processed are stored in respective column members of a column family;
and defining a function form of the delay influence factor, and acquiring no weight of the function influence factor changing along with time in different areas.
4. The big data processing method based on the convolutional neural network as claimed in claim 1, comprising:
step 004, initializing parameters including feature identification of big data after starting convolution to be processed, data to be processed, column members in a column family, processing depth and a queue, and completing dynamic processing of big data after convolution by a big data dynamic functionalization processing method after convolution of a learning model and a big data feature functionalization processing method after convolution of shadow tracking, further comprising the following steps:
marking the peak value of the scalar field corresponding to the density map as a target in a training set, and identifying a low-density target in the gray map by using the model according to a training result;
calculating optical flows of all pixels in the images based on a dense optical flow method, selecting an optical flow algorithm to obtain an optical flow field of a pair of images, and calculating a motion field between the pair of images;
setting parameters to generate dynamic Berlin noise movement, wherein the initial parameters generate a density map animation comprising a small number of density lumps, the significance of the objects is high, and the grabbing interval is short;
resetting parameters, increasing the size of a single density cluster and the number of density clusters contained in a single area, and expanding the grabbing range and the interval of data collection;
expanding the collected data set, wherein the expanded data set is also used for model training, and repeating the steps until enough data is collected after a certain amount of data is obtained;
decomposing a multi-target tracking task into target detection and motion detection through a multi-target tracking model, extracting image characteristics, and selecting ResNet to replace VGG in an original model;
predicting a possible state of the moving object, and comparing the predicted object state with a target measurement state at the next time. The algorithm corrects the matching relation according to the difference between the target predicted state and the real state, finally confirms the process that the position of the target moves along with the time, and then further describes the moving trend of the target.
5. The big data processing method based on the convolutional neural network as claimed in claim 1, comprising:
step 005, according to the big data feature identification after starting convolution for processing, querying the big data feature after starting convolution for processing, inserting the big data feature after starting convolution for processing into the queue, where the current processing depth is greater than the initialized processing depth and the queue is empty-time ending processing, and processing the column members in the column family in the Oracle table corresponding to the big data feature after first convolution, further comprising:
when the feature depth of the big data after the convolution of the queue head is larger than the processing depth, adding the big data feature after the convolution of the queue head, which is not accessed, of the big data after the convolution of the queue head, into the queue, recording the current processing depth of the big data feature after the convolution of the queue head and the big data feature after the convolution of the queue head, and deleting the big data feature elements after the convolution of the queue head;
and judging whether the current processing depth is greater than the initialized processing depth and the queue is empty or not, if so, finishing the processing, and otherwise, turning to the step 003.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210008247.2A CN114444580A (en) | 2022-01-06 | 2022-01-06 | Big data processing method based on convolutional neural network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210008247.2A CN114444580A (en) | 2022-01-06 | 2022-01-06 | Big data processing method based on convolutional neural network |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114444580A true CN114444580A (en) | 2022-05-06 |
Family
ID=81364987
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210008247.2A Pending CN114444580A (en) | 2022-01-06 | 2022-01-06 | Big data processing method based on convolutional neural network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114444580A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116684452A (en) * | 2023-08-04 | 2023-09-01 | 华云天下(南京)科技有限公司 | Knowledge center construction method and system based on AIGC large model |
-
2022
- 2022-01-06 CN CN202210008247.2A patent/CN114444580A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116684452A (en) * | 2023-08-04 | 2023-09-01 | 华云天下(南京)科技有限公司 | Knowledge center construction method and system based on AIGC large model |
| CN116684452B (en) * | 2023-08-04 | 2023-10-03 | 华云天下(南京)科技有限公司 | Knowledge center construction method and system based on AIGC large model |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109738970B (en) | Method, device and storage medium for realizing lightning early warning based on lightning data mining | |
| CN112526513B (en) | Millimeter wave radar environment map construction method and device based on clustering algorithm | |
| CN112116830B (en) | Unmanned aerial vehicle dynamic geo-fence planning method based on airspace meshing | |
| CN104977584B (en) | A kind of nowcasting method and system of convection weather | |
| CN108805350B (en) | Search and rescue range prediction method based on multi-dimensional Monte Carlo theory | |
| CN112101278A (en) | Hotel point cloud classification method based on k nearest neighbor feature extraction and deep learning | |
| CN110232398A (en) | A kind of road network sub-area division and its appraisal procedure based on Canopy+Kmeans cluster | |
| US20140088799A1 (en) | Systems and methods providing a fuel-efficient rta implementation with uncertain winds | |
| CN108520023A (en) | A kind of identification of thunderstorm core and method for tracing based on Hybrid Clustering Algorithm | |
| CN110009037B (en) | Short-term engineering wind speed prediction method and system based on physical information coupling | |
| CN113591380B (en) | Traffic flow prediction method, medium and equipment based on graph Gaussian process | |
| CN109061774B (en) | Thunderstorm core correlation processing method | |
| CN109961470B (en) | Living standing tree She Shuxing accurate estimation method based on laser point cloud | |
| CN111291098B (en) | Track mode mining method and device | |
| CN115098993A (en) | Unmanned aerial vehicle conflict detection method and device for airspace digital grid and storage medium | |
| CN114444580A (en) | Big data processing method based on convolutional neural network | |
| CN112200083A (en) | Airborne multi-spectral LiDAR data segmentation method based on multivariate Gaussian mixture model | |
| CN115752708A (en) | Airport single-point noise prediction method based on deep time convolution network | |
| Wu et al. | A non-rigid hierarchical discrete grid structure and its application to UAVs conflict detection and path planning | |
| CN114443763A (en) | Big data synchronization method based on distributed network | |
| CN107239559B (en) | Method for calculating position of space moving target based on vector grid | |
| JP2002257933A (en) | Meteorological radar data processor | |
| CN111881964A (en) | Linear building mode identification method and system based on Delaunay triangulation network | |
| CN115718623A (en) | Heterogeneous radar point trace fusion parallel processing method in dense clutter environment | |
| CN113867410B (en) | Unmanned aerial vehicle aerial photographing data acquisition mode identification method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |