CN110995815B

CN110995815B - Information transmission method based on Gaia big data analysis system

Info

Publication number: CN110995815B
Application number: CN201911181992.1A
Authority: CN
Inventors: 黄山; 房六一; 逯波; 段晓东
Original assignee: Dalian Minzu University
Current assignee: Dalian Minzu University
Priority date: 2019-11-27
Filing date: 2019-11-27
Publication date: 2022-08-05
Anticipated expiration: 2039-11-27
Also published as: CN110995815A

Abstract

An information transmission method based on a Gaia big data analysis system belongs to the field of distributed information transmission such as data mining, machine learning and application statistics under a big data environment. The technical points are as follows: marking key attributes of the data in the historical statistical information by using a data analysis tool; using the key attribute marked in step S1 to call the key attribute of each piece of data with the key attribute to the front; the client transmits the data after the order is adjusted in step S2 to the server; when receiving data, the server side firstly analyzes the first few key attributes, if no key attribute exists, the server side does not continue to analyze, and if the key attribute exists, the server side continues to analyze. Has the advantages that: according to the invention, by adjusting the sequence of the attributes in each piece of data transmitted by the client, the number of pieces of data required to be analyzed is reduced when the server analyzes the data, only the required data is analyzed, and the required time can be greatly reduced.

Description

Information transmission method based on Gaia big data analysis system

Technical Field

The invention belongs to the field of distributed information transmission such as data mining, machine learning and application statistics under a big data environment, and particularly relates to an information transmission method based on a Gaia big data analysis system.

Background

Gaia is a high-timeliness and extensible new generation big data analysis system oriented to multi-computing model mixing and coexistence. The method solves a series of key technical problems at several core levels of big data analysis systems such as self-adaption, telescopic big data storage, batch flow fusion big data calculation, high-dimensional large-scale machine learning, high-timeliness big data intelligent interaction guide and the like, constructs a new generation of autonomous and controllable high-timeliness telescopic big data analysis system, and masters the international leading big data analysis system core technology.

The novel big data computing system has a full-period multi-scale optimization and unified computing engine for batch flow mixing tasks. The existing big data computing system simulates the behavior of another type of framework by depending on a computing engine of the existing big data computing system, or defines a set of general interfaces to shield the difference of underlying computing engines, and has weak support for batch flow fusion. Meanwhile, the optimization is mostly located at a specific period or a specific level of execution, and the optimization capability for high-complexity tasks is insufficient. Aiming at the problems, a high-performance batch flow fusion big data computing technology based on a unified computing engine and full-period multi \ scale optimization is innovatively developed. The technology provides a unified expression logic support for batch flow fusion processing, and realizes the real fusion of batch and flow processing by integrating a calculation model, a data model, a transformation model and an action model of batch flow processing through unified expression modeling. Aiming at the characteristics of diversity, persistence, iteration and the like of the operation, optimization strategies facing to multi-operation, multi-task, iterative computation, persistent computation and the like are provided, and the optimization pertinence is stronger. Meanwhile, full-period optimization before and during execution is provided, and the optimization is subdivided into a plurality of scales such as a job level, a task level and a transformation level, so that extremely-fast response and massive throughput are realized.

Gaia is an open source computing platform facing distributed data stream processing and batch processing, and supports distributed parallel computing, including model parallel and data parallel. Big data computing engines are common and various in the society of today, and in recent years, the general computing framework comprising Hadoop and Spark and the special computing framework comprising TensorFlow are provided. This is a result of our diversification of the computational model requirements. In addition, large data frameworks based on resource management, support services and storage services are available. Big data is closely related to the development of people's survival. The content supported by the big data engine comprises various applications such as batch processing, flow calculation, machine learning and interactive analysis.

Two of the most important tasks are unique to data mining for large data. Firstly, real-time performance, such massive data scale needs real-time analysis and rapid result feedback. Secondly, accuracy is achieved, valuable information needed by users and hidden in the valuable information is required to be accurately extracted from massive data, the information obtained through mining is converted into organized knowledge to be expressed in modes of models and the like, and therefore the analysis models are applied to real life to improve production efficiency and optimize marketing schemes.

Therefore, whichever large data computing engine requires a fast response, i.e., low latency, there are many problems in life that require feedback in real time. In the case of a rapid increase in data, it is important to make a high throughput, low latency system.

Gaia integrates a plurality of advantages, including rapidness, reliability, expandability, complete compatibility with Hadoop, simple use and excellent performance. Using memory-based data flow and deep integration of iterative processing algorithms into the runtime of the system, Gaia enables the system to process data-intensive and iterative tasks at extremely fast speeds. Gaia also has high reliability and good scalability. And is Hadoop compatible. In addition, Gaia includes Java-based APIs for bulk and stream data-based analysis, optimizers, and distributed runtimes with custom memory management functions, etc., and thus can be fully compatible with Hadoop. In summary, Gaia has the high efficiency, flexibility and extensibility of a distributed MapReduce class of platform, and parallel database query optimization scheme, while it supports batch and stream-based data analysis, and provides Java-based APIs. In general, Gaia is an efficient, distributed, Java-based implementation of a general big data analytics engine.

One important characteristic of Gaia is low latency, but the original transmission method may waste a lot of resources during the information transmission process.

Disclosure of Invention

In order to enable each node to fully utilize resources and save the time for filtering information, the invention provides an information transmission method based on a Gaia big data analysis system, which can save time and improve efficiency when transmitting and filtering information.

The technical scheme is as follows:

an information transmission method based on a Gaia big data analysis system comprises the following steps:

s1, marking key attributes of the data in the historical statistical information by using a data analysis tool;

s2, using the key attribute marked in the step S1 to adjust the key attribute of each piece of data with the key attribute to the front;

s3, the client sends the data after the sequence is adjusted in the step S2 to the server;

s4, when the server receives the data, the first several key attributes are analyzed, if no key attribute exists, the analysis is not continued, and if the key attribute exists, the analysis is continued.

Further, a Socket is used for establishing communication connection between the client and the server.

Further, the client sends data to the server by using the IO stream.

Further, the data analysis tool is Python or Mysql.

Further, if there are several key attributes, the several key attributes are adjusted to the front according to the front-back order before adjustment.

According to the information transmission method based on the Gaia big data analysis system, the sequence of the attributes in each piece of data transmitted by the client is adjusted, so that the number of pieces of data to be analyzed is reduced when the server analyzes the data, only the needed data is analyzed, and the needed time can be greatly reduced; the method can filter out the unnecessary information, effectively save the time for transmitting the information, greatly improve the efficiency of information transmission and reasonably utilize resources.

Drawings

FIG. 1 is a process diagram for adjusting the key attribute order in an embodiment of the present invention;

fig. 2 is a basic framework diagram of information transmission in the embodiment of the present invention.

Detailed Description

The information transmission method based on the Gaia big data analysis system is further explained with reference to fig. 1-2.

Example 1

s1, marking key attributes of the data by using historical statistical information and a data analysis tool;

s2, using the key attribute analyzed in the step S1 to adjust the key attribute of each piece of data with the key attribute to the front;

s4, the server side analyzes the first few key attributes when receiving data, if no key attribute exists, the analysis is not continued, and if the key attribute is concerned, the analysis is continued;

further, the data analysis tool is Python or Mysql.

Further, if there are several key attributes, the several key attributes are adjusted to the front according to the front-back order before adjustment. The criteria for determining the key attributes is, regardless of the other attributes, as long as the attributes can identify the tag of the piece of data. For example, if a certain attribute in a piece of data is 1, the tag of the piece of data will not change regardless of other attributes.

Further, in step S1, the client and the server are connected to each other to ensure mutual communication.

Further, in step S2, each piece of data has one or more key attributes that can be determined for the information tag. Such as in a piece of intrusion detection data: 2, tcp, smtp, SF,1684,363,0,0,0,0,0,1,0,0,0,0,0,0, 1,1,0.00,0.00,0.00,0.00,1.00,0.00,0.00,104,66,0.63,0.03,0.01,0.00,0.00,0.00,0.00,0.00, normal. The last attribute is a label, the data is marked as a normal type and is not invaded, and the data is of the normal type if the 11 th attribute and the 12 th attribute are combined in a way of 01.

If bits 11 and 12 are other numbers, such as a piece of data: 0, tcp, private, REJ,0,0,0,0,0,0,0,0,0,0,0, 38,1,0.00,0.00,1.00,1.00,0.03,0.55,0.00,208,1,0.00,0.11,0.18,0.00,0.01,0.00,0.42,1.00, portsweet, this piece of data is labeled as portsweet (an attack) type.

Further, in step S3, the client sends the newly adjusted sequence of data to the server, where the data takes the first data in S2 as an example, and the data to be sent to the server after the adjustment is: 0,1,2, tcp, smtp, SF,1684,363,0,0,0,0,0, 0,0,0,0,1,1,0.00,0.00,0.00,0.00,1.00,0.00,0.00,104,66,0.63,0.03,0.01,0.00,0.00, 0.00. Since the normal is a label and is marked later, the real data is unmarked, so the normal does not need to be sent to the server side.

Further, in step S4, in the server-side parsing process, taking the previous intrusion detection data as an example: 0,1,2, tcp, smtp, SF,1684,363,0,0,0,0,0, 0,0,0,0,1,1,0.00,0.00,0.00,0.00,1.00,0.00,0.00,104,66,0.63,0.03,0.01,0.00,0.00,0.00, resolving the first two bits of data at the server end to be 0,1, then continuing to resolve the whole piece of data, if not 0,1, then not resolving the rest of data.

Example 2

Gaia supports high performance pipeline information transfer. In the information transmission process, a client and a server are firstly connected, the client adjusts the key attributes of each piece of information to the first few bits, and then the adjusted data is sent to the server. The client analyzes the attributes of the first data bits at first, if the attributes of the first data bits are related to the key attributes, the data is continuously analyzed, otherwise, the data is not continuously analyzed.

A data transmission method is mainly characterized in that a client side adjusts key attributes of each piece of data to be transmitted to the front before transmitting the data to a server, then the server can analyze the key attributes first when analyzing, judges whether the data is needed, and does not need to analyze the rest data if the data is not needed, so that the time is saved, the resources are saved, and the efficiency is improved.

The embodiment of the invention is described by taking information transmission in intrusion detection as an example, and the information transmission method based on the Gaia system comprises the following steps:

step 1: the client server establishes a connection. After establishing the connection, the client and the server may communicate.

Step 2: the order of the key attributes of each piece of information to be transmitted is adjusted. The key attribute needed to identify this piece of information is adjusted to the front.

And step 3: and sending the information. And establishing connection in the first two steps, and sending information to the server after adjusting the sequence of the key attributes.

And 4, step 4: and (6) analyzing the information. After the information sent by the client, the server firstly analyzes the first key attributes of the information, if the first key attributes are relevant, the whole information is continuously analyzed, otherwise, the information is not analyzed any more.

Example 3

Information transmission becomes an important approach for solving problems in the field of big data application, is frequently used in big data processing algorithms, is more important for novel big data computing platforms Gaia, and the important characteristic of Gaia is low delay. Gaia is used as a real-time stream processing system, and the parallel mechanism of the Gaia is carried out by slicing nodes, and the task of information transmission is completed by a job.

The invention completes the information transmission through the following steps, as shown in fig. 2, the specific steps are as follows:

step 1: and establishing connection between the server and the client. And establishing the connection between the server and the client by using the Socket. In order to communicate between two computers, a network line must be used to connect the two computers. The Server (Server) refers to a computer or a program for providing information, the Client (Client) refers to a computer or a program for requesting information, and the network is used for connecting the Server and the Client to realize the mutual communication, and a Socket is used for monitoring a fixed port.

Step 2: and adjusting the sequence of key attributes of the information sent by the client. In the vast amount of information, including both information that we need and information that we do not need, there must be key attributes that mark this information. These key attributes are brought to the front before sending the information to the server.

And step 3: the client sends information to the server. And sending data to the server side by using the IO stream. Figure 1 illustrates the process of adjusting key attributes before sending data.

And 4, step 4: the server parses the information. The server firstly analyzes the first few bits of the information, if the first few bits are the key attributes, the whole piece of data is continuously analyzed, otherwise, the analysis is not continuously performed.

In the Gaia system, the flow of information transmission is shown in fig. 2. Suppose that each piece of information has 1000 attributes and 2 key attributes, and the two key attributes are the values that we need, and the whole piece of data is the value we need. The first 2 bits of data are parsed first, and if the first 2 bits of data are needed, the parsing is continued, otherwise, the parsing is not performed. The algorithm pseudo code for the server to parse the information is as follows:

the above description is only for the preferred embodiment of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art should be able to cover the technical solutions and the inventive concepts of the present invention within the technical scope of the present invention.

Claims

1. An information transmission method based on a Gaia big data analysis system is characterized by comprising the following steps:

s1, marking key attributes of the data in the historical statistical information by using a data analysis tool:

in data of intrusion detection, if 11 th bit and 12 th bit are 01, the key attribute is the last attribute of the data, and the key attribute is marked as a normal type; if the 11 th bit and the 12 th bit are not 01 and the key attribute is the last attribute of the data, marking the key attribute as an attack type;

s2, adjusting the 11 th bit attribute and the 12 th bit attribute to the forefront of the data by using the key attribute marked in the step S1;

s3, the client sends the data after the sequence is adjusted in the step S2 to the server; establishing communication connection between the client and the server by using Socket sockets; the client sends data to the server by using the IO stream;

s4, when the server receives the data, the first two bits of the data are firstly analyzed, if the data are '01', the whole data are continuously analyzed; if not "01," the remaining data is not parsed.

2. The information transmission method based on the Gaia big data analysis system according to claim 1, wherein the data analysis tool is Python or Mysql.

3. The information transmission method based on the Gaia big data analysis system according to claim 1, wherein if there are several key attributes, the several key attributes are adjusted to the front in a front-to-back order before the adjustment.