CN111431902A - Big data all-in-one machine - Google Patents
Big data all-in-one machine Download PDFInfo
- Publication number
- CN111431902A CN111431902A CN202010214258.7A CN202010214258A CN111431902A CN 111431902 A CN111431902 A CN 111431902A CN 202010214258 A CN202010214258 A CN 202010214258A CN 111431902 A CN111431902 A CN 111431902A
- Authority
- CN
- China
- Prior art keywords
- data
- node
- state
- calculation
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
- H04L63/1408—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic by monitoring network traffic
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/12—Protocols specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computing Systems (AREA)
- Computer Security & Cryptography (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Computer Hardware Design (AREA)
- General Engineering & Computer Science (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention provides a big data all-in-one machine which is connected with a plurality of data storage nodes of a data storage module and a plurality of data calculation nodes of a data calculation module through an internet of things, which can effectively ensure that different data streams and different instruction streams can be quickly and accurately transmitted to corresponding node areas to ensure that target data can be efficiently processed, and, in addition, the big data all-in-one machine can also carry out security monitoring processing on data streams and/or instruction streams on a plurality of data storage nodes of the data storage module and a plurality of data calculation nodes of the data calculation module so as to improve the overall data security of the all-in-one machine and prevent the all-in-one machine from being attacked, the big data all-in-one machine can carry out corresponding data scheduling processing according to the internet of things state and/or the data security state in the big data all-in-one machine, and therefore the working stability and the safety of the big data all-in-one machine are improved.
Description
Technical Field
The invention relates to the technical field of big data processing, in particular to a big data all-in-one machine.
Background
With the development of electronic information technology and cloud data processing technology, the data volume of internet data not only shows explosive growth, but also the data types of the internet data are more and the data structure is more and more complex. With the rapid development and change of internet data, the traditional data processing system and data processing mode can not meet the requirements of the big data era on the speed and the precision of data processing. In order to meet the new requirements of the big data era on data processing, a big data all-in-one machine has been proposed in the prior art, and the big data all-in-one machine is a platform integrating big data storage and big data calculation, is a device specially designed for a specific application field, and can perform centralized optimization and provide a complete big data solution for the specific application field. Although the big data all-in-one machine has the advantages of high data processing efficiency, high speed and high accuracy, the existing big data all-in-one machine still has the problems of excessively complex data network layout structure, unstable software and hardware performance of the big data bureau, easy data loss, easy attack and ineffective guarantee of data security, which seriously restricts the popularization and application of the big data all-in-one machine.
Disclosure of Invention
Aiming at the defects in the prior art, the invention provides a big data all-in-one machine, which comprises a data storage module, a data calculation module, a data internet of things module, a data security module and a data scheduling module: the data storage module is used for providing a plurality of data storage nodes; the data calculation module is used for providing a plurality of data calculation nodes; the data internet of things module is used for performing communication connection on the internet of things for the data storage module and the data calculation module; the data security module is used for carrying out security monitoring processing on data streams and/or instruction streams in the data storage module and/or the data calculation module; the data scheduling module is used for carrying out adaptive data scheduling processing on the big data all-in-one machine according to the working state of the data internet of things module and/or the data security module; therefore, the big data all-in-one machine is connected with a plurality of data storage nodes of the data storage module and a plurality of data calculation nodes of the data calculation module through the Internet of things, which can effectively ensure that different data streams and different instruction streams can be quickly and accurately transmitted to corresponding node areas to ensure that target data can be efficiently processed, and, in addition, the big data all-in-one machine can also carry out security monitoring processing on data streams and/or instruction streams on a plurality of data storage nodes of the data storage module and a plurality of data calculation nodes of the data calculation module so as to improve the overall data security of the all-in-one machine and prevent the all-in-one machine from being attacked, the big data all-in-one machine can carry out corresponding data scheduling processing according to the internet of things state and/or the data security state in the big data all-in-one machine, and therefore the working stability and the safety of the big data all-in-one machine are improved.
The invention provides a big data all-in-one machine, which is characterized in that:
the big data all-in-one machine comprises a data storage module, a data calculation module, a data internet of things module, a data security module and a data scheduling module: wherein the content of the first and second substances,
the data storage module is used for providing a plurality of data storage nodes;
the data calculation module is used for providing a plurality of data calculation nodes;
the data internet of things module is used for performing communication connection on the internet of things for the data storage module and the data calculation module;
the data security module is used for carrying out security monitoring processing on data streams and/or instruction streams in the data storage module and/or the data calculation module;
the data scheduling module is used for carrying out adaptive data scheduling processing on the big data all-in-one machine according to the working state of the data internet of things module and/or the data security module;
further, the data storage module comprises a first data stream monitoring submodule, a first instruction stream monitoring submodule, a storage node submodule and a storage node management submodule; wherein the content of the first and second substances,
the storage node submodule is used for providing a plurality of data storage nodes so as to perform distributed node storage processing on target data;
the first data flow monitoring submodule is used for monitoring data flow information and/or data flow rate information corresponding to each data storage node;
the first instruction stream monitoring submodule is used for monitoring instruction stream transmission interval information and/or instruction stream transmission time sequence information corresponding to each data storage node;
the storage node management submodule is used for adjusting the data storage state of each data storage node according to at least one of the data flow information, the data flow rate information, the instruction stream transmission interval information and the instruction stream transmission timing sequence information;
further, the storage node submodule comprises a plurality of data storage node units and a storage node adjusting unit; wherein the content of the first and second substances,
each data storage node unit is used for providing a data storage element for performing the distributed node storage processing;
the storage node adjusting unit is used for adjusting the connection relation between different data storage node units so as to realize any physical combination processing of the data storage elements;
alternatively, the first and second electrodes may be,
the first data flow monitoring submodule comprises a first data flow information acquisition unit and a first data flow rate information acquisition unit; wherein the content of the first and second substances,
the first data flow information acquisition unit is used for acquiring data flow information corresponding to each data storage node;
the first data flow rate information acquisition unit is used for acquiring data flow rate information corresponding to each data storage node;
alternatively, the first and second electrodes may be,
the first instruction stream monitoring submodule comprises a first transmission interval information acquisition unit and a first transmission time sequence information acquisition unit; wherein the content of the first and second substances,
the first transmission section information acquisition unit is used for acquiring instruction stream transmission section information corresponding to each data storage node;
the first transmission timing information acquisition unit is used for acquiring instruction stream transmission timing information corresponding to each data storage node;
alternatively, the first and second electrodes may be,
the storage node management submodule comprises a storage node state determining unit and a storage node state adjusting unit; wherein the content of the first and second substances,
the storage node state determining unit is configured to determine a current data storage state of each data storage node according to at least one of the data traffic information, the data flow rate information, the instruction stream transmission interval information, and the instruction stream transmission timing information;
the storage node state adjusting unit is used for adjusting the data storage state of each data storage node in real time according to the state difference between the current data storage state and the expected data storage state;
further, the data calculation module comprises a second data stream monitoring submodule, a second instruction stream monitoring submodule, a calculation node submodule and a calculation node management submodule; wherein the content of the first and second substances,
the computing node sub-module is used for providing a plurality of data computing nodes so as to perform parallel node computing processing on target data;
the second data flow monitoring submodule is used for monitoring data flow information and/or data flow rate information corresponding to each data calculation node;
the second instruction stream monitoring submodule is used for monitoring instruction stream transmission interval information and/or instruction stream transmission time sequence information corresponding to each data calculation node;
the computing node management submodule is used for adjusting the data computing state of each data computing node according to at least one of the data flow information, the data flow rate information, the instruction stream transmission interval information and the instruction stream transmission time sequence information;
further, the computing node submodule comprises a plurality of data computing node units and a computing node adjusting unit; wherein the content of the first and second substances,
each data computation node unit is used for providing a data computation element for performing the parallel node computation processing;
the computing node adjusting unit is used for adjusting the operational level relation among different data computing node units so as to realize any physical combination processing of the data computing elements;
alternatively, the first and second electrodes may be,
the second data flow monitoring submodule comprises a second data flow information acquisition unit and a second data flow rate information acquisition unit; wherein the content of the first and second substances,
the second data flow information acquisition unit is used for acquiring data flow information corresponding to each data calculation node;
the second data flow rate information acquisition unit is used for acquiring data flow rate information corresponding to each data calculation node;
alternatively, the first and second electrodes may be,
the second instruction stream monitoring submodule comprises a second transmission interval information acquisition unit and a second transmission time sequence information acquisition unit; wherein the content of the first and second substances,
the second transmission section information acquisition unit is used for acquiring instruction stream transmission section information corresponding to each data calculation node;
the second transmission timing information acquisition unit is used for acquiring instruction stream transmission timing information corresponding to each data calculation node;
alternatively, the first and second electrodes may be,
the computing node management submodule comprises a computing node state determining unit and a computing node state adjusting unit; wherein the content of the first and second substances,
the computing node state determining unit is configured to determine a current data computing state of each data computing node according to at least one of the data traffic information, the data flow rate information, the instruction stream transmission interval information, and the instruction stream transmission timing information;
the computing node state adjusting unit is used for adjusting the data computing state of each data computing node in real time according to the state difference between the current data computing state and the expected data computing state, and specifically comprises,
step S1, pre-processing the current data calculation state according to the following formula (1), and correspondingly,
setting the data computation state to X, X ═ α1,α2,…,αn]Where n is the number of data compute nodes, αiCalculating status information for data corresponding to the ith data node, an
m is the amount of information contained in the data calculation status information, aijCalculating a state information value for the jth data of the ith data calculation node, wherein the data calculation state information comprises the data flow and the data flow rate of the data calculation node
In the above formula (1), a'ijCalculating a preprocessing value, a, corresponding to the state information for the jth data of the ith data calculation nodeijCalculating a state information value for the jth data of the ith data calculation node, a for minimum and maximum absolute value calculationpjCalculating a state information value, a, for the jth data of the pth data calculation nodeqjCalculating a state information value for the jth data of the qth data calculation node;
step S2, calculating a state difference between the current data calculation state and the desired data calculation state according to the following formula (2),
Δi＝α'i-χ (2)
in the above formula (2), ΔiFor a state difference between a current data compute state and an expected data compute state for an ith data compute node, χ is the expected data compute state, α'iCalculating a preprocessing value corresponding to the state information for the current data of the ith data calculation node, and
the expected data calculation state is formed by an information optimal value corresponding to the data state information;
step S3, adjusting the data computation state of the data computation node according to the state difference,
when the state difference between the current data computing state and the expected data computing state of the ith data computing node is less than 0, determining that the data computing state of the corresponding data computing node does not reach the expected state, and adjusting the data computing state of the data computing node; when the state difference between the current data calculation state and the expected data calculation state of the ith data calculation node is greater than or equal to 0, determining that the data calculation state of the corresponding data calculation node reaches the expected state, and not adjusting the data calculation state of the data calculation node;
further, the data internet of things network module comprises a storage node-calculation node matching submodule, a storage node-calculation node internet of things link construction submodule and a storage node-calculation node internet of things link adjustment optimization submodule; wherein the content of the first and second substances,
the storage node-calculation node matching submodule is used for determining the connection matching relationship between different data storage nodes and different data calculation nodes according to the data storage states corresponding to the data storage nodes and the data calculation states corresponding to the data calculation nodes;
the storage node-computing node Internet of things link construction sub-module is used for constructing an Internet of things data joint link between different data storage nodes and different data computing nodes according to the connection matching relation;
the storage node-computing node Internet of things link adjustment optimization submodule is used for adjusting and optimizing the Internet of things data joint link according to a preset storage node-computing node data interaction model;
further, the data security module comprises a data stream security state monitoring submodule, an instruction stream security state monitoring submodule and a data security evaluation submodule; wherein the content of the first and second substances,
the data stream safety state monitoring submodule is used for monitoring the data stream safety in the data storage module and/or the data calculation module;
the instruction stream safety state monitoring submodule is used for monitoring the safety of the instruction stream in the data storage module and/or the data calculation module;
the data security evaluation sub-module is used for evaluating the data security of the data storage module and the data security of the data computing module or the data security of the data storage module and the data security of the instruction stream according to the data stream security and/or the instruction stream security;
further, the data security evaluation sub-module comprises a data security model construction unit and a data security confidence calculation unit; wherein the content of the first and second substances,
the data security model building unit is used for building a data security deep learning neural network model related to the big data all-in-one machine;
the data security confidence degree calculation unit is used for calculating a data security confidence degree value related to the data storage module and/or the data calculation module according to the data security deep learning neural network model so as to execute the judgment processing;
further, the data scheduling module comprises a working state information acquisition submodule and a scheduling execution submodule; wherein the content of the first and second substances,
the working state acquisition submodule is used for acquiring working state information corresponding to the data Internet of things module and/or the data security module;
the scheduling execution submodule is used for performing the data scheduling processing on the big data all-in-one machine according to the working state information;
further, the scheduling execution submodule comprises a data stream guiding unit and an instruction stream guiding unit; wherein the content of the first and second substances,
the data stream guiding unit is used for carrying out transmission guiding processing on the data stream corresponding to the data storage module and/or the data calculation module according to the working state information;
and the instruction stream guiding unit is used for carrying out transmission guiding processing on the instruction stream corresponding to the data storage module and/or the data calculation module according to the working state information.
Compared with the prior art, the big data all-in-one machine is connected with a plurality of data storage nodes of the data storage module and a plurality of data calculation nodes of the data calculation module through the Internet of things, which can effectively ensure that different data streams and different instruction streams can be quickly and accurately transmitted to corresponding node areas to ensure that target data can be efficiently processed, and, in addition, the big data all-in-one machine can also carry out security monitoring processing on data streams and/or instruction streams on a plurality of data storage nodes of the data storage module and a plurality of data calculation nodes of the data calculation module so as to improve the overall data security of the all-in-one machine and prevent the all-in-one machine from being attacked, the big data all-in-one machine can carry out corresponding data scheduling processing according to the internet of things state and/or the data security state in the big data all-in-one machine, and therefore the working stability and the safety of the big data all-in-one machine are improved.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.
The technical solution of the present invention is further described in detail by the accompanying drawings and embodiments.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a schematic structural diagram of a big data all-in-one machine provided by the invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Fig. 1 is a schematic structural diagram of a big data all-in-one machine according to an embodiment of the present invention. The big data all-in-one machine comprises a data storage module, a data calculation module, a data internet of things module, a data security module and a data scheduling module: wherein the content of the first and second substances,
the data storage module is used for providing a plurality of data storage nodes;
the data calculation module is used for providing a plurality of data calculation nodes;
the data internet of things module is used for performing communication connection on the internet of things for the data storage module and the data calculation module;
the data security module is used for carrying out security monitoring processing on data streams and/or instruction streams in the data storage module and/or the data calculation module;
the data scheduling module is used for carrying out adaptive data scheduling processing on the big data all-in-one machine according to the working state of the data internet of things module and/or the data security module.
Preferably, the data storage module comprises a first data stream monitoring submodule, a first instruction stream monitoring submodule, a storage node submodule and a storage node management submodule; wherein the content of the first and second substances,
the storage node submodule is used for providing a plurality of data storage nodes so as to perform distributed node storage processing on target data;
the first data flow monitoring submodule is used for monitoring data flow information and/or data flow rate information corresponding to each data storage node;
the first instruction stream monitoring submodule is used for monitoring instruction stream transmission interval information and/or instruction stream transmission time sequence information corresponding to each data storage node;
the storage node management submodule is used for adjusting the data storage state of each data storage node according to at least one of the data flow information, the data flow rate information, the instruction stream transmission interval information and the instruction stream transmission timing sequence information.
Preferably, the storage node submodule includes a plurality of data storage node units and a storage node adjusting unit; wherein the content of the first and second substances,
each data storage node unit is used for providing a data storage element for carrying out storage processing of the distributed nodes;
the storage node adjusting unit is used for adjusting the connection relation between different data storage node units so as to realize any physical combination processing of the data storage elements.
Preferably, the first data flow monitoring submodule includes a first data flow information obtaining unit and a first data flow rate information obtaining unit; wherein the content of the first and second substances,
the first data flow information acquisition unit is used for acquiring data flow information corresponding to each data storage node;
the first data flow rate information acquisition unit is used for acquiring data flow rate information corresponding to each data storage node.
Preferably, the first instruction stream monitoring submodule includes a first transmission interval information acquisition unit and a first transmission timing information acquisition unit; wherein the content of the first and second substances,
the first transmission section information acquisition unit is used for acquiring instruction stream transmission section information corresponding to each data storage node;
the first transmission timing information acquisition unit is used for acquiring instruction stream transmission timing information corresponding to each data storage node.
Preferably, the storage node management submodule includes a storage node state determining unit and a storage node state adjusting unit; wherein the content of the first and second substances,
the storage node state determining unit is configured to determine a current data storage state of each data storage node according to at least one of the data traffic information, the data flow rate information, the instruction stream transmission interval information, and the instruction stream transmission timing information;
the storage node state adjusting unit is used for adjusting the data storage state of each data storage node in real time according to the state difference between the current data storage state and the expected data storage state.
Preferably, the data calculation module comprises a second data stream monitoring submodule, a second instruction stream monitoring submodule, a calculation node submodule and a calculation node management submodule; wherein the content of the first and second substances,
the computing node submodule is used for providing a plurality of data computing nodes so as to carry out parallel node computing processing on target data;
the second data flow monitoring submodule is used for monitoring data flow information and/or data flow rate information corresponding to each data calculation node;
the second instruction stream monitoring submodule is used for monitoring instruction stream transmission interval information and/or instruction stream transmission time sequence information corresponding to each data calculation node;
the computing node management submodule is used for adjusting the data computing state of each data computing node according to at least one of the data flow information, the data flow rate information, the instruction stream transmission interval information and the instruction stream transmission time sequence information.
Preferably, the computing node submodule comprises a plurality of data computing node units and a computing node adjusting unit; wherein the content of the first and second substances,
each data computation node unit is used for providing a data computation element for performing the parallel node computation processing;
the computing node adjusting unit is used for adjusting the operational level relation among different data computing node units so as to realize the random physical combination processing of the data computing element.
Preferably, the second data flow monitoring submodule includes a second data flow information obtaining unit and a second data flow rate information obtaining unit; wherein the content of the first and second substances,
the second data flow information acquisition unit is used for acquiring data flow information corresponding to each data calculation node;
the second data flow rate information acquisition unit is used for acquiring data flow rate information corresponding to each data calculation node.
Preferably, the second instruction stream monitoring submodule includes a second transmission section information acquisition unit and a second transmission timing information acquisition unit; wherein the content of the first and second substances,
the second transmission section information acquisition unit is used for acquiring instruction stream transmission section information corresponding to each data calculation node;
the second transmission timing information acquisition unit is used for acquiring the instruction stream transmission timing information corresponding to each data calculation node.
Preferably, the compute node management submodule includes a compute node state determining unit and a compute node state adjusting unit; wherein the content of the first and second substances,
the computing node state determining unit is configured to determine a current data computing state of each data computing node according to at least one of the data traffic information, the data flow rate information, the instruction stream transmission interval information, and the instruction stream transmission timing information;
the computing node state adjusting unit is used for adjusting the data computing state of each data computing node in real time according to the state difference between the current data computing state and the expected data computing state, and specifically comprises,
step S1, preprocessing the current data calculation state according to the following formula (1), and correspondingly,
the data calculation state is set to X, X ═ α1,α2,…,αn]Where n is the number of data compute nodes, αiCalculating status information for data corresponding to the ith data node, an
m is the amount of information contained in the data calculation status information, aijCalculating a state information value for the jth data of the ith data calculation node, wherein the state information comprises the data flow and the data flow rate of the ith data calculation node
In the above formula (1), a'ijCalculating a preprocessing value, a, corresponding to the state information for the jth data of the ith data calculation nodeijCalculating a state information value for the jth data of the ith data calculation node, a for minimum and maximum absolute value calculationpjCalculating a state information value, a, for the jth data of the pth data calculation nodeqjCalculating a state information value for the jth data of the qth data calculation node;
step S2, calculating a state difference between the current data calculation state and the desired data calculation state according to the following formula (2),
Δi＝α'i-χ (2)
in the above formula (2), ΔiFor the state difference between the current data compute state and the desired data compute state for the ith data compute node, χ is the desired data compute state, α'iCalculating a preprocessing value corresponding to the state information for the current data of the ith data calculation node, and
the expected data calculation state is formed by the information optimal value corresponding to the data state information;
step S3, adjusting the data computation state of the data computation node according to the state difference,
when the state difference between the current data computing state and the expected data computing state of the ith data computing node is less than 0, determining that the data computing state of the corresponding data computing node does not reach the expected state, and adjusting the data computing state of the data computing node; when the state difference between the current data calculation state and the expected data calculation state of the ith data calculation node is greater than or equal to 0, determining that the data calculation state of the corresponding data calculation node reaches the expected state, and not adjusting the data calculation state of the data calculation node;
through the process, the state difference between the current data calculation state and the expected data calculation state of the data calculation node can be obtained, the data calculation state of the data calculation node is further adjusted, the current data calculation state of the data calculation node can be preprocessed, the limitation of data can be removed, the data are uniformly mapped to the [0,1] interval, and the comparison with the expected data calculation state is facilitated.
Preferably, the data internet of things network module comprises a storage node-calculation node matching submodule, a storage node-calculation node internet of things link construction submodule and a storage node-calculation node internet of things link adjustment optimization submodule; wherein the content of the first and second substances,
the storage node-calculation node matching submodule is used for determining the connection matching relationship between different data storage nodes and different data calculation nodes according to the data storage states corresponding to the data storage nodes and the data calculation states corresponding to the data calculation nodes;
the storage node-computing node Internet of things link construction submodule is used for constructing an Internet of things data joint link between different data storage nodes and different data computing nodes according to the connection matching relation;
the storage node-computing node Internet of things link adjustment optimization submodule is used for adjusting and optimizing the Internet of things data joint link according to a preset storage node-computing node data interaction model.
Preferably, the data security module comprises a data stream security state monitoring submodule, an instruction stream security state monitoring submodule and a data security evaluation submodule; wherein the content of the first and second substances,
the data stream safety state monitoring submodule is used for monitoring the data stream safety in the data storage module and/or the data calculation module;
the instruction stream safety state monitoring submodule is used for monitoring the safety of the instruction stream in the data storage module and/or the data calculation module;
the data security evaluation sub-module is used for evaluating the data security of the data storage module and the data security of the data computing module or the data security of the data storage module and the data security of the instruction stream according to the data stream security and/or the instruction stream security.
Preferably, the data security evaluation sub-module comprises a data security model construction unit and a data security confidence calculation unit; wherein the content of the first and second substances,
the data security model building unit is used for building a data security deep learning neural network model related to the big data all-in-one machine;
the data security confidence calculation unit is used for calculating and obtaining a data security confidence value related to the data storage module and/or the data calculation module according to the data security deep learning neural network model so as to execute the judgment processing.
Preferably, the data scheduling module comprises a working state information acquisition submodule and a scheduling execution submodule; wherein the content of the first and second substances,
the working state acquisition submodule is used for acquiring working state information corresponding to the data Internet of things module and/or the data security module;
the scheduling execution submodule is used for carrying out the data scheduling processing on the big data all-in-one machine according to the working state information.
Preferably, the schedule execution submodule includes a data stream steering unit and an instruction stream steering unit; wherein the content of the first and second substances,
the data stream guiding unit is used for carrying out transmission guiding processing on the data stream corresponding to the data storage module and/or the data calculation module according to the working state information;
the instruction stream guiding unit is used for carrying out transmission guiding processing on the instruction stream corresponding to the data storage module and/or the data calculation module according to the working state information.
In brief, the purpose of the big data all-in-one machine is to set a plurality of data storage nodes and a plurality of data calculation nodes in a memory and a calculator in the big data all-in-one machine respectively, each storage node is equivalent to a data storage base unit, each data calculation node is equivalent to a data calculation unit, and the data storage nodes and the data calculation nodes are in data connection in the way of internet of things, so that massive and rapid storage and calculation of big data can be realized, and the big data all-in-one machine is also provided with a data security monitoring device for monitoring data flow and control instruction flow in each data storage node and each data calculation node, and the data security monitoring device can timely discover the data storage nodes and/or the data calculation nodes with abnormal states in the data flow and/or the control instruction flow, and corresponding scheduling processing is carried out on the data storage nodes and/or the data calculation nodes with abnormal states in time so as to ensure that the data storage and/or the data calculation of the whole big data all-in-one machine are normal.
From the content of the above embodiments, the big data all-in-one machine connects a plurality of data storage nodes of the data storage module and a plurality of data calculation nodes of the data calculation module through the internet of things, which can effectively ensure that different data streams and different instruction streams can be quickly and accurately transmitted to corresponding node areas to ensure that target data can be efficiently processed, and, in addition, the big data all-in-one machine can also carry out security monitoring processing on data streams and/or instruction streams on a plurality of data storage nodes of the data storage module and a plurality of data calculation nodes of the data calculation module so as to improve the overall data security of the all-in-one machine and prevent the all-in-one machine from being attacked, the big data all-in-one machine can carry out corresponding data scheduling processing according to the internet of things state and/or the data security state in the big data all-in-one machine, and therefore the working stability and the safety of the big data all-in-one machine are improved.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present invention without departing from the spirit and scope of the invention. Thus, if such modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include such modifications and variations.
Claims (10)
1. Big data all-in-one, its characterized in that:
the big data all-in-one machine comprises a data storage module, a data calculation module, a data internet of things module, a data security module and a data scheduling module: wherein the content of the first and second substances,
the data storage module is used for providing a plurality of data storage nodes;
the data calculation module is used for providing a plurality of data calculation nodes;
the data internet of things module is used for performing communication connection on the internet of things for the data storage module and the data calculation module;
the data security module is used for carrying out security monitoring processing on data streams and/or instruction streams in the data storage module and/or the data calculation module;
and the data scheduling module is used for carrying out adaptive data scheduling processing on the big data all-in-one machine according to the working state of the data internet of things module and/or the data security module.
2. The big data all-in-one machine of claim 1, wherein:
the data storage module comprises a first data stream monitoring submodule, a first instruction stream monitoring submodule, a storage node submodule and a storage node management submodule; wherein the content of the first and second substances,
the storage node submodule is used for providing a plurality of data storage nodes so as to perform distributed node storage processing on target data;
the first data flow monitoring submodule is used for monitoring data flow information and/or data flow rate information corresponding to each data storage node;
the first instruction stream monitoring submodule is used for monitoring instruction stream transmission interval information and/or instruction stream transmission time sequence information corresponding to each data storage node;
the storage node management submodule is used for adjusting the data storage state of each data storage node according to at least one of the data flow information, the data flow rate information, the instruction stream transmission interval information and the instruction stream transmission timing sequence information.
3. The big data all-in-one machine of claim 2, wherein:
the storage node submodule comprises a plurality of data storage node units and a storage node adjusting unit; wherein the content of the first and second substances,
each data storage node unit is used for providing a data storage element for performing the distributed node storage processing;
the storage node adjusting unit is used for adjusting the connection relation between different data storage node units so as to realize any physical combination processing of the data storage elements;
alternatively, the first and second electrodes may be,
the first data flow monitoring submodule comprises a first data flow information acquisition unit and a first data flow rate information acquisition unit; wherein the content of the first and second substances,
the first data flow information acquisition unit is used for acquiring data flow information corresponding to each data storage node;
the first data flow rate information acquisition unit is used for acquiring data flow rate information corresponding to each data storage node;
alternatively, the first and second electrodes may be,
the first instruction stream monitoring submodule comprises a first transmission interval information acquisition unit and a first transmission time sequence information acquisition unit; wherein the content of the first and second substances,
the first transmission section information acquisition unit is used for acquiring instruction stream transmission section information corresponding to each data storage node;
the first transmission timing information acquisition unit is used for acquiring instruction stream transmission timing information corresponding to each data storage node;
alternatively, the first and second electrodes may be,
the storage node management submodule comprises a storage node state determining unit and a storage node state adjusting unit; wherein the content of the first and second substances,
the storage node state determining unit is configured to determine a current data storage state of each data storage node according to at least one of the data traffic information, the data flow rate information, the instruction stream transmission interval information, and the instruction stream transmission timing information;
the storage node state adjusting unit is used for adjusting the data storage state of each data storage node in real time according to the state difference between the current data storage state and the expected data storage state.
4. The big data all-in-one machine of claim 1, wherein:
the data calculation module comprises a second data stream monitoring submodule, a second instruction stream monitoring submodule, a calculation node submodule and a calculation node management submodule; wherein the content of the first and second substances,
the computing node sub-module is used for providing a plurality of data computing nodes so as to perform parallel node computing processing on target data;
the second data flow monitoring submodule is used for monitoring data flow information and/or data flow rate information corresponding to each data calculation node;
the second instruction stream monitoring submodule is used for monitoring instruction stream transmission interval information and/or instruction stream transmission time sequence information corresponding to each data calculation node;
the computing node management submodule is used for adjusting the data computing state of each data computing node according to at least one of the data flow information, the data flow rate information, the instruction stream transmission interval information and the instruction stream transmission time sequence information.
5. The big data all-in-one machine of claim 4, wherein:
the computing node submodule comprises a plurality of data computing node units and a computing node adjusting unit; wherein the content of the first and second substances,
each data computation node unit is used for providing a data computation element for performing the parallel node computation processing;
the computing node adjusting unit is used for adjusting the operational level relation among different data computing node units so as to realize any physical combination processing of the data computing elements;
alternatively, the first and second electrodes may be,
the second data flow monitoring submodule comprises a second data flow information acquisition unit and a second data flow rate information acquisition unit; wherein the content of the first and second substances,
the second data flow information acquisition unit is used for acquiring data flow information corresponding to each data calculation node;
the second data flow rate information acquisition unit is used for acquiring data flow rate information corresponding to each data calculation node;
alternatively, the first and second electrodes may be,
the second instruction stream monitoring submodule comprises a second transmission interval information acquisition unit and a second transmission time sequence information acquisition unit; wherein the content of the first and second substances,
the second transmission section information acquisition unit is used for acquiring instruction stream transmission section information corresponding to each data calculation node;
the second transmission timing information acquisition unit is used for acquiring instruction stream transmission timing information corresponding to each data calculation node;
alternatively, the first and second electrodes may be,
the computing node management submodule comprises a computing node state determining unit and a computing node state adjusting unit; wherein the content of the first and second substances,
the computing node state determining unit is configured to determine a current data computing state of each data computing node according to at least one of the data traffic information, the data flow rate information, the instruction stream transmission interval information, and the instruction stream transmission timing information;
the computing node state adjusting unit is used for adjusting the data computing state of each data computing node in real time according to the state difference between the current data computing state and the expected data computing state, and specifically comprises,
step S1, pre-processing the current data calculation state according to the following formula (1), and correspondingly,
setting the data computation state to X, X ═ α1,α2,…,αn]Where n is the number of data compute nodes, αiCalculating status information for data corresponding to the ith data node, an
m is the amount of information contained in the data calculation status information, aijCalculating a state information value for the jth data of the ith data calculation node, wherein the data calculation state information comprises the data flow and the data flow rate of the data calculation node
In the above formula (1), a'ijCalculating a preprocessing value, a, corresponding to the state information for the jth data of the ith data calculation nodeijCalculating a state information value for the jth data of the ith data calculation node, a for minimum and maximum absolute value calculationpjCalculating a state information value, a, for the jth data of the pth data calculation nodeqjCalculating a state information value for the jth data of the qth data calculation node;
step S2, calculating a state difference between the current data calculation state and the desired data calculation state according to the following formula (2),
Δi＝α'i-χ (2)
in the above formula (2), ΔiFor a state difference between a current data compute state and an expected data compute state for an ith data compute node, χ is the expected data compute state, α'iCalculating a preprocessing value corresponding to the state information for the current data of the ith data calculation node, and
the expected data calculation state is formed by an information optimal value corresponding to the data state information;
step S3, adjusting the data computation state of the data computation node according to the state difference,
when the state difference between the current data computing state and the expected data computing state of the ith data computing node is less than 0, determining that the data computing state of the corresponding data computing node does not reach the expected state, and adjusting the data computing state of the data computing node; and when the state difference between the current data computing state and the expected data computing state of the ith data computing node is greater than or equal to 0, determining that the data computing state of the corresponding data computing node reaches the expected state, and not adjusting the data computing state of the data computing node.
6. The big data all-in-one machine of claim 1, wherein:
the data internet of things module comprises a storage node-calculation node matching submodule, a storage node-calculation node internet of things link construction submodule and a storage node-calculation node internet of things link adjustment optimization submodule; wherein the content of the first and second substances,
the storage node-calculation node matching submodule is used for determining the connection matching relationship between different data storage nodes and different data calculation nodes according to the data storage states corresponding to the data storage nodes and the data calculation states corresponding to the data calculation nodes;
the storage node-computing node Internet of things link construction sub-module is used for constructing an Internet of things data joint link between different data storage nodes and different data computing nodes according to the connection matching relation;
and the storage node-computing node Internet of things link adjustment optimization submodule is used for adjusting and optimizing the Internet of things data joint link according to a preset storage node-computing node data interaction model.
7. The big data all-in-one machine of claim 1, wherein:
the data security module comprises a data stream security state monitoring submodule, an instruction stream security state monitoring submodule and a data security evaluation submodule; wherein the content of the first and second substances,
the data stream safety state monitoring submodule is used for monitoring the data stream safety in the data storage module and/or the data calculation module;
the instruction stream safety state monitoring submodule is used for monitoring the safety of the instruction stream in the data storage module and/or the data calculation module;
the data security evaluation sub-module is used for evaluating the data security of the data storage module and the data security of the data computing module or the data security of the data storage module and the data security of the instruction stream according to the data stream security and/or the instruction stream security.
8. The big data all-in-one machine of claim 7, wherein:
the data security evaluation sub-module comprises a data security model building unit and a data security confidence calculation unit; wherein the content of the first and second substances,
the data security model building unit is used for building a data security deep learning neural network model related to the big data all-in-one machine;
the data security confidence degree calculation unit is used for calculating a data security confidence degree value related to the data storage module and/or the data calculation module according to the data security deep learning neural network model so as to execute the judgment processing.
9. The big data all-in-one machine of claim 1, wherein:
the data scheduling module comprises a working state information acquisition submodule and a scheduling execution submodule; wherein the content of the first and second substances,
the working state acquisition submodule is used for acquiring working state information corresponding to the data Internet of things module and/or the data security module;
and the scheduling execution submodule is used for carrying out data scheduling processing on the big data all-in-one machine according to the working state information.
10. The big data all-in-one machine of claim 9, wherein:
the scheduling execution submodule comprises a data stream guiding unit and an instruction stream guiding unit; wherein the content of the first and second substances,
the data stream guiding unit is used for carrying out transmission guiding processing on the data stream corresponding to the data storage module and/or the data calculation module according to the working state information;
and the instruction stream guiding unit is used for carrying out transmission guiding processing on the instruction stream corresponding to the data storage module and/or the data calculation module according to the working state information.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010214258.7A CN111431902B (en) | 2020-03-24 | 2020-03-24 | Big data all-in-one machine |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010214258.7A CN111431902B (en) | 2020-03-24 | 2020-03-24 | Big data all-in-one machine |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111431902A true CN111431902A (en) | 2020-07-17 |
| CN111431902B CN111431902B (en) | 2020-12-04 |
Family
ID=71549733
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010214258.7A Active CN111431902B (en) | 2020-03-24 | 2020-03-24 | Big data all-in-one machine |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111431902B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102169505A (en) * | 2011-05-16 | 2011-08-31 | 苏州两江科技有限公司 | Recommendation system building method based on cloud computing |
| CN103136101A (en) * | 2012-12-31 | 2013-06-05 | 博彦科技（上海）有限公司 | Software automated testing unified operation platform |
| CN103442076A (en) * | 2013-09-04 | 2013-12-11 | 上海海事大学 | Usability guarantee method for cloud storage system |
| CN104021194A (en) * | 2014-06-13 | 2014-09-03 | 浪潮(北京)电子信息产业有限公司 | Mixed type processing system and method oriented to industry big data diversity application |
| CN104268695A (en) * | 2014-09-26 | 2015-01-07 | 武汉大学 | Multi-center watershed water environment distributed cluster management system and method |
| US9363388B2 (en) * | 2013-02-18 | 2016-06-07 | Tekelec, Inc. | Methods, systems, and computer readable media for providing targeted services to telecommunications network subscribers based on information extracted from network signaling and data traffic |
| CN108200109A (en) * | 2016-12-08 | 2018-06-22 | 南京海道普数据技术有限公司 | Big data all-in-one machine |
-
2020
- 2020-03-24 CN CN202010214258.7A patent/CN111431902B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102169505A (en) * | 2011-05-16 | 2011-08-31 | 苏州两江科技有限公司 | Recommendation system building method based on cloud computing |
| CN103136101A (en) * | 2012-12-31 | 2013-06-05 | 博彦科技（上海）有限公司 | Software automated testing unified operation platform |
| US9363388B2 (en) * | 2013-02-18 | 2016-06-07 | Tekelec, Inc. | Methods, systems, and computer readable media for providing targeted services to telecommunications network subscribers based on information extracted from network signaling and data traffic |
| CN103442076A (en) * | 2013-09-04 | 2013-12-11 | 上海海事大学 | Usability guarantee method for cloud storage system |
| CN104021194A (en) * | 2014-06-13 | 2014-09-03 | 浪潮(北京)电子信息产业有限公司 | Mixed type processing system and method oriented to industry big data diversity application |
| CN104268695A (en) * | 2014-09-26 | 2015-01-07 | 武汉大学 | Multi-center watershed water environment distributed cluster management system and method |
| CN108200109A (en) * | 2016-12-08 | 2018-06-22 | 南京海道普数据技术有限公司 | Big data all-in-one machine |
Non-Patent Citations (1)
| Title |
|---|
| 李栋华等: "能源互联网形势下的电力大数据发展趋势", 《现代电力》 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111431902B (en) | 2020-12-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107292023A (en) | A kind of bridge structural state diagnostic method based on damage index system narrow characteristic | |
| CN111953757B (en) | Information processing method based on cloud computing and intelligent device interaction and cloud server | |
| CN113705929B (en) | Spring festival holiday load prediction method based on load characteristic curve and typical characteristic value fusion | |
| CN113759868A (en) | Industrial Ethernet fault prediction method based on neural network | |
| CN112884163B (en) | Combined service evaluation method and system based on federal machine learning algorithm and cloud feedback | |
| CN112396135A (en) | Method and system for detecting abnormal traffic of converged communication network | |
| CN106210129A (en) | The current-limiting method of a kind of sing on web server configures and system | |
| CN115829297A (en) | Method and device for generating work package of assembly type building, terminal and storage medium | |
| CN111431902B (en) | Big data all-in-one machine | |
| CN114996362A (en) | Data processing and storing method | |
| CN105933138B (en) | Space-time dimension combined cloud service credibility situation assessment and prediction method | |
| CN114610813A (en) | Distributed storage method, device, equipment and medium for federal learning | |
| CN113806094A (en) | Cloud platform resource dynamic scheduling method based on deep learning | |
| CN109117352B (en) | Server performance prediction method and device | |
| CN115794405A (en) | Dynamic resource allocation method of big data processing framework based on SSA-XGboost algorithm | |
| CN111953533B (en) | Method, device and equipment for regulating and controlling working duration of target network node and storage medium | |
| Wang et al. | Multi-objects scalable coordinated learning in internet of things | |
| CN113364806A (en) | Service interface migration verification method and device | |
| CN115208717B (en) | Edge computing gateway device | |
| CN113542276B (en) | Method and system for detecting intrusion target of hybrid network | |
| Yan | Adaptive scheduling of agricultural machinery equipment production lines for intelligent manufacturing | |
| CN109902831A (en) | Operational decision making processing method and device | |
| Zhao et al. | The game selection model for the payoff strategy optimization of mobile CrowdSensing task | |
| CN117237804B (en) | Pyrotechnical recognition system and method based on federal learning | |
| CN114138614A (en) | Running time prediction method of batch processing task and related device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |