CN117971173A

CN117971173A - Intelligent data convergence software system based on Internet of things

Info

Publication number: CN117971173A
Application number: CN202410144067.6A
Authority: CN
Inventors: 张志红; 范嫦; 张宇鸣; 王斌
Original assignee: Jiangxi Zhihong Digital Technology Co ltd
Current assignee: Jiangxi Zhihong Digital Technology Co ltd
Priority date: 2024-02-01
Filing date: 2024-02-01
Publication date: 2024-05-03

Abstract

The invention relates to the technical field of data processing, in particular to an intelligent data convergence software system based on the Internet of things, which comprises a data flow control module, an intelligent storage optimization module, a static path optimization module, an edge calculation coordination module, a priority scheduling management module, a time efficiency analysis module, a rule driving distribution module and a dynamic resource adjustment module. According to the invention, through a multidimensional data flow analysis algorithm and a real-time flow analysis, the system can effectively monitor and adapt to dynamic data flow in the environment of the Internet of things, data processing delay is obviously reduced, the self-adaptive storage allocation and storage space optimization algorithm enables storage management to be more efficient, the application of the static path optimization and dynamic routing adjustment algorithm optimizes the data flow path, the data transmission efficiency of the whole network is improved, and the system realizes full utilization of edge computing resources and effective cooperation of central data processing through efficient cooperation of the edge node management and central cooperation sub-modules.

Description

Intelligent data convergence software system based on Internet of things

Technical Field

The invention relates to the technical field of data processing, in particular to an intelligent data convergence software system based on the Internet of things.

Background

The data processing technology field is focused on processing and analyzing various data to extract valuable information, in the data processing technology field, a large amount of data can be effectively organized, stored and analyzed by using algorithms, a database management system and a data mining technology, and the core purpose of the field is to convert the data into usable information so as to better support decision making, predictive analysis and business optimization.

The intelligent data convergence software system is a system integrating data collection, processing and analysis functions, and mainly aims to realize rapid fusion and intelligent analysis of information by efficiently converging and processing data from different sources.

Although the prior art has achieved significant achievements in terms of accuracy of data processing, there are still problems of processing delay and inefficiency in coping with data fluidity and instantaneity in the environment of the internet of things, especially in terms of real-time monitoring and optimization of dynamic data flows, and rapid integration and intelligent analysis of large-scale heterogeneous data sources, the existing system faces challenges of processing speed and data throughput, and furthermore, the existing system also shows limitations in terms of full utilization of edge computing resources and effective cooperation of a central data processing system, which is particularly obvious in terms of strategic planning and execution efficiency of data distribution, and in terms of priority scheduling and time efficiency optimization of data processing, the prior art has difficulty in fully realizing adaptive adjustment of flows and optimization of time cost, resulting in limited efficiency in processing urgent or complex data.

Disclosure of Invention

The intelligent data convergence software system based on the Internet of things solves the problems that although the prior art has achieved remarkable achievement in the aspect of accuracy of data processing, the prior art has insufficient processing delay and efficiency in the aspect of coping with data mobility and instantaneity in the environment of the Internet of things, particularly in the aspects of real-time monitoring and optimization of dynamic data streams and rapid integration and intelligent analysis of large-scale heterogeneous data sources, the prior system faces the challenges of processing speed and data throughput, and in addition, the prior system also shows limitations in the aspects of full utilization of edge computing resources and effective coordination of a central data processing system, which is particularly obvious in strategic planning and execution efficiency of data distribution, and in the aspects of priority scheduling and time efficiency optimization of data processing, the prior art has difficulty in fully realizing self-adaptive adjustment of flow and optimization of time cost, so that the efficiency is limited in the process of urgent or complex data.

In view of the above problems, the present application provides an intelligent data convergence software system based on the internet of things.

The application provides an intelligent data convergence software system based on the Internet of things, wherein the system comprises a data flow control module, an intelligent storage optimization module, a static path optimization module, an edge calculation coordination module, a priority scheduling management module, a time efficiency analysis module, a rule driving distribution module and a dynamic resource adjustment module;

the data flow control module monitors data flow dynamics by adopting a multidimensional data flow analysis algorithm based on the data flow of the Internet of things, analyzes flow trend and mode, performs flow path planning according to the trend, and generates a data flow dynamic analysis result;

The intelligent storage optimization module analyzes data storage requirements by adopting a self-adaptive storage allocation algorithm based on a data flow dynamic analysis result, reorganizes a storage structure, optimizes storage position allocation and generates an optimized storage scheme;

The static path optimization module analyzes a data flow path by adopting a graph theory path optimization algorithm based on the optimized storage scheme, plans a path layout, adjusts path configuration and generates a static path optimization strategy;

The edge computing coordination module analyzes the cooperation of the edge computing nodes and the key data processing clusters by adopting a distributed resource coordination algorithm based on a static path optimization strategy, plans resource allocation, adjusts a task scheduling strategy and generates a coordination computing scheme;

the priority scheduling management module adopts a dynamic priority queuing algorithm based on a coordination calculation scheme to analyze data processing tasks, plan task priorities, adjust task execution queues and generate a priority scheduling plan;

The time efficiency analysis module analyzes the time distribution of the data processing flow by adopting a time window analysis method based on the priority scheduling plan, plans the flow optimization step, adjusts the execution sequence and generates a time optimization strategy;

The rule-driven distribution module analyzes data distribution requirements by adopting a logic reasoning distribution algorithm based on a time optimization strategy, distributes data tasks according to rules, formulates a distribution flow and generates a rule-driven distribution scheme;

the dynamic resource adjustment module adopts a load sensing resource allocation algorithm based on a rule-driven distribution scheme, analyzes the current load and data flow of the system, plans a resource allocation strategy, performs resource allocation adjustment, and generates a resource adjustment scheme.

Preferably, the data flow state analysis result comprises flow peak distribution, data transmission frequency and flow hot spot areas, the optimized storage scheme specifically refers to adjustment of data storage partition, backup strategy formulation and storage space redistribution, the static path optimization strategy comprises data transmission path determination, backup path planning and path redundancy assessment, the coordination calculation scheme specifically comprises resource sharing rules, data synchronization mechanisms and collaborative task distribution of edge computing nodes, the priority scheduling scheme specifically comprises task priority classification, execution queue arrangement and task dependency graph, the time optimization strategy specifically refers to a time compression method, critical path optimization and non-critical task postponement strategy of flow execution, the rule-driven distribution scheme specifically comprises data distribution rule setting, periodic assessment of task distribution logic and distribution efficiency, and the resource adjustment scheme specifically comprises resource redistribution based on load change, dynamic scaling strategy and resource usage optimization.

Preferably, the data flow control module comprises a flow monitoring sub-module, a trend analysis sub-module and a flow guiding sub-module;

The flow monitoring submodule captures and initially classifies data packets through a network flow real-time capturing technology based on the data flow of the Internet of things by adopting a real-time flow analysis algorithm, analyzes the real-time change of the network flow, analyzes the flow intensity according to the characteristics of the data packets, and generates a flow monitoring result;

The trend analysis submodule analyzes historical flow data by adopting a data trend mining algorithm and a statistical learning method based on the flow monitoring result, identifies a periodic mode and an abnormal trend of the flow, predicts the flow trend according to the statistical result and generates a flow trend analysis result;

The flow guiding submodule adopts an optimized path planning algorithm based on the flow trend analysis result, analyzes and selects a network topological structure and a path through a graph theory analysis technology, adjusts the routing and the path distribution of data flow, optimizes network congestion and delay and generates a data flow dynamic analysis result.

Preferably, the intelligent storage optimization module comprises a storage structure adjustment sub-module, a position optimization sub-module and a storage efficiency analysis sub-module;

The storage structure adjustment submodule adopts a hierarchical storage management algorithm based on the data flow analysis result, performs data layering through data classification and access frequency analysis, adjusts the storage format and access strategy of each layer of data according to the characteristics of each layer of data, and matches the access requirement of multiple data types to generate a storage structure adjustment scheme;

The position optimization submodule adopts a storage space optimization algorithm based on a storage structure adjustment scheme, adjusts the distribution of data on a physical storage unit by analyzing the utilization rate and the access mode of the storage unit, optimizes the data access delay and the storage efficiency, and generates a storage position optimization result;

The storage efficiency analysis submodule adopts a storage performance evaluation algorithm based on a storage position optimization result, evaluates the read-write efficiency of the storage equipment by monitoring and analyzing the response time and the data transmission rate of the storage operation, identifies and solves the performance bottleneck, and generates an optimized storage scheme.

Preferably, the static path optimization module comprises a path planning sub-module, a path efficiency evaluation sub-module and a path execution sub-module;

the path planning submodule adopts a network diagram path analysis algorithm based on the optimized storage scheme, identifies key nodes and links by constructing a topological diagram of network data flow, plans potential paths of the data flow, performs optimized layout of data transmission paths by referring to connectivity and path length among the nodes, and generates a data transmission path planning result;

The path efficiency evaluation submodule adopts a path load evaluation algorithm based on a data transmission path planning result, analyzes congestion conditions and data transmission efficiency of paths by simulating data flow on multiple planning paths, evaluates the efficiency of the planning paths by considering the stability and the transmission rate of the paths, and generates a path efficiency analysis result;

The path execution submodule adopts a dynamic route adjustment algorithm based on the path efficiency analysis result, and implements the configuration of the path by updating the network route and adjusting the data transmission mechanism, so that the data flows along the path in the network, and a static path optimization strategy is generated.

Preferably, the edge computing coordination module comprises an edge node management sub-module, a central coordination sub-module and a coordination strategy sub-module;

The edge node management submodule adopts a resource allocation and scheduling algorithm based on a static path optimization strategy, analyzes the computing capacity and the storage capacity of the edge node, combines the position information and the network connection state of the node, performs resource allocation, adjusts the computing and storage resources of each node, matches multiple data processing requirements, and generates an edge node resource allocation scheme;

The center collaboration submodule adopts a data synchronization and exchange algorithm based on an edge node resource allocation scheme, optimizes the synchronization mode of data between a center and an edge node by quantifying the data flow and the processing capacity between the center and the edge node, adjusts the flow of data processing, and generates a data interaction optimization scheme;

The coordination strategy sub-module adopts a calculation task allocation algorithm based on a data interaction optimization scheme, dynamically adjusts the allocation of tasks between the edge nodes and the center by monitoring the network state and the calculation load in real time, optimizes the priority and the sequence of task execution, and generates a coordination calculation scheme.

Preferably, the priority scheduling management module comprises a priority judging sub-module, a scheduling strategy sub-module and a task execution monitoring sub-module;

The priority judging submodule adopts a task importance analysis algorithm based on a coordinated calculation scheme, quantifies task priority by comprehensively evaluating the urgency degree, resource demand and influence range of a data processing task, classifies and prioritizes the task, and generates a task priority analysis result;

The scheduling strategy submodule adopts a dynamic resource allocation algorithm based on a task priority analysis result, adjusts the position of a task in an execution queue by monitoring the system resource condition and the task execution progress in real time, reallocates computing resources and storage resources, optimizes the execution sequence and resource utilization of the task, and generates a task scheduling optimization result;

The task execution monitoring submodule adopts a real-time monitoring and feedback adjustment algorithm based on a task scheduling optimization result, and carries out real-time adjustment and optimization on the task in execution by continuously tracking the task execution state and the system performance index, so as to generate a priority scheduling plan.

Preferably, the time efficiency analysis module comprises a time sequence analysis sub-module, a flow optimization sub-module and a flow time tracking sub-module;

the time sequence analysis submodule analyzes the historical data processing time by implementing trend analysis and seasonal adjustment by adopting a time sequence prediction algorithm based on the priority scheduling plan, identifies a periodic mode and abnormal fluctuation, evaluates the time point of future data processing and generates a time point prediction analysis result;

The flow optimization submodule adopts a flow efficiency optimization algorithm based on the time point prediction analysis result, identifies and eliminates redundant links by analyzing the time consumption and flow configuration of each data processing stage, reconstructs and adjusts the data processing flow, and generates a flow reconstruction optimization result;

The flow time tracking sub-module is based on a flow reconstruction optimization result, adopts a dynamic time tracking algorithm, tracks the execution time and the resource use condition of each step by continuously monitoring the real-time data processing flow, and instantly adjusts the flow execution sequence and the resource allocation to generate a time optimization strategy.

Preferably, the rule driving distribution module comprises a rule setting sub-module, a data distribution sub-module and an efficiency monitoring sub-module;

The rule setting submodule classifies the data tasks by analyzing data characteristics and requirements and utilizing attribute selection measurement and a branch generation strategy based on a time optimization strategy and adopts a decision tree algorithm to formulate a data distribution rule conforming to multi-class task characteristics and generate a preliminary data distribution rule scheme;

The data distribution sub-module analyzes the matching degree of the data task and the available resource by adopting a stable matching principle and a priority matching mechanism based on a preliminary data distribution rule scheme and adopting a matching theory algorithm to generate a refined task resource matching scheme;

The efficiency monitoring submodule monitors efficiency indexes of the data distribution process, including response time and resource utilization rate, by applying an autoregressive model and a moving average model and adopting a time sequence analysis method based on a refined task resource matching scheme, and instantly adjusts a distribution strategy to generate a rule-driven distribution scheme.

Preferably, the dynamic resource adjustment module comprises a load analysis sub-module, a resource allocation sub-module and an adjustment strategy sub-module;

The load analysis submodule adopts a time sequence analysis algorithm based on a rule-driven distribution scheme, analyzes the current processing capacity and data flow of the system by using an autoregressive comprehensive moving average model, and comprises comprehensive consideration of historical data and prediction of future trend, so as to generate a system load prediction result;

The resource allocation submodule optimizes the resource allocation scheme by adopting a genetic algorithm based on a system load prediction result and simulating natural selection and a genetic mechanism, and comprises the steps of initializing population, selecting, crossing and mutating, and performs resource allocation optimizing operation to generate a resource optimization allocation scheme;

the adjustment strategy submodule adopts a feedback control strategy based on a resource optimization allocation scheme, and adjusts and optimizes the resource allocation in real time by continuously monitoring the performance and the system performance of the resource allocation, so as to generate a resource adjustment scheme.

One or more technical schemes provided by the application have at least the following technical effects or advantages:

Through a multidimensional data flow analysis algorithm and real-time flow analysis, the system can effectively monitor and adapt to dynamic data flow in the environment of the Internet of things, remarkably reduce data processing delay and improve data throughput. The adaptive storage allocation and storage space optimization algorithm makes storage management more efficient, and further reduces the pressure of the storage system. The application of the static path optimization and the dynamic route adjustment algorithm optimizes the data flow path and improves the data transmission efficiency of the whole network. In the aspect of the cooperation of the edge computing and the central data processing, the system realizes the full utilization of the edge computing resources and the effective cooperation of the central data processing through the efficient cooperation of the edge node management and the central cooperation sub-module. Not only improves the real-time performance of data processing, but also reduces the communication cost and processing delay. The introduction of the priority scheduling management module, in particular to a dynamic priority queuing algorithm, enables the system to intelligently adjust the data processing sequence according to the emergency degree and importance of the task, thereby ensuring the timely processing of the key data. The time efficiency analysis module, in particular to a time sequence prediction algorithm and a flow efficiency optimization algorithm, enables the system to dynamically optimize time distribution in the processing flow, effectively reduces the total processing time and improves the efficiency of the whole data processing flow. The combination of the rule driving distribution module and the dynamic resource adjustment module provides a high-efficiency data distribution mechanism and a resource optimization strategy for the system, and further improves the overall performance and the resource utilization efficiency of the system.

The foregoing description is only an overview of the present application, and is intended to be implemented in accordance with the teachings of the present application in order that the same may be more clearly understood and to make the same and other objects, features and advantages of the present application more readily apparent.

Drawings

Fig. 1 is a block diagram of an intelligent data convergence software system based on the internet of things;

FIG. 2 is a system frame diagram of an intelligent data convergence software system based on the Internet of things;

FIG. 3 is a schematic diagram showing a specific flow of a data flow control module of an intelligent data convergence software system based on the Internet of things;

Fig. 4 is a schematic diagram of a specific flow of an intelligent storage optimization module of an intelligent data convergence software system based on the internet of things;

fig. 5 is a schematic diagram of a specific flow of the static path optimization module of the intelligent data convergence software system based on the internet of things according to the present invention;

fig. 6 is a schematic diagram of a specific flow of an edge computation coordination module of an intelligent data convergence software system based on the internet of things according to the present invention;

fig. 7 is a schematic diagram of a specific flow chart of a priority scheduling management module of an intelligent data convergence software system based on the internet of things;

Fig. 8 is a schematic diagram of a specific flow of an intelligent data convergence software system time efficiency analysis module based on the internet of things according to the present invention;

Fig. 9 is a schematic diagram of a specific flow of a rule-driven distribution module of an intelligent data convergence software system based on the internet of things;

fig. 10 is a schematic flow chart of a dynamic resource adjustment module of an intelligent data convergence software system based on the internet of things according to the present invention.

Detailed Description

The application provides an intelligent data convergence software system based on the Internet of things.

Summary of the application

Although the prior art has achieved significant achievements in terms of data processing efficiency and accuracy, there are still problems of processing delay and inefficiency in coping with data fluidity and instantaneity in the environment of the internet of things, especially in terms of real-time monitoring and optimization of dynamic data flows, and rapid integration and intelligent analysis of large-scale heterogeneous data sources, the prior art system faces challenges of processing speed and data throughput, and in addition, the prior art system also shows limitations in terms of full utilization of edge computing resources and effective cooperation of a central data processing system, which is particularly obvious in terms of strategic planning and execution efficiency of data distribution, and in terms of priority scheduling and time efficiency optimization of data processing, the prior art has difficulty in fully realizing adaptive adjustment of flows and optimization of time cost, resulting in limited efficiency in processing urgent or complex data.

Aiming at the technical problems, the technical scheme provided by the application has the following overall thought:

As shown in fig. 1, the application provides an intelligent data convergence software system based on the internet of things, wherein the system comprises a data flow control module, an intelligent storage optimization module, a static path optimization module, an edge calculation coordination module, a priority scheduling management module, a time efficiency analysis module, a rule driving distribution module and a dynamic resource adjustment module;

The data flow control module monitors data flow dynamics based on the data flow of the Internet of things by adopting a multidimensional data flow analysis algorithm, analyzes flow trends and modes, performs flow path planning according to the trends, and generates a data flow dynamic analysis result;

the static path optimization module analyzes a data flow path, plans a path layout and adjusts path configuration by adopting a graph theory path optimization algorithm based on the optimized storage scheme to generate a static path optimization strategy;

The data flow dynamic analysis result comprises flow peak distribution, data transmission frequency and flow hot spot areas, the optimized storage scheme specifically refers to adjustment of data storage partition, backup strategy formulation and storage space redistribution, the static path optimization strategy comprises data transmission path determination, backup path planning and path redundancy assessment, the coordination calculation scheme specifically comprises resource sharing rules of edge calculation nodes, a data synchronization mechanism and cooperation task allocation, the priority scheduling scheme comprises task priority classification, execution queue arrangement and a task dependency graph, the time optimization strategy specifically refers to a time compression method of flow execution, critical path optimization and non-critical task postponement strategy, the rule-driven distribution scheme comprises data distribution rule setting, periodic assessment of task allocation logic and distribution efficiency, and the resource adjustment scheme specifically comprises resource redistribution based on load change, dynamic scaling strategy and resource usage optimization.

In the data flow control module, the system receives the data flow of the Internet of things in a JSON format or an XML format through a multidimensional data flow analysis algorithm, and the data contains real-time information of various sensors. The algorithm firstly cleans and normalizes the data, and then analyzes the change trend of the data flow by using statistical analysis and pattern recognition technology. The algorithm can predict peak distribution and hot spot areas of the flow by calculating statistical indexes such as data flow mean values, variances and the like of different time periods. In this process, the algorithm also performs time series analysis in combination with the historical data to thereby plan the most efficient data flow path. The generated dynamic analysis result of the data stream is stored in a database table or a document form, and a decision basis is provided for a subsequent module.

The intelligent storage optimization module adopts a self-adaptive storage allocation algorithm based on the dynamic analysis result of the data flow, and dynamically adjusts the storage strategy according to the size, frequency and type of the data. The module first matches the data stream dynamic analysis result with the current storage system state, and identifies the use mode of the storage resource. Then, the machine learning algorithm is utilized to classify different types of data, such as time series data, event trigger data, etc., and the data storage structure is adjusted according to the classification result and the data access frequency. Finally, the module optimizes the distribution of the data on the physical storage unit through a linear programming algorithm, generates an optimized storage scheme, and implements the optimized storage scheme in a database update or configuration file mode, so that the data access speed and the storage efficiency are improved.

And after receiving the optimized storage scheme, the static path optimization module performs planning and optimization of the data flow path through a graph theory path optimization algorithm. The module first builds a graph model of the data flow network in which nodes represent data processing units and edges represent data transmission paths. The algorithm determines the optimal path of data transmission by calculating key indexes such as the shortest path, the minimum cut and the like of the network diagram. Based on this, the module also considers network congestion and delay factors to dynamically adjust the path configuration. Finally, the generated static path optimization strategy is stored in a configuration file form to guide the data to flow in the network so as to achieve the purposes of reducing delay and improving transmission efficiency.

The edge computing coordination module combines a static path optimization strategy and a distributed resource coordination algorithm to optimize the cooperation of the edge computing nodes and the central data processing unit. The module first analyzes the computing power and storage capacity of the edge nodes while taking into account the processing power of the central system. And then, dynamically distributing tasks based on the calculation complexity and the data dependency relationship of the tasks by using a distributed scheduling algorithm, and optimizing resource distribution and task scheduling strategies. The algorithm also takes network status and data synchronization requirements into account, ensuring efficient synchronization of data between the edge and the center. The generated coordination calculation scheme is output in a strategy document form and is used for guiding the cooperation of the edge calculation nodes and the central system, so that the processing capacity and the efficiency of the whole system are improved.

The priority scheduling management module utilizes a dynamic priority queuing algorithm to plan and adjust task priorities based on a coordination calculation scheme. The module classifies the tasks according to the urgency, resource demand and scope of influence of the tasks. And then, optimizing the task execution queue by using a queuing algorithm based on weight, so as to ensure the priority processing of the high-priority task. The module dynamically adjusts the task queue and the resource allocation by monitoring the system resource status and the task execution progress in real time. The generated priority scheduling plan is realized in the form of configuration files, so that timeliness of task processing and reasonable utilization of system resources are effectively balanced.

The time efficiency analysis module optimizes the time distribution of the data processing flow by adopting a time window analysis method based on the priority scheduling plan. The module first analyzes the time consumption patterns of different data processing tasks, and then optimizes the execution sequence and parallel processing strategy of each task by using a time window analysis method. The module also considers the dependency relationship among tasks and ensures the preferential execution of the critical path. The generated time optimization strategy is realized in a scheduling plan mode, so that the total processing time is reduced to the greatest extent, and the overall efficiency of the data processing flow is improved.

The rule-driven distribution module analyzes and plans the data distribution requirement by adopting a logic reasoning distribution algorithm based on a time optimization strategy. The module firstly carries out rule setting according to the type, the emergency degree and the target system of the data, and then uses a decision tree and a logical reasoning algorithm to generate and optimize the data distribution rule. The algorithm takes efficiency and accuracy in the data distribution process into consideration, and generates a rule-driven distribution scheme. The scheme is stored in a rule base form and is used for guiding efficient distribution of data in the system and ensuring that the data quickly and accurately reach a target processing unit.

The dynamic resource adjustment module receives a rule-driven distribution scheme, and adopts a load-aware resource allocation algorithm to dynamically optimize resource allocation. The module analyzes the current system load and data traffic, and then optimizes the resource allocation using a genetic algorithm according to the load prediction result. The algorithm finds the optimal resource allocation scheme by simulating the natural selection process. The generated resource adjustment scheme is output in a strategy document form, and the system resource allocation is adjusted in real time, so that the system can be ensured to operate efficiently under different load conditions.

Specifically, as shown in fig. 2 and 3, the data flow control module includes a flow monitoring sub-module, a trend analysis sub-module, and a flow guiding sub-module;

the flow monitoring submodule captures and initially classifies data packets through a network flow real-time capturing technology based on the data flow of the Internet of things by adopting a real-time flow analysis algorithm, analyzes real-time change of the network flow, analyzes flow intensity according to the characteristics of the data packets, and generates a flow monitoring result;

The trend analysis submodule adopts a data trend mining algorithm based on the flow monitoring result, analyzes historical flow data through a statistical learning method, identifies a periodic mode and an abnormal trend of the flow, predicts the flow trend according to the statistical result, and generates a flow trend analysis result;

The flow guiding submodule adopts an optimized path planning algorithm based on the flow trend analysis result, analyzes and selects a path of a network topological structure through a graph theory analysis technology, adjusts the route selection and the path distribution of data flow, optimizes network congestion and delay, and generates a data flow dynamic analysis result.

In the flow monitoring submodule, data processed by the system are mainly presented in a JSON format, and various sensor data collected from the internet-of-things equipment are covered. The submodule adopts a real-time flow analysis algorithm, and firstly carries out real-time capturing and preliminary classification on the data packet through a network flow real-time capturing technology. This process involves parsing the header information of the data packet, identifying its source and destination addresses, transport protocol type, etc., to initially classify and sort the data stream. The sub-modules then perform real-time variation analysis on these classified data streams, applying traffic intensity analysis techniques such as peak detection and outlier identification to monitor real-time variation of network traffic. In the process, the algorithm calculates the size, transmission speed and arrival frequency of each data packet to generate a flow monitoring result. The results are stored in a database, flow information of each time point is recorded, and a basis is provided for subsequent trend analysis.

The trend analysis sub-module receives the flow monitoring result from the flow monitoring sub-module and uses a data trend mining algorithm to conduct deep analysis. The submodule applies statistical learning methods such as linear regression and time series analysis to comprehensively analyze the historical flow data. It identifies periodic patterns and sudden anomaly trends in the data stream and uses this information to predict future traffic changes. For example, by calculating the moving average and standard deviation of the flow data, the module can identify long-term trends and seasonal fluctuations in flow. Meanwhile, the sub-module also adopts a machine learning algorithm, such as a random forest or a neural network, to improve the accuracy of prediction. The finally generated flow trend analysis results are stored in a report form, and the predicted flow mode and the potential hot spot areas are described in detail to provide data support for the decision making of the flow guiding sub-module.

The flow guiding sub-module applies an optimized path planning algorithm to optimize the route of the data flow based on the flow trend analysis result provided by the trend analysis sub-module. First, the submodule constructs a network topology structure diagram including a data center, edge nodes and transmission links. The best path is then found for the data stream using graph-theory analysis techniques, such as Dijkstra's algorithm or a-search algorithm. In the process, the algorithm takes factors such as the congestion degree, the transmission delay, the data priority and the like of the path into consideration to optimize the path. For example, for high priority real-time data, the algorithm will choose a shorter path, even though this means a higher risk of congestion; for non-urgent data, a longer but more stable path is selected. In this way, the dynamic analysis results of the data flow generated by the submodule contain routing suggestions for different data types and service requirements, and the results are stored as configuration files or policy documents for use by the network management system to optimize the data flow of the whole network and reduce congestion and delay.

Assume a temperature monitoring scenario in an internet of things environment, where internet of things devices send temperature data to a central system every minute. These data are packaged in JSON format, containing a timestamp, device ID and temperature value. For example, one piece of data is { "time stamp": "2023-04-05T12:00:00", "device_id": "sensor_001", "temperature":22.5}. In the traffic monitoring sub-module, a real-time traffic analysis algorithm captures and classifies these packets. Assume that within one hour, the system receives 60,000 packets from 100 different sensors. The algorithm analyzes the data and monitors a significant increase in the frequency of afternoon temperature data upload, resulting in an increase in network traffic from an average of 500 packets per minute to 800. The trend analysis sub-module then receives these flow monitoring results. By comparing and analyzing the data for the same time period of each day during the past week, the sub-module finds that similar peaks in flow occur every afternoon. For example, the average number of packets per minute remains above 800 continuously after noon, while other time periods are below 500. Based on this finding, the flow directing sub-module employs an optimized path planning algorithm to assign a more efficient transmission path for these high frequency temperature data. For example, it calculates that sending data directly to the nearest data processing center, rather than to the default remote cloud server, can reduce the average transmission delay from the original 200 milliseconds to 150 milliseconds. Updating the adjusted path planning result to the network management system in the form of a configuration file, wherein the configuration file is specified as follows: "sensor_001" to "sensor_100" are directly connected to "DATACENTER _a" between 12:00 and 14:00. In this way, the system not only monitors and analyzes data traffic in real time, but also intelligently optimizes the data transmission path. The adjustment obviously relieves network congestion, improves the data transmission efficiency and real-time performance of the whole network, and ensures the rapid processing of key data.

Specifically, as shown in fig. 2 and fig. 4, the intelligent storage optimization module includes a storage structure adjustment sub-module, a position optimization sub-module, and a storage efficiency analysis sub-module;

The storage structure adjustment submodule adopts a hierarchical storage management algorithm based on the dynamic analysis result of the data flow, performs layering on the data through data classification and analysis of access frequency, adjusts the storage format and access strategy of the data according to the characteristics of each layer of data, and matches the access requirement of multiple data types to generate a storage structure adjustment scheme;

the position optimization sub-module is based on a storage structure adjustment scheme, adopts a storage space optimization algorithm, adjusts the distribution of data on a physical storage unit by analyzing the utilization rate and the access mode of the storage unit, optimizes the data access delay and the storage efficiency, and generates a storage position optimization result;

In the storage structure adjustment sub-module, data mainly adopts a structured format, such as XML or CSV, and contains key information about the type, the size, the access frequency and the like of the data. The submodule adopts a hierarchical storage management algorithm, and firstly classifies data, such as real-time monitoring data, historical record data, log data and the like. Next, a data layering strategy is determined by analyzing the access frequency and processing requirements of the data. For example, frequently accessed data is assigned to the quick access layer, while infrequently accessed historical data is placed in deep storage. Next, the storage format and access policy are adjusted according to the characteristics of each layer of data, such as using a more efficient index structure for real-time data and compression technique for historical data to save space. Finally, the storage structure adjustment scheme generated by the submodule specifies the storage hierarchy and the corresponding access policy of various data, and the scheme is stored as a configuration file and is used for guiding the optimization and reorganization of the data storage system so as to improve the storage efficiency and the data access speed.

The position optimization sub-module further optimizes the physical storage position of the data by using a storage space optimization algorithm on the basis of the storage structure adjustment scheme. The sub-module analyzes the usage and data access patterns of the storage units, for example, by calculating the number of I/O operations and data access delays for each storage unit, and identifies storage hot spot areas and underutilized storage space. Then, based on the analysis results, the distribution of the data on the physical storage units is adjusted. For example, data accessed at high frequency is migrated to a better performing storage unit, while data accessed less frequently is migrated to a larger but lower performing storage unit. The storage location optimization results generated by the sub-modules specify the optimal physical storage locations for each type of data, and these results are implemented in the form of storage configuration instructions for improving data access efficiency and reducing storage latency.

And the storage efficiency analysis sub-module comprehensively evaluates the read-write efficiency of the storage device by using a storage performance evaluation algorithm based on the storage position optimization result. The submodule monitors response time and data transfer rate of the store operation, such as by tracking completion time and data transfer rate of the store request, and identifies performance bottlenecks. For example, if a memory cell is found to have a response time well above average, the submodule may further analyze the load condition and hardware state of the memory cell. Based on these analyses, the sub-module identifies and proposes policies that address performance issues, such as reallocating data to relieve overload storage units or proposing to upgrade hardware. Finally, the resulting optimized storage schemes detail how to improve the performance of the storage system, and these schemes are saved in the form of reports and execution instructions, which aim to improve the read-write efficiency and overall performance of the overall storage system.

An internet of things environment is assumed for monitoring and analyzing machine operation data within a plant. The environment generates a large amount of data including machine state, temperature readings, energy consumption, etc. daily, and the data is stored in a CSV format, wherein each data format is as follows: { "time stamp": "2024-01-08T10:00:00", "machine_id": "M001", "temperature":75,

"Energy_Condition": 500}. And in the storage structure adjustment sub-module, classifying and layering the data by adopting a hierarchical storage management algorithm. For example, frequently accessed machine state data is placed in the quick access layer, and temperature and energy consumption data before one week is archived into deep storage. Thus, the latest machine state data (one piece per minute, such as { "timestamp":

"2024-01-08T10:00:00", "machine_id": "M001", "status": "running" }) is quickly accessible, while outdated data is stored in a lower cost store. The location optimization submodule analyzes the usage of the storage units, and finds that the usage of one high-performance SSD storage unit (storage unit A) reaches 90%, while the usage of the other HDD storage unit (storage unit B) is only 30%. Based on this, the submodule adjusts the storage allocation to migrate part of the historical data from storage unit a to storage unit B to balance load and optimize performance. The storage efficiency analysis submodule finds through monitoring that the average read-write response time of the storage unit A is 15ms, and the storage unit B is 50ms. To increase efficiency, the submodule proposes to retain data in the last week in storage unit a, while earlier data is migrated to storage unit B. After this adjustment, the response time of memory cell a is reduced to 12ms, while memory cell B is maintained at 50ms, and overall memory efficiency is optimized. Finally, these storage optimization measures are integrated into a detailed storage optimization scheme, including storage hierarchy adjustment, data migration instructions, and performance improvement measures. Through the adjustment, the data storage system of the factory not only realizes the efficient management and access of the data, but also remarkably improves the overall storage performance and the data processing speed.

Specifically, as shown in fig. 2 and 5, the static path optimization module includes a path planning sub-module, a path efficiency evaluation sub-module, and a path execution sub-module;

The path planning submodule adopts a network diagram path analysis algorithm based on an optimized storage scheme, identifies key nodes and links by constructing a topological diagram of network data flow, plans potential paths of the data flow, performs optimized layout of data transmission paths by referring to connectivity and path length among the nodes, and generates a data transmission path planning result;

The path efficiency evaluation submodule adopts a path load evaluation algorithm based on the data transmission path planning result, analyzes the congestion condition and the data transmission efficiency of the path by simulating the data flow on multiple planning paths, and evaluates the efficiency of the planning paths by considering the stability and the transmission rate of the path to generate a path efficiency analysis result;

And in the path planning sub-module, analyzing the topological structure of the data flow of the Internet of things through a network diagram path analysis algorithm. The algorithm receives as input a network topology graph of the data flow, the nodes in the graph representing devices in the network, and the edges representing paths of the data flow. The algorithm first performs a depth-first search of the network topology to identify key nodes and links, which are the core part of the data flow. The shortest path between nodes is then calculated using the bellman-ford algorithm, providing an optimal path layout for the data flow. In the path optimization process, the algorithm also considers the connectivity and path length between nodes so as to ensure the data transmission efficiency. Through the steps, the path planning submodule generates detailed data transmission path planning results, and a foundation is provided for subsequent path efficiency evaluation.

The path efficiency evaluation sub-module analyzes the data flow condition on the planned path by using a path load evaluation algorithm on the basis of the path planning sub-module. The algorithm first simulates the data flow on the planned path, using Monte Carlo simulation to predict the flow distribution and congestion on different paths. The algorithm also combines real-time network traffic data, and dynamically evaluates the load condition of the path by monitoring the network state in real time. In addition, the algorithm also applies queue theory to analyze the waiting time of data transmission so as to evaluate the transmission efficiency of the path. Through the calculation, the submodule can propose specific path optimization suggestions, generate detailed path efficiency analysis results and indicate which paths can provide more efficient and stable data transmission.

The path execution submodule implements path configuration by using a dynamic routing adjustment algorithm based on the analysis results of the first two submodules. The algorithm takes the path efficiency analysis result as input, and uses a shortest path algorithm of graph theory, such as Di Jie St-Law algorithm, to update the network route in real time, so as to ensure that the data flows along the optimal path. In the algorithm execution process, a load balancing technology is combined, and the distribution of data on different paths is dynamically adjusted so as to cope with the change of network states. In addition, the algorithm also realizes a self-adaptive adjustment mechanism, and automatically adjusts a data transmission mechanism according to network congestion and delay conditions, such as changing the size of a data packet or adjusting the transmission frequency. Through the operations, the path execution sub-module not only ensures the high efficiency of data transmission, but also increases the stability and the robustness of the network, and generates a specific executable static path optimization strategy.

Assuming that there are A, B, C nodes in the internet of things network, the network topology shows that a to C have two paths, one through B (a-B-C) and the other directly connected (a-C). In the path planning sub-module, the algorithm analyzes the two paths and determines that the a-B-C path, although longer, is more efficient in data transmission due to the high processing power of the node B. Specific data shows that the processing speed of the node A, B, C is 100Mbps, 300Mbps and 100Mbps, and the storage capacity is 50GB, 150GB and 50GB, respectively. The total length of paths a-B-C is 200km, while the total length of paths a-C is 100km. During the high traffic period, the data transmission rate of the path a-C drops to 70Mbps, while the path a-B-C is maintained at a transmission rate close to 290Mbps due to the high processing power of the node B. The path efficiency evaluation submodule discovers through analog data flow that the A-C path is easy to be jammed during a high-traffic period, and the A-B-C path is more stable. Finally, the path execution submodule decides to use the A-B-C path preferentially in the high traffic period and to use the A-C path in the low traffic period. This dynamic adjustment ensures an efficient data transmission and a stable network. Finally, the system generates a detailed data flow policy report containing path selection, traffic distribution, and expected transmission efficiency.

Specifically, as shown in fig. 2 and fig. 6, the edge computing coordination module includes an edge node management sub-module, a central coordination sub-module, and a coordination policy sub-module;

The central collaboration sub-module adopts a data synchronization and exchange algorithm based on an edge node resource allocation scheme, optimizes the synchronization mode of data between a center and an edge node by quantifying the data flow and the processing capacity between the center and the edge node, adjusts the flow of data processing, and generates a data interaction optimization scheme;

In the edge node management sub-module, the computing capacity and the storage capacity of the edge node are carefully analyzed and optimally configured through a resource allocation and scheduling algorithm. The submodule receives data from the static path optimization strategy, including the connection state among nodes, the network traffic distribution and the performance index of each node. The algorithm first evaluates the computing power and storage capacity of each edge node, quantifying node performance using a weight-based scoring system. This process involves calculating metrics such as processing speed, memory usage, and response time for each node. The algorithm then uses a linear programming approach to optimize the resource allocation in combination with the geographical location of the node and the network connection status. In this process, the algorithm considers the diverse needs of data processing and the cooperative work between nodes to ensure that the resource allocation maximally improves the overall network performance. Finally, the sub-module generates a detailed edge node resource allocation scheme including resource allocation details and expected performance improvements for each node.

The central collaboration sub-module adopts a data synchronization and exchange algorithm to optimize data interaction between the center and the edge nodes on the basis of the edge node resource allocation scheme. The submodule firstly acquires data flow and processing capacity information between the edge node and the central node, wherein the data flow and processing capacity information comprises data packet size, transmission frequency and delay time. The algorithm applies graph-based data flow analysis techniques to construct a model of the data flow to quantify and visualize the pattern of the data flow. The data synchronization and exchange process is then optimized using a dynamic programming algorithm that dynamically adjusts the distribution of data between the center and edge nodes taking into account the priority of the data, the transmission rate, and the processing power of the nodes. In the process, the algorithm also applies a caching technology to reduce data transmission delay and improve data processing efficiency. Finally, the submodule generates a data interaction optimization scheme, and the improved data synchronization mode and expected performance improvement are described in detail.

The coordination strategy sub-module dynamically adjusts the allocation of tasks between the edge nodes and the center by using a computing task allocation algorithm based on the data interaction optimization scheme. The submodule collects real-time network state and calculation load data firstly, wherein the real-time network state and calculation load data comprise network bandwidth use conditions, CPU (Central processing Unit) of the node and memory use rate. The algorithm then uses constraint-based optimization techniques to formulate task allocation policies, taking into account the urgency of the task, resource requirements, and availability. The algorithm balances the priority of task execution and the resource utilization rate by constructing a multi-objective optimization model, ensures that the task with high priority obtains enough resources, and improves the resource utilization efficiency of the whole network. In the process, the algorithm also realizes a self-adaptive adjustment mechanism, and dynamically adjusts task allocation according to real-time data so as to respond to the change of the network state. Finally, the sub-modules generate a coordinated computing scheme detailing the policies and expected effects of task allocation, including order of task execution, resource allocation, and performance improvement.

Assuming three edge nodes A, B, C and a central node, node A has strong computing power but limited storage space, and B and C have moderate computing power but sufficient storage space. The specific node performance data are: the processing speed of the node A is 1.5GHz, and the storage space is 256GB; the processing speed of the node B and the node C is 1GHz, and the storage space is 500GB. After the analysis of the edge node management submodule, the edge node management submodule decides to allocate more calculation tasks for the node A, and mainly allocates data storage tasks to the nodes B and C, so that the advantages of each node are fully utilized. The resource allocation and scheduling algorithm determines a resource allocation scheme according to the high processing speed and low memory usage of the node a, and the high storage capacity and stable connection state of the nodes B and C. This scheme details the resource usage plan for each node, including the type of processing task and the amount of data stored. The central collaboration sub-module then optimizes the data interactions between node A, B, C and the central node. According to the data synchronization and exchange algorithm, the submodule finds that the data transmission between the node A and the center is frequent but small in quantity, and the data transmission between the node B and the center is large but low in frequency. Thus, the algorithm adjusts the data synchronization policy, setting up more frequent small packet synchronization for node a, while enforcing a large packet but less frequent synchronization policy for B and C. This optimization reduces the processing pressure of the center node while maintaining the real-time and accuracy of the data between the edge node and the center node. The coordination policy sub-module monitors the data in real time, and finds that during high load periods, the processing power of the central node is insufficient to handle all data processing requests from the edge nodes. Thus, the computing task allocation algorithm decides to transfer part of the computationally intensive tasks to the edge nodes, in particular node a, which is more computationally intensive during high load periods, while more of the processing power of the central node is utilized during low load periods. The dynamic adjustment strategy ensures the timely processing of the task and avoids the waste of resources. The coordinated computing scheme details the mechanism of this dynamic task allocation, including the kind of task, allocation time, and expected processing efficiency. These schemes not only improve the overall network performance, but also ensure high efficiency and low latency of data processing.

Specifically, as shown in fig. 2 and fig. 7, the priority scheduling management module includes a priority determining sub-module, a scheduling policy sub-module, and a task execution monitoring sub-module;

In the priority judging submodule, the emergency degree, the resource requirement and the influence range of the data processing task are comprehensively evaluated through a task importance analysis algorithm. The data received by the sub-module includes the type of task, the expected time required, the resources required and their impact on the system operation, etc. The algorithm first inputs the data into a rule-based scoring system that scores each task for urgency, resource demand, and impact. The emergency degree score is based on the adjacency of the task deadline, the resource demand score is quantified according to the computing capacity and the storage space required by the task, and the influence range score considers the influence of the task completion on the overall operation of the system. The algorithm then integrates these scores using a weighted sum method to determine the overall priority of each task. Finally, the submodule generates a task priority analysis result, and the result lists the priority orders of all the tasks in detail, so that a basis is provided for a subsequent scheduling strategy.

The scheduling policy submodule optimizes the task execution sequence and the resource utilization by using a dynamic resource allocation algorithm based on the task priority analysis result. The submodule firstly acquires current resource status data of the system, including CPU and memory use conditions of each computing node and execution progress information of tasks. The algorithm applies a priority-based queue scheduling mechanism that adjusts the position of tasks in the execution queue according to their priorities. Meanwhile, the algorithm adopts a dynamic adjustment strategy, and the computing resources and the storage resources are redistributed according to the real-time resource use condition and the task progress. For example, for high priority tasks, the algorithm may add more computing resources, while for low priority long-term tasks, the resource allocation may be reduced to increase the efficiency of resource usage. Finally, the submodule generates a task scheduling optimization result, and the result describes the optimized task execution sequence and resource allocation scheme in detail, so that the task processing efficiency and the utilization rate of system resources are improved.

The task execution monitoring submodule adopts a real-time monitoring and feedback adjustment algorithm to ensure the high efficiency of task execution based on the task scheduling optimization result. The submodule continuously tracks the task execution state and the system performance index, including the completion degree, the execution time, the resource use condition and the like of the task. Algorithms use real-time data analysis techniques, such as time series analysis, to evaluate the real-time efficiency of task execution and resource usage. If the task execution efficiency is low or the resource usage is unbalanced, the algorithm immediately adjusts, such as reallocating the resource or adjusting the priority of the task. This feedback mechanism ensures a fast response and efficient handling of dynamic changes by the system. Finally, the sub-module generates a priority scheduling plan that includes the final execution order of the tasks, the resource allocation, and the expected completion time, ensuring efficient operation of the overall system.

Assuming that there are three tasks to be processed in the system, task 1 is a high priority urgent task, task 2 is a medium priority regular task, and task 3 is a low priority long term task. The priority judging submodule scores and sorts the three tasks, and determines that the priority of the task 1 is highest and the task 3 is lowest. The specific data are as follows: task 1 has an urgency score of 9 (full score 10), a resource demand score of 7, an impact range score of 8, and a total priority score of 8.0; the emergency degree score of the task 2 is 5, the resource demand score is 6, the influence range score is 5, and the total priority score is 5.3; task 3 has an urgency score of 2, resource demand score of 4, impact range score of 3, and total priority score of 3.0. The scheduling policy sub-module allocates more computing resources to task 1 according to the priority order and the resource status of the system, and reduces the resource allocation of task 3. Initially, task 1 allocated 50% of CPU resources and 40% of memory resources, task 2 allocated 30% of CPU resources and 35% of memory resources, and task 3 allocated 20% of CPU resources and 25% of memory resources. The task execution monitoring submodule continuously monitors the execution states of the three tasks, and finds that the execution efficiency of the task 2 is lower than expected, and the completion degree of the task 2 reaches 40% only at 60% of the expected time. The resource allocation of task 2 is then adjusted, increasing its CPU resources to 40% and decreasing the CPU resources of task 3 to 10%. The finally generated priority scheduling plan details the resource allocation and the expected completion time of each task, and effectively improves the overall performance of the system and the task processing efficiency.

Specifically, as shown in fig. 2 and 8, the time efficiency analysis module includes a time sequence analysis sub-module, a flow optimization sub-module, and a flow time tracking sub-module;

the flow time tracking sub-module adopts a dynamic time tracking algorithm based on a flow reconstruction optimization result, tracks the execution time and resource use condition of each step by continuously monitoring the real-time data processing flow, and instantly adjusts the flow execution sequence and resource allocation to generate a time optimization strategy.

In the time series analysis sub-module, historical data processing time is analyzed by a time series prediction algorithm to evaluate a point in time of future data processing. The data format received by the sub-module includes the time stamp of the historical data processing, the time used, the type and size of the associated task. The algorithm first uses an autoregressive moving average model to trend the historical data, identifying a periodic pattern of data processing time. Next, a time series analysis of seasonal decomposition is applied to adjust for seasonal variations, more accurately predicting future data processing time points. The algorithm takes into account the unusual fluctuations of the historical data during the analysis and employs outlier detection techniques to correct these deviations. Through the steps, the submodule generates time point prediction analysis results which show the expected data processing time of each time point in the future in detail, and the system is helped to adjust the resource allocation and the task scheduling in advance.

The flow optimization submodule reconstructs and adjusts the data processing flow by applying a flow efficiency optimization algorithm based on the time point prediction analysis result. The data collected by the sub-modules includes time consumption, resource usage, and flow configuration information for each data processing stage. The algorithm firstly analyzes the time consumption of the existing flow, and identifies the bottleneck link in the flow by using a key path method. Value flow analysis is then employed to identify and eliminate redundant elements in the flow, such as invalid data transmissions or unnecessary processing steps. When the process is reconstructed, the algorithm also considers the resource utilization rate and the processing efficiency, and adopts linear programming to optimize the resource allocation and the process layout. Through the operations, the submodule generates a flow reconstruction optimization result, and the result details the optimized data processing flow, including the time consumption and the resource requirement of each stage, so that the efficiency of the whole processing flow is obviously improved.

The flow time tracking sub-module monitors the real-time data processing flow by adopting a dynamic time tracking algorithm based on the flow reconstruction optimization result. The sub-modules constantly collect time records, resource usage and execution status of real-time data processing. Algorithms apply real-time data analysis techniques, such as stream processing and event driven programming, to keep track of the execution time and resource consumption of each step. Based on this information, the algorithm dynamically adjusts the flow execution order and resource allocation to accommodate real-time workload and system performance. For example, if the execution time of a certain data processing step continues beyond expectations, the algorithm may immediately adjust the resource allocation or reorder the execution order of other steps. Finally, the sub-module generates a time optimization strategy detailing decisions and expected effects of real-time adjustment, such as reduced processing time and increased resource utilization.

Assuming that there is a data processing task in the system, the history data shows that the processing time is in a mode of weekend decrease and workday increase in the past month, and the average processing time is 2 hours. The time series analysis submodule analyzes the historical data and predicts that the processing time in the next week will rise to 2.5 hours on weekdays and fall to 1.5 hours on weekends. Flow optimization submodule analysis found that redundant operations exist in the data preprocessing stage, resulting in an increase in average processing time of 30 minutes. The treatment time at this stage was reduced to 20 minutes by optimization. The flow time tracking submodule monitors the data processing flow in real time according to the changes, and discovers that after the optimization, the processing time of the overall task is reduced to be 1 hour and 45 minutes on average, so that the efficiency is improved and the resource consumption is reduced.

Specifically, as shown in fig. 2 and 9, the rule driving distribution module includes a rule setting sub-module, a data distribution sub-module, and an efficiency monitoring sub-module;

The rule setting submodule classifies the data tasks by analyzing the data characteristics and the requirements and utilizing the attribute selection measurement and the branch generation strategy based on the time optimization strategy, and formulates the data distribution rules conforming to the multi-class task characteristics to generate a preliminary data distribution rule scheme;

In the rule setting submodule, the data tasks are classified through a decision tree algorithm, and data distribution rules are formulated. The data format received by the sub-module comprises the attributes of the type, the size, the processing duration, the emergency degree and the like of the task. The decision tree algorithm first uses the information gain or the genie uncertainty as an attribute selection metric to select the most appropriate attribute to branch. In constructing the decision tree, the algorithm progressively refines each node until each leaf node represents a particular task class. In the branch generation process, the algorithm considers the diversity and complexity of the tasks to ensure that the generated rules can cover the characteristics of various tasks. Finally, the submodule generates a preliminary data distribution rule scheme, and the scheme lists the distribution rules of various tasks in detail, including the priority of the tasks, target resources and processing modes, and provides clear guidance for data distribution.

The data distribution sub-module adopts a matching theory algorithm to optimize the matching of tasks and resources based on a preliminary data distribution rule scheme. The data received by the sub-module includes the type, quantity, performance index and current state of the available resources. The match theory algorithm first builds a task and resource preference list that includes the preference ordering of each task for the resources and the preference ordering of each resource for the tasks. Then, the algorithm uses a stable matching principle and a priority matching mechanism, and gradually optimizes the matching between the task and the resource through an iterative process until a stable matching state is achieved, wherein no task or resource is willing to change the current matching result. Finally, the submodule generates a refined task resource matching scheme, the scheme details the specific resources allocated to each task, the reasons and the expected effects of matching, and the utilization efficiency of the resources and the accuracy of task processing are improved.

The efficiency monitoring submodule monitors the efficiency of the data distribution process by adopting a time sequence analysis method based on a refined task resource matching scheme. The data collected by the sub-modules includes response time, completion time, and utilization of the resources for the task. The time series analysis method applies an autoregressive model and a moving average model to analyze time series data of the efficiency index and identify trends and seasonal patterns of the efficiency. Based on these analysis results, the algorithm adjusts the data distribution policy in real time, such as adjusting the priority of tasks or reallocating resources. Finally, the submodule generates a rule-driven distribution scheme which contains the adjusted data distribution rules and the expected efficiency improvement, and ensures the high efficiency of the data distribution process and the optimal utilization of system resources.

Assume that there are three types of data tasks in the system: emergency processing tasks, routine analysis tasks, and long-term storage tasks. The task data received by the rule setting submodule is displayed, the emergency processing task is smaller than 1GB, and the processing time is smaller than 30 minutes; the size of the conventional analysis task is between 1GB and 10GB, and the treatment time is between 30 minutes and 2 hours; the long-term storage task is more than 10GB, and the processing time is longer than 2 hours. The decision tree algorithm generates distribution rules of three tasks according to the attributes, the emergency processing tasks are preferentially distributed to high-performance computing resources, the conventional analysis tasks are distributed to standard computing resources, and the long-term storage tasks are distributed to mass storage resources. The matching theory algorithm of the data distribution sub-module further refines the rules, and optimal matching is carried out according to the current state of the resource and the emergency degree of the task. The efficiency monitoring submodule analysis finds that the average response time of the emergency processing task is 20 minutes, the average completion time of the conventional analysis task is 1.5 hours, and the resource utilization rate of the long-term storage task is 75%. Based on the data, the sub-module adjusts the distribution strategy, improves the priority of the emergency processing task, and reallocates part of the storage resources to improve the efficiency of long-term storage tasks. The resulting rule-driven distribution scheme details the adjusted distribution rules and the expected efficiency improvement.

Specifically, as shown in fig. 2 and 10, the dynamic resource adjustment module includes a load analysis sub-module, a resource allocation sub-module, and an adjustment policy sub-module;

The resource allocation submodule optimizes a resource allocation scheme by adopting a genetic algorithm based on a system load prediction result and simulating natural selection and a genetic mechanism, and comprises the steps of initializing population, selecting, crossing and mutating, and performs resource allocation optimizing operation to generate the resource optimization allocation scheme;

The adjustment strategy submodule is based on a resource optimization allocation scheme, adopts a feedback control strategy, and carries out real-time adjustment and optimization on the resource allocation by continuously monitoring the performance and the system performance of the resource allocation, so as to generate a resource adjustment scheme.

In the load analysis submodule, the current processing capacity and data flow of the system are deeply analyzed through a time sequence analysis algorithm, in particular an autoregressive comprehensive moving average model. The data format received by the sub-module includes processing capacity parameters of the system such as CPU utilization, memory occupation, network bandwidth usage, and history of data traffic such as data volume, processing time and frequency of each task. The execution flow of the time series analysis starts with the integration and preprocessing of the historical data to form a continuous time series. Next, an autoregressive integrated moving average model is used to analyze the trends and seasonal patterns of these data, while taking into account the autocorrelation and partial autocorrelation of the historical data. The setting of model parameters, such as autoregressive terms, differential times and moving average terms, is optimized based on the red-cell information criterion. And finally, predicting future load trend by the model, and generating a system load prediction result. These results detail the expected processing power requirements and data traffic, making resource management more proactive and adaptive.

The resource allocation sub-module adopts a genetic algorithm to optimize a resource allocation scheme based on a system load prediction result. The input data for the sub-modules includes system resource configuration information such as available CPU cores, memory size, and storage space. The implementation of the genetic algorithm involves initializing a population of solutions, each solution representing a resource allocation scheme. Next, the performance of each solution is evaluated by a fitness function, which is defined based on the resource utilization and the task completion rate. Solutions in the population are continually evolved through selection, crossover and mutation operations to explore more optimal resource allocation schemes. The selection operation is based on the fitness, and a scheme with better performance is reserved; crossover and mutation operations introduce new schemes that increase diversity of populations. After multiple generations of iteration, the algorithm converges to an optimal solution to generate a resource optimization allocation scheme. This scheme specifies in detail how system resources are allocated to meet predicted load demands, thereby improving resource utilization efficiency and task processing speed.

The adjustment strategy submodule is used for carrying out real-time resource adjustment by applying a feedback control strategy based on a resource optimization allocation scheme. The submodule continuously collects resource use condition and system performance data, such as real-time CPU and memory use rate, response time of tasks and completion rate. The execution of the feedback control strategy includes analyzing the real-time data to evaluate the performance of the current resource allocation scheme. If resource utilization is found to be uneven or tasks are delayed, the policy may dynamically adjust resource allocation, such as increasing CPU and memory allocation for high load tasks, or decreasing allocation of idle resources. This adjustment is a continuous process aimed at constantly optimizing the resource configuration according to the actual operating conditions of the system. Finally, the sub-module generates a resource adjustment scheme, including adjusted resource allocation details and expected performance improvements. The scheme not only improves the responsiveness and adaptability of the system, but also ensures the efficient utilization of resources.

Assuming a data center running multiple parallel tasks, the historical data shows that CPU usage averages 80% during weekday peak hours (e.g., 9 am to 5 pm) and drops to 40% during the night and weekends. The load analysis sub-module uses the ARIMA model to analyze these data, predicting CPU usage trends in the next week, and finding that CPU demand at the peak of the weekday will increase to 85%. The genetic algorithm of the resource allocation submodule optimizes the allocation of CPU cores based on the prediction, allocates more cores for high-demand tasks, and reduces the allocation of low-demand time periods. The adjustment strategy sub-module finds that certain tasks often have delays during peak periods according to the real-time monitoring data, and then dynamically increases the resource allocation of the tasks. The resulting resource adjustment scheme details the resource allocation of each task at different time periods, and the expected performance improvement, such as reduced task response time during peak hours, and increased resource utilization during nighttime and weekends.

Although the application has been described in connection with specific features and embodiments thereof, it will be apparent that various modifications and combinations can be made without departing from the spirit and scope of the application. Accordingly, the specification and drawings are merely exemplary illustrations of the present application as defined in the appended claims and are considered to cover any and all modifications, variations, combinations, or equivalents that fall within the scope of the application. It will be apparent to those skilled in the art that various modifications and variations can be made to the present application without departing from the scope of the application. Thus, it is intended that the present application cover the modifications and variations of this application provided they come within the scope of the appended claims and their equivalents.

Claims

1. Intelligent data gathers software system based on thing networking, its characterized in that: the system comprises a data flow control module, an intelligent storage optimization module, a static path optimization module, an edge calculation coordination module, a priority scheduling management module, a time efficiency analysis module, a rule driving distribution module and a dynamic resource adjustment module;

2. The intelligent data convergence software system based on the internet of things as set forth in claim 1, wherein: the data flow dynamic analysis result comprises flow peak distribution, data transmission frequency and flow hot spot areas, the optimized storage scheme specifically refers to adjustment of data storage partition, backup strategy formulation and storage space redistribution, the static path optimization strategy comprises data transmission path determination, standby path planning and path redundancy assessment, the coordination calculation scheme specifically comprises resource sharing rules, data synchronization mechanisms and collaborative task distribution of edge calculation nodes, the priority scheduling scheme specifically comprises task priority classification, execution queue arrangement and task dependency graph, the time optimization strategy specifically refers to a time compression method, key path optimization and non-key task postponement strategy of flow execution, the rule-driven distribution scheme comprises data distribution rule setting, task distribution logic and periodic assessment of distribution efficiency, and the resource adjustment scheme specifically comprises resource redistribution based on load change, dynamic scaling strategy and resource use optimization.

3. The intelligent data convergence software system based on the internet of things as set forth in claim 1, wherein: the data flow control module comprises a flow monitoring sub-module, a trend analysis sub-module and a flow guiding sub-module;

4. The intelligent data convergence software system based on the internet of things as set forth in claim 1, wherein: the intelligent storage optimization module comprises a storage structure adjustment sub-module, a position optimization sub-module and a storage efficiency analysis sub-module;

5. The intelligent data convergence software system based on the internet of things as set forth in claim 1, wherein: the static path optimization module comprises a path planning sub-module, a path efficiency evaluation sub-module and a path execution sub-module;

6. The intelligent data convergence software system based on the internet of things as set forth in claim 1, wherein: the edge computing coordination module comprises an edge node management sub-module, a center coordination sub-module and a coordination strategy sub-module;

7. The intelligent data convergence software system based on the internet of things as set forth in claim 1, wherein: the priority scheduling management module comprises a priority judging sub-module, a scheduling strategy sub-module and a task execution monitoring sub-module;

8. The intelligent data convergence software system based on the internet of things as set forth in claim 1, wherein: the time efficiency analysis module comprises a time sequence analysis sub-module, a flow optimization sub-module and a flow time tracking sub-module;

9. The intelligent data convergence software system based on the internet of things as set forth in claim 1, wherein: the rule driving distribution module comprises a rule setting sub-module, a data distribution sub-module and an efficiency monitoring sub-module;

10. The intelligent data convergence software system based on the internet of things as set forth in claim 1, wherein: the dynamic resource adjustment module comprises a load analysis sub-module, a resource allocation sub-module and an adjustment strategy sub-module;